Sporadic Failover Calls in the past couple of months

[GreenLantern]

Hey guys,

Some of my customers have started experiencing sporadic failover calls in the past couple of months.

It is happening with new customers with new equipment, as well as with long time customers who had been working great (other than known outage events).

I've experienced unusual amount of failovers as well, despite having superfast internet, with hardly anything using it but my phones.

I'm going to try registering to the alternate data center, to see if that helps.

Just wanted to see if I'm the only one experiencing this, or if it might be a wider problem (indicating that the problem may be on Voipo's side).

Let me know if you've had more than usual failovers lately.

Thanks!

[racerdude]

I've been experiencing intermittent incoming calls going to VM instead of ringing on my side lately even though my ATA was online and registered. Seems to have just started recently and like you mentioned nothing has changed on my side. I've been using sip.voipwelcome and switched to sip-central01.voipwelcome over the weekend to see if that will make a difference but have had only a few incoming calls this week which worked ok, but not sure if this server is better or not yet since the issue is intermittent.

Which server are you using?

[racerdude]

I spoke too soon.. just had an incoming call that didn't ring on my side and went to VM (I can see the call in the call history). The person calling redialed and the call came through on my side the second time. So it's happening for me on both sip and sip-central01 servers.

[GreenLantern]

Racer, I agree that it doesn't seem to be server specific.

It's hard to determine what the cause is.

Some of my larger clients do thousands of calls per month, with no issues. So it doesn't seem to be a global issue.

I suspect it is some kind of port timeout issue on certain routers.

If you have an ATA or IP phone, be sure to change the registration timeout, aka server timeout, or whatever it is called in your device, to something other than the usual 3600 second default. It needs to be more like 300 seconds. That solves the random inbound call failure most of the time.

Setting up port forwarding also helps, if you haven't done that.

And make sure you don't have a double NAT (one router/firewall behind another router/firewall).

My problem, is that I've already done all the usual stuff... eliminated double nat, setup port forwarding, corrected server timeout, etc. But I still get some random failovers to voicemail on certain user accounts. These accounts are usually numbers that had some issue during porting in, so it just makes me wonder if there's some kind of provisioning issue going on with these new numbers, so it doesn't affect older accounts.

But it sounds like I'm the only one with this specific issue, so my guess is it must be pretty "user specific", such as some particular carrier glitch, or maybe just the router the end users have.

I'll be doing some remote sessions with these users, so maybe I'll spot something.

Thanks for the info. :)

[racerdude]

Yeah, my settings are good, I've been a long time customer (going on 6th year) with BYOD, first on PAP2T and now on SPA2102 and have not noticed this problem until the past few weeks. It sounds to me like we have the same issue, my failover setting is to go to VM which is exactly what is happening on sporadic incoming calls. Not sure if something changed on Voipo's side that would be causing this, but there is another thread here in the forums with 2 others having the same issue.

[GreenLantern]

Racer, tech support did something today to one of my accounts. We're going to test and see if that resolves it.

One point of frustration is that they don't tell me what they did. They just reply with something like, "try it again now."

Would be nice to know what they are doing, so I can track it from my side, and help them identify similar issues more quickly in the future.

Anyway, I'll let you know if their changes work.

[GreenLantern]

Ok, tech support's change didn't help.

However, I switched the customer over to the new Chicago server and everything started working much better right away.

Incidentally, I've noticed that most (but not all) of the clients having trouble are using Comcast Business Class for internet service.

In the past, I've had trouble between Comcast users and the Texas server. I moved a ton of clients over to the Michigan server (sip7) to resolve. I had started moving them back to Texas when Voipo announced they were dropping the Michigan server. I think some of these issues have now returned. So I'll likely have to start moving people away from the Texas server again.

[racerdude]

That's interesting, I use TWC not Comcast and am still having the issue. I tried sip, sip7 and sip-central01 and have the problem on all of them, looks like I am intermittently losing registration since in Control Panel it sometimes shows "no devices connected" which is causing incoming calls to go to failover when this happens . I even tried setting the registration interval to 30 seconds and I noticed that sometimes I still lose registration before the 30 seconds is up. I know it sounds like a router issue, but I have keepalive on the ATA set to 15 secs and my udp timeouts on my router are 10 secs for unreplied and 300 for assured which has been working fine until recently. These settings also work fine with other providers, it's only with Voipo that I'm seeing the issue.

What is the dns name for the Chicago server so I can give that a try?

[GreenLantern]

Yes, I should say most of the clients are using Comcast, Time Warner, Cox and Charter Cable. Most of those are using Comcast, but all are using some form of Cable, except for one client who has Verizon FiOS.

I have Comcast, and yesterday, one of my clients could not call me from her cell phone. She got various messages over a 30 minute period, before finally getting through.

I was using the north reseller server at the time, and no other issues that I was aware of.

[chevyman]

I have a customer that has a Grandstream PBX and voipo set as one of the trunks. in the logs it has showed voipo has went down for a minute here and there. So i add a free callcentric line as a sip trunks so i can trouble shoot if only voipo goes down or both. That way i will know if both go down its has something to do with the internet and not voipo.

[GreenLantern]

I've had to switch a few clients to the alternate voipo data center to resolve registrations dropping for no reason.

Only need to move a small percentage of clients, and it does seem to resolve the registration issues.

No solid pattern, but seems to affect a small segment of cable internet users, mostly Comcast business class or Time Warner business class. Maybe there's something different about those networks?

We don't have the problem with other voip services that we have at the same locations as backup.

[GreenLantern]

Getting busy signals or "all circuits are busy" on pretty much all of my accounts today.

Anyone else?

[VOIPoTim]

We had a hardware failure earlier this morning that was causing issues for some customers this morning. It has been resolved.

[tritch]

Resellers....just curious to see if this issue has gone away.

I'm not a reseller, but I've been trying to resolve this same issue for my dad on BYOD. For whatever reason, random incoming calls seem to go straight to failover when absolutely nothing seems to be wrong with his network or equipment. (ATA registered, Cable ISP is up). This has been happening a lot over the past month with no obvious explanation. I've had him change his ATA registration back and forth between sip.voipwelcome and sip7.voipwelcome with no improvement. He's cable ISP is Fidelity Communications which doesn't seem to have a large footprint, so I'm wondering if there's some routing issues going on between Voipo's Sip servers and his house. I'm going to have him try swapping out his ATA next. Currently, his ATA is temporarily in front of the router with no help either.

Has this issue occurred lately with your clients? If so, what has been your solution.....Thanks!

[GreenLantern]

I may be on to something with this issue. I want to see if anyone else can verify what I'm seeing.

It appears that sometimes the registration times out early, causing a brief period for every registration cycle, where calls will failover.

Example, if my registration timeout is 10 minutes, the voipo server seems to timeout at around 9 min 20 sec. So there's about 40 seconds where calls failover.

I can increase or decrease the timeout with the same result... just more or less frequent, but still leaving a downtime every cycle.

To verify this, I simply log into a control panel and hit refresh while watching the registration as the timeout approaches. Then sure enough, I see "no devices registered" message towards the end of the registration period.

Anyone else experiencing this?

[chpalmer]

To verify this, I simply log into a control panel and hit refresh while watching the registration as the timeout approaches. Then sure enough, I see "no devices registered" message towards the end of the registration period. Anyone else experiencing this? I just did this and see that my devices actually update earlier than the timeout period by a minute or so.. Both my Voipo provided device and my BYOD PAP2. I have no issues with my lines.

[GreenLantern]

I just did this and see that my devices actually update earlier than the timeout period by a minute or so.. Both my Voipo provided device and my BYOD PAP2. I have no issues with my lines.

Good info.

I see that the Grandstream ATA's have a "re-register before expiration" setting.

This likely helps in situations exactly like this, where the service provider has an early timeout.

I suspect this issue has been affecting lots of BYOD users with devices that don't have a "re-register before expiration" setting.

Seems to have started in the last couple of months, perhaps after some voipo software changes/updates.

So hopefully voipo can do something on their side to set expirations correctly.

[GreenLantern]

late last night, voipo resolved a clock synchronization issue, that was causing registration to expire prematurely.

registrations are now expiring accurately.

many thanks to the voipo team for working on that and getting it fixed!

[tritch]

GreenLantern, thanks for taking a look and your help with Voipo's team to resolve this issue. We'll see how things go from here...


Update 9/7/16: 100% fixed...not a single failover issue since. Great work!

[tritch]

I'm seeing this same issue again (on my father's BYOD ATA). SIP server registrations expiring 20-30 seconds prematurely causing calls to fail.

GreenLatern, would you mind confirming this is happening again? If so, could you please contact the same support person again to get this fixed...much appreciated.!!

FYI: I had him switch over to sip7 and it is occurring on there as well.

[GreenLantern]

Yes. Looks like their clocks are off again, by almost a minute. So registrations are expiring early, causing inbound calls to failover.

Outbound calls work, because they open connection at time of the call.

I've reported it, but you should too.

[tritch]

Thanks for confirming, I will open a ticket as well (on behalf of my father). I see BYOD users complaining about the registration issue over on DSL reports. I've confirmed this is not occurring on their SIP servers for provisioned users (sip-central01).

I'm assuming this is being worked by Tier 2 support. Would you happen to have the name of the Voipo tech working your support ticket?

FYI: I've complained to Tim before about their lack of oversight on these BYOD servers before, so this is just another classic example.

[tritch]

Update 4/10: (BYOD sip) seems to be fixed to a certain degree. The ATA registration value needs to be set lower than 4 minutes to avoid a brief 2 second premature expiration from server. For example, an ATA registration setting of 3 minutes, shows a server expiration in vPanel of 3 minutes and 58 seconds. This allows the ATA to re-register in plenty of time before the server expiration (a 58 second cushion). As of yesterday, BYOD sip7 was still screwed up. Server registrations were prematurely expiring anywhere from 20 to 90 seconds depending upon the ATA registration settings. I have not checked sip7 today.

[GreenLantern]

yeah, ticket got escalated to Tim and Brandon. They seem to have sorted it out. Though if what you say about 2 seconds is true, that is still a problem in my mind.

I agree it is frustrating that byod and resellers seem to be a bit of an afterthought lately.

https://memegenerator.net/img/instances/62996464/today-we-test-in-production.jpg