Calling problems 5-7-09 AM

[Xponder1]

I am having all kinds of problems with calls this morning. Incoming calls are going to fail over and voicemail. Outbound call dropped during the call. Unable to answer even when using X-Lite instead of phones attached to the ATA. A moment ago only 2 of the 4 lights on the ATA were on and the internet light is flashing very fast in bursts for no apparent reason.

Anyone else seeing a issue? I submitted a ticket. XHK-385766

[scott2020]

I am seeing a major latency problem over 1000ms to central01 right now and incoming calls are not working. Several times it has been unreachable. My backup numbers on other providers are OK.

[VOIPoTim]

There was a loop that lasted for a few minutes that caused hundreds of thousands of connections which resulted in a load spike. A user had registered to multiple servers (with the same account) and was attempting to use both which caused the packets to loop between the two servers.

Since we applied the VOIPo-VOIPo internal calling fix, the system behaves differently and is not designed for multiple server registrations. Usually this isn't an issue since we provision everything. With that said, today we're going to work on some security measures to prevent users from registering to multiple servers. Going forward, if you are registered to one server, the system will only allow an additional registration (such as softphone) on the same server.

[Xponder1]

I am having all kinds of problems with calls this morning. Incoming calls are going to fail over and voicemail. Outbound call dropped during the call. Unable to answer even when using X-Lite instead of phones attached to the ATA. A moment ago only 2 of the 4 lights on the ATA were on and the internet light is flashing very fast in bursts for no apparent reason.

Anyone else seeing a issue? I submitted a ticket. XHK-385766

I did a life chat with support and my X-Lite was configured to use sip. but they say that is no longer in use. So I changed to central. Still does not explain the ATA rebooting. Plus its been configured this way for months.
EDIT - Tim posted before I hit submit lol. I hope I am not that lone user....
I had a voice mail and deleted it but I still have the message waiting dial tone. I have tried the reset from vpanel twice. It will not reset.

Edit - Logged in voicemail again and its says I have 0 messages. Hit pound and still have mw tone. I am seeing ping times of 14 to 30ms.

[Xponder1]

There was a loop that lasted for a few minutes that caused hundreds of thousands of connections which resulted in a load spike. A user had registered to multiple servers (with the same account) and was attempting to use both which caused the packets to loop between the two servers.

Since we applied the VOIPo-VOIPo internal calling fix, the system behaves differently and is not designed for multiple server registrations. Usually this isn't an issue since we provision everything. With that said, today we're going to work on some security measures to prevent users from registering to multiple servers. Going forward, if you are registered to one server, the system will only allow an additional registration (such as softphone) on the same server.

Any thoughts on the message waiting tone issue? Also I would like to suggest sending a mass email asking customers to use central for all their needs. When you implement the changes above some folks will be contacting support if you do not notify them in advance.

[Russell]

Glad the check for the same server will be put in place. Anyway, please consider displaying the server connected to (the new Preferences -> ATA would not be inappropriate). That way those who want to use a softphone don't have to open a ticket to find the server they're assigned to.

[scott2020]

There was a loop that lasted for a few minutes that caused hundreds of thousands of connections which resulted in a load spike. A user had registered to multiple servers (with the same account) and was attempting to use both which caused the packets to loop between the two servers.

Since we applied the VOIPo-VOIPo internal calling fix, the system behaves differently and is not designed for multiple server registrations. Usually this isn't an issue since we provision everything. With that said, today we're going to work on some security measures to prevent users from registering to multiple servers. Going forward, if you are registered to one server, the system will only allow an additional registration (such as softphone) on the same server.

Thanks Tim. It is looking much better now. Was this a BYOD person I assume? If a person uses Asterisk for example, the config should be to only one server (ie central01.voipwelcome.com) I am guessing. Actually in asterisk I don't know how to register to multiple servers with the same service so I hope it wasn't me that broke it!

[Xponder1]

I have tried from two different cell phones and can not get any calls to complete to my voipo number. No voice mail, no fail over, nothing at all. I can dial out.

EDIT- MWI issue still there and has been escalated to tier 2.
The two calls I made from cell phones both show in vpanel even though the call never rang on anything.

[Xponder1]

Thanks Tim. It is looking much better now. Was this a BYOD person I assume? If a person uses Asterisk for example, the config should be to only one server (ie central01.voipwelcome.com) I am guessing. Actually in asterisk I don't know how to register to multiple servers with the same service so I hope it wasn't me that broke it!

I have you tried any incoming calls to see if they work?

[scott2020]

Yes, my incoming calls seem to work fine.

[Xponder1]

Removed because I am sure Tim will take care of this.

[dswartz]

Scott, is your outgoing working? I cannot make calls, but can receive them. I sniffed the traffic and saw my invites being ignored, so the outbound times out. I opened a ticket but was told that they were able to make a call using my credentials so it must be at my end.

[VOIPoBrandon]

Scott, is your outgoing working? I cannot make calls, but can receive them. I sniffed the traffic and saw my invites being ignored, so the outbound times out. I opened a ticket but was told that they were able to make a call using my credentials so it must be at my end.

Currently no known network issues, if you open a ticket or shoot me a PM I can grab a network trace and see whats going on, thanks!

[VOIPoTim]

Well... thats how I would of handled it. While I was at it I would put a note on the account in case said customer contacted support so my peers knew what was going on so they could help this customer.

On a side note if my X-Lite setting somehow caused 1000's of customers service to have issues for a moment I apologize for their inconvenience. I was unaware of the problem which should be obvious from reading this thread.


What basically happened is that a VM notification went out from our VM servers and since the system detected you were on two servers, it would send to one and that one would send to the other, on and on.

This used to not happen, but since the VOIPo to VOIPo internal calling fix, things are organized a little differently in terms of how on network packets are handled and this was an unforseen effect of that.

The loop created some sporadic behavior on central01 and central02 since those were the servers involved. In a situation like this, the top priority is to minimize impact and restore service as a whole. From what I understand they put the temporary block on was they were concerned that once they stopped the loop, the ATA and softphone would be "fighting with each other" to re-register and both would end up doing so then if a VM MWI notification was triggered somehow, it'd just happen again. So that's why they blocked it.

I agree though that you should have been notified and am going to look into it a little more to make sure this stuff is handled better in the future. Sorry for the inconvenience.

We also have a few changes that are being finalized/tested in terms of the way multiple registrations are handled and also how we deal with softphone/grandfathered BYOD accounts so that they're not able to impact the "general population" as a whole.

[Xponder1]

What basically happened is that a VM notification went out from our VM servers and since the system detected you were on two servers, it would send to one and that one would send to the other, on and on.

This used to not happen, but since the VOIPo to VOIPo internal calling fix, things are organized a little differently in terms of how on network packets are handled and this was an unforseen effect of that.

The loop created some sporadic behavior on central01 and central02 since those were the servers involved. In a situation like this, the top priority is to minimize impact and restore service as a whole. From what I understand they put the temporary block on was they were concerned that once they stopped the loop, the ATA and softphone would be "fighting with each other" to re-register and both would end up doing so then if a VM MWI notification was triggered somehow, it'd just happen again. So that's why they blocked it.

I agree though that you should have been notified and am going to look into it a little more to make sure this stuff is handled better in the future. Sorry for the inconvenience.

We also have a few changes that are being finalized/tested in terms of the way multiple registrations are handled and also how we deal with softphone/grandfathered BYOD accounts so that they're not able to impact the "general population" as a whole (whether intentionally or unintentionally).

Thank you for taking the time to reply Tim. I feel better about it already.
Have a great day.

[scott2020]

Scott, is your outgoing working? I cannot make calls, but can receive them. I sniffed the traffic and saw my invites being ignored, so the outbound times out. I opened a ticket but was told that they were able to make a call using my credentials so it must be at my end.

Everything is working as normal for me. I was having some break up on some calls which is not normal but I believe it is cleared up.

One thing I noticed on my asterisk box, with inbound calls they used to come from a Level3 IP address 4.68.250.xxx and now incoming calls show a 192.168.27 address? I think calls used to be handed off to Level3 and my box would talk to Level3's media gateway directly and now the audio is going through VOIPo's IP's when I look at connections on my firewall. This started around 9:00 this morning.
Scott

[Xponder1]

Everything is working as normal for me. I was having some break up on some calls which is not normal but I believe it is cleared up.

One thing I noticed on my asterisk box, with inbound calls they used to come from a Level3 IP address 4.68.250.xxx and now incoming calls show a 192.168.27 address? I think calls used to be handed off to Level3 and my box would talk to Level3's media gateway directly and now the audio is going through VOIPo's IP's when I look at connections on my firewall. This started around 9:00 this morning.
Scott

That's about the time my service went out and apparently caused issues for 1000's of people. They probably made some kind of change when they were trying to figure it out.

[VOIPoBrandon]

Everything is working as normal for me. I was having some break up on some calls which is not normal but I believe it is cleared up.

One thing I noticed on my asterisk box, with inbound calls they used to come from a Level3 IP address 4.68.250.xxx and now incoming calls show a 192.168.27 address? I think calls used to be handed off to Level3 and my box would talk to Level3's media gateway directly and now the audio is going through VOIPo's IP's when I look at connections on my firewall. This started around 9:00 this morning.
Scott

Scott,

That 192.168.27 address you are seeing is the local CLEC that is delivering your origination (for that phone number). As for your media path being directed to our gateway, this change has been for quite some time now (since we've had better reports of us handling the audio). This is not a result from any changes this morning.

Are you continuing to see any call quality issues? Please submit a ticket and do let us know if you are so we can further look into this for you.

Thanks!

[scott2020]

I see. The local CLEC used to send the 4.68.250.xxx address and that is what would connect to me, but it changed this morning. On the audio path, I am almost positive my audio was always going direct to the provider media gateway but maybe I was wrong. The audio quality is fine now. thanks!

[sr98user]

192.168.x.x is a Class C private network and it is not routable on the internet. Unless your ISP is sending it or your ATA is trying to contact, you should not be seeing 192.168.x.x address.

[scott2020]

192.168.x.x is a Class C private network and it is not routable on the internet. Unless your ISP is sending it or your ATA is trying to contact, you should not be seeing 192.168.x.x address.

Sorry, I should have mentioned that I am seeing the 192.168 address on incoming connections on my Asterisk box, as in the incoming SIP messages whereas it used to show the 4.68.250 addresses. On the actual firewall I am seeing connections to the Internet IP's.

[VOIPoBrandon]

192.168.x.x is a Class C private network and it is not routable on the internet. Unless your ISP is sending it or your ATA is trying to contact, you should not be seeing 192.168.x.x address.

That's correct, although he is talking about the Call-ID (of the call) where it was originated.
________
Babe vid (http://www.fucktube.com/categories/6/babe/videos/1)

[voipinit]

I noticed my cloned line was showing 192.168.x.x which was strange as normally both lines show the internet IP, it was like this for several days. When my ISP went down for several hours (hardware broke) they forced a reasignment of IP's which forced me to reboot the ATA (as well as everything else), and it came back normally not showing my private network IP anymore.