Anyone else down?

[burris]

Both my boxes just went down...no registration..

[TomP]

I'm not at home to confirm, but vPanel says no devices registered.

[burris]

OK now....after a power reboot..

[abward]

Yep. The daily VOIPo problem. Only 2 lights on the adapter...rebooted, again...seems registered. Who knows how long for.

Tried calling an 866 number this morning and it would not go through on VOIPo.

OK, I give up. Bailing as soon as I can get another provider and port out. Sorry, no more "tickets".

[ksarpong]

My PAP2 just went down 2 minutes ago..

[bretski]

Yup. I'm down as well. Just power-cycled, and seem to have re-registered.

[slimone]

My papt just lost registration.
I had to reboot for the 1st time. :eek:

[VOIPoBrandon]

Hey guys -- looks like there was a load issue on sip.voipwelcome.com, looks like a possible loop like scenario, if your device lost connection, it should automatically reconnect within 5 minutes. We have corrected this and rebalanced the load, thanks and sorry for any inconvenience!
________
Creampie Porn (http://www.fucktube.com/categories/16/creampie/videos/1)

[VOIPoTim]

Just a sidenote...this loop is a prime and timely example of why we're ending BYOD support.

BYOD users can easily cause a lot of issues like this and do so on a regular basis when they misconfigure things like Asterisk no matter how many safeguards you put in place. Today is was an SPA3000.

If we wouldn't have allowed BYOD, this would not have happened.

[ptrowski]

My wife was just trying to call my sister back and was about to throw the phone through the TV. 4 dials out, nothing. I was wondering why my parents called my cell phone earlier.

[sr98user]

I am on east01 and I don't believe my ATA went down.

[usa2k]

Never noticed anything (looks fine here at this moment!)

EDIT:
If you want BYOD, (and its not cost prohibitive)
perhaps they could be on their own server? (sandbox)

Or perhaps they could graduate from a sandbox?

Just dreaming up options ... you don't really need to be all things to everyone!

[dbmaven]

I'm totally down right now - and vPanel will not load.
And I'm on EAST01

[digger16309]

Yep. The daily VOIPo problem. Only 2 lights on the adapter...rebooted, again...seems registered. Who knows how long for.



This is my EXACT problem. First noticed it today but it dates back to yesterday upon further review.

PAP2T shows as registered - does not lose registration according to VPanel.

But the 2 phone lights go out and there is no dial tone or incoming.

Then everything works, then it doesn't, over and over - without losing registration.

Reboot changes nothing.

Ticket submitted but PAP2T is registered so.....

WAF (Wife Aggravation Factor) very high.

[Bobman]

I just had a quick outage around 7:30 PM Eastern. Couldn't get to voice mail with 123 or *123, then no dial tone then couldn't get to vpanel . Now, all seems to be ok.

[VOIPoTim]

We have seen several cases of massive floods of traffic causing latency a few times tonight. The Planet is working to control it with firewalls.

traceroute to 174.132.131.132 (174.132.131.132), 30 hops max, 40 byte packets
1 41.38.344a.static.theplanet.com (74.52.56.65) 916.325 ms 916.529 ms 916.741 ms
2 91.fe.5746.static.theplanet.com (70.87.254.145) 0.474 ms 0.460 ms 0.659 ms
3 e2.fe.5746.static.theplanet.com (70.87.254.226) 0.895 ms 0.896 ms 0.885 ms

[VOIPoTim]

We are seeing another huge flood of traffic in which is causing major network latency. This is similar to last night.

traceroute to 174.132.131.132 (174.132.131.132), 30 hops max, 40 byte packets
1 41.38.344a.static.theplanet.com (74.52.56.65) 916.325 ms 916.529 ms 916.741 ms
2 91.fe.5746.static.theplanet.com (70.87.254.145) 0.474 ms 0.460 ms 0.659 ms
3 e2.fe.5746.static.theplanet.com (70.87.254.226) 0.895 ms 0.896 ms 0.885 ms

They're working to firewall it now.

[Huffy]

I'm in Southern California.

Both lines lost registration at around 4:30pm PDT with a status of "Can't connect to login server" on the PAP2T status page. I could not get to vPanel.

THe PAP2T was able to re-register on it's own within 5 to 10 minutes. And I could then get into vPanel.

[digger16309]

Tim,

I'm not up on the technical stuff. Why does that cause the two phone lights on my PAP2T to keep going out?

[VOIPoTim]

It looks like someone does not want our service to stay up and just keeps sending these floods of traffic (almost to the point of being a DOS attack).

We're evaluating putting in some more powerful hardware firewalls tonight.

[voipinit]

Just a sidenote...this loop is a prime and timely example of why we're ending BYOD support.

BYOD users can easily cause a lot of issues like this and do so on a regular basis when they misconfigure things like Asterisk no matter how many safeguards you put in place. Today is was an SPA3000.

If we wouldn't have allowed BYOD, this would not have happened.


Don't shoot us all, some of us leave a pretty lite footprint on your network. I do understand your frustration though.

[digger16309]

It looks like someone does not want our service to stay up and just keeps sending these floods of traffic (almost to the point of being a DOS attack).

We're evaluating putting in some more powerful hardware firewalls tonight.

I'd say that's an accurate description because my service is being denied.

[voipinit]

It looks like someone does not want our service to stay up and just keeps sending these floods of traffic (almost to the point of being a DOS attack).

We're evaluating putting in some more powerful hardware firewalls tonight.



If you think you are being attacked, check out:

http://www.stopddos.org/

These guys are sharp and in most cases can identify the source of an active attack within minutes.

[gbh2o]

FWIW, I did not notice any outage, the Zoom indicates that it's happy with all appropriate lights lit and I still have service; no reboot was required, so I would guess no outage occurred here. It's still set to register to the default sip.voipwelcome.com, where ever that puts me at a given moment.

Please do not shoot all the BYOD users. If you believe there is a problem my equipment, simply turn me off and send me note to fix it! I listen well to criticism before counter-attacking! ;-)

[ptrowski]

My lights on the PAP2 never went off, but the service was going to dead air, the CP showed the device connected.

Maybe a denial of service, how about all of the obviously fake profiles that were being generated and finally had to be brought to Voipo's attention?

[sal]

I noticed no devices registered around 4:20 pm and my pap2t showed can't logon to server. However, I left it alone and pap2t re-registered within a few minutes. Everything works fine. Didn't miss a call.

I believe I am on sip. I am a light footprint. :) I check and double check and then check here and BBR if something is not right. I always figure it will correct itself if left alone.

[VOIPoTim]

My lights on the PAP2 never went off, but the service was going to dead air, the CP showed the device connected.

Maybe a denial of service, how about all of the obviously fake profiles that were being generated and finally had to be brought to Voipo's attention?

Forum users wouldn't have anything to do with the other portions of the service. They're not even in the same datacenters.

The attacks do seem to be under control, but we're working on making some changes tonight to beef up our protection since they do seem to come back.

Even more interesting in where we think they originate, but this isn't really the place to go into that.

[ptrowski]

I know that, but all of a sudden there are an influx here and now you are being targeted. Either a huge coincidence....

[VOIPoTim]

I noticed no devices registered around 4:20 pm and my pap2t showed can't logon to server. However, I left it alone and pap2t re-registered within a few minutes. Everything works fine. Didn't miss a call

Glad to hear. This is exactly how things are supposed to work with the re-registration.

[VOIPoTim]

I know that, but all of a sudden there are an influx here and now you are being targeted. Either a huge coincidence....

Based on what we think is the origination of the network traffic, we're definitely being targeted and I think everyone on this forum would connect the dots very quickly if I mentioned a certain city. I just can't get into specifics here.

[ptrowski]

They can't really be that stupid if that is what you are inferring....

[Russell]

They can't really be that stupid if that is what you are inferring....
Now my curiosity is really aroused. Are we referring to a competitor?

[ptrowski]

Not being as technical as some, I have heard of a BYOD loop before but don't really know too much about it.

[Russell]

Not being as technical as some, I have heard of a BYOD loop before but don't really know too much about it.
I'm as technical (or lacking those skills) as you. Someone care to elaborate?

[Xponder1]

Sounds like a denial of service attack to me.

Someone with nothing better to do is sitting in a irc chat room with 1000's of bots typing commands like !pingfast (insert one of voipo's many ips here)

I have been in several bot net channels. Used to run a large irc network. Had more than one guys bot net put out of commission permanently. You really should not be able to tell where its originating because they would just be infected pc's. Unless they are morons using dos software on their own pc's (most are smarter than that).

Wish they would really crack down on that stuff. The FBI seems to only focus on the people who profit from it. Good luck fighting them off.

[dbmaven]

I don't know if this mess is still going on or not.
But I can't stay registered at all.

If I reboot my router, then re-apply power to the PAP2, both lines register.

On the first re-registration (260 secs?) both fail - lights out - vPanel empty.

I'm going to submit a ticket - but I can't reboot or do anything until tomorrow night - because I don't have time to mess with it in the AM.

Edit: Ticket opened - HEK-343939: Complete Registration failure

[digger16309]

I don't know if this mess is still going on or not.
But I can't stay registered at all.

If I reboot my router, then re-apply power to the PAP2, both lines register.

On the first re-registration (260 secs?) both fail - lights out - vPanel empty.

I'm going to submit a ticket - but I can't reboot or do anything until tomorrow night - because I don't have time to mess with it in the AM.

Edit: Ticket opened - HEK-343939: Complete Registration failure

Same here this morning but VPanel still shows me as registered even though I am really not.

The pissed off factor is starting to rise now....

[VOIPoTim]

Same here this morning but VPanel still shows me as registered even though I am really not.

The pissed off factor is starting to rise now....

When you say you're really not what's happening? All the graphs look fine on our end so there aren't any system-wide issues that we can tell.

Did all of this start for you when the firmware was upgraded the other day?

[VOIPoTim]

I don't know if this mess is still going on or not.
But I can't stay registered at all.

If I reboot my router, then re-apply power to the PAP2, both lines register.

On the first re-registration (260 secs?) both fail - lights out - vPanel empty.

I'm going to submit a ticket - but I can't reboot or do anything until tomorrow night - because I don't have time to mess with it in the AM.

Edit: Ticket opened - HEK-343939: Complete Registration failure

We're not even seeing attempts from your account at all which is very odd.

I know in the past you mentioned having to reboot your router for it to connect, have you tried that time time?

[ptrowski]

Would this DDOS attack in theory affect people that are not having their audio relayed though the Voipo servers? I would think in theory it shouldn't.

[VOIPoTim]

Would this DDOS attack in theory affect people that are not having their audio relayed though the Voipo servers? I would think in theory it shouldn't.

Yes. All the signaling would still have to go through our network to establish the calls, so major latency there would cause an issue.

[ptrowski]

Ah ok. I am a better resource manager than VoIP engineer.

[sr98user]

Would this DDOS attack in theory affect people that are not having their audio relayed though the Voipo servers? I would think in theory it shouldn't.

I am on east01 and my audio is not relayed through Voipo servers. My ATA did not go down yesterday and when people had problems, I tried making some outgoing calls and it worked fine. ATA and vPanel showed I was registered. I am not sure if this was isolated to the Dallas servers.

[ptrowski]

I am on east01 and my audio is not relayed through Voipo servers. My ATA did not go down yesterday and when people had problems, I tried making some outgoing calls and it worked fine. ATA and vPanel showed I was registered. I am not sure if this was isolated to the Dallas servers.


That's why I asked as I am supposed to be on East01 also and I was getting dead air.

[VOIPoTim]

That's why I asked as I am supposed to be on East01 also and I was getting dead air.

We're showing you're using a HT502 on Central02 in Dallas.

[ptrowski]

We're showing you're using a HT502 on Central02 in Dallas.


That must be because I took the pap2 out and put the 502 back in after the wife was done giving me dirty looks.

[dbmaven]

We're not even seeing attempts from your account at all which is very odd.

I know in the past you mentioned having to reboot your router for it to connect, have you tried that time time?

Every time I reboot the router, it registers successfully. At the first expiration (which looks like 250+ seconds) - *BAM* = FAIL. Cannot connect to registration server (or whatever the exact message is in the PAP2 status page), and vPanel shows nothing.

My ticket update by support shows a suspicion that this was related to issues at your datacenter.

I will reboot everything tonight when I get home and we'll go from there.

Brandon also indicated that he was reviewing my router logs (from the earlier ticket) and (paraphrased) 'was seeing some very weird things'....

If it's all related to issues at your data center I'll be happy. For most of the prior week - it just worked. No port forwarding or anything at all in the router setup - almost an 'out of the box' router config - which is the way it should be... :)

[VOIPoLukeBotha]

Hi guys,

There was a network failure earlier which affected registrations - This has since been resolved.

More information regarding the network failure can be found using the URL listed below:

http://forums.voipo.com/showthread.php?p=11222#post11222

If you're still experiencing any issues please direct a ticket to the Support department by emailing support@voipo.com.

We'll gladly look into this for you.

Thanks

[Dave_N]

Every time I reboot the router, it registers successfully. At the first expiration (which looks like 250+ seconds) - *BAM* = FAIL. Cannot connect to registration server (or whatever the exact message is in the PAP2 status page), and vPanel shows nothing.

My ticket update by support shows a suspicion that this was related to issues at your datacenter.

I will reboot everything tonight when I get home and we'll go from there.

Brandon also indicated that he was reviewing my router logs (from the earlier ticket) and (paraphrased) 'was seeing some very weird things'....

If it's all related to issues at your data center I'll be happy. For most of the prior week - it just worked. No port forwarding or anything at all in the router setup - almost an 'out of the box' router config - which is the way it should be... :)

I can confirm this behavior as well. Prior to this week, it was just working, didn't have to port forward on my router or anything. As of late Tuesday, things got bad again. Unfortunately, there were "two changes". The firmware upgrade and the network instability. I also was usually connected after the reboot and then unable to connect. This of course led me to reconfigure my router for the port forwarding recommended. Change #3. As of now things are working but the WAI (and index of the current Wife Acceptance Factor) is at a new low and the installation might not survive another extended outage.

[dbmaven]

Hi guys,

There was a network failure earlier which affected registrations - This has since been resolved.

More information regarding the network failure can be found using the URL listed below:

http://forums.voipo.com/showthread.php?p=11222#post11222

If you're still experiencing any issues please direct a ticket to the Support department by emailing support@voipo.com.

We'll gladly look into this for you.

Thanks

HEK-343939

[VOIPoTim]

HEK-343939

We think this is a seperate issue due to the router logs you'd sent. I have someone looking at it for now you.

[VOIPoBrandon]

Craig,

I have looked at both your syslogs and your network traces that are currently being sent to the server it looks like you have a few issues going on. The first being A) loosing registration, I sent you a call and as soon as it was over it seems as if your registration has dropped. After doing extensive research with your syslog information you sent over to me, I found an error that looks to be involved between your router / firmware: "ip_nat_mangle_udp_packet: undersized packet" -- doing a quick google on this quickly brings up results with routers having odd results such as calls actually rebooting their router.

B) The other issue at hand as far as why you are not re-registering properly is that it seems you are not receiving our "401 Unauthorized" Response to your original SIP REGISTER request.

The way it works is as follows:
A) PAP2T sends registration to SERVER without "realm" and "credentials" i.e. your "SIP Authentication Information"
B) SERVER sends "401 Unauthorized" -- which means "Give me my credentials to authorize you"
C) You receive the 401 and send back a REGISTER request with Proxy-Authorization header / credentials & realm set.

This process is not occurring? Why one may ask... the answer is simple. The packet is not making it to your PAP2T, as for the reason to that, there can be a myriad of reasons, to name a few: firewall, ports not forwarded, router / firmware issues, etc.

Is it possible for you to setup a network sniffer on your side and see if you are indeed getting these 401 responses to your registration request? If you are then perhaps you have a defective ATA -- however I'd bet it's a 99% chance that the PAP2T is not receiving this message.

As for going back to Issue A -- I'd like to continue to work with you and perhaps get some more syslog information perhaps at the precise time, of a call being placed / etc.

This is why we tell users to forward ports -- (even if their router allows it when the device is DMZ'ed) -- since it is all to common that most routers do not interact properly, and forwarding ports will more than likely fix your registration issues.

[dbmaven]

Not having seen this - I updated the ticket by sending a new - small - router log.

Last week this was working fine with a pretty "stock" router config.
No ports forwarded.
SPI Firewall enabled.
Ports 5062 and 5079.

Now I'm seeing 5061 and 5079.
Coincidentally, perhaps, with the datacenter/latency issues, it started failing again.
I forwarded ports. No love.

For the log file - the SPI was disabled and no ports were forwarded.

Unfortunately, I don't have a sniffer to capture packets. But I'll do whatever diagnostics you want.

If we really believe that the router is the issue - intermittently or regularly - I can retrieve my 'old' WRT54GS and set it up as the internet router, and hang the PAP2 off it, with my WRT610N as an internal router, and a static custom route from the 54 to the 610N. It's a PITA, because I have N devices that I need - but I'll do it.

If that resolves the issue - somebody needs to beat Cisco/Linksys with a very big stick - 'cause that means as more people go to their higher end "n" routers you're going to see a lot of unhappy customers...

Let me know how you want to proceed.

[sr98user]

dbmaven, Here are few trouble shooting steps..

* Update latest firmware ( you probably already did this). Sometimes going back one rev will fix things too.

* Try turning off STUN on PAP2T and disable SPI on router. No port forwarding. See if it works. When you are doing these steps, keep the registation interval to 1 minute, so that the registration expires quickly. Also wait for 1 minute before you goto the next step.

* If it doesn't work, turn on STUN in PAP2T, disable SPI on router, See if it works. There are lot of parameters for STUN in PAP2T. You might have to change things few times.

* Try port forwarding. 5061 and 5062 are PAP2T defaults. I am not sure if they changed one of those to 5079 for you. 5079 was used by GS 502. Since you had problems with registration, I think just forwarding those ports will check if the registration will stick.

* See this thread (see sipmaster posting) http://forums.linksysbycisco.com/linksys/board/message?board.id=Wireless_Routers&message.id=126691&query.id=99574#M126691

He seems to say 610N modifies SIP packets and maybe doing it incorrectly.

http://forums.linksysbycisco.com/linksys/board/message?board.id=VoIP_Routers&message.id=3724&query.id=99574#M3724

* If you are able to get it to work, then you can add SPI firewall and see if it breaks things.

* To sniff the packets, you need wireshark and a hub. Put the pap2t and pc behind the hub. Connect the hub to the router. Use Wireshark to capture the packets.

[dbmaven]

http://forums.linksysbycisco.com/linksys/board/message?board.id=Wireless_Routers&message.id=126691&query.id=99574#M126691

That seems to validate what Brandon has been seeing in my logs and/or the diagnostics/traces that have been done.

Thank you for finding that - the search function on their forums is horrible - I should have been able to find that before I posted a thread about the same topic that got zero (useful) responses.

I don't think I'm going to waste any more time with this - probably going to go get my WRT54GS and set it up as the main internet router and use the 610N as an internal router.

[dbmaven]

I want to post a closing comment to my ridiculous saga.

I spent quite a bit of time this afternoon with Brandon - trying all sorts of interesting router and pap2/server configurations.

I even set up my 'old' WRT54GS as the internet router, and hung the PAP2 off it, taking the WRT610N out of the picture.

The behavior was identical.

I ended up flashing the 54GS to DD-WRT.
And.....

IT JUST FREAKIN' WORKS

No port forwarding
No port triggering
No DMZ

No nothing
As basic a router config as you can get.

It just works.

Somebody needs to take whoever is responsible for the NAT section of the LINKSYS firmware, and put him in a room with a couple dozen recalcitrant Linksys routers and PAP2s. Tell him/her they can come out when they just work properly all the time.

Many thanks to Brandon, and Tim - for their dedication and patience.

Now to compose a nastygram for the President of Cisco...... :p

[VOIPoTim]

I want to post a closing comment to my ridiculous saga.

I spent quite a bit of time this afternoon with Brandon - trying all sorts of interesting router and pap2/server configurations.

I even set up my 'old' WRT54GS as the internet router, and hung the PAP2 off it, taking the WRT610N out of the picture.

The behavior was identical.

I ended up flashing the 54GS to DD-WRT.
And.....

IT JUST FREAKIN' WORKS

No port forwarding
No port triggering
No DMZ

No nothing
As basic a router config as you can get.

It just works.

Somebody needs to take whoever is responsible for the NAT section of the LINKSYS firmware, and put him in a room with a couple dozen recalcitrant Linksys routers and PAP2s. Tell him/her they can come out when they just work properly all the time.

Many thanks to Brandon, and Tim - for their dedication and patience.

Now to compose a nastygram for the President of Cisco...... :p

This is what we fight every day with getting through it through to users...but most don't believe us and insist that it must just be that the service is unreliable for everyone.

The majority of the users with problems just don't realize they're being caused by their routers.

I know it's crazy, but the newer, more advanced the routers are, the more problems they seem to have.

Users think that because they don't have anything forwarded on them the router won't mess with them. The new ones mostly seem to rewrite ALL traffic though so it causes issues.

Glad to hear you're up and running. :)

[dbmaven]

I know it's crazy, but the newer, more advanced the routers are, the more problems they seem to have.

Users think that because they don't have anything forwarded on them the router won't mess with them. The new ones mostly seem to rewrite ALL traffic though so it causes issues.

Glad to hear you're up and running. :)

I'm glad too - and I know you and Brandon are ;)

Sooner or later you and other VoIP providers are going to have to deal with Cisco on stuff like this - along with those of us having the issues and fighting through them. The average guy/gal really does want this stuff to be plug and play - and if it isn't - they'll bail. The router works for internet and everything else - but it doesn't work for VoIP - must be VoIP's problem.....

I'll be happy to help in any way I can.

[ptrowski]

I followed dbmaven's lead and went DD-WRT across the boards. Running a WRT54G v4 into the cable modem and a WRT54GL as a wireless bridge.

I also learned it's not best to do it late night after some drinks.

[Xponder1]

I want to post a closing comment to my ridiculous saga.

I spent quite a bit of time this afternoon with Brandon - trying all sorts of interesting router and pap2/server configurations.

I even set up my 'old' WRT54GS as the internet router, and hung the PAP2 off it, taking the WRT610N out of the picture.

The behavior was identical.

I ended up flashing the 54GS to DD-WRT.
And.....

IT JUST FREAKIN' WORKS

No port forwarding
No port triggering
No DMZ

No nothing
As basic a router config as you can get.

It just works.

Somebody needs to take whoever is responsible for the NAT section of the LINKSYS firmware, and put him in a room with a couple dozen recalcitrant Linksys routers and PAP2s. Tell him/her they can come out when they just work properly all the time.

Many thanks to Brandon, and Tim - for their dedication and patience.

Now to compose a nastygram for the President of Cisco...... :p

Linksys firmware is crap on purpose. Lets see you have a company making money and doing really well selling $50.00 routers that people in turn take and install other firmware on and they make it work like a 300.00 (or more) router. This in turn spreads by word of mouth and a company (Cisco) starts losing business on its "high end" products.

So if you can not beat them buy them out, cut out some of the extra space, memory, install crappy firmware , and while your at it lets stop using open source firmware and instead we will use something we can (try unsuccessfully ) to keep a secret.
EDIT -- I forgot to mention the really crappy tech support.

My WRT54G V2 was great. It died and I went to Walmart and purchased another (V6). It would not work out of the box so I returned it and got another (surely it must of been a bad apple right). The replacement (WRT54GS) did not work either.

I had to connect directly to the internet, download the newest firmware, install it just to get dns to work. Then ssh connections would die after about 10 minutes of use. I repaid Linksys for my wasted time by taking my dead V2 and sticking it in the box the new one came in and returning it to Walmart. I mean I can not be expected to pay for something that does not work right (lol).

Eventually they came out with DD-WRT micro and instructions on how to install it on the V6. Installed it and never looked back. Lots of control, ssh never disconnects (been connected for days at a time), and so many new features.

If you follow the directions flashing your Linksys with DD-WRT is safe. I know some folks here love tomato but its not compatible with any of the newer Linksys models except (maybe) the WRT54GL which still has the Linux based open source firmware.

[dbmaven]

I've got the best of both worlds right now.

Since I have, in effect, a dual network.

The 54GS with DD-WRT keeps PAP2/Voipo happy (and reduces support costs ;) ).
The 610N keeps me happy with all of my N remote devices (wireless media bridge) and everything else. And I've effectively added an extra layer of protection by having an "inside" network.

Should they release a "stable" version of DD-WRT for the 610N - I may go for it. Time will tell.

If not - I may just go for a D-Link dual-N router - people aren't seeing the issues with the D-Link routers from what I can gather.

BTW - if you're a DSLR person, post in the 'Equipment Central' thread with your setup. The more people who contribute "what works" helps people as they come along looking for service or trouble-shooting info.

http://www.dslreports.com/forum/r22180539-Equipment-Equipment-Central-informal-poll