Outage? and can't get to secure.voipo.com to open a ticket either

[Dave_N]

Hi,

Anyone else experiencing an inability to call out or in. I would open a ticket but I can't get to secure.voipo.com either.

N/NW chicago suburbs if that matters.

[scott2020]

I have servers in The Planet in Dallas and they have disappeared off the face of The Planet... no pun intended. What's going on at The Planet? Just a black hole right now. support.voipo.com I believe is there. My servers are unreachable.

[VOIPoTim]

Are you using BYOD?

If you're having a service outage, please contact support for assistance.

As for vPanel, please see this announcement: http://forums.voipo.com/showthread.php?p=11931#post11931

[Xponder1]

The problem is some kind of internet outage. I was unable to load Google, then my own site at hostgator went down, hostgator was down, VOIPo was down.

Tried calling and was unable to call out.
Ya something is going on.

Edit - since I got this site to load I can dial out but hostgator.com and my sites at the planet are not loading.

Edit again Tim knows lol http://forums.voipo.com/showthread.php?t=1428

[scott2020]

Incoming calls on two VOIPo numbers are not going through. My backup SIP trunks are working now. Just getting DISCONNECT messages now calling VOIPo DID's.

I am on central01 and it is showing UNREACHABLE.

About 10 minutes later the phone starts ringing. No failover. I am BYOD, but it shows unreachable or high latency on central01. Tried opening a ticket but not sure if it made it.

Also cannot call out on VOIPo trunk.

[VOIPoTim]

Incoming calls on two VOIPo numbers are not going through. My backup SIP trunks are working now. Just getting DISCONNECT messages now calling VOIPo DID's.

I am on central01 and it is showing UNREACHABLE.

About 10 minutes later the phone starts ringing. No failover. I am BYOD, but it shows unreachable or high latency on central01. Tried opening a ticket but not sure if it made it.

Also cannot call out on VOIPo trunk.

Central01 is not in The Planet. Can you post a traceroute?

[Xponder1]

I checked The Planets own forums and the last update posted a moment ago says "We may have a lead, specific to a certain subnet range. This is still unconfirmed, but the network guys are all on the conference bridge working on different parts."

So in English they dunno what is wrong lol.

I still can not get Google to load. Off topic but is anyone else able to pull up google.com?

[VOIPoTim]

I checked The Planets own forums and the last update posted a moment ago says "We may have a lead, specific to a certain subnet range. This is still unconfirmed, but the network guys are all on the conference bridge working on different parts."

So in English they dunno what is wrong lol.

I still can not get Google to load. Off topic but is anyone else able to pull up google.com?

A specific subnet? :)

Kind of ironic that that would affect multiple datacenters with thousands of servers in hundreds of IP blocks. That's just my opinion though :)

[Xponder1]

A specific subnet? :)

Kind of ironic that that would affect multiple datacenters with thousands of servers in hundreds of IP blocks. That's just my opinion though :)

Ya thats why I threw in the comment about them not knowing what is wrong.

I was bored so I did a tracert for you note the part in red.

Tracing route to Central01.voipwelcome.com [67.228.251.106]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms Access_Denied [192.168.10.1]
2 18 ms 6 ms 8 ms 10.33.128.1
3 9 ms 9 ms 11 ms COX-68-12-8-157-static.coxinet.net [68.12.8.157]

4 10 ms 8 ms 7 ms ge-7-0-1-mx02.mtc1.tul.ok.cox.net [68.12.18.246]

5 9 ms 8 ms 9 ms xe-0-3-0-dsr02.rd.tul.ok.cox.net [68.12.18.4]
6 15 ms 16 ms 14 ms dalsbbrj01-ae0.r2.dl.cox.net [68.1.0.142]
7 158 ms 15 ms 203 ms po-2.r03.dllstx09.us.bb.gin.ntt.net [129.250.4.3
8]
8 15 ms 15 ms 17 ms te1-1.cer01.dal01.dallas-datacenter.com [157.238
.228.10]
9 17 ms 15 ms 16 ms po1.dar02.dal01.dallas-datacenter.com [66.228.11
8.203]
10 38 ms 16 ms 40 ms po2.fcr04.dal01.dallas-datacenter.com [66.228.11
8.218]
11 16 ms 17 ms 17 ms central01.voipwelcome.com [67.228.251.106]

Trace complete.

[Dave_N]

Are you using BYOD?

If you're having a service outage, please contact support for assistance.

As for vPanel, please see this announcement: http://forums.voipo.com/showthread.php?p=11931#post11931

No, I am not using BYOD. I am not at home but if I dial my home number it is now being answered by voice mail. It was Ring no answer.

[VOIPoTim]

No, I am not using BYOD. I am not at home but if I dial my home number it is now being answer by voice mail. It was Ring no answer.

Go ahead and open a ticket or give us a call so we can look into this for you. Everything should be normal at this time.

[Dave_N]

No, I am not using BYOD. I am not at home but if I dial my home number it is now being answered by voice mail. It was Ring no answer.

And I can get to vpanel now and it show my devices connected.

[zcnkac]

I can also confirm a blip on central01 between 3:05 PM and 3:15 PM PDT when my asterisk had to re-attempt registrations because of timeouts. My traceroute is,

traceroute to central01.voipwelcome.com (67.228.251.106), 30 hops max, 40 byte packets
1 unknown (192.168.1.1) 3.673 ms 3.657 ms 3.640 ms
2 * * *
3 te-3-4-ur03.santaclara.ca.sfba.comcast.net (68.85.190.249) 26.345 ms 26.344 ms 26.330 ms
4 te-8-2-ur01.santaclara.ca.sfba.comcast.net (68.87.226.74) 26.315 ms 27.291 ms 27.275 ms
5 be-30-ar01.sfsutro.ca.sfba.comcast.net (68.87.226.185) 30.355 ms 30.344 ms 30.327 ms
6 pos-1-8-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.91.229) 31.318 ms 23.092 ms 20.929 ms
7 pos-0-8-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.85.82) 31.414 ms 31.398 ms 31.384 ms
8 pos-0-14-0-0-cr01.dallas.tx.ibone.comcast.net (68.86.85.142) 66.382 ms 66.373 ms 71.508 ms
9 softlayer-cr01.dallas.tx.ibone.comcast.net (75.149.228.34) 71.495 ms 68.406 ms 69.397 ms
10 po2.dar02.dal01.dallas-datacenter.com (66.228.118.207) 69.383 ms 69.410 ms 70.981 ms
11 po2.fcr04.dal01.dallas-datacenter.com (66.228.118.218) 65.547 ms 67.151 ms 67.109 ms
12 central01.voipwelcome.com (67.228.251.106) 67.097 ms 58.515 ms 65.943 ms

[Xponder1]

Tim could a Vpanel outage cause other servers to have a problem even if for just a short time. It's kind of odd how the moment "The Planet" went down I was unable to call out and others had issues too.

[scott2020]

During the time when I could not make or receive calls, I was able to PING and Trace to central01 without any problems. However I got many registration errors on my asterisk box. I'll look in the log file and see what it thought was wrong.

Several of these in my log files around 17:12 central time:

[2009-05-12 17:11:09] NOTICE[2911] chan_sip.c: -- Registration for 'mydidnumber@central01.voipwelcome.com' timed out, trying again (Attempt #2)

Seems like maybe a bigger Dallas based outage or something?

[voipinit]

Is VPanel still having issues? I can't get into the 'contacts' panel (under the features tab). Anyone else?

[burris]

I checked The Planets own forums and the last update posted a moment ago says "We may have a lead, specific to a certain subnet range. This is still unconfirmed, but the network guys are all on the conference bridge working on different parts."

So in English they dunno what is wrong lol.

I still can not get Google to load. Off topic but is anyone else able to pull up google.com?

I have ATT DSL and that the internet was never a problem.

VOIPo had me on East and I went down. I moved to Central and all is well..

[Xponder1]

Is VPanel still having issues? I can't get into the 'contacts' panel (under the features tab). Anyone else?


I just loaded contacts with no issue.

[Xponder1]

I have ATT DSL and that the internet was never a problem.

VOIPo had me on East and I went down. I moved to Central and all is well..

Well I promise you that the outage was real.

http://forums.theplanet.com/index.php?showtopic=91526&pid=601238&st=60&#entry601238

[eagle 1]

I also show registration failed from 17:52 to 18:12 est for central01. everything fine now
also timeout for registration from 14:12 to 14:21, then again 14:26 to 14:42
Tracing route to central01.voipwelcome.com [67.228.251.106]
over a maximum of 30 hops:

1 10 ms 8 ms 7 ms wsip-68-99-159-33.ri.ri.cox.net [68.99.159.33]
2 8 ms 7 ms 8 ms clvdcmtb04.cl.ri.cox.net [98.173.132.5]
3 9 ms 6 ms 8 ms ip98-173-132-97.cl.ri.cox.net [98.173.132.97]
4 16 ms 15 ms * chgobbrj01-ae0.r2.ch.cox.net [68.1.0.222]
5 16 ms 18 ms 16 ms ae-0.r21.chcgil09.us.bb.gin.ntt.net [129.250.3.98]
6 42 ms 38 ms 43 ms as-1.r21.dllstx09.us.bb.gin.ntt.net [129.250.3.17]
7 * 45 ms 39 ms po-3.r03.dllstx09.us.bb.gin.ntt.net [129.250.4.182]
8 43 ms 41 ms 37 ms te1-1.cer01.dal01.dallas-datacenter.com [157.238.228.10]
9 79 ms 210 ms 42 ms po1.dar01.dal01.dallas-datacenter.com [66.228.118.201]
10 * 43 ms 38 ms po1.fcr04.dal01.dallas-datacenter.com [66.228.118.214]
11 39 ms 42 ms 39 ms central01.voipwelcome.com [67.228.251.106]

Trace complete.

[VOIPoTim]

We're looking into why some people still had a failure. It looks like a few dozen people went down during that time.

We'll post more info later after we've looked into it more.

[tritch]

There are still lingering problems with vPanel. Contacts are not showing, E911 is missing data, Support tab shows "This feature is currently under development."

[wishfull1]

I'm seeing the same issues as tritch.


There are still lingering problems with vPanel. Contacts are not showing, E911 is missing data, Support tab shows "This feature is currently under development."

[VOIPoTim]

There are still lingering problems with vPanel. Contacts are not showing, E911 is missing data, Support tab shows "This feature is currently under development."

It sounds like you have a cached DNS entry. Can you confirm what IP secure.voipo.com is going to for you?

[tritch]

75.126.21.109-static.reverse.softlayer.com


Full trace route:

Tracing route to secure.voipo.com [75.126.21.109]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 7 ms 7 ms 6 ms tx-67-76-237-65.sta.embarqhsd.net [67.76.237.65]
3 7 ms 7 ms 7 ms tx-76-4-42-29.sta.embarqhsd.net [76.4.42.29]
4 15 ms 13 ms 14 ms ge-6-12.car1.Houston1.Level3.net [4.78.8.13]
5 15 ms 15 ms 15 ms ae-2-5.bar1.Houston1.Level3.net [4.69.132.230]
6 14 ms 14 ms 18 ms ae-13-13.ebr1.Dallas1.Level3.net [4.69.137.138]
7 14 ms 13 ms 17 ms ae-61-61.csw1.Dallas1.Level3.net [4.69.136.122]
8 14 ms 14 ms 13 ms ae-1-69.edge3.Dallas1.Level3.net [4.68.19.8]
9 14 ms 16 ms 16 ms te2-1.cer01.dal01.dallas-datacenter.com [4.71.198.18]
10 16 ms 14 ms 15 ms po1.dar02.dal01.dallas-datacenter.com [66.228.118.203]
11 15 ms 14 ms 14 ms po2.fcr01.dal01.dallas-datacenter.com [66.228.118.158]
12 15 ms 14 ms 14 ms 75.126.21.109-static.reverse.softlayer.com [75.126.21.109]

Trace complete.

[VOIPoTim]

Looks like you had DNS cached routing you to Softlayer instead of The Planet from failover, but that wasn't supposed to happen actually since we hadn't fully migrated vPanel out of The Planet.

So it was actually hitting Softlayer in your case and accessing an older version.

Can you try it now?


75.126.21.109-static.reverse.softlayer.com


Full trace route:

Tracing route to secure.voipo.com [75.126.21.109]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 7 ms 7 ms 6 ms tx-67-76-237-65.sta.embarqhsd.net [67.76.237.65]
3 7 ms 7 ms 7 ms tx-76-4-42-29.sta.embarqhsd.net [76.4.42.29]
4 15 ms 13 ms 14 ms ge-6-12.car1.Houston1.Level3.net [4.78.8.13]
5 15 ms 15 ms 15 ms ae-2-5.bar1.Houston1.Level3.net [4.69.132.230]
6 14 ms 14 ms 18 ms ae-13-13.ebr1.Dallas1.Level3.net [4.69.137.138]
7 14 ms 13 ms 17 ms ae-61-61.csw1.Dallas1.Level3.net [4.69.136.122]
8 14 ms 14 ms 13 ms ae-1-69.edge3.Dallas1.Level3.net [4.68.19.8]
9 14 ms 16 ms 16 ms te2-1.cer01.dal01.dallas-datacenter.com [4.71.198.18]
10 16 ms 14 ms 15 ms po1.dar02.dal01.dallas-datacenter.com [66.228.118.203]
11 15 ms 14 ms 14 ms po2.fcr01.dal01.dallas-datacenter.com [66.228.118.158]
12 15 ms 14 ms 14 ms 75.126.21.109-static.reverse.softlayer.com [75.126.21.109]

Trace complete.

[tritch]

Just checked....Contacts and Support tab are working now. E911 still missing data.

Still going to same IP.

[VOIPoTim]

Just checked....Contacts and Support tab are working now. E911 still missing data.

E911 information is not stored on our end. It's all with a third-party E911 vendor. So when you click that in vPanel it queries their database. Since the failover server your vPanel went to is not really a production server, it looks like its IP is not authorized with them to do the ookup so no info is returned.

Once your DNS cache clears and you're back on the production one, you'll be able to see it in there.

[scott2020]

We're looking into why some people still had a failure. It looks like a few dozen people went down during that time.

We'll post more info later after we've looked into it more.

That is the response I received from support. It was something along the lines of some people not being able to reach VOIPo's centralized servers. It is strange that some people were taken out by something at The Planet when we are all moved away from it.

[wishfull1]

All is well for me now.