PDA

View Full Version : The Planet (Datacenter) Network Outage



VOIPoTim
04-02-2009, 09:07 AM
We are seeing a complete network failure in the datacenter at The Planet where the bulk of our core equipment has always been located. They are determining the cause now.

As you know, we announced our plans to migrate out of that facility a few months ago. Due to ongoing network issues this week, we expedited the move last night.

»forums.voipo.com/showthread.php?t=1341 (http://forums.voipo.com/showthread.php?t=1341)

During that maintenance window, we were able to mirror most everything at the new facility.

The Planet is working on this outage as we speak and service should be restored shortly.

Given this new network outage, we've determined that our best possible course of action is to immediately begin re-routing as much traffic to the new facility to take The Planet out of the picture. The recent network issues have been completely unacceptable. There has been at least one every day this week with this one being the largest.

VOIPoTim
04-02-2009, 09:09 AM
The Planet has restored network access at this time.

Due to the chronic nature of this, we're continuing our plans to begin moving all traffic out immediately.

quattrohead
04-02-2009, 10:54 AM
WTF is going on at The Planet ? Are Hostgator customers affected too ?

VOIPoTim
04-02-2009, 05:44 PM
WTF is going on at The Planet ? Are Hostgator customers affected too ?

Yes it is.

VOIPoTim
04-06-2009, 10:57 PM
Just as an update, we're still seeing very short periods of extremely high latency within The Planet.

Just a few minutes ago, it was over 2000ms on the internal network there.

We've been moving things one server at a time out and are continuing to work on this throughout the week.

If this does not clear up shortly, we're going to begin migrating remaining users out tonight to put an end to this.

VOIPoTim
04-06-2009, 11:34 PM
If this does not clear up shortly, we're going to begin migrating remaining users out tonight to put an end to this.

This seems to have cleared up, but it should not be happening in the first place and continues to come up.

We are migrating users all users out tonight and will continue to migrate any remaining components out this week.

HelpNYC
04-07-2009, 03:31 PM
Great work Tim.