Page 1 of 9 123 ... LastLast
Results 1 to 10 of 89

Thread: Outage - 1AM Central

  1. #1
    Join Date
    Feb 2007
    Location
    Irvine CA
    Posts
    1,542,128,044

    Default Outage - 1AM Central

    When it rains, it pours. As much as I hate to say it, we had another outage right now.

    The datacenter which we have some of our core equipment in (along with many thousands of HostGator servers) had a network switch that went down for scheduled maintenance tonight. Our DNS servers are both in that zone so some users undoubtedly ran into issues connecting.

    Their scheduled maintenance was communicated to HostGator and HostGator communicated it to affected customers, but there was a major major breakdown in communication in that it wasn't communicated to us since we're not really a HostGator customer. We're increasing the communication between the two companies so this never happens again.

    Once we realized it was down and found out about this maintenance, we were able to get things back online very quickly.

    It looks like nearly all PAP2 users reconnected immediately and nearly all Grandstream HT502 users directly connected to a modem reconnected almost immediately. Unfortunately, in terms of Grandstream users behind a router/NAT, it seems like about 90% of them reconnected at this point and 10% did not. If you are one of those users (a few hundred) you may need to reboot.

    We'll be sending out a system-wide e-mail shortly so everyone is aware of the situation. We apologize for any inconvenience has caused and we're confident that these outages will become less frequent.
    Last edited by VOIPoTim; 03-04-2009 at 02:48 AM.
    Timothy Dick
    Founder/CEO
    VOIPo.com

    Interact with VOIPo: Twitter, Facebook

  2. #2
    Join Date
    Feb 2007
    Location
    Irvine CA
    Posts
    1,542,128,044

    Default Re: Outage - 1AM Central

    Since network blips, etc will always occur with a service that uses the public internet, it's critical that we get the remaining Grandstream adapters used behind a NAT to ALL be re-registering consistently.

    Again, we're now shipping PAP2s out by default and only sending the HT502s out by request so in the future this would not come up.

    We're evaluating options for dealing with the 10% of so of our HT502s that are still not re-registering. We've considered possible replacing them all with PAP2s but are working with Grandstream to try to exhaust other options before doing this. They've asked us to give them a week and they're going through logs. We're going to let them work on it and will continue to evaluate all options for dealing with them.

    Again virtual all PAP2 users and Grandstream HT502 users NOT behind a router will automatically re-register and these outages will be self-correcting.

    It's the Grandstream HT502s behind a router that still have about 10% not doing it.
    Timothy Dick
    Founder/CEO
    VOIPo.com

    Interact with VOIPo: Twitter, Facebook

  3. #3
    Join Date
    Apr 2008
    Location
    Aventura Fl
    Posts
    860

    Default Re: Outage - 1AM Central

    Yep..

    Just got up and found both my PAP2s online as you described, with no problems.

  4. #4
    Join Date
    Dec 2008
    Location
    Tulsa, Oklahoma
    Posts
    538

    Default Re: Outage - 1AM Central

    Thank you for the heads up. I am one of the few it seems.
    "You do not currently have any devices registered to our servers! "

    Thanks Tim,

  5. #5
    Join Date
    Dec 2008
    Posts
    200

    Default Re: Outage - 1AM Central

    I lost registration overnight. It did not reconnect. I had to reboot.

    Is it a problem with the GS firmware? Why is it stopping the retrys?

  6. #6
    Join Date
    Feb 2007
    Location
    Michigan
    Posts
    2,220

    Default Re: Outage - 1AM Central

    You guys who lost registration are behind NAT?
    (Mine did great behind the Cable Modem.)


    Using VOIPo services since February 2007
    Beta Tested the VOIPo Reseller Plan.
    A happy VOIPo Residential Customer

    Using VoIP devices since 12-2002
    Companies I've tried
    iConnectHere|Vonage|BroadvoxDirect|Vonage|Packet8| VOIPo
    VOIPo is a keeper!


  7. #7
    Join Date
    Dec 2008
    Posts
    200

    Default Re: Outage - 1AM Central

    usa2k,

    I am behind the router. But it still doesn't explain why the GS 502 is stopping the re-registration tries. Unless there is a problem with the new firmware. With the old firmware I was running flawlessly for 5-6 weeks without any issues (that was during Dec-Jan month).

  8. #8

    Default Re: Outage - 1AM Central

    So this affected devices even using the updated DNS servers 156.154.70.1 or 156.154.71.1?

    FYI, best practices for deploying DNS servers is to have them different subnets (which it appears have been done) and geographically or logically different network paths than the primary. If a single switch took down both of your DNS servers, then something should be done to resolve this issue, otherwise, when an unscheduled outage happens with that same switch, the same thing will happen again.

  9. #9
    Join Date
    Jan 2009
    Posts
    14

    Default Re: Outage - 1AM Central

    I have a 502, and I have it behind my router, count me in as one of the few who lost their registration.

  10. #10
    Join Date
    Dec 2008
    Posts
    200

    Default Re: Outage - 1AM Central

    Quote Originally Posted by bubbanc View Post
    So this affected devices even using the updated DNS servers 156.154.70.1 or 156.154.71.1?
    Yes. My adapter is using these DNS servers. From what Tim has said, it looks like the DNS servers were unavailable during the outage.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •