Printable View
We're adding more database servers today.
This should not cause any issues, but I wanted to post ahead of time and let you know that if you see any quirkiness in vPanel, it could be related.
Again, no issues are expected from this and it'll increase stability quite a bit.
Tim,
Is there any particular reason why you don't do this sort of thing during an off-hours maintenance window, like 03:00 Sunday morning, or something like that? Seems to me that you really don't want to me generating more possible complaints these days.
IMHO, no change (unless a sev 1, of course) should be made during production hours.
I wonder if that's the result of the outages many are experiencing now..Quote:
Originally Posted by abward
When we added new database servers to the mix a few minutes ago, there was a table that was not fully synced for some reason and left about 3000 records out for a few minutes.
It is being restored/pushed to all database servers right now and those clients should be back up now.
This was a pretty critical change (will eliminate the latency issues that have come up a lot lately).Quote:
Originally Posted by abward
Some people don't agree with the timing, but I don't want to do something at night and then risk something happening when support is not fully staffed to handle the flood of calls.
The main reason we do this stuff during the day is so that our support center is fully staffed and available if issues do come up.
I think the results today speak for themselves. I rest my case, but I won't rub it in anymore.
Cut them some slack. Tim doesn't have to post anything in the forums about it. At least they make announcements so people know what is going on, instead of just doing something and denying anything has changed if there are problems.
Sorry, but no slack. As a paying customer, and a guy in the IT business for 27 years, I expect maintenance windows (potential outages) to occur during off hours, and I expect (of course hope against) only sev 1 or unplanned outages during normal hours.
That is just the way it is done, period.
There is insufficient amount of paper in the world to list the number of times when IT people go make a change that is not supposed to break anything, but does anyway. I even periodically do this wonderful practice myself! This is why, repeat after me, we never change production systems during normal operational hours.
Amen, brother.Quote:
Originally Posted by abward
My telephone is needed 24/7. I never know when I will need it or when an emergency will come up. My telephone doesn't know banker's hours. I know I do as much maintenance after hours as possible, but sometimes the benefit of making changes, all things considered, can outweigh the risk. With VOIPo being relatively new with a very small staff of tech people, and since I am saving lots over POTS, I am willing to cut them some slack, paying or not.Quote:
Originally Posted by abward
99.5% of people (maybe even you) probably would not have known the difference on any database problems, yet Tim found it appropriate to post that information in a public forum. I give him a lot of credit for that.
Agreed. Time to move on to true enterprise "production" of services. Change windows need to be planned around "your time", not customers' time. Like the idea of lots of info to customers, but, time to minimize downtime or even potential down time to customers -- which I am one of.
"Beta" is one thing. I would think production would be 99.5% uptime (no, not 99.9% or 99.99%)
Keep up the good work -- please condsider your maintenenace windows after hours.
-bird
OK, I understand your reason for making changes like this during the day. However, abward brings up an excellent point--more people are going to be using their phone at 4pm on a weekday than at 3am on a Sunday. The "flood" of calls, therefore, would more likely (than at 4pm on a weekday) be a "trickle".Quote:
Originally Posted by VOIPoTim
Scott2020 also brings up a good point that the phone should work 24/7. After all, you never know when an emergency call will need to be made. Well, you do have to perform your maintenance sometime, and when your call volume is the lowest is the best time for it to be done, even if your support staff is all sleeping (OFF the clock, of course! :D ).
Perhaps the best way to handle these potential issues is to have a maintenance recording for your support line. In the case of this database server addition/reconfiguration, it could have said, "On Sunday March 15 at 3AM Central time, Voipo began scheduled maintenance. These activities were anticipated to be complete by 4AM and should not affect anyone's service. If your service is not working, we apologize for any inconvenience, and we invite you to submit a ticket by typing in your voipo number now. We will restore your service as soon as we can."
You could then log all tickets submitted at that time so you can be sure any affected customers (who reported problems) are back online.
Good thoughts. My ISP often has recorded messages on their support line.
"If your service is in Livonia, WideOpenWest is aware of a cable service issue ..."
Agree................
Quote:
Originally Posted by VOIPoTim