Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > HostDime is out
HostDime is out
| Posted by Mix, 08-03-2009, 12:26 PM |
Sigh 
Apparently an APC hard reset the whole datacenter? |
| Posted by proclanh, 08-03-2009, 12:33 PM |
|
Yeah thats odd 2 of my severs are up and my main one is down. |
| Posted by proclanh, 08-03-2009, 12:35 PM |
|
There phone service is down too now mine is ringing. |
| Posted by Mix, 08-03-2009, 12:36 PM |
|
I just talked to Miguel on the phone and he said there was a fire? He's not sure all the details yet but he was running into the NOC, and promised me a call back. |
| Posted by Mix, 08-03-2009, 12:52 PM |
|
Manny is telling me that a transformer exploded in a nearby building, and people may be dead. The fire department made them shut down. Everything should be coming up now he says. |
| Posted by Thomas, 08-03-2009, 12:53 PM |
|
Down from several locations still. I guess they will slowly bring it all back up! |
| Posted by larry2148, 08-03-2009, 12:53 PM |
|
Wow, hope for the best. What city is their DC located in? |
| Posted by Mix, 08-03-2009, 01:09 PM |
Quote:
|
Originally Posted by larry2148
Wow, hope for the best. What city is their DC located in?
|
Orlando, Fl |
| Posted by eshutter, 08-03-2009, 04:10 PM |
|
Down 4 hours and counting |
| Posted by Maupinf, 08-03-2009, 04:38 PM |
|
Still down here as well. |
| Posted by stablehost, 08-03-2009, 04:50 PM |
|
Why didn't the generators kick in, once you turned off power? |
| Posted by eshutter, 08-03-2009, 04:50 PM |
|
We're back up here. Total service downtime = 4h24m. Last outage like this I experienced was May 2008 which lasted >6 hours. |
| Posted by Maupinf, 08-03-2009, 04:56 PM |
|
Just popped back up as well. |
| Posted by evilelmo, 08-03-2009, 07:57 PM |
Quote:
|
Originally Posted by nerdie
Why didn't the generators kick in, once you turned off power?
|
Excellent question. Let's see if Mr. HostDime is as eager to answer this. Something doesn't quite add up with the "We had to shut down as a pre-cautionary measure" explanation insofar as some of their NOC racks remained up, powered, and accessable during the entire outage event.
From http://forums.hostdime.com/showthread.php?t=7780 :
Quote:
How Long to the Real Cause of Today's Problems Comes Out?
I understand there was an incident regarding the data center external power supply, but how does a facility with its own back up power generator and UPS, have a huge issue when their external power is disrupted, and how does Hostdime plan to prevent this occurrence in the future. Did someone forget to pay the fuel bill or not put in the proper disconnection switch to remove the data center from the grid?
Im running out of fingers on a hand to count the number of major power issues in the data center in less than two years.
|
|
| Posted by subigo, 08-03-2009, 08:21 PM |
I asked about their backup generator in the past when this happened and I never received a response.
Also, we're going on 12 hours now and a lot of my servers are still slow as hell. Their main site is currently down and the packet loss to ANY server in their datacenter is very, very high. |
| Posted by Maupinf, 08-03-2009, 08:42 PM |
|
Seeing that as well. Almost every refresh of my site results in "The connection was reset" Firefox error. |
| Posted by Paul, 08-04-2009, 01:45 AM |
subigo,
I was speaking with you on aim. The last report I got was "yep, they all look good again". Are you seeing issues again?
Maupinf,
If you are seeing issues. Can you send me a pm and I'll contact you personally? |
| Posted by kenbiz, 08-04-2009, 02:18 AM |
My server was down for more than 12 hours and submitted a few tickets without any resolution.
What's going on with hostdime?
Really a big disappointment seeing this happen on a data center and as wondering how come power generator backup never switch over.
I had a big question mark on this ??? |
| Posted by HostcrateCEO, 08-04-2009, 06:05 AM |
|
wow that is crazy and what is even crazier is that we live about an hour from Orlando too so im surprised i didn't hear it on the local news. |
| Posted by Maupinf, 08-04-2009, 09:26 AM |
Paul, this morning things look good. No connective issues so far.
Would be curious as well why the backup generators weren't used yesterday to keep the data center running if external power was cut/had to be cut. |
| Posted by Mix, 08-04-2009, 09:44 AM |
Quote:
|
Originally Posted by nerdie
Why didn't the generators kick in, once you turned off power?
|
They did kick in. There was a guy on a transformer so the fire department to be safe made the data center kill the generator in case there was back feed while attempting the rescue. |
| Posted by HostDime, 08-04-2009, 03:08 PM |
|
As Mix said above, a freak incident was a chain effect to something many didn't have control over. We are glad no one was hurt and the person, who was mentally ill didn't die. We sincerely apologize for any downtime you may experienced. I assure we had 25+ staff working around the clock to get things back to normal. We will happily apply any credit based on our SLA http://www.hostdime.com/policies/sla/ please file a ticker or a call so we can apply this for you. |
| Posted by Sekweta, 08-04-2009, 05:28 PM |
Quote:
|
Originally Posted by Mix
They did kick in. There was a guy on a transformer so the fire department to be safe made the data center kill the generator in case there was back feed while attempting the rescue.
|
So does that mean anytime there are linemen working on the poles, such as repairing downed cables after a storm, the DC (and anyone else in the area with generators) will have to shut down?
Something just doesn't smell right about that. |
| Posted by actnow, 08-04-2009, 05:57 PM |
Quote:
|
Originally Posted by HostDime
As Mix said above, a freak incident was a chain effect to something many didn't have control over.
|
Why does it feel like -
"teacher, the dog ate my homework." |
| Posted by HostDime, 08-04-2009, 08:15 PM |
Quote:
|
Originally Posted by Sekweta
So does that mean anytime there are linemen working on the poles, such as repairing downed cables after a storm, the DC (and anyone else in the area with generators) will have to shut down?
Something just doesn't smell right about that.
|
Perhaps you didn't read the events as they happened. We posted direct links from the media sources a long with videos. When the person from the mental institute was stuck between two transformers after being electrocuted the fired dept shut down and made everyone within the Plaza ( our building is within this plaza) shut down and evacuate as precautionary. The pictures posted taken on site show this as it happened.
http://www.orlandosentinel.com/news/...,5160170.story
http://www.wftv.com/video/20268586/index.html |
| Posted by Sekweta, 08-04-2009, 08:20 PM |
Quote:
|
Originally Posted by HostDime
Perhaps you didn't read the events as they happened. We posted direct links from the media sources a long with videos.
|
Actually, I did. Read the story, watched the video.
Quote:
|
Originally Posted by HostDime
When the person from the mental institute was stuck after being electrified the fired dept shut down and made everyone within the 2 buildings next door shut down and evacuate.
|
I asked that question because I've not heard of grid AND generator power being ordered OFF for a building except when there are environmental hazards such as gas leaks, flooding, etc.
Power feedback onto the utility is a new one on me, especially for a facility that isolates itself from the grid when on generator power. |
| Posted by evilelmo, 08-04-2009, 09:47 PM |
HostDime - Since you are kind enough to reply here, can you give a direct answer to the question many of your customers are now asking:
Why didn't the generators kick in, once you turned off power?
Honestly, trying to get an answer to this question is like pulling teeth. Your customers are concerned - rightly concerned - that HostDime does not have a working system in place to deal with power outages. I remember when Hurricane Charlie hit Orlando in August 2004 and a caused a power outage at DimeNOC. Your generators did not work then either, and HostDime/SurpassHosting customers had to deal with a major and extended service outage as a result.
Put yourself in your customers shoes. It appears (again) that you have no working backup power system in place to keep the servers functioning without hiccups when there's a power failure. |
| Posted by UNIXy, 08-04-2009, 10:13 PM |
How often do extended disruptions like this one happen at HD? Someone mentioned that a similar one took place in May. When did the previous disruption besides may happen and what is the duration? Also, some pointed out that their servers were kept up throughout. How come? Any answers to these questions is appreciated.
Thanks! |
| Posted by HostDime, 08-04-2009, 10:42 PM |
Quote:
|
Originally Posted by evilelmo
HostDime - Since you are kind enough to reply here, can you give a direct answer to the question many of your customers are now asking:
Why didn't the generators kick in, once you turned off power?
Honestly, trying to get an answer to this question is like pulling teeth. Your customers are concerned - rightly concerned - that HostDime does not have a working system in place to deal with power outages. I remember when Hurricane Charlie hit Orlando in August 2004 and a caused a power outage at DimeNOC. Your generators did not work then either, and HostDime/SurpassHosting customers had to deal with a major and extended service outage as a result.
Put yourself in your customers shoes. It appears (again) that you have no working backup power system in place to keep the servers functioning without hiccups when there's a power failure.
|
evilelmo,
There is actually a formal announcement on what happened being posted tonight to clients from a company official which is why nothing has been posted here yet. It was a priority to restore systems as priority as am sure you can appreciate. I do apologize the full explanation hasn't been released yet but it will be tonight.
I am not sure if you realize this but we go on UPS/Generator about 2-3 times a month here half the year in florida due to the summer storms giving brown outs. Do you ever notice issue during those? Of course not because there is systems and place and surely you have seen the videos of it? http://www.hostdime.com/facilities/
If indeed we didn't have a working system in place we would not be able to meet our 99.9% uptime each year which we have met every year since we been in business. No downtime is acceptable however it appears you are shinning on the times there is issues and I do not blame you. We are here to be nothing but perfect and we will continue to strive and do whatever it takes to keep everyone happy and push for 100% uptime year after year. |
| Posted by HostDime, 08-04-2009, 10:52 PM |
Quote:
|
Originally Posted by UNIXy
How often do extended disruptions like this one happen at HD? Someone mentioned that a similar one took place in May. When did the previous disruption besides may happen and what is the duration? Also, some pointed out that their servers were kept up throughout. How come? Any answers to these questions is appreciated.
Thanks!
|
Before this outage it was May 2008 approximately 15 months ago when we had a UPS battery malfunction. The duration of disruption has been less then 1 hour in most cases, although some servers experienced more due to the issues which came about when trying to bring them up. Everyone company has issues, how they are addressed is what makes the issue/mistake into something to be learned from and be better. We are continuously striving to nothing but perfect and we will continue to do this. |
| Posted by Mix, 08-05-2009, 03:21 AM |
Guys,
Without trying to look like a HostDime/DimeNOC cheerleader I was (probably/more than likely) affected a lot worse then most/all of you. I have A LOT of servers in dedicated cabinets with these guys, and I deployed my techs to the building at the same time.
I have only been with these guys for literally a couple of weeks; but they showed a ton of professionalism, and gave us a very experienced Sr. Technician to use at our disposal to get our cabinets online. (Thank You Paul <3)
I am with you guys in being upset about ANY down time but at the same time I would love to see any of you write a contingency plan that involves someone with a mental health disorder climbing a power pole.
The main thing here is that it is VERY easy for us all to play Monday morning quarterbacks; but we were not the technicians that were on staff trying to make decisions while a guy was on the power pole being electrocuted. I think that we should all take a step back and be thankful that a fellow human even survived such an incident. I challenge you as a human being to watch the video - put yourself in the technicians place and be able to make such life changing decisions.
At the end of the day it is my personal feeling whether you share it or not that a human life is far more important than a few minutes or even an hours worth of downtime.
I shared these same thoughts with our customers and the response has been very overwhelming. Here is a link to our announcements page:
http://www.made2own.com/clients/announcements.php?id=4 |
| Posted by Sekweta, 08-05-2009, 08:48 AM |
Quote:
|
Originally Posted by HostDime
the fired dept shut down and made everyone within the Plaza ( our building is within this plaza) shut down and evacuate as precautionary.
|
I'm having problems with this statement. I'm not saying it isn't true becasue I wasn't there. It does make sense that they would cut utility power to rescue someone from amongst the power lines.
What does not add up is the alleged mandatory evacuation of all buildings in the plaza. It was a power outage, not a gas leak with risk of explosion. If anything, they would want people to stay INSIDE the buildings to keep the area clear of spectators.
In fact, that is precisely what was said in the news article-- "Police and fire workers cleared observers out of a rear parking lot at the request of Lakeside Behavioral Healthcare administrators before the final rescue effort began."
[remove spectators from parking lot] != [evacuate buildings] |
| Posted by FastServ, 08-05-2009, 09:03 AM |
|
So any time someone wants to shut down hostdime they climb the utility pole outside the building and sit there? Seems like a major security problem any way you look at it. |
| Posted by UNIXy, 08-05-2009, 09:07 AM |
Quote:
|
Originally Posted by HostDime
Before this outage it was May 2008 approximately 15 months ago when we had a UPS battery malfunction. The duration of disruption has been less then 1 hour in most cases, although some servers experienced more due to the issues which came about when trying to bring them up. Everyone company has issues, how they are addressed is what makes the issue/mistake into something to be learned from and be better. We are continuously striving to nothing but perfect and we will continue to do this.
|
Thanks for taking the time to answer my questions. I hope all is back to normal now. Frankly, I would rather co-locate at a facility that is prepared for storms and flooding as opposed to one that isn't well prepared because it thinks its FEMA 500-years flood plain is safe.
Regards |
| Posted by HostDime, 08-06-2009, 12:04 AM |
Quote:
|
Originally Posted by Sekweta
I'm having problems with this statement. I'm not saying it isn't true becasue I wasn't there. It does make sense that they would cut utility power to rescue someone from amongst the power lines.
What does not add up is the alleged mandatory evacuation of all buildings in the plaza. It was a power outage, not a gas leak with risk of explosion. If anything, they would want people to stay INSIDE the buildings to keep the area clear of spectators.
In fact, that is precisely what was said in the news article-- "Police and fire workers cleared observers out of a rear parking lot at the request of Lakeside Behavioral Healthcare administrators before the final rescue effort began."
[remove spectators from parking lot] != [evacuate buildings]
|
Perhaps this image http://www.hostdime.com/images/Orlando.png will help you understand more. Evacuation was NEVER mandatory but we were advised of what was being done and as a precaution and for the well being of everyone we had over 50% of our staff evacuated and only kept key people in the facility. The proximity of the main issue where this guy was at was so close, as you can see in the picture from an aerial view!, it was pretty close that if an explosion with those two transformers could have caused some serious damage. The newspaper isn't aware of everything that is done or said in these type of media as you know. Perhaps if you were here you would understand what transpired easier. |
| Posted by Sekweta, 08-06-2009, 08:50 AM |
If the concern was the risk of transformer explosion, then it's making a lot more sense now (except why would they be worried about that after the power was cut). Since I was not there, the only information is that which comes from WHT posts, media coverage, and "this doesn't immediately make sense" speculation.
I also agree that authorities often over-react to circumstances out of what they consider an abundance of caution. How often have we seen an interstate highway shut down in both directions after a serious accident, even though the crash was contained to one side, separated by a 30+ foot median? IMO, the risk of transformer explosion was essentially negated when power was cut.
I'm glad to hear your ops are back to normal, and am sorry anytime I see folks inconvenienced and/or incur financial loss because of some criminal, careless moron, or (in this case) mentally ill person, over whom you had no control. It stinks even more if you had the ability to cover the power outage via on-site generator, and weren't permitted to use it. |
Add to Favourites 
Print this Article
Also Read
Enotch down (Views: 1076)
Rackspace down (Views: 1074)
Ezzi.net Down (Views: 896)
Shared Hosting
Shared Hosting
