Hurricane Electric FMT1 Outage :(

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Hurricane Electric FMT1 Outage :(

Posted by DeltaAnime, 11-21-2010, 12:49 AM
Hi everyone,

I can confirm HE FMT1 just had a power disruption.

The tech I talked to mentioned they were doing some work on their UPS's

Francisco
Posted by CNSERVERS, 11-21-2010, 01:04 AM
all down or partial down?
Posted by qwizie, 11-21-2010, 01:19 AM
Oh, that explained why none of your pages loaded.
Posted by DeltaAnime, 11-21-2010, 01:38 AM
It wasn't just us, the entire DC just lost it's power.

At first I thought it was a peering issue until I found out the HE website was down as well.

We're spinning up everything as quickly as possible on our end, but waiting on HE to finish a bunch of other work :/

Francisco
Posted by Aaronn, 11-21-2010, 02:59 AM
Glad your fixing this!
Posted by cedricd, 11-21-2010, 05:05 AM
Ah, so that's what happened -- had all of my VPSes go down for about 3-4 hours and was quite confused
Posted by Aaronn, 11-21-2010, 06:07 AM
Everything seems quite stable now, and seems fran got everything back running, as i'm idling his IRC to monitor update and speak to him.
Posted by Trix, 11-23-2010, 12:03 PM
Is it down for anyone else (again)?
Posted by OverlordQ, 11-23-2010, 12:16 PM
Yes, seems to be (another) power issue.
Posted by Eric - Zoidial, 11-23-2010, 12:23 PM
Yep, all my equipment rebooted as well - didn't last long, last log was 10:25 prior to power cut, restart of logging was 10:32 (EST).

Is there still bad weather in Fremont, or are they working on UPS/power generators and something went wrong?
Posted by jvasa, 11-23-2010, 01:49 PM
What is causing these outages? It happened at a bad time this morning at 7:30am PST. The entire facility rebooted as it did on Saturday, so it seems they haven't figured out the problem. I hope it doesn't happen again during the holidays. Weather could be an issue. Should I be concered everytime there is a little rain?
Posted by DeltaAnime, 11-23-2010, 06:11 PM
I got word back and it seems it's like this:

- Big thunderstorm and power surge damaged 2x UPS's Saturday night causing outage #1
- HE was forced into 'stand by' mode to get things online and away from the fried UPS's
- PG&E had a power blip this morning which caused outage #2. Since the UPS's are still conked, they had nothing to jump onto
- Replacement parts for the failed UPS's arrived today and is supposedly getting installed as we speak

I know I called them a few hours after the power blip and it went directly to voicemail so i'm assuming they got all hands on deck resolving this.

So far I think Linode has had the worst of it. Frantech & BuyVM have been forced into FSCKing some of our nodes, but they've suffered from physically
damaged boxes it seems

Francisco
Posted by DeltaAnime, 11-23-2010, 08:11 PM
Here is the RFO we've been given by HE for those curious.

Francisco
Posted by LoginTech, 11-23-2010, 09:34 PM

Quote:

Originally Posted by DeltaAnime

Here is the RFO we've been given by HE for those curious.

Francisco

I can't help but wonder if they need a new service company...

We have MGE units (Schneider Electric / APC) and Schneider is able to get parts the same day.. regardless of the part (including weekends).
Posted by DeltaAnime, 11-23-2010, 10:13 PM

Quote:

Originally Posted by LoginTech

I can't help but wonder if they need a new service company...

We have MGE units (Schneider Electric / APC) and Schneider is able to get parts the same day.. regardless of the part (including weekends).

I don't know but at this point we're going to have to plop down a few thousand just to get UPS units in our racks. Both outages were short on the HE side of things, but the FSCK's we had todo were far from easy on us. I'd much rather have the network drop for a couple minutes on their end than our boxes get dropped on their heads like that :/

Either way, i'm not pleased but like everyone else, we have a business to run and clients to protect

Francisco
Posted by romanrm, 11-24-2010, 03:37 AM
Maybe you should consider using JFS as the filesystem, see this article:
linux dot com /archive/feature/119025
The guy tried to kill it by pulling the power cord, but couldn't, and the longest fsck time was 3 seconds))
Posted by woods01, 11-24-2010, 04:21 AM
You probably shouldn't of posted that.

However if a company with 'electric' in it's name can't keep the power on i'd be looking for perhaps a company named "Tornado Outage".

Maybe they believe in N+1 and don't have single points of failure.

Add to Favourites Print this Article

Knowledgebase

Hurricane Electric FMT1 Outage :(

Our Services

Client Menu

Legal