iWeb CL Datacenter Down

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > iWeb CL Datacenter Down

Posted by tamouh, 11-04-2010, 12:05 AM
Accroding to iWeb Blog:

http://blog.iweb.com/en/2010/11/inci...comment-258787

"Incident: Power distribution Data center CL

Start: ~11:00PM EDT

Estimated time of resolution: N/A

We are currently investigating a situation that might affect a segment of our electrical alimentation infrastructure on data center CL Our technical team is actively working on finding a solution to this problem. We will regularly communicate with you about the status of this operation.

We apologize for any inconvenience this situation may cause. If you have any questions, please do not hesitate to contact our support team (http://www.iweb.com/contacts/). Thank you for your patience and understanding.
"
Posted by tamouh, 11-04-2010, 12:19 AM
Problem resolved, most servers appear back up.
Posted by iWeb-Foo, 11-04-2010, 12:21 AM
Hi,

Please note that as of 00:07AM EST the power is back.

An update has been posted on our blog:
http://blog.iweb.com/en/2010/11/inci...r-cl/5978.html

We are currently investigating the root cause of the issue and will be able to provide a complete report. For the time being, if any of your servers are still unreachable, please contact our support team by opening a server outage ticket through your customer hub.
Posted by JohnnyUtah, 11-04-2010, 09:53 AM
The CL datacenter is a piece o crap! Its out in the east end of Montreal. That part of the city is run down. I doubt they have backup power over there... The Nuns Island Datacenter is far better in my books.
Posted by The Universes, 11-04-2010, 01:51 PM

Quote:

Originally Posted by JohnnyUtah

The CL datacenter is a piece o crap! Its out in the east end of Montreal. That part of the city is run down. I doubt they have backup power over there...

That actually doesn't appear to be the case, from their blog:

Quote:

What happened?

Around 11pm, a fire near the iWEB-CL datacenter caused power disruptions. As electrical power was unstable, all power was automatically switched to generator power. The power transfer went normally for two our of three generators that power iWEB-CL when electrical power is not available. The third generator started but the transfer switch did not transfer power to the generator as it should have. We suspect the transfer switch control logic has been disrupted by the kind of power outage (the fact that it was related to a fire nearby) we are still investigating to find out why the power has not been transfered to the third generator.

Generator tests were completed successfully yesterday (Tuesday 2/2010) without load and successfully with full load last week (26/10/2010) with no indications of any potential problem. We run generator tests each week in order to prevent unplanned issues.

Posted by atchoooo, 11-09-2010, 11:11 PM

Quote:

I doubt they have backup power over there... The Nuns Island Datacenter is far better in my books.

St-Leonard and Nun's Island are different facilities. Nuns Island is N+1 and colocation only, CL is mostly dedicated. CL in St-Leonard is fully redundant with 3 x 1.5MW generators and it is a very well designed power redundant high density datacenter (for the dedicated server part) that holds 12,000 dedicated servers.
Posted by JohnnyUtah, 11-09-2010, 11:19 PM

Quote:

Originally Posted by atchoooo

St-Leonard and Nun's Island are different facilities. Nuns Island is N+1 and colocation only, CL is mostly dedicated. CL in St-Leonard is fully redundant with 3 x 1.5MW generators and it is a very well designed power redundant high density datacenter (for the dedicated server part) that holds 12,000
dedicated servers.

I guess thats why it went down then? The city cutting power should have no effect on your dstacenter right?
Posted by tamouh, 11-10-2010, 01:06 AM
@JohnnyUtah: Instead of unknowingly ranting and making false presumptions, why don't you go down to the CL datacenter and find out for yourself! Heck, I've been to both CL & HD DCs in Montreal and have seen the power generators. Further, I've actually been present at one time when they were a daily test in the morning at HD DC.

Just to add: Datacenters do go down, not often, but every 3 years or so you can expect either a power or network issue to occur. In my opinion iWeb DC runs very solid.
Posted by vtjballeng, 11-11-2010, 02:08 PM
When the CL datacenter went down (late on Weds Nov 3rd), our web cluster at iweb went down with it. Apparently an APC backup failed during the problem. Despite having their level 5 (highest available) managed service, they did not bring our cluster back up. We had to alert them on Monday morning (Nov 8th) before they brought our cluster back up. This, coupled with a few other issues has already put us below two 9 reliability in 2010. We will likely be moving from iweb.

I could accept the datacenter downtime and their explanation for the problem. What I cannot accept is our cluster going down and not being brought back to life until we asked. This despite us paying a hefty premium for their level 5 managed cluster as well as adding other boxes and paying heavily to maintain the highest level of server management with them. Who knows how long our cluster would have been down had we not alerted them specifically?

We have spoken to them about the issue, they have not yet determined an action to take and we are looking into other hosts. We do have boxes at other companies that have had 99.999% or better and we even have a colo box at another iweb facility that has been very reliable. For our part, we are now questioning the pricing for "managed" servers/clusters as there seems to be little benefit for the significant additional cost.
Posted by Scott.Mc, 11-11-2010, 02:22 PM

Quote:

Originally Posted by vtjballeng

When the CL datacenter went down (late on Weds Nov 3rd), our web cluster at iweb went down with it. Apparently an APC backup failed during the problem. Despite having their level 5 (highest available) managed service, they did not bring our cluster back up. We had to alert them on Monday morning (Nov 8th) before they brought our cluster back up. This, coupled with a few other issues has already put us below two 9 reliability in 2010. We will likely be moving from iweb.

I could accept the datacenter downtime and their explanation for the problem. What I cannot accept is our cluster going down and not being brought back to life until we asked. This despite us paying a hefty premium for their level 5 managed cluster as well as adding other boxes and paying heavily to maintain the highest level of server management with them. Who knows how long our cluster would have been down had we not alerted them specifically?

We have spoken to them about the issue, they have not yet determined an action to take and we are looking into other hosts. We do have boxes at other companies that have had 99.999% or better and we even have a colo box at another iweb facility that has been very reliable. For our part, we are now questioning the pricing for "managed" servers/clusters as there seems to be little benefit for the significant additional cost.

Just out of interest why did you wait 5 days to notify them you were down? I am not excusing them in any way at all, I am simply curious.
Posted by vtjballeng, 11-11-2010, 05:17 PM

Quote:

Originally Posted by Scott.Mc

Just out of interest why did you wait 5 days to notify them you were down? I am not excusing them in any way at all, I am simply curious.

I was out of the office and we had not completed our migration to this cluster so it was not an active / in use box for us. Some of our programmers noticed it went down but did not choose to alert me because I already have enough to deal with and because it did not affect any critical systems. When I got back to the office on Monday and looked through my email, I saw that it went down and investigated before asking iweb to take action. Had this been a production box, we would have been monitoring it and created a ticket quickly as our staff would have been notified on their cell phones.
Posted by Scott.Mc, 11-11-2010, 07:48 PM

Quote:

Originally Posted by vtjballeng

I was out of the office and we had not completed our migration to this cluster so it was not an active / in use box for us. Some of our programmers noticed it went down but did not choose to alert me because I already have enough to deal with and because it did not affect any critical systems. When I got back to the office on Monday and looked through my email, I saw that it went down and investigated before asking iweb to take action. Had this been a production box, we would have been monitoring it and created a ticket quickly as our staff would have been notified on their cell phones.

Right I was just going on the uptime mention but I see you mean for this cluster at iweb. Thanks for the info and sorry for poking my nose in, but it did make me wonder

Add to Favourites Print this Article

Knowledgebase

iWeb CL Datacenter Down

Our Services

Client Menu

Legal