Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > FDC Denver Outage
FDC Denver Outage
| Posted by Ilya74, 12-05-2013, 11:05 AM |
|
Looks like FDC Denver location is down. Anyone experiencing the issue? |
| Posted by rnavarro, 12-05-2013, 11:31 AM |
I have some servers colo'd there and they shutdown because of thermal overload....they were reporting 114F+ on the ambient temp....
I wonder if their CRAC shut down |
| Posted by Ilya74, 12-05-2013, 11:33 AM |
Did you get any response from techs? I sent two tickets, one to Denver 1 hour ago and one to Chicago. Chicago tech moved ticket to Denver, then Azam from Denver moved it to Chicago. However, I still have nothing in both tickets.
Btw, Denver looking glass is UP... |
| Posted by rnavarro, 12-05-2013, 11:33 AM |
|
I have not, opened an emergency ticket in Denver 14 minutes ago asking what's going on. |
| Posted by BeZazz, 12-05-2013, 11:34 AM |
|
I have 2 servers coloed at Denver and they are both fine. |
| Posted by rnavarro, 12-05-2013, 11:35 AM |
Quote:
|
Originally Posted by BeZazz
I have 2 servers coloed at Denver and they are both fine.
|
Can you check the ambient temperatures for the two boxes you have colo'd? Via IPMI or the OS?
Just want to compare notes. |
| Posted by Ilya74, 12-05-2013, 11:39 AM |
Still down, 1 hour.
I am wondering why techs are not responding, they are there for sure and they saw my tickets. |
| Posted by rnavarro, 12-05-2013, 11:50 AM |
Just got a response to my emergency ticket:
Quote:
There is outage in denver datacenter, we are working on restoring service, that
you for your patience.
|
|
| Posted by BeZazz, 12-05-2013, 11:58 AM |
Quote:
|
Originally Posted by rnavarro
Can you check the ambient temperatures for the two boxes you have colo'd? Via IPMI or the OS?
Just want to compare notes.
|
Here is the best I can do.
Server 1
[root@main ~]# sensors
k8temp-pci-00c3
Adapter: PCI adapter
Core0 Temp: +61.0°C
Core1 Temp: +62.0°C
k8temp-pci-00cb
Adapter: PCI adapter
Core0 Temp: +54.0°C
Core1 Temp: +55.0°C
Server 2
(Been running scripts flat out on it for hours)
[root@host ~]# sensors
k8temp-pci-00c3
Adapter: PCI adapter
Core0 Temp:
+72°C
Core1 Temp:
+72°C
k8temp-pci-00cb
Adapter: PCI adapter
Core0 Temp:
+68°C
Core1 Temp:
+71°C |
| Posted by Ilya74, 12-05-2013, 12:01 PM |
|
rnavarro, can you please check where exactly your servers located? I mean rack ID, row ID or VLAN ID. Maybe we are located in the same segment and the segment is experiencing power/networking/cooling issue. |
| Posted by rnavarro, 12-05-2013, 12:05 PM |
Quote:
|
Originally Posted by Ilya74
rnavarro, can you please check where exactly your servers located? I mean rack ID, row ID or VLAN ID. Maybe we are located in the same segment and the segment is experiencing power/networking/cooling issue.
|
PM'd . |
| Posted by rnavarro, 12-05-2013, 12:09 PM |
@BeZazz - Thanks for that....those CPU temps look a little high for my blood...but I'm glad your gear is working  |
| Posted by BeZazz, 12-05-2013, 12:12 PM |
Quote:
|
Originally Posted by rnavarro
@ BeZazz - Thanks for that....those CPU temps look a little high for my blood...but I'm glad your gear is working
|
Yes they are a bit high but not bad enough that I would shut them down.
thanks |
| Posted by rnavarro, 12-05-2013, 12:46 PM |
Hmmm it appears temperatures are still going up....my server is shutdown but the ambient is now reading:
Ambient Temp: 120.20F/49.00C
Edit
08:52 PST - The networking for my IPMI just went down.... *sad panda* |
| Posted by BeZazz, 12-05-2013, 12:53 PM |
I just lost connection to one server.
EDIT: Back up. |
| Posted by Ilya74, 12-05-2013, 01:01 PM |
BeZazz, may I know where is that server located? I am trying to understand if there is issue in one segment or it is global Denver issue.
Since Denver looking glass is still online and loads fast, it seems not all DC is experiencing the issue.
Edit:
It seems routers are also online, since all gateways are pinging. |
| Posted by BeZazz, 12-05-2013, 01:06 PM |
Both are in
Basement: B-200 |
| Posted by Ilya74, 12-05-2013, 01:09 PM |
|
I meant rack ID or row ID. As far I know, "Basement: B-200" is just the name of the DC space. |
| Posted by BeZazz, 12-05-2013, 01:11 PM |
The one that down ofr about 15 mins
VLAN 140 - 10/100 Racks: Shelf 92
8
The other
VLAN 140 - 10/100 Racks: Shelf 101
35 |
| Posted by Ilya74, 12-05-2013, 01:15 PM |
|
Thank you. We both located in the neighboring VLANs and racks. So the problem may be in there somewhere. |
| Posted by BeZazz, 12-05-2013, 01:19 PM |
|
I have just lost connection to both servers I have there. |
| Posted by neXeon, 12-05-2013, 01:36 PM |
|
Hopefully they fix this soon. We haven't had issues with them for ages in Denver. |
| Posted by Ilya74, 12-05-2013, 01:40 PM |
The tech just replied:
Quote:
|
Due to extreme weather conditions in the Denver area, we are are currently experiencing equipment failure. We hope to get it resolved asap.
|
|
| Posted by rnavarro, 12-05-2013, 01:55 PM |
Quote:
|
Originally Posted by Ilya74
The tech just replied:
|
I just got the same response....canned message! haha |
| Posted by rnavarro, 12-05-2013, 02:37 PM |
Quote:
|
Originally Posted by spork985
|
Haha I took a look at the weather map and asked the same question!
I can't even reach my IPMI interfaces anymore, so I'm completely in the dark....
I opened my ticket 3 hours ago... |
| Posted by rnavarro, 12-05-2013, 02:45 PM |
Well it is pretty cold in Denver right now relative to the average....but that should make it easier to cool the servers! haha
Just open the doors |
| Posted by Ilya74, 12-05-2013, 02:49 PM |
|
Perhaps the cooling system has failed (frozen? heh), which is why there was so hot. If your sensors were correct, then it was +50C in the DC and -24C outside. Looks bad. |
| Posted by rnavarro, 12-05-2013, 02:51 PM |
SWEEEET I think things are coming back online.
My IP just came back, and IPMI is now reporting:
Ambient Temp: 69.80F/21.00C
Which is much more inline with expectations |
| Posted by Ilya74, 12-05-2013, 02:52 PM |
|
Yeah, glad you are online now. Our switches are still down |
| Posted by spork985, 12-05-2013, 02:53 PM |
|
Mine's back online as well. I didn't get any explanation as to the problem though. |
| Posted by deriklogov1983, 12-05-2013, 03:25 PM |
|
Any news ? |
| Posted by deriklogov1983, 12-05-2013, 03:29 PM |
|
3 mines are still down |
| Posted by Ilya74, 12-05-2013, 03:30 PM |
|
Our switches are still down too. No updates from support. |
| Posted by rnavarro, 12-05-2013, 03:43 PM |
I haven't heard anything back with respect to cause, but the temperatures seems to be holding steady at Ambient Temp: 82.40F/28.00C
I've gone ahead and powered on my servers again and i'm running some consistency checks on the drives and file systems. |
| Posted by deriklogov1983, 12-05-2013, 03:49 PM |
|
Any idea what happens over there ? |
| Posted by ayksolutions, 12-05-2013, 03:51 PM |
|
I believe this is a very isolated issue in the basement, I was told that AC failed there, no ETA yet. We only have 2 or 3 machines down, which is a small fraction of what we have there. Still not fun though. Just have to wait. |
| Posted by Ilya74, 12-05-2013, 04:57 PM |
They told
Quote:
We believe all connectivity has now been restored. Please reopen this ticket if
you are still experiencing issues.
Thanks
|
However, we are still down. Anyone else? |
| Posted by deriklogov1983, 12-05-2013, 05:00 PM |
2 of 3 come back online, 1 still down
the one in : Shelf 1 |
| Posted by deriklogov1983, 12-05-2013, 05:03 PM |
|
where yours located ? |
| Posted by Ilya74, 12-05-2013, 05:05 PM |
|
We are using rack 108. I believe it is Shelf 108 in their numeration. |
| Posted by deriklogov1983, 12-05-2013, 05:07 PM |
|
Looks like last one come back online, check if your server power on |
| Posted by Ilya74, 12-05-2013, 05:11 PM |
Our switches are still down, so I can not check anything
Can you please go to Device Manager and see if bandwidth graphs are live? |
| Posted by Ilya74, 12-05-2013, 05:52 PM |
|
Finally, we are back. |
| Posted by studclub, 12-05-2013, 10:05 PM |
|
VLAN 110 - 1Gb: Shelf 1 |
| Posted by neXeon, 12-05-2013, 10:42 PM |
|
Yep, still down in some areas of the DC it seems. |
| Posted by dlewis23, 12-05-2013, 11:29 PM |
|
Because of this my power supply failed on the server I have with FDC and now they say the server gives a Boot Failure and the only thing they say they can do is a OS reload. Fantastic. |
| Posted by CBH-Nick, 12-05-2013, 11:46 PM |
Quote:
|
Originally Posted by neXeon
Yep, still down in some areas of the DC it seems.
|
I opened a ticket with you before coming across this thread. Do you have an ETA on when the server will be back online? |
| Posted by allanespinoza, 12-06-2013, 01:41 AM |
|
I like know, what FDC says?, that is my question. At this forum is no one from FDC?. In how many time the servers will go online?. My server have right now 13 HOURS OFFLINE. |
| Posted by NYCServers-Nick, 12-06-2013, 02:24 AM |
Quote:
|
Originally Posted by allanespinoza
At this forum is no one from FDC?
|
http://www.webhostingtalk.com/member.php?u=12276 is a rep from FDC. I would suspect that he is busy taking care of the issue, and the best way would be to contact them via their official support channels. |
| Posted by CBH-Nick, 12-06-2013, 02:26 AM |
|
Just wondering, how often do outages occur at this DC? I'm hoping this a rare occurrence. |
| Posted by neXeon, 12-06-2013, 02:40 AM |
|
This is the first we've seen in ages. It's just too bad they can't respond in a timely manner when some parts of the issue are an easy fix. |
| Posted by allanespinoza, 12-06-2013, 02:54 AM |
Quote:
|
Originally Posted by neXeon
This is the first we've seen in ages. It's just too bad they can't respond in a timely manner when some parts of the issue are an easy fix.
|
The really question i have is, that datacenter works 24x7 support? Phone, Email Support? You have received any reply with information about this problem?.
They are working RIGHT NOW to RESOLVE this problem?. Trust me i cant sleep if my server is not online.
This is so crazy, a Datacenter dont have a "BLOG" or any information about the problems. I have see a lot of PROFESSIONAL Datacenters and every one have "Forums, Business Blog, Status Page, etc" but this FDC only have PICS an old forum without any updates and nothing more. |
| Posted by neXeon, 12-06-2013, 03:35 AM |
|
Still major issues there. |
| Posted by CBH-Nick, 12-06-2013, 03:39 AM |
|
Right after my other server got disconnected and I was going to move it to this one, this happens. Just my luck! |
| Posted by TorSnipe, 12-06-2013, 04:37 AM |
|
Have a full on "grand opening" for one of my sites.. nek minnit FDC Down for a day.. -.- |
| Posted by dominum, 12-06-2013, 05:27 AM |
 almost 24 hours now... no news from FDC? |
| Posted by allanespinoza, 12-06-2013, 06:05 AM |
|
24 Hours and DOWN. This is a sh*t trust me. I cant sleep. I need that dedicated up. Why FDC dont give us any update?. |
| Posted by dominum, 12-06-2013, 06:16 AM |
|
Well, $h!t does happen. We cant blame anybody except the dc. My server is there too,from instantdedis. All we can do is hope for our data to be intact. |
| Posted by Ilya74, 12-06-2013, 06:17 AM |
|
When they stopped to answer you? |
| Posted by dominum, 12-06-2013, 09:43 AM |
|
My server is up. Finally. |
| Posted by allanespinoza, 12-06-2013, 09:53 AM |
|
Thanks god!!!! Servers back. |
| Posted by johnnydus3d, 12-06-2013, 08:59 PM |
|
Are your servers online now? Mine is still down for more than 24 hours now.. or 33 hours to be exact. I already opened a ticket yesterday but not a single reply. Not even a "copy & paste" reply or a bot reply. |
| Posted by johnnydus3d, 12-06-2013, 09:59 PM |
|
After opening another ticket that is not related to my server problem. I finally got a reply saying they are hopeful it will be fixed by tomorrow. That would be 3 days downtime. I never experience 3 days downtime from other hosting before so I will never accept that. I will transfer to another company once I get my data. |
| Posted by CBH-Nick, 12-06-2013, 10:06 PM |
Quote:
|
Originally Posted by johnnydus3d
After opening another ticket that is not related to my server problem. I finally got a reply saying they are hopeful it will be fixed by tomorrow. That would be 3 days downtime. I never experience 3 days downtime from other hosting before so I will never accept that. I will transfer to another company once I get my data.
|
Just curious what host your with? Did they say why yours is taking longer to get back online? Mine went back online this morning. |
| Posted by Ilya74, 12-06-2013, 11:45 PM |
|
Our servers are offline again. Tech support telling us that everything is fine, however, even their switch that is providing connection for our rack is down. |
| Posted by Ilya74, 12-07-2013, 12:07 AM |
The FDC switch that is providing connection was unplugged. Or like tech said "it wasn't like, hanging out, just barely out of the port".
Looks like there is real mess in the datacenter. |
Add to Favourites 
Print this Article
Also Read
Shared Hosting
Shared Hosting
