Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Tailormadeservers / Colo4Dallas down?
Tailormadeservers / Colo4Dallas down?
| Posted by xnpu, 08-10-2011, 12:22 PM |
Hi,
Anyone else having trouble reaching tailormadeserver or colo4dallas?
Thanks. |
| Posted by xnpu, 08-10-2011, 12:23 PM |
|
Noticed the Knownhost thread also discusses this just now. Maybe this one should be closed. |
| Posted by avizzle, 08-10-2011, 12:27 PM |
|
Colo4 is experiencing a power plant issue causing the facility to go down. Vendors are working to restore power. |
| Posted by daviolinist12, 08-10-2011, 12:27 PM |
Yes, TMS and Colo4 are down as we have servers in the datacenter as well.
All of our servers are down. Apparently this is due to a power issue at the DC which is affecting everyone. |
| Posted by andryus, 08-10-2011, 12:32 PM |
|
Many of my servers with tms are down. |
| Posted by Tarak, 08-10-2011, 12:35 PM |
|
Strange, datacenter like Colo4 is having power issue. |
| Posted by cpservr, 08-10-2011, 12:36 PM |
yeah, i can confirm that too.. and my browser keep loading after clicking submit ticket button.  |
| Posted by dexx, 08-10-2011, 12:45 PM |
some of our servers are down too. no news yet??
Colo4 is one of the worst experiences, and seems to be down with DOS attacks, and now for the power issue?? |
| Posted by ASIMichael, 08-10-2011, 12:49 PM |
We are also down and seems the colo4.com website also.
Phones are all busy. I think they have voip so if its power or network related thats why the phones are down. Gives thought to having backup #s with cell phones.
Hang in folks..might be a long day.
I myself might have to drive down just for an update |
| Posted by KevinJ5, 08-10-2011, 12:49 PM |
Same with Colo4.com
Anyone have a better solution?
Back when it was Colo4Dallas.com I don't remember having a single bit of downtime. |
| Posted by Hamid Reza, 08-10-2011, 01:09 PM |
|
Anyone can confirm its a power failure? |
| Posted by Hamid Reza, 08-10-2011, 01:10 PM |
|
I should note that all servers didn't go down at the same time, Thats why I wonder whether its a Power Failure (which would take all servers down at the same time) |
| Posted by Patrick, 08-10-2011, 01:11 PM |
Quote:
|
Originally Posted by Hamid Reza
Anyone can confirm its a power failure?
|
Confirmed by Jose. |
| Posted by Speckz, 08-10-2011, 01:11 PM |
I was told this came from Colo4:
Quote:
We have an issue with one phase of our power plant.
We have a vendor in route now to deal with this issue.
We will provide more information as soon as we are able to.
|
|
| Posted by TMS - JoseQ, 08-10-2011, 01:17 PM |
|
What was posted above, is all the info we have as well... we're awaiting a resolution. |
| Posted by xnpu, 08-10-2011, 01:20 PM |
Quote:
|
Originally Posted by TMS - JoseQ
What was posted above, is all the info we have as well... we're awaiting a resolution.
|
Thanks for that. Good to know you're aware and standby. |
| Posted by linux-image, 08-10-2011, 01:21 PM |
|
Any update ? |
| Posted by ! Unique !, 08-10-2011, 01:21 PM |
Quote:
|
Originally Posted by TMS - JoseQ
What was posted above, is all the info we have as well... we're awaiting a resolution.
|
Hello !
we are lost a lot of money !
what can we do now !!
if u can't solve this problem please give me my server backup and recharge back my money ! |
| Posted by andryus, 08-10-2011, 01:23 PM |
Quote:
|
Originally Posted by ! Unique !
Hello !
we are lost a lot of money !
what can we do now !!
if u can't solve this problem please give me my server backup and recharge back my money !
|
This problem is not from TMS , is from their datacenter , Colo4. |
| Posted by Patrick, 08-10-2011, 01:23 PM |
I've been with TMS since late 2006 or early 2007 and this is the first time something like this has happened. Colo4 is considered one of the best data centers in the world, sometimes things just go bad and redundancies fail... sh*t just happens? If you're new to TMS, don't let this outage ruin the fun. It's definitely one of those one in a million freak incidents and it could happen to anyone.  |
| Posted by MikeTrike, 08-10-2011, 01:26 PM |
I can also confirm a colo4 power issue, ColocateUSA and ApertureHost are also impacted. My web sites be down to ongoing colo4 power issues.  |
| Posted by elmister, 08-10-2011, 01:27 PM |
Quote:
|
Originally Posted by ! Unique !
Hello !
we are lost a lot of money !
what can we do now !!
if u can't solve this problem please give me my server backup and recharge back my money !
|
Stay calm, server will be back sooner or later, if you are loosing "a lot" then you should consider a multiple locations setup.
Failures happen sometimes, i've been with TMS for 3 years and this is an unusual issue, i'm not worried, i trust the people behind TMS and Colo4, we will be up again soon |
| Posted by ! Unique !, 08-10-2011, 01:31 PM |
Quote:
|
Originally Posted by elmister
Stay calm, server will be back sooner or later, if you are loosing "a lot" then you should consider a multiple locations setup.
Failures happen sometimes, i've been with TMS for 3 years and this is an unusual issue, i'm not worried, i trust the people behind TMS and Colo4, we will be up again soon
|
I'm afraid
if i lose backup lose all thing ! |
| Posted by Patrick, 08-10-2011, 01:34 PM |
Quote:
|
Originally Posted by ! Unique !
I'm afraid
if i lose backup lose all thing !
|
Relax, your server is OK. You're not going to lose anything. Take a deep breathe, that's all you can do at the moment. No point in freaking out when you can't control the outcome. Colo4 is working as fast as humanely possible to get everything back up, and then any lingering issues related to TMS will be taken care of by Jose and his crew. You're in good hands, relax. |
| Posted by ! Unique !, 08-10-2011, 01:35 PM |
Quote:
|
Originally Posted by Patrick
Relax, your server is OK. You're not going to lose anything. Take a deep breathe, that's all you can do at the moment. No point in freaking out when you can't control the outcome. Colo4 is working as fast as humanely possible to get everything back up, and then any lingering issues related to TMS will be taken care of by Jose and his crew. You're in good hands, relax.
|
ok .. well done .. i am wait .. Am Relax now |
| Posted by elmister, 08-10-2011, 01:36 PM |
Quote:
|
Originally Posted by ! Unique !
I'm afraid
if i lose backup lose all thing !
|
Always keep local backups, external USB hard drives are cheap enough, download them twice a month with rsync and you'll sleep fine ;-)
Don't worry, probably all our servers are safe. |
| Posted by dexx, 08-10-2011, 01:47 PM |
|
Jose, Any ETA? |
| Posted by colnector, 08-10-2011, 01:54 PM |
Our TMS server is also down. I created a ticket with the highest priority that, as written, should be answered within 15 minutes. I got no response although an hour passed. There's no indication on http://www.tailormadeservers.com that something is wrong and without WHT I would be out in the dark. Why is that?
Any idea when power should be back up? |
| Posted by Patrick, 08-10-2011, 01:58 PM |
Quote:
|
Originally Posted by colnector
Our TMS server colnect.com is also down. I created a ticket with the highest priority that, as written, should be answered within 15 minutes.
|
You're probably being left in the dark because they have hundreds of other open tickets regarding the same matter. There is nothing for them to report. The data center has just stated:
I don't have an ETA on this plant returning. The contractors are going through the systems now to determine what caused the outage. This must be done before restoring a 2000 amp ATS (that's 2000 amps at 480 volt).
Monitor this thread for updates from Colo4:
http://www.webhostingtalk.com/showthread.php?t=1072692 |
| Posted by cpservr, 08-10-2011, 01:58 PM |
Quote:
|
Originally Posted by colnector
Our TMS server colnect.com is also down. I created a ticket with the highest priority that, as written, should be answered within 15 minutes. I got no response. There's no indication on http://www.tailormadeservers.com that something is wrong and without WHT I would be out in the dark. Why is that?
Any idea when power should be back up? |
i close my priority 1 ticket after seeing this thread. hope he'll answer you shortly, at least my ticket not in queue list.. |
| Posted by EGXHosting, 08-10-2011, 02:04 PM |
|
This explains why my sites are down now. Thanks everyone for the information. |
| Posted by jackpx, 08-10-2011, 02:11 PM |
|
I have 2 servers in Tailor ...1 Online, 1 Offline |
| Posted by uberhostNET, 08-10-2011, 02:15 PM |
As if things weren't bad enough, we have been getting calls from Colo4's customers because they can't answer them, lol. On my tracking screen right now I see several or more people on our site because they searched for "Colo4Dallas" or "Colo4" using Google. I guess we come up fairly high in the search results so they call us on our toll-free number to get answers. How many ways do we have to pay for this outage, lol?
Cheers all,
Rick |
| Posted by colnector, 08-10-2011, 02:24 PM |
and I'm happy this thread exists so at least I can find out what the problem is. I guess hosting providers should have some form of automatically informing all their customers of critical errors (power failure being one). I do get emails when a planned hardware upgrade is suspected to create a shortage so why can't I get an email that an unexpected shortage happens?
I hope to soon see the servers up and running. Crossing my fingers. Thanks |
| Posted by solidhostingph, 08-10-2011, 02:25 PM |
|
I just received an email from Jose confirming power issue at Colo4. |
| Posted by TMS - JoseQ, 08-10-2011, 02:28 PM |
I'm sorry I can't update the helpdesk... the server is using resolvers that are down due to the power issue. On top of that, the server only allows ssh connections from certain hosts, all of which are down, so I can't fix the resolver issue either.
Unfortunately I'm getting my updates just as you are, via Colo4's WHT thread. Just like us, they're probably being flodded with requests for updates, and they're mostly focused on getting things resolved rather than delay the fix to provide updates to the hundreds of people asking.
Rest assured the problem will be fixed as soon as it is humanly possible. |
| Posted by colnector, 08-10-2011, 02:41 PM |
|
Thanks Jose for your response |
| Posted by elmister, 08-10-2011, 04:23 PM |
Quote:
|
Originally Posted by TMS - JoseQ
I'm sorry I can't update the helpdesk... the server is using resolvers that are down due to the power issue. On top of that, the server only allows ssh connections from certain hosts, all of which are down, so I can't fix the resolver issue either.
Unfortunately I'm getting my updates just as you are, via Colo4's WHT thread. Just like us, they're probably being flodded with requests for updates, and they're mostly focused on getting things resolved rather than delay the fix to provide updates to the hundreds of people asking.
Rest assured the problem will be fixed as soon as it is humanly possible.
|
Let me suggest you to add those ips to other server meanwhile your dns servers are down |
| Posted by albatroz, 08-10-2011, 06:44 PM |
Everything is still down?
I can´t believe that it is almost a 12 hours outage. |
| Posted by Speckz, 08-10-2011, 06:48 PM |
Quote:
|
Originally Posted by albatroz
Everything is still down?
I can´t believe that it is almost a 12 hours outage.
|
It started around 12:05 PM EST. It's just past 6 hours now. |
| Posted by Patrick, 08-10-2011, 06:49 PM |
Quote:
|
Originally Posted by albatroz
Everything is still down?
I can´t believe that it is almost a 12 hours outage.
|
Colo4 is starting the process of restoring power... they have given an ETA of 18:00 -> 19:00 CDT for everything but that could change. |
| Posted by elmister, 08-10-2011, 07:40 PM |
|
2 of my servers are up again! |
| Posted by TMS - JoseQ, 08-10-2011, 07:43 PM |
Switches and services are coming back. Please note not all servers will be coming online on their own, so they will need manual assistance. We will be doing our best to get everyone back online as soon as possible.
Since not all switches are yet online, please refrain from opening tickets just yet. We'll update here and twitter once all of the network is ready to go. |
| Posted by jackpx, 08-10-2011, 08:02 PM |
|
I hope that my server will be online soon. |
| Posted by VPSForge-Ray, 08-10-2011, 08:14 PM |
|
My server is still offline |
| Posted by sodapopinski, 08-10-2011, 08:17 PM |
One server is up and one server is still down. The problem is the server which is up is remote backup server and the helpdesk system while the other one which is live server for clients still down  |
| Posted by Patrick, 08-10-2011, 08:55 PM |
From their Twitter feed:
Several switches and PDUs are still offline. Please be patient. I see a ton of tickets, but I can't respond until we're online.
Stop submitting tickets people until Jose gives the OK. |
| Posted by solidhostingph, 08-10-2011, 08:58 PM |
|
Patience is a virtue! :-) |
| Posted by forumtalk, 08-10-2011, 09:06 PM |
|
good I am almost dead |
| Posted by elmister, 08-10-2011, 09:24 PM |
4 servers recovered
9 more and i can go to sleep (3:24am here) |
| Posted by solidhostingph, 08-10-2011, 09:32 PM |
|
How long is the wait for these 9 servers? |
| Posted by solidhostingph, 08-10-2011, 09:36 PM |
|
@elmister are you from TMS? |
| Posted by uberhostNET, 08-10-2011, 09:39 PM |
Quote:
|
Originally Posted by Patrick
From their Twitter feed: Several switches and PDUs are still offline. Please be patient. I see a ton of tickets, but I can't respond until we're online.
Stop submitting tickets people until Jose gives the OK.
|
From the Colo4 thread:
Quote:
If you heard that some PDUs are still down you got misinformation. Some individual power strips in cabinets tripped when all servers tried to come up at once, but the PDUs are up. Also, there are 6 generators operational.
Here is the latest update:
As of 6:30PM CDT all PDUs are online supplying power to our customers. The UPSs are currently in bypass mode while the batteries charge. During this time we are operating on generator power, as is the best practice. Once the UPS batteries are fully charged, we will migrate the PDUs to the UPSs and then put power back on the utility. This process will be short in duration and begin around 9:00PM CDT. We do not expect any issues or impact to customers tonight while these transitions happen. We also have extra staff that will remain on-site throughout the evening.
We have technicians working with our customers to help bring any equipment online that did not come back on with the power restore. There are also a few customers who tripped breakers while the equipment was powering up, and we are working to reset those devices. We appreciate your patience as we continue to bring this issue to closure. If you encounter any problems with your equipment or access, please open a help ticket so that we may respond in the fastest manner.
We will update customers with an official reason for outage (RFO) once we assess the reports that were generated today.
|
http://www.webhostingtalk.com/showpo...&postcount=317 |
| Posted by forumtalk, 08-10-2011, 09:42 PM |
" Current Update: As of 6:30PM CDT all PDUs are online supplying power to our customers. "
then why still down  |
| Posted by Patrick, 08-10-2011, 09:44 PM |
Quote:
|
Originally Posted by uberhostNET
|
Yeah, I saw that. Who knows.. |
| Posted by elmister, 08-10-2011, 09:45 PM |
Quote:
|
Originally Posted by solidhostingph
@elmister are you from TMS?
|
I'm a customer.
Last server was powered on 30 minutes ago.
Colo4 may be giving power to customers since 2 hours ago, but each customer, (and TMS is colo4 customer) may have their own PDU that may need to be switched on individually
In other server i noticed the server got power 1 hour before it got network access, server was powered on but switch was off |
| Posted by elmister, 08-10-2011, 10:06 PM |
|
Another 7 servers went up, only 2 to go ;-) |
| Posted by Patrick, 08-10-2011, 10:08 PM |
@elmister:
Were your servers rebooted or did they maintain their uptimes? |
| Posted by elmister, 08-10-2011, 10:24 PM |
Quote:
|
Originally Posted by Patrick
@elmister:
Were your servers rebooted or did they maintain their uptimes?
|
at this exact time i'm looking at 2 servers with 3h 6 min uptime, but they started to respond to ping about 25 minutes ago.
So i think all servers were powered on, but switches weren't or maybe switches needed some kind of configuration |
| Posted by elmister, 08-10-2011, 10:26 PM |
|
Of the 2 remaining servers one is online but having big packet loss, i'll have to open a ticket, i'm afraid Jose might have thousands of tickets and it will take a lot to get assistance on this |
| Posted by VPSForge-Ray, 08-10-2011, 10:43 PM |
|
This is taking too long. My server is still down and ticket still unanswered |
| Posted by jackpx, 08-10-2011, 10:50 PM |
|
yes,... my server online. |
| Posted by AWH - Kalpesh, 08-10-2011, 10:55 PM |
|
Still waiting for our server to kick back into life as well. Hopefully it won't be too much longer after hearing some of yours are slowly responding. |
| Posted by VPSForge-Ray, 08-10-2011, 10:57 PM |
Still waiting for mine... |
| Posted by elmister, 08-10-2011, 11:00 PM |
|
I got answer to one of my tickets, be patient |
| Posted by daviolinist12, 08-10-2011, 11:00 PM |
|
Stay patient everyone. Jose is working very hard to get everyone's servers back online - just spoke with him on MSN. |
| Posted by elmister, 08-10-2011, 11:07 PM |
i think we shouldn't speak with him on msn/icq/etc, it's not a good time for speaking with him.
Having multiple message windows increases stress |
| Posted by daviolinist12, 08-10-2011, 11:08 PM |
Quote:
|
Originally Posted by elmister
i think we shouldn't speak with him on msn/icq/etc, it's not a good time for speaking with him.
Having multiple message windows increases stress
|
In order for my servers to get online, he needed to manually get them up. - that's why. |
| Posted by elmister, 08-10-2011, 11:25 PM |
last 2 servers fixed.
Dont worry, JoseQ will answer your tickets, he is fast.
He fixed them but didn't answer the ticket, just check your server from time to time, maybe he fixed them while fixing other servers at the same rack. |
| Posted by AWH - Kalpesh, 08-10-2011, 11:33 PM |
I haven't submitted a ticket to let them know our server isn't responding. Should I let him know or wait? I've been quite patient thus far and I don't want to disturb him..
I'm just not sure if they are checking ALL servers or just the ones there is a ticket for. |
| Posted by Patrick, 08-10-2011, 11:36 PM |
|
Yes, put a ticket in. |
| Posted by forumtalk, 08-10-2011, 11:48 PM |
still down here 3 more to go  |
| Posted by Hamid Reza, 08-10-2011, 11:48 PM |
Quote:
|
Originally Posted by Patrick
@elmister:
Were your servers rebooted or did they maintain their uptimes?
|
I have some affected servers without uptime change. That means the switch/router behind the server were affected by this power failure? |
| Posted by AWH - Kalpesh, 08-10-2011, 11:56 PM |
|
We are up! Thanks to Jose and the rest of the team. |
| Posted by forumtalk, 08-11-2011, 01:41 AM |
|
one more to go |
| Posted by TMS - JoseQ, 08-11-2011, 03:36 AM |
Sorry I couldn't keep checking here once we got things going... As Murphy's Laws would have it, it wasn't a simple flip of a switch. A bunch of our Remote Reboot PDUs decided to power on with all of their outlets turned off, and no response on their respective IPs. A tech had to manually go through each of them, reset the config and then set it to power on all of the outlets. Some of these PDUs had more than one switch attached to them, so many servers came up but had no connectivity due to this issue. Likewise, some servers were up through the entire ordeal, but the switches they were connected to had no power.
Most of the hardware is back to normal. There were some server casualties, some of which are still being dealt with. If your server is still offline, please don't hestitate to contact me on IM or open a ticket.
I offer my apologies for this event. Unfortunately it's one of those things that although rarely, they can happen. I appreciate your patience and understanding during this whole ordeal. |
Add to Favourites 
Print this Article
Also Read
Shared Hosting
Shared Hosting
