WebNX down?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > WebNX down?

Posted by zymic, 07-16-2010, 07:16 PM
http://www.webnx.com

Fails to load, ticket system down, my servers also down.
Posted by jandres4, 07-16-2010, 07:18 PM
Yep, same here. I have 4 boxes down, 7 minutes and counting.
Posted by Mivo, 07-16-2010, 07:19 PM
Everything seems down. There site is not accessible as well.
Posted by 123finder.com, 07-16-2010, 07:19 PM
All 5 boxes down. Drop off the net 100%. Serious stuff
Posted by zymic, 07-16-2010, 07:21 PM
Down since 00:09:48 UTC
Posted by jandres4, 07-16-2010, 07:22 PM
Their website is up now, but all my boxes are still down.
Posted by jandres4, 07-16-2010, 07:24 PM
2 boxes up now, 2 still down!
Posted by jandres4, 07-16-2010, 07:27 PM
All servers up now !
Posted by Dan_EZPZ, 07-16-2010, 07:27 PM
Looks to have been a power outage but everything is back online here.
Posted by zymic, 07-16-2010, 07:29 PM
Now to fix corrupted databases, great
Posted by Mivo, 07-16-2010, 07:31 PM
mine is still down.
Posted by zymic, 07-16-2010, 07:32 PM
Yes some of ours are still down

Pinging 67.220.217.226 with 32 bytes of data:
Request timed out.
Posted by forasse, 07-16-2010, 07:34 PM
Probably doing fsck
Posted by Gearbox, 07-16-2010, 07:56 PM
Yes, my box is also down.
Posted by Gearbox, 07-16-2010, 08:02 PM
My server is back online after around 30 minutes downtime.
Posted by jackpx, 07-16-2010, 08:05 PM
yes, online now
Posted by jumpinjack, 07-16-2010, 08:08 PM
Still down here. I thought my server had crashed and they didn't even respond to my ticket or reboot request.

Good thing we have WHT to see what is actually going on.
Posted by 123finder.com, 07-16-2010, 08:12 PM
My 5 boxes are now back online. Power loss. Let's see what they say. It's always good to use multiple dedicated providers.
Posted by zymic, 07-16-2010, 09:09 PM
All up now ... after fsck's
Posted by Josh|Servlix, 07-16-2010, 09:13 PM
Did they say what the issue was?
Posted by jumpinjack, 07-16-2010, 09:17 PM

Quote:

Originally Posted by Josh|Servlix

Did they say what the issue was?

Power outage.

Still waiting on fsck.
Posted by cyb, 07-16-2010, 11:32 PM
My server is still down and WebNX didn't respond to my ticket. Very bad support.
Posted by JLHC, 07-16-2010, 11:36 PM
All came back up here. Longest downtime is 44 minutes for one server while the others average around 10 to 15 minutes only.
Posted by -OY-, 07-16-2010, 11:38 PM
Mine's been down for over 2 hours now, and no reply to the ticket - but that's a given with a flood of tickets.
Posted by cyb, 07-16-2010, 11:49 PM
Their live support said that they are manully working on the servers one by one and a tech will be checking my server shortly. I think this means they doesn't have enough on-site support so my server been down for over 4 HOURS and no one has time to check it
Posted by Scott.Mc, 07-17-2010, 12:06 AM

Quote:

Originally Posted by cyb

Their live support said that they are manully working on the servers one by one and a tech will be checking my server shortly. I think this means they doesn't have enough on-site support so my server been down for over 4 HOURS and no one has time to check it

How do you come to that conclusion? If your server is having issues that require more extensive troubleshooting it's going to be left till last. It's a simple choice of spend ~30 minutes troubleshooting your system or restore ~30 other customers. If the shoe was on the other foot and you were one of those other customers what would you expect done?

They have restored all of our priority servers and only ~9 servers are still down, we have several hundred that have been recovered, troubleshooted any lingering issues and back online. Most were back online after a few minutes and another huge bulk after ~15 minutes [ once the remaining vlan routers were restored (this is only an assumption)].
Posted by WebPerez, 07-17-2010, 12:15 AM

Quote:

Originally Posted by cyb

Their live support said that they are manully working on the servers one by one and a tech will be checking my server shortly. I think this means they doesn't have enough on-site support so my server been down for over 4 HOURS and no one has time to check it

Do you have a lower service level with webnx or is it that your system needed more extensive work to bring it up? Was your business offline for these four+ hours?

Thanks and hope things are better!
Posted by cyb, 07-17-2010, 12:32 AM

Quote:

Originally Posted by Scott.Mc

How do you come to that conclusion? If your server is having issues that require more extensive troubleshooting it's going to be left till last. It's a simple choice of spend ~30 minutes troubleshooting your system or restore ~30 other customers. If the shoe was on the other foot and you were one of those other customers what would you expect done?

They have restored all of our priority servers and only ~9 servers are still down, we have several hundred that have been recovered, troubleshooted any lingering issues and back online. Most were back online after a few minutes and another huge bulk after ~15 minutes [ once the remaining vlan routers were restored (this is only an assumption)].

I'll wait for it if they updated me that my server needs some time to troubleshooting. Now it has been over 5 HOURS, but my server isn't checked yet and they can't give me a update on the status. I think this is unacceptable.
Posted by Scott.Mc, 07-17-2010, 12:36 AM

Quote:

Originally Posted by cyb

I'll wait for it if they updated me that my server needs some time to troubleshooting. Now it has been over 5 HOURS, but my server isn't checked yet and they can't give me a update on the status. I think this is unacceptable.

I know it doesn't exactly help because for you why should you care about other customers but they have been very responsive to us and are bringing up our remaining servers (which all have different issues, from simply requiring the root pass to fsck to actual hardware issues). I wouldn't assume your server isn't checked yet, if it's down it's most likely got another issue which needs investigated and has been moved to the back of the queue. Not sure where you are not getting updates but if it's a ticket chances are they simply have thousands of tickets to get through (most for servers that are already resolved) and are concentrating on actually finishing the rest of the servers off.

If they aren't doing that they should and then mass reply to tickets stating for it to be reopened if there is still a problem, works best to speed things up for everyone.
Posted by jumpinjack, 07-17-2010, 12:56 AM

Quote:

Originally Posted by cyb

I'll wait for it if they updated me that my server needs some time to troubleshooting. Now it has been over 5 HOURS, but my server isn't checked yet and they can't give me a update on the status. I think this is unacceptable.

Same here man, they haven't replied to my ticket in over 3 hours, so I don't know what the heck is going on.

Last update was from me telling them the root pass for fsck, and then complete silence.

I see other people have their servers up and are getting some sort of support. I guess with only one server with them I'm not important enough.

An update or even some sort of acknowledgement that they are working on my server would be nice. I'm counting 6 hours of downtime now.

And it's not even the downtime that's the problem, it's the utter lack of communication. Having been with them for 9+ months with a $300 server I don't think my expectations are too high for them to acknowledge that I exist.
Posted by WebPerez, 07-17-2010, 01:08 AM

Quote:

Originally Posted by cyb

Their live support said that they are manully working on the servers one by one and a tech will be checking my server shortly. I think this means they doesn't have enough on-site support so my server been down for over 4 HOURS and no one has time to check it

For an event this big, you call your staff members at home to help manage the updates to customers while those on site do the manual work. Oh yeah, after five hours, they needed to bring folks in as well. If you work in a data center environment, you're always on call. Friday night is not exactly Monday morning in terms of webnx staffing perhaps?

Anyone has an idea who is their largest customer/hosted site?

Thx!
Posted by syang3, 07-17-2010, 01:11 AM
My box is still down. They are working on individual boxes that doesn't come up after the power outage.
Posted by MattE, 07-17-2010, 01:18 AM
Most of our servers came up relatively quickly after the outage, unfortunately there is one that has a failed drive in the RAID array, we've been waiting for an update on the ticket since the outage occurred, so far only a canned response about manual FSCKs..
Posted by inteltechs, 07-17-2010, 01:34 AM
one of our server is still down. Can't get a hold of support...this is just bad.
Posted by Peter-SexyWing, 07-17-2010, 01:37 AM
down MANY HOURS!!! still DOWN

They are SO SHORT of staffs
Posted by robbyhicks, 07-17-2010, 01:39 AM
all of my boxes still down as well.
Posted by WebNX, 07-17-2010, 01:44 AM
Quick update. We're actively working on getting everyones servers back online. Your not being ignored, we're focusing on bringing the servers up. Please try to keep phone calls to a minimum as this interrupts technicians and slows down our ability to help you. Thank you for your patience and understanding.
Posted by -OY-, 07-17-2010, 02:20 AM
Thanks for the update! The most we can do as a client to speed this up is not to give them additional workload by phoning them or replying/updating endlessly on the ticket.
Posted by syang3, 07-17-2010, 02:22 AM
I am waiting patiently but my clients are throwing tickets at me with angry words. Oh well..
Posted by cyb, 07-17-2010, 03:40 AM
Server down over 8 hours and counting...
Posted by Peter-SexyWing, 07-17-2010, 03:59 AM
why power issue causes so much downtime????
Posted by syang3, 07-17-2010, 04:44 AM
Server is still down..
Posted by FHDave, 07-17-2010, 05:17 AM

Quote:

Originally Posted by Peter-SexyWing

why power issue causes so much downtime????

Power issue can cause long downtime especially if your hard drive is large and the OS forces fsck/disk check. It may even cause your server not to come up (e.g., failed HD), in which case you would e down much longer.

Best of luck.
Posted by cyb, 07-17-2010, 05:24 AM

Quote:

Originally Posted by FHDave

Power issue can cause long downtime especially if your hard drive is large and the OS forces fsck/disk check. It may even cause your server not to come up (e.g., failed HD), in which case you would e down much longer.

Best of luck.

My disk usage is low so I don't think fsck will cause so long downtime. Server down over 10 hours and counting...
Posted by jumpinjack, 07-17-2010, 05:32 AM
Still down as well. 10+ hours.

Shouldn't datacenters have some sort of protection for power outages. What a mess.
Posted by FHDave, 07-17-2010, 05:34 AM

Quote:

Originally Posted by cyb

My disk usage is low so I don't think fsck will cause so long downtime. Server down over 10 hours and counting...

It is possible that the server is prompting for somebody to press a key before continue booting. Hope somebody there can hook up a KVM and see what's actually is happening. Whatever the reason is, let's hope it is not your HD getting toasted.
Posted by Peter-SexyWing, 07-17-2010, 07:15 AM
This is going to be rediculous

9 HOURS now!
Posted by Scott.Mc, 07-17-2010, 07:17 AM

Quote:

Originally Posted by Peter-SexyWing

This is going to be rediculous

9 HOURS now!

and they haven't updated your ticket? Have you tried replying to the ticket because they have replied to all of mine. It's only the servers that have hardware issues which are remaining as far as I can tell.
Posted by echipvina, 07-17-2010, 07:22 AM
Yes, my box is also down.
Posted by Khell, 07-17-2010, 09:39 AM
Two of my boxes are still down, one I fortunately wasn't using yet, but the other that wouldn't boot up has had chkdsk on Windows delete most of the data that mattered. Anyone had any experience recovering data chkdsk has deleted? Program(s) used?
Posted by WebNX, 07-17-2010, 02:36 PM
Hello,

Thank you for your patience during this, all client issues show resolved. If you have any oustanding issues please raise a new ticket. A RFO will be sent out to clients early next week.
Posted by echipvina, 07-17-2010, 03:07 PM
You can check server ip 173.231.6.130
Posted by Peter-SexyWing, 07-17-2010, 09:41 PM
what was the "exact" issue then?

We have used many providers in many years, never see a power failure causes so much long downtime, do you really have good UPS then???
Posted by WebNX, 07-18-2010, 02:02 AM

Quote:

Originally Posted by Peter-SexyWing

what was the "exact" issue then?

We have used many providers in many years, never see a power failure causes so much long downtime, do you really have good UPS then???

Hello Peter,

The ups's were fine, they held the load for over 10 minutes. The swapover to generator power didn't go smoothly. The total power problems lasted just over 15 minutes.

Thousands of servers did come back within minutes(total downtime for them was under 10 minutes). Unfortunately there were some servers with extended downtimes. These were related to fsck's and or hardware failures. If you're in this group, please raise a ticket titled "SLA Credit" and the proper credit will be added to your account.

I do sincerely apologize for the inconvenience.

Add to Favourites Print this Article

Knowledgebase

WebNX down?

Our Services

Client Menu

Legal