Did Cloudive Evaporate?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Did Cloudive Evaporate?

Posted by oldsmoboi, 06-24-2013, 08:38 PM
My sites are down. Cloudive.com is down. Can't get anyone on the phone. No replies to emails. Been having shaky performance for a month... I wanted to stay with them, but I guess it's time to leave.
Posted by serverian, 06-24-2013, 08:39 PM

Quote:

Originally Posted by oldsmoboi

My sites are down. Cloudive.com is down. Can't get anyone on the phone. No replies to emails. Been having shaky performance for a month... I wanted to stay with them, but I guess it's time to leave.

We are working on the issues, please stand by.
Posted by oldsmoboi, 06-24-2013, 08:42 PM
Once you're back, I plan to talk to you about your 99.9% uptime guarantee, as you're clearly below that threshold over the past 30 days.
Posted by serverian, 06-24-2013, 09:01 PM

Quote:

Originally Posted by oldsmoboi

Once you're back, I plan to talk to you about your 99.9% uptime guarantee, as you're clearly below that threshold over the past 30 days.

This is the reply we got from the datacenter. "We experienced a blackout during a thunderstorm and went over to generator, however it appears that your machines did not transfer over properly. Cabinets on the same phase transferred over just fine."

We are getting the servers up soon.
Posted by k2host, 06-24-2013, 09:03 PM
My machine has been offline for over 1.5 hours now.
Posted by oldsmoboi, 06-24-2013, 09:43 PM
How long does it take to get a guy to walk over and plug the servers into a different PDU?
Posted by k2host, 06-24-2013, 10:54 PM
Cloudive, need an update, what's the status?
Posted by NetworkPanda, 06-24-2013, 11:12 PM
Is this in Illinois? I have heard that a thunderstorm in Illinois has caused an extended power outage and hundreds of servers operated by various hosting companies went offline. So, it is an extended outage and not affecting only Cloudive.
We have some servers there for testing purposes only (fortunately, not used for web hosting and not affecting any customers) and they were offline for some time due to the thunderstorm. For some reason, the datacenter's backup power generators didn't work.

Actually, the servers are now powered on again, but they are running an fsck (file system check) which could take some time to complete.
Posted by Eased, 06-24-2013, 11:16 PM

Quote:

Originally Posted by NetworkPanda

Is this in Illinois? I have heard that a thunderstorm in Illinois has caused an extended power outage and hundreds of servers operated by various hosting companies went offline. So, it is an extended outage and not affecting only Cloudive.
We have some servers there for testing purposes only (fortunately, not used for web hosting and not affecting any customers) and they were offline for some time due to the thunderstorm. For some reason, the datacenter's backup power generators didn't work.

Actually, the servers are now powered on again, but they are running an fsck (file system check) which could take some time to complete.

Confirmed power outage in Lombard, IL. We have servers out too (non Cloudive). Datacenter has reported that the servers didn't get picked up by the generator automatically. Due to power failure from a thunderstorm, many servers are executing forced file system checks and this will take a couple of hours.
Posted by serverian, 06-25-2013, 04:08 AM
Our SSD SAN's operating system died because of the power failure. We are working hard to restore it for the last hours. We'll update you once it's restored.
Posted by serverian, 06-25-2013, 07:30 AM
The drive that holds the operating system on our SSD SAN has died. We are getting a replacement in 2-3 hours.
Posted by oldsmoboi, 06-25-2013, 10:17 AM
Any update? It's been 3 hours.
Posted by serverian, 06-25-2013, 10:19 AM

Quote:

Originally Posted by oldsmoboi

Any update? It's been 3 hours.

We have got the new drives, installing the OS and setting up the system. This should take a few more hours. I'm so sorry for the issues but this is just unfortunate as you see. We'll be applying a full month of credit to your account.
Posted by TravisJeffries, 06-25-2013, 11:38 AM
5 straight minutes of 30%+ packet loss.

http://i.imgur.com/6IOFQxU.png

I would normally be a little sympathetic because of the storm that supposedly knocked out power? It hasn't been storming for a month straight though.
Posted by blogmaster2003, 06-25-2013, 12:32 PM
I agree with you completely. My cloud is offline for maybe 14 hours and last month the performance is poor
Posted by serverian, 06-25-2013, 12:35 PM
We are going to restore the system in a few hours and then upgrade the infrastructure on Thursday without any downtime.
Posted by MrGeneral, 06-25-2013, 12:52 PM
Folks, let's give them a few more hours :-)
These kind of failures do happen, to any hosting company.

The compensatin you get is quite fair, support your hosts as well, come on, you're all nice :=)
Posted by k2host, 06-25-2013, 01:21 PM
serverian, can you please give us another update.
Posted by serverian, 06-25-2013, 01:23 PM

Quote:

Originally Posted by k2host

serverian, can you please give us another update.

We are compiling dependencies for the infiniband stack. It's taking looong.
Posted by oldsmoboi, 06-25-2013, 03:57 PM
Do you have a percentage?
Posted by k2host, 06-25-2013, 04:47 PM
Any update? This is killing me, losing money...
Posted by serverian, 06-25-2013, 04:48 PM

Quote:

Originally Posted by k2host

Any update? This is killing me, losing money...

Almost there. Need 2 more hours.
Posted by k2host, 06-25-2013, 07:00 PM
Well, it's been 2 hours... where are we at now?
Posted by oldsmoboi, 06-25-2013, 07:18 PM
It has now been over 24 hours since initial outage... where is my cloud?
Posted by blogmaster2003, 06-25-2013, 07:19 PM
Im also at same situation, but people try to keep calm.
At least Serverian is posting here something.
Let him work so we can have our cloud again (fast)
Posted by oldsmoboi, 06-25-2013, 07:21 PM
It's been 24 hours since initial outage...
Posted by oldsmoboi, 06-25-2013, 09:51 PM
I guess they went to bed...
Posted by k2host, 06-26-2013, 12:55 AM
Please give us an update... this is ridiculous.

And for those of you saying just have patience, you clearly don't have a profitable business that is completely shutdown because of this outage. If it was just some blog of mine that wasn't online, no big deal, but we're talking thousands of dollars.
Posted by rehansiddiq, 06-26-2013, 02:30 AM
its to late now, Please do the needful ASAP.
Posted by techjr, 06-26-2013, 03:09 AM

Quote:

Originally Posted by k2host

Please give us an update... this is ridiculous.

And for those of you saying just have patience, you clearly don't have a profitable business that is completely shutdown because of this outage. If it was just some blog of mine that wasn't online, no big deal, but we're talking thousands of dollars.

If you are risking thousands of dollars on a single host, you need to come up with a redundancy plan or have a failover in place to get online as soon as possible. Not wait 24+ hours to come back online.

Quote:

Originally Posted by serverian

We are going to restore the system in a few hours and then upgrade the infrastructure on Thursday without any downtime.

Since I'm unable to load your site or contact you still. You may want to try using your twitter or facebook page for updates if you have the time. As I recall the website links to both but neither are actively being used.

You could also stick blog.cloudive.com with blogger or something similar so it stays up for status updates.
Posted by larishosting, 06-26-2013, 03:13 AM

Quote:

Originally Posted by techjr

If you are risking thousands of dollars on a single host, you need to come up with a redundancy plan or have a failover in place to get online as soon as possible. Not wait 24+ hours to come back online.

Since I'm unable to load your site or contact you still. You may want to try using your twitter or facebook page for updates if you have the time. As I recall the website links to both but neither are actively being used.

You could also stick blog.cloudive.com with blogger or something similar so it stays up for status updates.

I agree, it's amazing how many people don't use offsite backups for this reason alone.
Posted by serverian, 06-26-2013, 05:30 AM
Due to latest infiniband stack being out of date for the latest CentOS, all our efforts failed yesterday. We are going back to 2 CentOS versions earlier in order to get this done ASAP. This should take another 2 hours unfortunately.
Posted by rehansiddiq, 06-26-2013, 06:09 AM
when cloudive 'll be up and running hope fully?
Please update with time slice?
Posted by serverian, 06-26-2013, 07:06 AM
All is up now. We are still going to upgrade the whole infrastructure on Monday.
Posted by oldsmoboi, 06-26-2013, 10:55 AM
Do you plan to share with us what the upgrades will be?
Posted by serverian, 06-26-2013, 11:05 AM

Quote:

Originally Posted by oldsmoboi

Do you plan to share with us what the upgrades will be?

We are going to use 10GE storage network instead of Infiniband and we are going to use E5 CPUs.
Posted by rehansiddiq, 06-26-2013, 12:07 PM
thanks! its up and running, but the performance is very poor now
Posted by serverian, 06-26-2013, 12:10 PM

Quote:

Originally Posted by rehansiddiq

thanks! its up and running, but the performance is very poor now

Hold tight for a few days and i'll be solved!
Posted by oldsmoboi, 06-26-2013, 07:37 PM
My disk access seems snappier now.. but I'm still looking forward to the upgrades to decide if I'll stay or not.

That said, here are a list of things I'd like to see serverian:

1. A status page NOT hosted in the same data center your clouds are in. As someone mentioned earlier, put it on Blogger or something else not at all in your infrastructure. As many troubles as I had with VPS.net, I could always go to status.vps.net to find out what was broken, and I appreciated that.

2. Have failover redundancy between your Chicago data center and the second datacenter you use for Backupsy, for Cloudive.com so that if there is some major outage we can still get to our control panels and chat with your support if needed.

3. This is your second power issue in about 45 days, the first being a bad PDU the second being a failure to switch to generator power during a power event. I want a detailed description of what you plan to do to prevent these issues. There is simply no way a single failed PDU should take out the cloud like it did, nor is there an excuse for not being on battery/generator power.
Posted by oldsmoboi, 07-01-2013, 05:14 PM
Well did you complete the storage upgrade?
Posted by serverian, 07-01-2013, 05:20 PM

Quote:

Originally Posted by oldsmoboi

Well did you complete the storage upgrade?

Will be done on Thursday.
Posted by Nick B, 07-02-2013, 01:11 PM
Ahh, the joys of not being in your own datacenter, where you control your power and know that everything will switch over seamlessly.
Posted by WireNine, 07-02-2013, 06:11 PM
Which data center is Cloudive hosted in?
Posted by oldsmoboi, 07-08-2013, 11:53 AM

Quote:

Originally Posted by serverian

Will be done on Thursday.

How did things go?
Posted by blogmaster2003, 07-08-2013, 12:11 PM
Are you Going to give one full Month of credit Like you wrote some posts ago? I just received the invoice. Also did the upgrade happened?
Posted by DavidBee, 07-08-2013, 02:15 PM

Quote:

Originally Posted by k2host

Please give us an update... this is ridiculous.

And for those of you saying just have patience, you clearly don't have a profitable business that is completely shutdown because of this outage. If it was just some blog of mine that wasn't online, no big deal, but we're talking thousands of dollars.

I'm sure we all know how it feels on this forum having downtime.

Don't you have a backup? If you're so reliant on earning from your website than I'm sure you would have a back up plan just incase something like this happens? Maybe you could spin another cloud VM from another provider? As the staff member said compensation will be applied so you wont really lose out.
Posted by oldsmoboi, 07-08-2013, 02:18 PM
The problem for us Cloudive clients was the lack of information coming from the provider. Unless you happen to also be a WHT member, you'll have no idea what happened during this outage.... Cloudive simply "vanished" in the outage period with no ability to contact or get updates.
Posted by DavidBee, 07-08-2013, 02:21 PM

Quote:

Originally Posted by oldsmoboi

The problem for us Cloudive clients was the lack of information coming from the provider. Unless you happen to also be a WHT member, you'll have no idea what happened during this outage.... Cloudive simply "vanished" in the outage period with no ability to contact or get updates.

I do agree with you there about just vanishing. Things like this would really make me wonder: https://twitter.com/Cloudive
Posted by serverian, 07-08-2013, 02:25 PM
I'm very sorry about the lack of communication in these past weeks. We were constantly working on with the upgrade and we were just very unlucky. We had lots of failed hardware and some weird software issues.

This last power failure has wounded us deeply as well.

We will be sending a detailed email to all of our customers after the upgrade is done.

To sum up why upgrade is delayed:

- Delayed shipments due to 4th July and the next day
- Hardware failures
- Weird hardware issues that even Supermicro does not know what would be the cause
Posted by Computaholic, 07-08-2013, 11:03 PM
So has it been confirmed that they vanished? Has anyone made phone calls, disputes, emails?
Posted by serverian, 07-09-2013, 01:50 AM

Quote:

Originally Posted by Computaholic

So has it been confirmed that they vanished? Has anyone made phone calls, disputes, emails?

I'm not sure where did you get that conclusion. We have been online for a long time.
Posted by serverian, 07-09-2013, 03:06 AM
I'm again sorry about the lack of communication. We have sent an update to all our customers and handed out the credits generously.

Add to Favourites Print this Article

Knowledgebase

Did Cloudive Evaporate?

Our Services

Client Menu

Legal