Shared Hosting
Shared HostingKnowledgebase
Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Rackspace Cloud Server Outage
Rackspace Cloud Server Outage
 |
 |
 |
Posted by Joshua, 12-23-2010, 02:14 PM One of Rackspace's Cloud Host Servers has failed. This is the second major outage on this server since June, when it was down for 8 hours. Their SLA policy is as follows, though 5% refund for every hour of downtime beyond the first is basically nothing.Quote:
Dear Customer,
Today December 23rd 2010, at approximately 11:02:35 UTC, your host server became unresponsive and after extensive troubleshooting and attempts to bring it back online, we determined that it experienced a catastrophic failure. All customer data on that host has been lost. If you have backups, we strongly urge you to create a new server from that backup, and we will share the old server's IP address over to the new one.
We sincerely apologize for any inconvenience this may have caused you. If you have any further questions, please contact a member of our support team by visiting us in live-chat or by calling 1.877.934.0407 or INTL +1.210.581.0407. Thank you for your support.
Best Regards,
The Rackspace CloudGlad to have a failover backup hosted at BurstNet for times like this.Quote:
Cloud Server Hosts
We guaranty the functioning of all cloud server hosts including compute, storage, and hypervisor. If a cloud server host fails, we guaranty that restoration or repair will be complete within one hour of problem identification.
Posted by media-hosts_com, 12-23-2010, 02:24 PM In the email you quoted, they refer to the "host server" in singular... This is what's sparking the questions in my mind.
Correct me if I'm wrong... Isn't the whole idea of a "Cloud" to ensure that if one piece of hardware experiences "catastrophic" failure, the cloud can stay online? Basically eliminating the single point of failure?
If there's a single point of failure, then it's not really a cloud... Unless there was a giant power surge or something that fried a whole rack of servers..
Posted by dotHostel, 12-23-2010, 02:44 PM
Correct me if I'm wrong... Isn't the whole idea of a "Cloud" to ensure that if one piece of hardware experiences "catastrophic" failure, the cloud can stay online? Basically eliminating the single point of failure?
If there's a single point of failure, then it's not really a cloud... Unless there was a giant power surge or something that fried a whole rack of servers..
I couldn't agree more.
If it is a cloud there is no oportunity to "catastrophic failures" -- except natural catastrophes of course, nor things like "All customer data on that host has been lost" as there isn't "that host". It seems a serious issue with their SAN (or a Sata Hdd) and no resilient/robust architecture adopted building their "cloud".
Posted by Joshua, 12-23-2010, 03:36 PM In this case, it seems the "cloud" is simply a standard VPS host server.
Posted by media-hosts_com, 12-23-2010, 03:39 PM They also mention their cloud consists of "compute, storage, and hypervisor" nodes...
So loosing all of the customer data would mean the storage node was wiped. But this should be a redundant storage array.
Then they could just re-create your account on a different hypervisor node from the same filesystem. But in no way should the data be non-recoverable requiring an off-site backup restoration.
Unless the storage array is causing the failure...
Posted by dotHostel, 12-23-2010, 03:49 PM
Posted by dotHostel, 12-23-2010, 04:02 PM http://ir.rackspace.com/phoenix.zhtm...761&highlight= Quote:
SAN ANTONIO, Jul 19, 2010
Rackspace(R) Hosting (NYSE:RAX) today announced the launch of OpenStack(TM), an open-source cloud platform designed to foster the emergence of technology standards and cloud interoperability. Rackspace, the leading specialist in the hosting and cloud computing industry, is donating the code that powers its Cloud Files and Cloud Servers public-cloud offerings to the OpenStack project. The project will also incorporate technology that powers the NASA Nebula Cloud Platform. Rackspace and NASA plan to actively collaborate on joint technology development and leverage the efforts of open-source software developers worldwide.
...
OpenStack will feature several cloud infrastructure components including a fully distributed object store based on Rackspace Cloud Files, available today at OpenStack.org. The next component planned for release is a scalable compute-provisioning engine based on the NASA Nebula cloud technology and Rackspace Cloud Servers technology. It is expected to be available later this year. Using these components, organizations would be able to turn physical hardware into scalable and extensible cloud environments using the same code currently in production serving tens of thousands of customers and large government projects.
...
Rackspace and NASA have committed to use OpenStack to power their cloud platforms, and Rackspace will dedicate open-source developers and resources to support adoption of OpenStack among enterprises and service providers. An OpenStack Design Summit hosted by Rackspace was held July 13-16 in Austin, where more than 100 technical advisors, developers and founding members joined to validate the code and ratify the project roadmap. More than 25 companies were represented at the Design Summit including AMD, Autonomic Resources, Citrix, Cloud.com, Cloudkick, Cloudscaling, CloudSwitch, Dell, enStratus, FathomDB, Intel, iomart Group, Limelight, Nicira, NTT DATA, Opscode, PEER 1, Puppet Labs, RightScale, Riptano, Scalr, SoftLayer, Sonian, Spiceworks, Zenoss and Zuora.
Posted by Rens, 12-23-2010, 04:21 PM Host server - data lost - cloud. Can't be combined.
Posted by JohnnyUtah, 12-23-2010, 05:08 PM This is actually unreal considering that they are one of the most expensive provider out there...
Posted by ObjectZone, 12-23-2010, 05:19 PM Way too often do I see "cloud" thrown around in place of a VPS account. Tell customers what it really is. Not saying VPS is bad, just that you shouldn't give your customers an impression that their hosting account is indestructible. I've heard numerous supposedly clued in IT professionals tell me that they don't need backups because they're using a cloud service -- after a few questions about their setup, I discover they're just on a glorified VPS account and have to give them a shocking wake-up call.
--Chris
Posted by dotHostel, 12-23-2010, 05:27 PM Taking in account the press-release Rackspace is running a "true" cloud. It seems an issue with the storage system (hardware or the "Cloud Files" file system).
Posted by bizwizkid, 12-23-2010, 06:12 PM What annoys me is that they notify us by email and ignore the latest communication vehicle ... you know, text messaging!! grrrr
I was just told they will tell us later in the afternoon what happened - so they must've been attacked.
What irritates me to no end is when we set this site up they didn't even automatically do a backup image nor make it clear we should do this. I literally bumped into this feature (thank God) as I would be face-down right now with a potential loss of devastating proportions (major database for a non-profit).
I've never been a fan of RS and will be shopping this service out. I can't stand buying into 'cloud' hype when in essense I bought server space where these rocket scientists probably put my db and website on the same server. Oh, to boot we didn't backup my .svn site (if it matters) and I'm betting we lost tons there too. Hopefully we can restore that from our pc's but, just another pain.
Posted by layer0, 12-23-2010, 06:58 PM
Their cloud is in fact nothing more than a glorified VPS last I checked.
The host server in this case had its own storage array fail I am guessing.
Posted by Mxhub, 12-23-2010, 09:02 PM Lot of disappointment with big names these few days.. first is skype's 24 hours downtime.. now, rackspace cloud couldn't promise 100% uptime or even have a proper backup procedure to keep data safe.
-joseph
Posted by bizwizkid, 12-23-2010, 09:26 PM I gathered from my chats with tech 'support' that there were external forces (they referred to legal issues) that helped them crash. Who knows. All I know is thank goodness I had a backup and no help to them for our having discovered that on our own.
I'll be looking to pay a bit more and move from these waters. They're so caught up on their 'fanatical support' they actually believe it. Too much Kool-Aid is my take.
Posted by Scott.Mc, 12-23-2010, 09:55 PM
The incorrect use of the terminology rubs me up the wrong way but when a provider has to launch their product to jump on the bandwagon they have a choice to use the incorrect terminology to pickup more customers, why would they not use it.
If people actually bothered to question their "cloud" host they will find it's nothing more than a VPS with shared storage and a pretty interface in the overwhelming majority of cases. This was not the original defintion of a cloud.
Posted by DeltaAnime, 12-23-2010, 10:36 PM
The incorrect use of the terminology rubs me up the wrong way..
Francisco
Posted by Scott.Mc, 12-23-2010, 10:42 PM
Francisco
While I agree in principle, I am sure you know exactly what I meant.
Posted by RossH, 12-23-2010, 10:42 PM
Francisco
http://csrc.nist.gov/groups/SNS/cloud-computing/
Posted by plumsauce, 12-24-2010, 01:25 AM
When you go into a used car dealership, do you actually have to ask the used car salesman if the car is actually a car?
Hosts are becoming, or already are, the aluminum siding boiler rooms of the world.
Posted by plumsauce, 12-24-2010, 01:28 AM
"legal issues" can be anything from "we don't want to get sued" to "we are planning to sue our vendor".
Posted by bizwizkid, 12-24-2010, 02:38 PM Here's my letter to serickso@rackspace.com after their weak explanation of this incident.
This is a very weak explanation. What actually happened?
This whole RS 'cloud' is a J.O.K.E.. You sold me a cloud solution that would ensure 100% up-time because of the load-sharing strategies of your software. Now, I find out the hard way that you've essentially hosted my website and my database on a single machine.
You then mock our intelligence by telling me that 'a' host server went down. Do you seriously mean to tell me that of the tens of thousands of servers you host that only one went down? Or is that cloud-speak to several thousands of us on far many more servers that our websites and respective databases were at times permanently lost? If you can't get past that question just how many websites are you implying went down when this one server went down? You lie.
Thank God I had a backup of this site. No thanks to you actually as I discovered this feature purely accidentally when I created our account. I wonder how many of your clients lost their entire sites because of your inability to communicate, nor your automatically providing a backup plan. I doubt you'll be throwing that number around any time soon.
Your quickness to grab my money, deliver an inferior interface, provide an outdated knowledge base materials -- and then email me patronizingly simplistic explanations -- are all reasons I am entirely dissatisfied with Rackspace. I cannot trust my business in your hands. Your 'fanatical' marketing plan was developed and implemented by a fanatic - not a realist and certainly not one who delivered your marketing schemes on integrity.
Thanks too for your concern about the holiday season as well. It's great to know how much you care. My site was down the entire day yesterday - in the one season people donate the most. We probably only lost a thousand or two but, you lost a loyal customer.
Get off the cloud fantasy and sell your services for what they are. Virtual hosting.
Here's their letter 'describing' what happened:
Dear Rackspace Customer,
We’re reaching out to you again in to sincerely apologize for the failure we experienced in one of our host servers on the morning of December 23, 2010. Those that were affected by this hardware malfunction suffered a period of downtime as well as data loss. Please know that we take your trust in our organization very seriously, especially when times are adverse. We will work hard to regain that trust in the face of this incident.
We understand there are many demands on your time with the holidays approaching. We want you to be assured Fanatical Support does not take hiatus on the holidays. If you have any remaining questions or concerns about the incident yesterday, we have support staff standing by 24/7 who are ready to assist you via chat or tickets at [rackspacecloud.com] or via phone at 1-877-934-0407.
We wish you & yours a happy holiday season from the Rackspace family,
Scott Erickson
Rackspace Cloud Servers
Posted by GCM, 12-24-2010, 02:42 PM
This is a very weak explanation. What actually happened?
This whole RS 'cloud' is a J.O.K.E.. You sold me a cloud solution that would ensure 100% up-time because of the load-sharing strategies of your software. Now, I find out the hard way that you've essentially hosted my website and my database on a single machine.
You then mock our intelligence by telling me that 'a' host server went down. Do you seriously mean to tell me that of the tens of thousands of servers you host that only one went down? Or is that cloud-speak to several thousands of us on far many more servers that our websites and respective databases were at times permanently lost? If you can't get past that question just how many websites are you implying went down when this one server went down? You lie.
Thank God I had a backup of this site. No thanks to you actually as I discovered this feature purely accidentally when I created our account. I wonder how many of your clients lost their entire sites because of your inability to communicate, nor your automatically providing a backup plan. I doubt you'll be throwing that number around any time soon.
Your quickness to grab my money, deliver an inferior interface, provide an outdated knowledge base materials -- and then email me patronizingly simplistic explanations -- are all reasons I am entirely dissatisfied with Rackspace. I cannot trust my business in your hands. Your 'fanatical' marketing plan was developed and implemented by a fanatic - not a realist and certainly not one who delivered your marketing schemes on integrity.
Thanks too for your concern about the holiday season as well. It's great to know how much you care. My site was down the entire day yesterday - in the one season people donate the most. We probably only lost a thousand or two but, you lost a loyal customer.
Get off the cloud fantasy and sell your services for what they are. Virtual hosting.
Here's their letter 'describing' what happened:
Dear Rackspace Customer,
Were reaching out to you again in to sincerely apologize for the failure we experienced in one of our host servers on the morning of December 23, 2010. Those that were affected by this hardware malfunction suffered a period of downtime as well as data loss. Please know that we take your trust in our organization very seriously, especially when times are adverse. We will work hard to regain that trust in the face of this incident.
We understand there are many demands on your time with the holidays approaching. We want you to be assured Fanatical Support does not take hiatus on the holidays. If you have any remaining questions or concerns about the incident yesterday, we have support staff standing by 24/7 who are ready to assist you via chat or tickets at [rackspacecloud.com] or via phone at 1-877-934-0407.
We wish you & yours a happy holiday season from the Rackspace family,
Scott Erickson
Rackspace Cloud Servers
Posted by bizwizkid, 12-24-2010, 06:02 PM These morons offered me $15 credit after I spent approximately $8,000 in hosting over the last two years. Run from Rackspace... I'll be right behind you...
Posted by Joshua, 12-24-2010, 11:05 PM
Posted by mainarea, 12-24-2010, 11:17 PM Here's RackSpace's "cloud" configuration, from my chat with support after the server failed: A single server hosting VMs, using 8 HDs configured in RAID 10, is now considered "cloud"? I expected a lot more from Rackspace.Quote:
They're in 3U cynix boxes, and the RAID is generally pretty resilient (8 drives, Raid 10)
Posted by cartika-andrew, 12-24-2010, 11:41 PM
A single server hosting VMs, using 8 HDs configured in RAID 10, is now considered "cloud"?
We have been selling VPS and Cloud solutions for awhile and try to differentiate the 2. It is a HUGE uphill battle. The issue is that a xen/vmware/kvm vps is significantly different then a virtuozzo/openvz vps - and a "cloud" - at least how "I" would define it, is again different from a xen/vmware local storage vps.
Posted by UH-Matt, 12-25-2010, 04:26 AM Completely agree with Andrew above.
We have not launched our cloud just yet, we waited to see people make mistakes and for it to be further defined by other peoples money before investing in what we felt to be the best system (which is coming soon). It's a shame that companies with the biggest budgets (Rackspace and others) seem happy to use the word cloud but in reality not invest and do it the right way!
Posted by winlinuxadmins, 12-25-2010, 05:57 AM When i was thinking for cloud i was come around the Cartika hosting which is great to deal with you can check with it.
Posted by HostingFuze, 12-25-2010, 06:18 AM
But on a more serious note, couldn't legal action be taken? There seems to be some serious conflicts with their SLA, 100% uptime, and.. what $15 that was given as credit for entire data loss?
And wait, why weren't they taking backups in the first place if it's just a simple "VPS" solutions? Is rsync hard to setup? Are they not a huge corporate brand with their own hardware, etc?
Hmm
Posted by HostingFuze, 12-25-2010, 06:22 AM
All these things are questionable. And honestly, I doubt NASA would care if they just saw some random hosting company claiming they work with them. Maybe they host a few NASA backup dns servers, heh.
Posted by dotHostel, 12-25-2010, 08:35 AM
Posted by layer0, 12-25-2010, 04:32 PM
All these things are questionable. And honestly, I doubt NASA would care if they just saw some random hosting company claiming they work with them. Maybe they host a few NASA backup dns servers, heh.
http://www.nasa.gov/centers/ames/new...10/10-172.html
http://www.openstack.org/
NASA isn't necessarily a customer of Rackspace's cloud.
Posted by Winstyn, 12-25-2010, 05:45 PM Cloud means a lot of things right now and I think that will continue for quite some time unti we get it right. Generally in practice new products like this take a lot longer than this to really get off the ground but I believe that a major role is the serious resource and cost savings on all ends that have really pushed the cloud into the mainstream market and an unreal pace.
Nobody wants downtime, and no one wants data loss, that's a given. But, we also want a completely elastic solution to computing at an enterprise level as soon as we caught wind of the idea. So, yes, companies rushed it because they had too, growing pains are expected and natural selection will occur, it's part of the industry.
Now, from the code I have read and the way understand what they have released to the public, rackspace is closer to a true cloud than most. Everyone says cloud is a about redundancy, while I believe it has nothing to do with it. Redundancy should be left up to highly available systems that are designed with redundancy in mind. Not using standard single minded software to fake a redundant solution with underlaying infrastructure.
Cloud in my understanding is seamless distributed computing that allows the service customers to experience a seemless experience without limits. Most providers that I have used that claim cloud allow you to do this. I think they are living up to their text.
Everyone believes that their virtual machines should never go down in the event of any type of hardware failure and if they do then clearly the provider is at fault for having poor infrastructure, hardware, or software. This is simply not the case. The customers have access to virtually unlimited resources at a fraction of the cost of a few years ago, at the same time customers are wanting an unrealistic amount of features and functionality to come with this cost decrease? I suppose a lot of that comes down to market overplay, but without it there would be no hype and no one would want to switch.
Consider also that the hardware in a cloud environment is high density and under high load at all times, the amount of failures that these providers tolerate without customers even noticing is astronomical. The infrastructure is doing it's job. If a node dies and it takes an hour for your machines to come back online that's amazing, a colocated server in the same situation could be offline a week or more.
Also keep in mind that when you as the customer may need a machine with 32gb memory, access to 16 CPU cores, 2tb of storage, you only pay <$1000/mo. Meanwhile to expand cloud infrastructure is on average $5,000 per node. To me that is what makes cloud so useful and cost effective.
Finally, in my vision cloud is this.
Providers build a scalable infrastructure. Users build highly available systems to operate in the space.
Posted by cartika-andrew, 12-26-2010, 08:26 PM Hello Winstyn,
I really liked your post and I agree with a lot of what you have said, but, unfortunately - this area is pretty grey at best. Companies offering a local storage xen/vmware/kvm vps "should", at least in my opinion, advertise it as such. heck, in some instances, this offers some benefits (ie for very heavy DB service, etc) - but, it is different and should be worded as such.Here is where I start to disagree with you. There is no "faking" redundant infrastructure. It is either redundant, or it isnt. This doesnt mean its perfect, and doesnt even mean you will actually get better uptime 100% of the time. But, a redundant infrastructure is fundamentally different then a non redundant infrastructure..Quote:
Now, from the code I have read and the way understand what they have released to the public, rackspace is closer to a true cloud than most. Everyone says cloud is a about redundancy, while I believe it has nothing to do with it. Redundancy should be left up to highly available systems that are designed with redundancy in mind. Not using standard single minded software to fake a redundant solution with underlaying infrastructure.agreed, except how is a user going to have access to 16 CPU cores, 32GB RAM, 2 TB storage, etc on a local storage host node? Chances are, the provider has sold other customers instances on that node. What happens if you try and scale beyond what is available on the node? manual migration correct? extended downtime correct? this is fundamentally different then a pool of resources in which an individual VM is not tied to a specific physical node - and in my mind, this is what differentiates a "cloud" from a normal local storage VM.Quote:
Also keep in mind that when you as the customer may need a machine with 32gb memory, access to 16 CPU cores, 2tb of storage, you only pay <$1000/mo. Meanwhile to expand cloud infrastructure is on average $5,000 per node. To me that is what makes cloud so useful and cost effective. I find it hard to imagine that it is the users responsibility to build highly available systems. I do agree the users need access to various types of infrastructure in order to build out applications to meet their specific requirements. ideally, they could use a cloud infrastructure for specific services (ie httpd, or IIS, or DNS, etc) and a local storage infrastructure, load balancers, etc to build out DB clusters.. end of the day though, simply throwing a fancy portal on top of local storage VMs, at least in my opinion, does not meet any requirements for a "cloud" (which include redundant, high availability infrastructure without single points of failure, real elasticity and most importantly, separating VMs reliance to specific physical servers).Quote:
Finally, in my vision cloud is this.
Providers build a scalable infrastructure. Users build highly available systems to operate in the space.
Anyway, I really enjoyed your post and for the most part, I do agree with what you are saying conceptually - I just think its really important to distinguish between local storage VMs and Cloud instances - and the most important differentiator is separating individual VMs from single physical servers - as once you do that, you have enabled the potential for high availability, fluidity, elasticity, etc... which to me anyway, are the most important characteristics cloud providers need to be continually working towards..
Thanks again and happy holidays...
Add to Favourites 
Print this Article
Also Read
