rapidswitch down?

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > rapidswitch down?

Posted by zoizo, 09-24-2009, 06:26 PM
Is rapidswitch down?
Posted by Steven, 09-24-2009, 06:27 PM
Yes it is.
Posted by CretaForce, 09-24-2009, 06:30 PM
Here is the message they sent few hours ago:
Posted by zoizo, 09-24-2009, 06:34 PM
@CretaForce, thanks.
Posted by ServiceLegacy, 09-24-2009, 06:46 PM
I'm with FutureHosting for my London services (they use RapidSwitch), and they're being useless right now (to put it lightly). They posted this not too long before it happened in their client forum, which is great - apart from I don't check it by habit every day. In this case, the notice was NOT emailed to me so I could not inform my clients of the upcoming maintenance, so I will have many questions to field over the next few hours. Yes, they did try communicating with customers, but only a certain number of their clients are members of the FH forum, so email would have been a far better means of contact. The quicker this is resolved the better.
Posted by futurehosting, 09-24-2009, 06:47 PM
An e-mail was sent to all customers when it was posted in the forum. The e-mail and post to the forum was within 5 minutes of being notified by RapidSwitch.
Posted by kparks, 09-24-2009, 06:52 PM
Core router maintenance annoys me, can't data centers have redundant hardware? The fact their own site isn't online is also pretty bad. If the outage continues how do we get any further updates. All services still appear down. 47 minutes and counting...
Posted by KJKingJ, 09-24-2009, 06:59 PM
Approaching 60 minutes, perhaps the upgrade has gone wrong? Strangely enough, their site and MyServers are down, I thought they hosted that elsewhere?
Posted by CretaForce, 09-24-2009, 06:59 PM
I had the same question and here is their reply:
Posted by ServiceLegacy, 09-24-2009, 07:00 PM
With all due respect, I have a smartphone and have emails pushed to it. I've just checked all my recent emails and haven't got anything from you guys. Are you sure that this mail went out to everyone?
Posted by Steven, 09-24-2009, 07:00 PM
Are you sure it's not your server denying it?
Posted by Bloory, 09-24-2009, 07:03 PM
I didn'r receive notice from Rapidswitch
Posted by ServiceLegacy, 09-24-2009, 07:05 PM
I'm pretty positive that it wasn't me - I've opened a ticket with them, and am getting responses. I always receive bills from the same system which I presume is used to mass mail clients. The email which should have been sent to me would have been forwarded to my Gmail account anyway, and I've checked to see if anything is there - and there isn't. I am assuming that forwarding is working as I've received plenty of emails forwarded to it today.
Posted by kparks, 09-24-2009, 07:09 PM
In their recent sales pitch, they state, at the end of the last financial year (31st March 2009) they made over 11 million pounds profit. Sound like they should invest some more money in their core infrastructure... 65 minutes still down...
Posted by Bloory, 09-24-2009, 07:09 PM
Well it's been 68 mins so far. Is that more than 45 minutes? Wonder if Poundhost will offer me six months free hosting and some champagne to migrate! pmsl!
Posted by Steven, 09-24-2009, 07:18 PM
/me pokes a switch at rapidswitch
Posted by BELLonline, 09-24-2009, 07:24 PM
any minute now aaaaany minute NOW nope now hmm any minute!
Posted by Veltrap, 09-24-2009, 07:25 PM
Steven, you could probably fix it faster. Haha .
Posted by HostThree, 09-24-2009, 07:32 PM
Gutted! I put an ad up on a couple of sites advertising my hosting then the site went down without being told anything about it. Pulling my hair out right about now.
Posted by kparks, 09-24-2009, 07:35 PM
92 minutes and counting... what are they doing?
Posted by paul_davis, 09-24-2009, 07:36 PM
Has anyone telephone them to get an update?
Posted by ValueVPS-Dave, 09-24-2009, 07:41 PM
Only if you can CC them the entire RapidSwitch database.... oh, you can't... their servers are down :-(
Posted by ValueVPS-Dave, 09-24-2009, 07:42 PM
Is this the first bit of outage you guys have experienced then?
Posted by HostThree, 09-24-2009, 07:43 PM
Nah, they've been down a number of times for me.
Posted by gone-afk, 09-24-2009, 07:44 PM
do believe their phones are network based...
Posted by gone-afk, 09-24-2009, 07:45 PM
Theres been a few small issues in the past few months, but usually less than 5 minutes with a weeks notice. Just recently an hours downtime for a datacenter move. Generally good service.
Posted by paul_davis, 09-24-2009, 07:46 PM
Yes - generally they've been great. No problems at all until this - just concerned at how long this is taking, as they are normally true to their stated estimate times...
Posted by yello, 09-24-2009, 07:49 PM
I think someone from rapidswitch should post here to inform us. We are many and there is no other way to communicate with them.
Posted by DATARTIM, 09-24-2009, 07:51 PM
They have just moved to a new DC and so it's not totally surprising that things don't always work exactly as planned. Hopefully things will be back soon.
Posted by kparks, 09-24-2009, 07:51 PM
Nods, have been good for me in the past, this is pretty rough though, communication has sucked.
Posted by jandia, 09-24-2009, 07:55 PM
Whilst we are waiting for iomart/rapidswitch to fix the router maybe we should start considering the compensation levels to be payable. Tomorrow morning our inbox will be filled with requests for compensation owing to our inability to provide a reasonable level of hosting service. Maybe as a goodwill gesture iomart/rapidswitch should consider providing all customers with at the very least a credit equal to say a month’s invoicing. We are in a tough competitive business and this outage by the morning will most likely cost us hosting clients as a result of this appalling situation.
Posted by kparks, 09-24-2009, 07:55 PM
Maybe they didn't actually move centers and they have just sold off all the servers and done a runner. ;-D
Posted by datajack, 09-24-2009, 07:58 PM
I've never had a major problem. They had a couple of minor unexpected outages today (10-15 mins) and I did get an email warning of a 45 min outage but did not expect it to be this long.
Posted by ValueVPS-Dave, 09-24-2009, 07:59 PM
Don't hold your breath for compensation....
Posted by Mxhub, 09-24-2009, 08:00 PM
It is over an hour now.. -joseph
Posted by openmind, 09-24-2009, 08:02 PM
It's actually 121 minutes but I'm not counting, honest...
Posted by Mxhub, 09-24-2009, 08:02 PM
yea yea. lol
Posted by BudgetVPS, 09-24-2009, 08:03 PM
Well..... you mean 2hours! It's now 1am
Posted by paul_davis, 09-24-2009, 08:05 PM
Its suprising there is no status update anywhere on this.
Posted by dacapt, 09-24-2009, 08:10 PM
Evenin folks. Yeah....their VOIP has been clobbered.... "dooo...dooo....doooo....the number you have called is out of service". Oh dear. My 6 Xen Dom0 boxes are feeling pretty lonely in Maidenhead.... Capt.
Posted by SlAiD, 09-24-2009, 08:10 PM
Does anyone have a real ETA? FutureHosting is not helping, but they don't have any answers too, I don't blame then. SL
Posted by datajack, 09-24-2009, 08:11 PM
I suppose it depends on your contracts with them. When I first commissioned a server with them (back when they were 49pence), the box they gave me had faulty RAM resulting in an intermittently unstable system. I had a week or so of instability before it was finally diagnosed. Although they didn't offer compensation, they were happy to cancel a chunk of my fees as a goodwill gesture.
Posted by jandia, 09-24-2009, 08:12 PM
I wanted to watch the James Bond movie tonight and I am dicking about fending off screaming customers. Two of these customers are UK solicitors who are not happy bunnies as they wanted to know why their emails are not working. I am going to be hanging from a lamppost real soon! This maintenance should have been planned more diligently. We all acknowledge that things don’t always go as planned but this situation appears reckless without proper steps in place to ensure such outage would be minimal.
Posted by dacapt, 09-24-2009, 08:14 PM
You and me both Jandia. We have VPN tunnels running thru these guys from everywhere. My monitoring is having a nose bleed. No sleep for us it seems - until red turns to green T.
Posted by datajack, 09-24-2009, 08:22 PM
As far as I can tell, the upgrade was in response to a previously unknown bug in the software running on their core switches. The prognosis being 'change now or probably die soon anyways'. Clearly the upgrade process has not gone to plan for some reason. I've been with them for several year and they have impressed me with their responsiveness, openness to problems, resolutions and upcoming work. The UPS problems a couple of years ago were the only other major outage I have had with them and their response to that gave me lots of confidence with them. Occasionally the brown stuff hits the rotating object and unexpected situations occur. Even the best fault tolerent solutions fail and clusters fall over. They've never given me reason to panic before so I am assuming that this is unforseeable catastrophic technical failure rather with the Cisco kit rather than systemic incompetence by Rapidswitch
Posted by Steven, 09-24-2009, 08:22 PM
Meh... any time now. I have emails to tend too.
Posted by dacapt, 09-24-2009, 08:23 PM
Seriously, what if this ain't fixed by the morning? I for one will be queuing up outside their building with a van to collect my servers. What are we now? Almost 1.5 hours? Anyone else colocating?? T
Posted by BELLonline, 09-24-2009, 08:23 PM
It would be nice if they'd kept us updated, something hasn't gone to plan. I can understand that they're busy but it only takes a minute to bang out an email.
Posted by ServiceLegacy, 09-24-2009, 08:26 PM
Well, this looks like it'll be a sleepless night! @BELLonline, I totally agree. It would be great to hear from someone...
Posted by CretaForce, 09-24-2009, 08:26 PM
Let's hope they will fix it in few more minutes.
Posted by dacapt, 09-24-2009, 08:31 PM
I just have a bad bad feeeling that this sw upgrade on the Cisco router has gone very bad, and that being so shockingly expensive, they only have one of em. Let us all prey to the gods that they have an old knackered x86 kicking around with a spare copy of Smoothwall Coffee time... T
Posted by Cats-Computing, 09-24-2009, 08:35 PM
I concur. Saying to customers "we're waiting for an update from the datacentre" is getting a bit tiresome
Posted by ValueVPS-Dave, 09-24-2009, 08:38 PM
Can't take your servers if there are any unpaid invoices... and even if i did want to pay my current invoice, i couldn't because their website is down....
Posted by dacapt, 09-24-2009, 08:42 PM
You're right! This happened to me a few weeks ago. Planned taking out a server with a week's notice, and the courier was turned away because of their billing system. Invoice Raised 3 days delay Payment Taken So during this interim, their systems think you're bad debtor. A bit shocking that.... T
Posted by EastCoast, 09-24-2009, 08:48 PM
www.rapidswitch.com is loading now!
Posted by BELLonline, 09-24-2009, 08:48 PM
I've got one box up! It looks like things are coming back on now.
Posted by Steven, 09-24-2009, 08:48 PM
my stuffs up.
Posted by londoh, 09-24-2009, 08:48 PM
its all down - phones, network. theres nothing from rapidswitch and its 2hrs 45 mins now. they should be sending out update emails from somewhere by now which could mean that its all gone very wrong. I found this thread looking for info at google (at least google is on the ball) does anybody on here have a contact in rs with a mobile phone by any chance?
Posted by FS - Mike, 09-24-2009, 08:48 PM
Their site might well be up, but our servers there are in the red still. Coming up to the big 3.
Posted by dacapt, 09-24-2009, 08:49 PM
Mine too. Phew. Night night. Cue the grovelling emails...... Cheers T
Posted by ValueVPS-Dave, 09-24-2009, 08:50 PM
Yeah right... it will be someone elses fault.
Posted by Cats-Computing, 09-24-2009, 08:50 PM
Some is back up here as well...
Posted by ValueVPS-Dave, 09-24-2009, 08:51 PM
none of ours are up yet :-(
Posted by HostThree, 09-24-2009, 08:52 PM
I'm back.. w00t... w00t!
Posted by dacapt, 09-24-2009, 08:52 PM
Ooops...spoke too soon. Just 2 boxes back up...
Posted by CretaForce, 09-24-2009, 08:53 PM
Mine are UP again. Let's hope they successfully upgrade the router and they didn't forced to switch back to the old version.
Posted by londoh, 09-24-2009, 08:59 PM
I can login to cp but still waiting for my box in there to come back online. 1:59 bst and nothing yet
Posted by BudgetVPS, 09-24-2009, 09:00 PM
Seem's only a selected few servers are online! What's going on
Posted by datajack, 09-24-2009, 09:00 PM
Half show red, half show green. None are accessible
Posted by richardwasf, 09-24-2009, 09:01 PM
we are at 3 hours. Now in most books this would be called a shambolic attempt at network upgrade. I know other people who use Rapidswitch and this has been going on now for over 24 hours. Should someone get sacked .. oh yes ... should they publish the procedures they are going to use in the future ... oh yes ... R
Posted by datajack, 09-24-2009, 09:04 PM
My server on their '87' network that got moved over from their other hosting facility is what is now affected and still down completely. It is all my fault - I was commenting in the office earlier today on how smoothly that whole migration went.
Posted by ValueVPS-Dave, 09-24-2009, 09:05 PM
we have some servers on the 87.117 and they have never been right since the move
Posted by ValueVPS-Dave, 09-24-2009, 09:06 PM
on 25th of last month I opened a ticket with them about it and they could not be bothered to get off their backsides and investigate - they expected me to do all the running around.
Posted by londoh, 09-24-2009, 09:06 PM
in every book I think seems a lucky few are back but I'm still offline They nned to have a bit of transparency in there methinks. iirc RS are one of the biggest in the UK. must be a lot of customers in there but there's no public forum, which might help to keep em in check
Posted by Nailz, 09-24-2009, 09:08 PM
Two servers I frequently use with IP's in the range 83.142.22x.xxx are still down, however servers in 82.136.xxx.xxx are up?
Posted by richardwasf, 09-24-2009, 09:10 PM
everyone now blames you ....
Posted by dacapt, 09-24-2009, 09:10 PM
You don't think? NO surely not? You don't think maybe they copied off their config.....applied new sw.....rebooted....and the old config didn't restore back in? And that the reason we're seeing a slow return of subnets/boxes is that some poor bugger is currently punching in network routing by hand? Oh boy. What a terrible thought.... As a wise man said...."I pity the fool".... T
Posted by Scott.Mc, 09-24-2009, 09:11 PM
For a provider that is very quick to jump all over mistakes of others and spam how great they are, their communication during this outage is nothing short of pathetic - you'd expect their representatives to be all over it with frequent updates (even if it just says "we are working on it"). That's not even getting started on how long this "upgrade" has taken. There are plenty of better hosts out there, gyron.net is one and poundhost.com is another.
Posted by StreamGBcom, 09-24-2009, 09:14 PM
i got 1 server on 78.129 up and 1 down on 78.129 myservers monitoring isnt working either
Posted by Nailz, 09-24-2009, 09:14 PM
BTW - I can't see Cisco accepting any form of liability for this amount of downtime. If this has happened then it's simply down to poor implementation in the first place without proper testing. How come this wasnt backed out in the maintenence window that was given? They've got a few essay's to write after this!
Posted by yello, 09-24-2009, 09:16 PM
Down again... only few minutes of uptime
Posted by ValueVPS-Dave, 09-24-2009, 09:17 PM
all of out 87.117 range are down, all of our 95.154 range is down - and the RS site is down too
Posted by Cats-Computing, 09-24-2009, 09:18 PM
Yeah everything is down again
Posted by datajack, 09-24-2009, 09:18 PM
Hmm ..their main site is down again. How likely is it that a reversion plan is now in operation. "Quick guys! Put it back how it was and pretend you didn't touch it!"
Posted by StreamGBcom, 09-24-2009, 09:20 PM
mine down again aswell
Posted by ValueVPS-Dave, 09-24-2009, 09:21 PM
I am seriously starting to wonder if the hosting business is for me. No matter how many good decisions I make, i am still at the mercy of someone elses bad ones.
Posted by ServiceLegacy, 09-24-2009, 09:22 PM
Again? I was writing a cheery "all is okay" message to clients as well...spoke too soon.
Posted by DATARTIM, 09-24-2009, 09:23 PM
Everything down again.. I really hope for everyone here that they get it resolved by 8am. These things do happen and I'm sure they are all working hard to resolve it, but communication is key.
Posted by ValueVPS-Dave, 09-24-2009, 09:24 PM
I have to admire your positive attitude - you just made me put the razor blade away ;-) Thanks!
Posted by openmind, 09-24-2009, 09:24 PM
MyServers now back up, no activity on ports though...
Posted by StreamGBcom, 09-24-2009, 09:25 PM
Does anyone think they will offer any money back for this? would be nice to pass something down the line to hang on to customers.
Posted by richardwasf, 09-24-2009, 09:25 PM
the information i have received from someone who got a more personalized reply today is that they are running quite 'new' hardware and a specific IOS for them. this could go on for hours .... R
Posted by ValueVPS-Dave, 09-24-2009, 09:26 PM
Clicking on peoples sigs in here has made me realise just how many of us keep our websites in the same place as our servers.....
Posted by datajack, 09-24-2009, 09:27 PM
Thankfully, I have no paying clients directly affected by this, though plenty of 'community' users who will be baying for blood right now which is probably worse (these guys moan when the server clock is five minutes out). I'm off to bed and will hope it seems all like a bad dream when I get up. Good luck all!
Posted by ValueVPS-Dave, 09-24-2009, 09:31 PM
we have 2 boxes in the 95.154 range up
Posted by ValueVPS-Dave, 09-24-2009, 09:32 PM
forget that :-(
Posted by jandia, 09-24-2009, 09:33 PM
This reminds me of the saying in the movie Armageddon “We’re On . . We’re Off”! Our server is currently up but I am too tired and have lost the will to care until the morning!.
Posted by ValueVPS-Dave, 09-24-2009, 09:33 PM
I just got an email....
Posted by ValueVPS-Dave, 09-24-2009, 09:33 PM
Never thought i'd be happy to see SPAM :-)
Posted by StreamGBcom, 09-24-2009, 09:34 PM
the ones that were on earlier are still down , but the ones thet were down are now up
Posted by ValueVPS-Dave, 09-24-2009, 09:35 PM
Polaris (95.154.xxx.xxx) is UP again at 25/09/2009 02:28:10AM, after 3h 25m of downtime. Bellatrix (SolusVM) (95.154.xxx.xxx) is UP again at 25/09/2009 02:28:25AM, after 3h 26m of downtime.
Posted by zibby, 09-24-2009, 09:37 PM
^ mine was down before and is still down
Posted by ValueVPS-Dave, 09-24-2009, 09:39 PM
A communication at last.....
Posted by ValueVPS-Dave, 09-24-2009, 09:39 PM
sounds to me like they have reverted so we'll probably have to go through this all over again.
Posted by richardwasf, 09-24-2009, 09:43 PM
test rig to try it out on first ? perhaps not then. R
Posted by ValueVPS-Dave, 09-24-2009, 09:44 PM
That would seem like the most obvious way to do it - we did it with servers for the 'millenium bug' and it saved us from potential problems....
Posted by ValueVPS-Dave, 09-24-2009, 09:46 PM
i'm just waiting for the 87.117 range now. Anyone else still got servers down?
Posted by kparks, 09-24-2009, 09:49 PM
Likewise still waiting for 87.117 range.
Posted by dacapt, 09-24-2009, 09:50 PM
More green lights than red for me now.... You guys popping back online?
Posted by ValueVPS-Dave, 09-24-2009, 09:50 PM
everything in the 95.154 range is up 87.117 range is still down
Posted by StreamGBcom, 09-24-2009, 09:50 PM
78.129 all up now
Posted by PCS-Jason, 09-24-2009, 09:51 PM
We're up at the minute but how long that will last is another matter
Posted by ServiceLegacy, 09-24-2009, 09:51 PM
I'm back...
Posted by ValueVPS-Dave, 09-24-2009, 09:51 PM
/me needs sleep but has to wait for a particular server in the 87.117 range to come to life so he can migrate a very important client away from RapidSwitch :-(
Posted by zibby, 09-24-2009, 09:53 PM
my VPS in the 95.154 range isn't up yet
Posted by dacapt, 09-24-2009, 09:56 PM
I'm hitting sack. I'll deal with the fallout in the morning. At least I won't need to set my alarm... NNight all...nice meeting ya T
Posted by magdesign, 09-24-2009, 09:57 PM
I'm on 87.117 range and all is back up for me.
Posted by PCS-Jason, 09-24-2009, 09:57 PM
/Sigh... The joyful life of a Hosting Provider. Night Mate
Posted by ValueVPS-Dave, 09-24-2009, 09:57 PM
I was told by Francesca at RS today that their normal SLA is 1 day for every hour of downtime (if you have an SLA). They state they have over 4000 servers, and we've experienced over 4 hours of downtime in a 24hour period... so thats 4 x 4000 = 16000 hours of downtime = 43.83 years of refund...WOW - i wouldn't like to have to foot that bill (if there were any SLA's in place)
Posted by StreamGBcom, 09-24-2009, 10:01 PM
If they have reverted back there probaly wait to we all go to bed then try again lol
Posted by ValueVPS-Dave, 09-24-2009, 10:02 PM
87.117 range is still not up - servers have been down over 4 hours now.
Posted by richardwasf, 09-24-2009, 10:06 PM
it all just went away yet again. R
Posted by ValueVPS-Dave, 09-24-2009, 10:08 PM
This is an absolute joke
Posted by richardwasf, 09-24-2009, 10:12 PM
i think everyone who has posted here should send rapidswitch a book, say 'Idiots guide to network administration' On a more serious note, I just hope they get this fixed before 5am now, otherwise real traffic will start to be effected at which point a torrent of crap will flood their already broken phone system. R
Posted by PCS-Jason, 09-24-2009, 10:15 PM
They are just closing all tickets now about this as well. Nice one...
Posted by FS - Mike, 09-24-2009, 10:15 PM
We've still got a server down. Primaries are up though, that's what counts I guess.
Posted by ValueVPS-Dave, 09-24-2009, 10:16 PM
I still need to move this client - i am desperately trying to cling onto my clients but i know i will lose a load more tomorrow (today now). This is the part that annoys me about all of this. RS know that a lot of small businesses depend on them yet they seem to have little regard for them. I had to stop advertising as it just wasn't a good idea to bring new clients in with the state of things for this past month. Now i just need to worry about paying the RS bill at the end of the month :-(
Posted by StreamGBcom, 09-24-2009, 10:18 PM
mine are all back up now trouble is if i go bed now and it goes down again and the myservers sms monitoring goes down i wont know.Think i need to look into 3rd party monitoring.
Posted by ValueVPS-Dave, 09-24-2009, 10:19 PM
Try Pingdom - we've been using them a while now
Posted by ValueVPS-Dave, 09-24-2009, 10:19 PM
http://www.pingdom.com/reports/67hjby7nys9j/ a few of our monitors are set to public so feel free to take a look.
Posted by ycnhosting, 09-24-2009, 10:21 PM
I've got a ping
Posted by ValueVPS-Dave, 09-24-2009, 10:22 PM
Looks like all of ours are up now too :-)
Posted by PCS-Jason, 09-24-2009, 10:23 PM
Question is dare you move anyone incase the network goes again mid-transfer
Posted by ValueVPS-Dave, 09-24-2009, 10:23 PM
Mars (XEN Node) (87.117.xxx.xxx) is UP again at 25/09/2009 03:21:36AM, after 4h 19m of downtime. Eris (OVZ Node) (87.117.xxx.xxx) is UP again at 25/09/2009 03:21:35AM, after 4h 19m of downtime. Neptune (XEN Node) (87.117.xxx.xxx) is UP again at 25/09/2009 03:22:06AM, after 4h 20m of downtime. Yay!
Posted by ValueVPS-Dave, 09-24-2009, 10:24 PM
if i don't move him, he will leave - he is as fed up with Rapidswitch as I am
Posted by StreamGBcom, 09-24-2009, 10:24 PM
thx startech will have a pat butchers
Posted by ValueVPS-Dave, 09-24-2009, 10:30 PM
You are welcome - they do a free trial as well.
Posted by richardwasf, 09-24-2009, 10:31 PM
i spent nearer 5k a month with rs. I cant move as quickly as some of you seem to be able to, but by the end of the year, perhaps Feb/Mar i will be spending no more than 1k. I just cant take the hit like this. R
Posted by richardwasf, 09-24-2009, 10:57 PM
the network is back up, but packet loss as before .. oh joy ... i sense another outage at some point. R
Posted by ValueVPS-Dave, 09-24-2009, 11:32 PM
i've not seen any packet loss yet
Posted by richardwasf, 09-24-2009, 11:41 PM
seems to be happening on the 87.117 network, specifically 87.117.211.61 drops 10% now and then, but not all the time. Hopefully it is just a tranistory problem as other networks are restored. R
Posted by chrisbyrd, 09-25-2009, 04:14 AM
Sounds like a stressful night for all concerned
Posted by CretaForce, 09-25-2009, 04:18 AM
New update:
Posted by paul_davis, 09-25-2009, 04:36 AM
RapidSwitch have always been great for us, and this event is the first that has been really concerning. Any company that makes £11 million profit but doesnt invest in a backup core router (cost ~£25,000) for instances like this is cutting a very big corner. It makes no sense. What happens if this current router fails at some point in the future - they'll have to order a new one in? How long would that take!
Posted by Oldboy, 09-25-2009, 05:18 AM
They've just lost me as a potential customer. I was researching dedicated server hosting last night and it seemed like they had some good deals, then their site went down... Well, all hosts can have brief outages I thought, no biggy, I'll check back a little later. Two hours later, their site still unavailable, and their domain names would no longer resolve. So, I thought I'd do a little research. My first concern came when I noticed that they only use their own DNS resolvers, inside their own domain to resolve their domain and the two DNS servers have very close IP addresses so are probably located on the same physical network... bad network design, black mark 1... They also didn't seem to have any status service or page hosted outside their own nework... black mark 2... Then I found this thread and discovered that this was planned (if you can call 45 minutes planned) emergency maintenance on their single core router. What is the point of backup generators, multiple peering agreements etc, if your network has a single point of failure like this??? Black marks 4 to 10! Despite their attractive pricing and generous machine specs and bandwidth allowances, there is no way I'd sign up with a company that is as clueless as this.
Posted by tekky, 09-25-2009, 05:37 AM
To be fair they have not to long ago moved into a new much bigger site, I'm sure they are planning on installing a secondary backup core router soon. I've been a customer at rapidswitch for quite some time now and this is the only major outage they've had in years and i've never had any problems, hopefully all the existing problems will resolve soon.
Posted by Oldboy, 09-25-2009, 06:02 AM
I don't really see moving to a new facility as mitigation. If anything it is yet another black mark. Why didn't plan their new facility with a least two routers from the start and test everything thoroughly ? To say they might be adding a secondary router 'soon' just suggests a continuing period of outages while they reconfigure their network.
Posted by londoh, 09-25-2009, 06:04 AM
yep likewise, but there have been lots of niggles and smaller issues lately and I have to agree strongly with paul_davis here: and Oldboy has it right on the money: Its hard for anybody to disagree with those points and imho Rapidswitch absolutely need to address them. I just read somewhere there are 4000 servers in there - its a ridiculous position to be in. and at 9:01 RS stated: "we do not anticipate" = "actually we dont whats going on" so reality is it looks like it aint fixed yet
Posted by tekky, 09-25-2009, 06:10 AM
I'm sure they are aware of this thread and will act on it accordingly. I got this message:
Posted by richardwasf, 09-25-2009, 06:37 AM
they have packet loss within their own network at the moment, which has been on going since the changes. The 'high CPU' notification is misleading. I am trying to capture this so I can pass it to them. If I do i'll post it here as well. R
Posted by CretaForce, 09-25-2009, 06:42 AM
I have 2 servers there and I don't see packet loss. I ping each other.
Posted by datajack, 09-25-2009, 07:07 AM
Their emails indicate that they are running a cluster of routers. Clearly something caused that to fail catastrophically. The initial mail seemed to imply that the initial problem was previously unknown and that the 'fixed' firmware was put together specifically for them so there was little or no chance for them to have predicted this problem. I have also been a customer for a long time. They did have a major UPS problem a few years back. It is the way they handled the aftermath that impressed me. As a matter of course, I was sent a fairly detailed explanation of the problem and how the UPS manufacturer was flying over some engineers from a far away land to investigate how the supposedly impossible happened, and the steps they were planning to try and ensure that this problem wouldn't impact service in future. It's that honesty and openness that impressed me and I hope we will see similar attention to detail now that they are much bigger. The lack of communication during the outage was worrisome, but it is quite possible that their network failed across multiple sites. I am more than willing to give them the benefit of the doubt at the moment.
Posted by ValueVPS-Dave, 09-25-2009, 07:10 AM
Yeah it's difficult to fire up a laptop with USB broadband dongle to fire off a few messages on WHT, WHC, Twitter etc.
Posted by datajack, 09-25-2009, 07:11 AM
They openly admit that in the same email you have quoted that last paragraph from. The previous paragraph reads :
Posted by SubS[T], 09-25-2009, 07:23 AM
I hope they get everything fixed pretty soon :\
Posted by richardwasf, 09-25-2009, 07:38 AM
it seems I can not upload files, but I have a pingplot showing 10% packet loss within the rs network. looks like the problems from yesterday have not been resolved at all. Just spoken to two other customers of rs and they say this has been on going for over 48 hours now, blimey. R
Posted by CretaForce, 09-25-2009, 07:51 AM
In what subnet is your server?
Posted by richardwasf, 09-25-2009, 08:02 AM
they are in the 87.117 ranges, yours ? it looks like the rs network just went away again, or has got really bad all of a sudden. R
Posted by CretaForce, 09-25-2009, 08:08 AM
78.129.242.xxx and 95.154.227.xxx
Posted by NETlayer, 09-25-2009, 08:13 AM
This is unacceptable. Last night everything was offline for more than 2 hours. Now again everything is offline. Totally unacceptable for a web hosting company.
Posted by londoh, 09-25-2009, 08:18 AM
I dont know anything about Cisco VSS 1440 but however it works, the email advising of the need for maintainence stated: "We are using A Cisco VSS-1440" and... "The maintenance is to perform an emergency upgrade of Cisco software" So it looks like its only one controller even if that is a cluster, and anyway in my book a hi tec datacenter housing 4000 machines shouldnt ever be in a position where it has to do "emergency" work on the only route in/out - which clearly this was. yep you're correct, I agree. Its still ongoing. and I agree with this too. Its very difficult to do - unless you've put a contingency in place beforehand. Sh1t does happen. dont get me wrong here - I've trusted RS for several years now, and want to continue to do so, but this morning I have to explain why things didnt work as they should last night, and why I aren't on plan B. Truth is I dont want to move out, but if goes wrong again...? I need to understand a little of whats wrong and what the solution is. l.
Posted by Votii, 09-25-2009, 08:20 AM
Do you think it is possible to claim a discount for next month due to the downtime over the last 2 days? Or does RapidSwitch not have a SLA policy?
Posted by BELLonline, 09-25-2009, 08:22 AM
I absolutely agree, I was thinking about that last night; it only costs £20 for a broadband dongle and a few minutes to write a message to post out. I'd suggest that it's extremely unlikely that people without an SLA agreement in place will get anything. Their network is generally great, they always respond to support tickets in good time, but one thing I've learned about them after being with them for about 3 years is that they don't give anything away and are very geared towards making as much money as possible (who can blame them). Anyway, it's impossible to stream at the moment from their network - I'm glad I spread my servers around datacentres and only have a couple with them. I wouldn't want to have all my eggs in one basket right now. Last edited by BELLonline; 09-25-2009 at 08:28 AM.
Posted by Votii, 09-25-2009, 08:30 AM
Would the customers which own a 100% SLA server from RapidSwitch be refunded for the whole month or just the time the server was down?
Posted by garrence, 09-25-2009, 08:32 AM
We decided to try out RapidSwitch so took out a new server. It was online for just two hours before we got a mail saying: "We are currently experiencing some connectivity issues" And it's been down for the last half hour.
Posted by NETlayer, 09-25-2009, 08:36 AM
I'm sorry that doesn't justify anything. They probably loose money from things like this, instead of making. I'm sure lots of their customers will be displeased from this situation (including myself). The only way to make money, is to spend money, not to make cutoffs and not buy a spare router. At least that is how i see it. This issue will be heard, it will only do harm to RS.
Posted by BELLonline, 09-25-2009, 08:36 AM
It sounds like you've joined at a bad time, the uptime is generally good (apart from once last year and the last month) http://www.hyperspin.com/publicrepor.../16394/2/53416 I didn't really mean to justify it (although it sounded like I was lol), just saying that's how they are. I'm not sure what SLA customers will get in terms of compensation, but the rest of us will probably have to take the hit ourselves (I'm sure most people here, like me, offer compensation to their customers for downtime)
Posted by datajack, 09-25-2009, 08:47 AM
The largest (so far) explanation mail explicitly mentions a cluster.. I suppose that the first mail could mean that they only detected a problem in one of the cluster nodes maybe. I dunno. Hopefully we'll get a fuller picture soon.
Posted by NETlayer, 09-25-2009, 08:54 AM
How long has everything been offline this time?
Posted by garrence, 09-25-2009, 09:00 AM
One hour now. More last night I understand.
Posted by CretaForce, 09-25-2009, 09:12 AM
It should be a part of their network down. My two servers are up and also 2 of their own nameservers are up too. Only one of their nameservers is down.
Posted by ChrisGlo, 09-25-2009, 09:21 AM
We've just joined them and this is really unacceptable. 3 dedicated servers, a load of cash paid to them and masses of work carried out last night. Their redundancy has obviously failed or was never in place in the first place, that was something sold to us over the last few days. We have an SLA agreement so will be asking for cash. In fact thinking of transferring back to other host, demanding a refund and suing them for miss-selling and loss of earnings. They'll have professional indemnity insurance so we should all hit it as this is far from professional ! Interestingly they are no longer taking phone calls just hanging up after 20seconds ! We were about to move the other 6 servers to them, glad we didn't yet.
Posted by jarimh1984, 09-25-2009, 09:23 AM
Subject: Connectivity Issue - RSH North: 25/09/2009 Sent: 25/09/2009 This message is an update to our previous announcement regarding a connectivity issue in RSH North. Following consultation with Cisco and works to alleviate the high CPU usage affecting the router cluster in RSH North, some subnets in this zone will be experiencing an interruption in connectivity. Only RSH North is affected by this issue. We are working as fast as possible to restore all network functionality. Our monitoring services do inform us when servers are unavailable and we will always respond as proactively as possible. In the mean time we will likely not be able to respond to individual tickets or phone calls with our normal response time or provide further information than we have in this message. As soon as we have more information we will provide it to you. Regards, RapidSwitch Support Subject: Emergency Maintenance: Connectivity Interruption - 25/09/09 Sent: 25/09/2009 In order to try and correct the problems being experienced by the RSH North router cluster, we are performing changes to the configuration. These changes are fundamental configuration changes and will result in a loss of conncetivity to all servers within the RSH North zone. This could last for anything up to 90 minutes. We recognise that this is a very significant action but we also need to address the seriousness of the situation. It requires prompt and effective action which is why we are taking this step. As soon as we have updates on the effectiveness of the solution we will provide them to you. Regards, RapidSwitch Support
Posted by jarimh1984, 09-25-2009, 09:25 AM
Guess in what RSH section all my servers are
Posted by BELLonline, 09-25-2009, 09:26 AM
Ah great, so aLL SERVERS are going to be down for over 90 minutes now with no notice at all (it looks like the poster above was lucky and actually got an email).
Posted by StreamGBcom, 09-25-2009, 09:29 AM
Here we go again day 2, definitely going to be loosing people after this.
Posted by NETlayer, 09-25-2009, 09:30 AM
Does anyone know if they actually HAVE redundancy?
Posted by CretaForce, 09-25-2009, 09:31 AM
The servers down again :/
Posted by HostThree, 09-25-2009, 09:32 AM
99.423% *Cries even more*
Posted by londoh, 09-25-2009, 09:35 AM
I know the chief engineer and md should GET redundancy - lol
Posted by kparks, 09-25-2009, 09:36 AM
this is now getting beyond a joke...
Posted by blueskimonkey, 09-25-2009, 09:40 AM
This is very annoying now, I suspect us colocation customers will get no compensation for this outage but lots of our customers expecting one. That said rapidswitch has been pretty solid for the last three years, lesson learnt from this one though not to have all my servers in one DC, once we are back online I will be distributing them across the UK. Does anyone have any public uptime stats for the rapidswitch network which I can share with my customers?
Posted by ValueVPS-Dave, 09-25-2009, 09:46 AM
Nope but i have some terrible uptime stats for my own servers on their network :-(
Posted by kparks, 09-25-2009, 09:47 AM
nothing public to view, but for the last 30 days, uptime for me as only been 99.14%.... over the last year 99.93%
Posted by StreamGBcom, 09-25-2009, 09:48 AM
Welcome back to the drawing board startech
Posted by PCS-Jason, 09-25-2009, 09:48 AM
Here we go again...
Posted by kparks, 09-25-2009, 09:49 AM
I think this is karma, for Rapidswitch's recent poundhost migration sales email ;-D
Posted by blueskimonkey, 09-25-2009, 09:49 AM
Really? Mine are quite ok http://www.pingdom.com/reports/n2ry6...=Neptune+Web01 However i screwed up last night I paused the stats during maintenance which now make the records look false. What does everyone else do in terms of stats? i.e. maintenance windows do you let stats continue to track downtime or pause them because its planned maintenance? Thanks Aaron
Posted by KJKingJ, 09-25-2009, 09:51 AM
While their performance has been good while i've been with them, i've noticed a significant increase in downtime and network instability recently. I already committed not to give them any further business when they mis-used the accidental leak of PoundHost's mailing list, i'm now seriously considering moving all my services away from them. Their VPS infrastructure, while very powerful, is very unreliable compared to the servers.
Posted by kparks, 09-25-2009, 09:51 AM
Any outage is an outage in my clients mind, as they can't access their server. So I track any and all downtime. I understand pausing stats for scheduled maintenance, but I don't.
Posted by PCS-Jason, 09-25-2009, 09:52 AM
We're back
Posted by Bloory, 09-25-2009, 09:53 AM
No notice again here, little interest in my support ticket this morning when I could access there system. Great stuff, wonder how I get my colo box deracked and moved. Lots of downtime (I didn't ask them to migrate me to their new facility) lately. Good for my business Thanks
Posted by kparks, 09-25-2009, 09:54 AM
87.117 network still looks partially down. I suspect things may all go down again like yesterday
Posted by CretaForce, 09-25-2009, 09:55 AM
Servers back online after 27 minutes of downtime.
Posted by blueskimonkey, 09-25-2009, 09:55 AM
Mine are all back up. I'm still fairly happen with rapidswitch the past 3 years have been bliss although there has been some increased activity recently especially the recent data centre move. My next plan is to now distribute all my colocated servers in rapidswitch across the UK.
Posted by daveharris, 09-25-2009, 09:57 AM
Everything including their own sites offline at 14:50
Posted by garrence, 09-25-2009, 09:57 AM
Ours is back too. RS' monitor tool shows downtime was 2 hr 01 min.
Posted by ValueVPS-Dave, 09-25-2009, 09:57 AM
30 Days of stats - monitored every 1 minute - 5 servers - 2 different subnets Uptime 99.16% Downtime 6h 24m 28s The average downtime length is 34m 57s Number of downtimes 11 The longest downtime was 4h 20m on 24/09/2009 11:02:06PM The shortest was 1m 47s on 06/09/2009 3:58:08PM ------------------------------------------------------ Uptime 99.12% Downtime 6h 39m 40s The average downtime length is 1h 6m 36s Number of downtimes 6 The longest downtime was 3h 25m on 24/09/2009 11:03:10PM the shortest was 5m on 09/09/2009 11:23:10PM --------------------------------------------------------- Uptime 98.90% Downtime 8h 19m 19s The average downtime length is 55m 28s Number of downtimes 9 The longest downtime was 4h 19m on 24/09/2009 11:02:36PM the shortest was 1m on 24/09/2009 3:01:36PM ---------------------------------------------------------- Uptime 98.71% Downtime 9h 47m The average downtime length is 8m 53s Number of downtimes 66 The longest downtime was 4h 19m on 24/09/2009 11:02:35PM the shortest was 23s on 19/09/2009 1:55:23PM ----------------------------------------------------------- Uptime 99.31% Downtime 5h 13m 5s The average downtime length is 44m 43s Number of downtimes 7 The longest downtime was 3h 26m 1s on 24/09/2009 11:02:24PM the shortest was 2m on 06/09/2009 3:59:24PM
Posted by Bloory, 09-25-2009, 10:02 AM
My boxes are still down, at least their site is up I suppose
Posted by jarimh1984, 09-25-2009, 10:05 AM
My one server located at "RSH North Upr" is still down other servers that are located in "RSH North Lwr" are working fine.
Posted by KJKingJ, 09-25-2009, 10:06 AM
My server in RSH North Upr has just come back up (78.129.*.*)
Posted by colodedi, 09-25-2009, 10:07 AM
My servers at RapidSwitch are also offline. I can't get through to them so I have called iomart head office and was advised by the receptionist that they are too busy fixing the problem to answer the phone? What is that all about? I would have expected them to have disaster recovery procedures such as calls being routed out to other iomart offices so at least us people with thousands of pounds worth of kit in Maidenhead have some peace of mind. I did find out that the operations director at iomart is Sarah Haran so if we dont get results soon we can try to get them from the top. I was advised that Sarah can be contacted by email on the address sarah [at] iomart [dot] com
Posted by kparks, 09-25-2009, 10:09 AM
Very unprofessional.
Posted by londoh, 09-25-2009, 10:13 AM
yep 79.129 is back but still a lot of probs... 373 packets transmitted, 206 received, 44% packet loss, time 373110ms and latency all over the place from 25 (usual) - 250ms from here
Posted by PCS-Jason, 09-25-2009, 10:14 AM
Yer ping times are all over the place here too, better than nothing at all though
Posted by eatmyd0g, 09-25-2009, 10:15 AM
first post. struggled with this the last few hours. i have 2 servers with rapidswitch .. just wondered if anyone thinks this could have something to do with the iomart purchase?
Posted by blueskimonkey, 09-25-2009, 10:18 AM
Not sure with the iomart purchase i was and i think still am looking forward to distributing my servers across the UK with them. I am a bit concerned though to why a planned maintanance window last night took down the network surely they had a spare and the redundency
Posted by blueskimonkey, 09-25-2009, 10:22 AM
What do you guys have in terms of sla policies for your customers? I think I need to have one setup and on this occassion offer my customers something in return for todays problems. Thanks
Posted by StreamGBcom, 09-25-2009, 10:29 AM
Most of the Emails i have had are regarding compensation.Looks like this months profit has gone. Rapid need to reimburse customers for this sla or no sla.
Posted by colodedi, 09-25-2009, 10:30 AM
Maybe not directly to do with the iomart purchase but quite possibly due to a rushed data center migration. The data center was moved from the priors way center on the blue square site across the other side of town to Spectrum House. I personally think they rushed the migration and did not plan ahead.
Posted by blueskimonkey, 09-25-2009, 10:33 AM
Yep it's tough luckily a lot of my customers are very understanding and have noticied a problem of this scale in the past years, I have just checked my site and it looks like I'm missing an SLA agreement although on a few pages i do quote to a certain uptime percent. What is the web hosting standards in terms of compensation is it one more free hosting or a percentage? I'm willing to take a hit in the pocket as a good will gesture for my customers along with details of how I am to avoid such issues in the future.
Posted by garrence, 09-25-2009, 10:34 AM
RapidSwitch offer a "100% SLA" on dedicated servers. What does this mean in practice?
Posted by colodedi, 09-25-2009, 10:37 AM
Are you sure that is not a hardware SLA?
Posted by eatmyd0g, 09-25-2009, 10:40 AM
i've read that that is only for high end servers
Posted by garrence, 09-25-2009, 10:42 AM
Quite possibly. The offer we signed up to just says "100% SLA". I've mailed asking for details.
Posted by kparks, 09-25-2009, 10:43 AM
Ok my servers are back now. 1Hr 10 Minutes of downtime.
Posted by jarimh1984, 09-25-2009, 10:47 AM
Subject: Emergency Maintenance: Connectivity Interruption - 25/09/09 *UPDATE* Sent: 25/09/2009 In order to try and correct the issues affecting the RSH North router cluster we have made some fundamental changes to the configuration. Whilst making these changes we were forced to reboot the cluster and this caused a loss of connectivity to all servers within the RSH North zone. This work has now been completed and we are focusing on restoring all connectivity. Over 80% of the servers in the RSH North zone are already back online and we are working on the remainder. Please bear with us whilst we restore your connectivity. The process has removed the VSS cluster configuration from the RSH North zone routers. These routers are now performing their role in a similar configuration to the one which served RHC stably for over two years. However, this means routing is operating in a non-redundant manner for the time being. Until we have had more time to assess the impact of this work we cannot issue an all clear on the problems experienced in the last two days. Once again, as soon as we have further information it will be provided. Regards, RapidSwitch Support
Posted by jarimh1984, 09-25-2009, 10:50 AM
Some info I found in my documents about SLA: IP transit, network services and colocation 100% Network Availability and Quality 1 day service credit for each hour when the service is not available The maximum credit in any one month is 100% of the monthly fee for the contracted service If there are simultaneous problems that two or more SLAs apply to, only one SLA claim may be made for the incident. EG If there is a power failure that creates a network failure, only claims on the power SLA may be made. Dedicated Servers 100% Network Availability and Quality 0.5 days service credit for each hour when the service is not available The maximum credit in any one month is 50% of the monthly fee for the contracted service If there are simultaneous problems that two or more SLAs apply to, only one SLA claim may be made for the incident. EG If there is a power failure and a network failure, only claims on the one SLA may be made.
Posted by garrence, 09-25-2009, 10:53 AM
Gah, I took my eye off it and now it's down again. For 13 minutes according to RS' monitor.
Posted by garrence, 09-25-2009, 10:55 AM
Ah no... RS monitor says it's been down (for 16 min now) but it's not.
Posted by ValueVPS-Dave, 09-25-2009, 10:57 AM
we still have three boxes down
Posted by CretaForce, 09-25-2009, 10:57 AM
Maybe the monitor is down.
Posted by StreamGBcom, 09-25-2009, 10:59 AM
My rs monitor is working
Posted by richardwasf, 09-25-2009, 11:12 AM
so is mine. But alas all servers are down. R
Posted by ValueVPS-Dave, 09-25-2009, 11:12 AM
Popped out and don't know the score - is everything supposed to be up and running now?
Posted by richardwasf, 09-25-2009, 11:15 AM
the score is 10 complete failures to rapidswitch, 0 service to customers. R
Posted by StreamGBcom, 09-25-2009, 11:16 AM
Still appears to be boxes down but most are running star tech there working on the rest
Posted by ValueVPS-Dave, 09-25-2009, 11:17 AM
Damn - thanks for the update though. I am starting to feel like i am telling lies to my clients when i say "it's not us" - i wish RS would get this sorted.
Posted by StreamGBcom, 09-25-2009, 11:21 AM
Yeah customers dont care or even understand sometimes bout whos fault it is, they pay us so we are the ones there ganna moan at for being off.Trouble is hosting accounts are easy to change, more of a drama moving a whole server(s) unless your colo. Last edited by StreamGBcom; 09-25-2009 at 11:22 AM. Reason: typo
Posted by BELLonline, 09-25-2009, 11:29 AM
My thoughts exactly, it's ironic really when you consider that I'm having to move some servers from RS to PH (and I'm sure many others are cancelling servers to move as we speak). I probably wouldn't have moved any servers over if they hadn't sent that email, but they need to understand that people will be less willing to deal with them if they are known to use underhand business practices like that.
Posted by ValueVPS-Dave, 09-25-2009, 11:43 AM
I don't know if i'll need to move any servers away. at this rate i'll have no clients left to put on them.
Posted by blueskimonkey, 09-25-2009, 11:53 AM
My servers are currently up at the moment so all customers happy again. I'm now working on a plan and will be sending out a communication to all my customers. I had a total loss of service due to have 3 servers in the same DC, similar to RS and I could not inform customers because contacts were in webmail / imap client (not good when offline) and the billing system. For me I am now planning the following: 1) Move 2 of the 3 servers to different DC's 2) Local copies of client contact details 3) an always available hosted elsewhere server status page 4) to get my customers involved with twitter to get instant details Most of the above is achievable no1 will be the most costly for me, as I will need to buy a new server to colocate and then migrate customers from old DC Server to new one thats if I still have all those customers. Aaron
Posted by StreamGBcom, 09-25-2009, 11:54 AM
lol startech. I tracked down an article on the rapid vs poundhost as i didnt have a clue what it was about. Article here for anyone else. https://secure.grepular.com/blog/ind...s-rapidswitch/
Posted by BELLonline, 09-25-2009, 11:56 AM
Yeah, that's best - for a while now I've hosted the websites and support/status sites at a different datacentre to any of the servers I use for customers - that way if anything goes down, customers will always be able to get in touch -- eggs and baskets
Posted by bobarm, 09-25-2009, 11:58 AM
Wow, completely annoyed and also upset about this major problem which Rapidswitch have caused me today, I host quite a large number of clients on a server I have with them (luckily I have some clients elsewhere who pay for a better class of hosting) who I have had to explain throughout the day as sites come and go what the problem is and how unfortunate the situation is, something I shouldn't have to be doing. Blaming Cisco or whatever lacking excuse they have come up with so far is just totally unacceptable, at the end of the day I pay them money to enable me to host websites on a dedicated server, so far today they have failed on this big time. Whoever made the comment earlier about them investing some of the multi-millions that they claim to be making back into the infrastructure was spot on. How they couldn't design a network which not only has a high level of resillience for the some 5000+ servers they host (if you believe that!) but also to keep some sort of spares onsite or even a contingency plan to get them out of trouble is beyond me, it's what they should do and keep doing wether it's 5 servers or the alleged 5000+. You only have to check Twitter (twitter.com/#search?q=RapidSwitch) to see how many people this is affecting. It also seems to be carrying on with no information coming from rapidswitch since 15:35, that's nearly an hour and half old - and my box has just gone offline for a few minutes again @ around 16:54 - FAIL! They also seem to be switching the monitoring off every so often - please tell me they don't think they can hide this? So, who's going to start the recommendations thread for who we should be all moving our servers to????
Posted by StreamGBcom, 09-25-2009, 11:58 AM
Mines gone down again i give up
Posted by blueskimonkey, 09-25-2009, 12:05 PM
Yep lesson learnt on this one primary web site or just a support status url will certainly be elsewhere in the future, twitter also seems to be good for keeping users up to date.
Posted by Blanks, 09-25-2009, 12:10 PM
Seems PoundHost are offering something similar. Rapidly switch? That is pretty subtle compared to someother ISPs... See thread 892612
Posted by garrence, 09-25-2009, 12:15 PM
Definitely down again this time...
Posted by Cooply, 09-25-2009, 12:23 PM
I have 1 server up and the other down
Posted by BudgetVPS, 09-25-2009, 12:24 PM
It's deffo down YET again. This is beyond a joke now! It reminds me of when Burst.net had all the issues! SHAMBLES!
Posted by ValueVPS-Dave, 09-25-2009, 12:27 PM
I've just asked them for an update... getting fed up being treated like a mushroom
Posted by ValueVPS-Dave, 09-25-2009, 12:28 PM
Acid tabs might sell a bit better ;-)
Posted by StreamGBcom, 09-25-2009, 12:28 PM
Think im going to give up hosting and start selling headache tabs on here
Posted by BudgetVPS, 09-25-2009, 12:31 PM
Give up hosting, And go show RS how it's really done someone PLEASE.
Posted by blueskimonkey, 09-25-2009, 12:32 PM
I was thinking the same earlier but then a lot of my customers were very understanding and know of our excellent uptime in the past, for me i am just rethinking my hosting strategy and will not be having all eggs in one basket again.
Posted by slev, 09-25-2009, 12:32 PM
its a joke .. past 24 hours as been up and down like erm yeah well getting annoyed now with their service since they moved data centres
Posted by ndn23, 09-25-2009, 12:32 PM
I was pretty willing to put up with this last night - I mean, emergency networking, had to be done, out of hours etc, but it's getting a bit out of hand now. Luckily my income isn't dependent on this stuff, it's all hobbyist stuff really, but I think I'll be getting a cheap backup VPS to host some essential utilities to keep my people updated in future...
Posted by BELLonline, 09-25-2009, 12:33 PM
Yeah, not as bad as before - 1 out of 5 servers down at the moment for me.
Posted by slev, 09-25-2009, 12:34 PM
will be nice to see what compensation we get offered for all this downtime ..
Posted by zymic, 09-25-2009, 12:34 PM
My server is down too.
Posted by ndn23, 09-25-2009, 12:38 PM
I think we'll be lucky to see anything unless you have an SLA. I'm on colo so doubt I'll see anything. Would be nice to see something as a gesture of goodwill though, especially since they're turning such a large profit.
Posted by Votii, 09-25-2009, 12:41 PM
If we all request a 10% cut on next months invoice we might just get it. Power to the people!
Posted by StreamGBcom, 09-25-2009, 12:44 PM
I think 20% would be more appropiate.
Posted by zymic, 09-25-2009, 12:47 PM
B5 @ RSH North Lwr = down D3 @ RSH North Upr = up for me ...
Posted by daveharris, 09-25-2009, 12:52 PM
B5 @ RSH North Lwr Up H5 @ RSH North Lwr Down With zymic showing B5 Lwr Up and me showing it down this is still routing messed up!
Posted by ndn23, 09-25-2009, 12:53 PM
My server on 78.129.* is still down.
Posted by daveharris, 09-25-2009, 12:57 PM
78.129.201.* Down 78.129.242.* Up ????
Posted by ndn23, 09-25-2009, 01:05 PM
I'm on 78.129.223.*. Downtime is getting a bit difficult to explain. What I can't understand is why, given they are hosting a crapload of servers, they only have a single core router? I think they need to seriously rethink their network infrastructure after this.
Posted by flinthosts, 09-25-2009, 01:06 PM
It's across all ranges - I've get about 40 servers with them and it's about 70% up 30% down - then it swaps - but it's accross all range so dont pin your hopes on them getting a range sorted and then that will be it, as it will probably go down again....
Posted by zymic, 09-25-2009, 01:08 PM
78.129.242.*** down too
Posted by slev, 09-25-2009, 01:10 PM
well looks like a change of dedicated hosting provider is in order and now I see poundhost has got a good deal on now from their post an hour ago. and still no updates from Rapidswitch, they could at least post on here with an update on what they are doing with their router
Posted by garrence, 09-25-2009, 01:15 PM
It took me a while to find this. Yes the Poundhost offer is very good! A great discount on monthly price for everyone. 2 months extra free if you switch. Here's a link: Rapidly switch to Poundhost
Posted by flinthosts, 09-25-2009, 01:17 PM
but theres no guarantee that this wont happen to another host is there - I think people talking about jumping ship are not looking at this issue correctly. This was probably human error - in terms of not having the foresight to get backup hardware in place - so I'm sure it wont happen again. And it will make them even more determined for it to never happen again. Other DC's who haven't experienced this might make the same mistake....
Posted by mattyfc, 09-25-2009, 01:26 PM
Finally got a response about the current issue: Hi, The connectivity situation is still ongoing - our network engineers are still working on this as a matter of priority. We will continue to send annoucements whenever we have any more information. Unfortunately we do not have an ETA on the resolution at this time, but please be assured we are doing everything within our power to restore full service. We will of course send a full explanation when it is available. Thank you for your patience
Posted by slev, 09-25-2009, 01:28 PM
yeah they posted an announcement in the ticket / message system which doesnt make me feel any happier as the server is still down.
Posted by flinthosts, 09-25-2009, 01:29 PM
Below is a summary update of the issues experienced by clients in RSH-North today and yesterday, 25th and 24th September 2009. Fundamentally we have experienced serious issues that have affected all clients in RSH-North. This has been due to the Cisco equipment at the core of our Spectrum House services not responding as per specification and documentation. As has already been explained, we have been in direct contact with Cisco, working on a Level 1 priority request to solve the issues that have affected our clients today. We take full responsibility for our vendor selection and do not wish to appear to be passing blame "conveniently". We pride ourselves in the level of service that we provide and also in the quality of communication that we send to clients. We accept that neither have been anywhere close to our usual standard during this prolonged incident. However, we would like to take this opportunity to clarify a few points that we are aware have been questioned and discussed by our clients and competitors: - RapidSwitch's Spectrum House routers were configured in a redundant VSS cluster - CPU usage was recorded as being very high for normal usage - Cisco have offered two possible solutions to the issue, including a firmware update provided to RapidSwitch yesterday. These solutions have failed to resolve the issues experienced - RapidSwitch's priority has, at all times, been to provide stable service for as much time as possible, to as many clients as possible. This has been the endeavour and sometimes this has not been possible. We do not shirk the responsibility for this matter and recognise the impact that it has on our clients' businesses and both our and their reputations. - We will be looking at the issue of compensation next week. Obviously all of our clients will have been financially affected by the issues this week and we will be taking a detailed look at what we can do for our clients, but the primary issue will always be the continuity of service. Our network team are continuing to investigate why some of our clients are still experiencing outages, interrupted service and packet-loss and are doing so with Cisco. We will endeavour to provide further updates and detail to all clients affected but we will ultimately also provide a full explanation next week. Regards, Paul Tacey-Green on behalf of the RapidSwitch Team
Posted by slev, 09-25-2009, 01:36 PM
Yeah I just read that in the message section, Which is all well and good if it's a sensible offer and not an insult, at the moment looking at some of the deals going in the past hour on here and twitter I would save more moving unless an offer is made prior to the new invoice which is due Wednesday.
Posted by ndn23, 09-25-2009, 01:40 PM
I'm seriously considering moving. Poundhost have an excellent offer on there - Basically £35/month inc. VAT, which would save me over £10/month hosting fees with RS. Doesn't sound like a whole lot but for my hobby server that's a significant saving! Anyone know more about the technicalities of pulling a RS server on short notice?
Posted by StreamGBcom, 09-25-2009, 01:46 PM
30 days notice normaly
Posted by admind, 09-25-2009, 01:48 PM
this is a nightmare!! "this problem was out of our control caused by Cisco software" yes right!! Cisco this Cisco that... blame Cisco now... not making proper tests or planning and not have a backup plan just in case something goes wrong is... you know...
Posted by BudgetVPS, 09-25-2009, 01:58 PM
So they never made an backup before this Patch? If they never then person that done it needs Sacking.
Posted by awww, 09-25-2009, 02:01 PM
The last update stated "Our network team are continuing to investigate why some of our clients are still experiencing outages, interrupted service and packet-loss and are doing so with Cisco." I guess some clients are up then? Is there anyone up here? We own 40+ servers there and none is online...
Posted by admind, 09-25-2009, 02:03 PM
yes, I have servers UP and servers DOWN
Posted by flinthosts, 09-25-2009, 02:04 PM
23 Servers up 15 Servers down
Posted by PCS-Chris, 09-25-2009, 02:04 PM
All of ours are up and running at the moment. I'm really not impressed with how things have been going. We were very happy in BSQ2 (the floor which rapidswitch leased), since everything been moved into Spectrum House it seems we have had endless amounts of planned maintainence and endless problems. Hopefully things will look up. It still boggles the mind that after being told upgrades were being done for a "High Availability Network", there is still a SPOF in this router which has killed everyones connectivity today.
Posted by Votii, 09-25-2009, 02:05 PM
My server is down and has been for the last 5+ hours!
Posted by ValueVPS-Dave, 09-25-2009, 02:05 PM
we're still down
Posted by techwacky, 09-25-2009, 02:07 PM
i am on 78.129.xxx.xxx ip my server is still down.....
Posted by slev, 09-25-2009, 02:08 PM
still down .. interesting reading on twitter , alot of posts on there to and many angry customers
Posted by Jon-RackSRV, 09-25-2009, 02:22 PM
Given the circumstances I'm sure you could push to leave early assuming you can pursue 'breach of contract' due to the extensive downtime (assuming you have an SLA!)
Posted by ValueVPS-Dave, 09-25-2009, 02:28 PM
We have no SLA's on our colocated servers :-(
Posted by Lewcy, 09-25-2009, 02:40 PM
Our servers and vps's are up (only short periods of downtime). Fingers crossed everyone else gets theirs back shortly.
Posted by slev, 09-25-2009, 02:45 PM
well at this rate it would of been quicker to pick the server up and move it back to the old RS data center or at least one that is working !
Posted by richardwasf, 09-25-2009, 03:01 PM
good point, i wonder, is that what they are doing ? R
Posted by slev, 09-25-2009, 03:09 PM
well they are now updating via twitter to if you use it with the username rapidswitch
Posted by Votii, 09-25-2009, 03:10 PM
http://twitter.com/rapidswitch Last edited by Votii; 09-25-2009 at 03:18 PM.
Posted by mattyfc, 09-25-2009, 03:19 PM
Managed to get a ping reply 1/4, some progress seems to being made....
Posted by ndn23, 09-25-2009, 03:20 PM
No progress here yet. Server still down and unresponsive. Submitted a ticket... seriously contemplating moving now.
Posted by ValueVPS-Dave, 09-25-2009, 03:22 PM
I submitted a ticket an hour ago asking if they can de-rack our servers so we can move them to another DC - still no reply. I am expecting the '30 days notice' reply from them...
Posted by kparks, 09-25-2009, 03:27 PM
Sounds like they need to get some new router equipment in as soon as possible... I have a horrible feeling this is going to run and run, all over the weekend....
Posted by Votii, 09-25-2009, 03:30 PM
Yes thats exactly what I was thinking. There is little hope of them getting new equipment tomorrow and defiantly not Sunday.
Posted by StreamGBcom, 09-25-2009, 03:32 PM
my server that was up has gone down now aswell.
Posted by kparks, 09-25-2009, 03:38 PM
If RS make a public apology for their underhand sales email sent earlier in the week re: poundhost migration offer, I think the gods will forgive them and all will fix itself ;-D
Posted by CretaForce, 09-25-2009, 03:38 PM
Both servers down again...
Posted by daveharris, 09-25-2009, 03:39 PM
Yes likewise 50% were working now 100% offline! This is just the way I planned to spend My Friday, explaining to clients that I have not got a clue when their sites will be back up!
Posted by Lewcy, 09-25-2009, 03:40 PM
down again!
Posted by StreamGBcom, 09-25-2009, 03:41 PM
Pound host offer is finishing within a couple of hours if anyone wants to jump on
Posted by Votii, 09-25-2009, 03:43 PM
I love the PoundHost offer title "Rapidly Switch". I might switch to them simply for there ingenuity.
Posted by jarimh1984, 09-25-2009, 03:44 PM
All the servers down again I published link to this thread to my service users.
Posted by kparks, 09-25-2009, 03:46 PM
Servers gone completely again for me..
Posted by CretaForce, 09-25-2009, 03:46 PM
They fix some subnets and they **** some other subnets...
Posted by Lewcy, 09-25-2009, 03:48 PM
Can we not just go back to how it was 24 hours ago a dodgy network is better than no network
Posted by kparks, 09-25-2009, 03:49 PM
I wish I hadn't let them move my server to Spectrum house....
Posted by richardwasf, 09-25-2009, 03:49 PM
i never thought i would say this but it does appear they have no idea what they are doing. Perhaps their head engineer is on holiday or they have sold the equipment they were using previously. Either way, I have to agree with a previous poster, getting fixed with new equipment at the weekend ... slim chance at best. R
Posted by BELLonline, 09-25-2009, 03:50 PM
this is an absolute joke
Posted by ValueVPS-Dave, 09-25-2009, 03:50 PM
You didn't have a choice. You are only a mere client of theirs, why should you have a say?
Posted by daveharris, 09-25-2009, 03:50 PM
Here. Here. Enough is enough. Just restore the original settings and get the network up again.
Posted by nilocj-d, 09-25-2009, 03:54 PM
Our servers were on and off during the night (and the night before) and then went again early this afternoon. All but one (the most important one, obviously) came back up about 90 minutes later... I've seen no sign off the most important server (or its Cisco ASA firewall) for over 7 hours now. And now all our servers have gone down again.... about 20 minutes ago. Absolute nightmare.
Posted by slev, 09-25-2009, 03:55 PM
Just take all the servers back to the old data center at least that one worked ! and it was less downtime to move them !!
Posted by FcUK_A, 09-25-2009, 03:58 PM
Had one working fine on a different subnet and rack and now that one has gone as well !!!
Posted by FcUK_A, 09-25-2009, 04:01 PM
Simple Cisco router config update !!! Mmmmmm
Posted by techwacky, 09-25-2009, 04:01 PM
damn man this is horrible...... my server is down for almost 6 hrs now..... i am on 78.129 ip....... HELPPPPPPPPP!!!! someone make my server uppppp.... am sure i have to shutdown my business bcoz of this down time......
Posted by nilocj-d, 09-25-2009, 04:04 PM
Interesting traffic usage... routing is well and truly screwed. Interesting that the RS website and control panel are still up this time. Attached Images Screenshot.png (13.0 KB, 56 views)
Posted by Mojo29, 09-25-2009, 04:05 PM
Down again!
Posted by FcUK_A, 09-25-2009, 04:08 PM
Thier site in on a different ip range and i would say not in the same building coming off the same Router !!
Posted by XFactorServers, 09-25-2009, 04:10 PM
Yup, indeed down. Glad I didn't go with the clowns at RS.
Posted by FcUK_A, 09-25-2009, 04:11 PM
Who ya hosting ya servers with ?
Posted by Votii, 09-25-2009, 04:12 PM
They must have moved it, because rapidswitch.com was down last night and some of today.
Posted by XFactorServers, 09-25-2009, 04:14 PM
We have a ventrilo reseller account and they use rapidswitch so I am stuck with having using them. We just got a machine from poundhost. A+
Posted by brycecohen, 09-25-2009, 04:15 PM
We've been with these guys a few months now and they've been fantastic until now....this is just APPALLING!! I've never experienced anything like this in 6 years of using multiple hosting facilities - except for the clowns at 1&1 of course but that was in the early days when I didn't know any better ;(. We've been down for 5 hours now at peaktime on a Friday. Unfrigging believable. Some test servers on a shared LAN came back up for an hour randomly but our main net is out. We'll have to move away from these guys as being out for 5 hours like this should never be possible.
Posted by daveharris, 09-25-2009, 04:15 PM
their own site is also down again!
Posted by slev, 09-25-2009, 04:16 PM
well it aint working for me now thats gone offline for me now.
Posted by kparks, 09-25-2009, 04:24 PM
I would love to hear what their plan is to get out of this mess...
Posted by FS - Mike, 09-25-2009, 04:25 PM
Our uptime has gone to the rocks. This is not good for us. I don't think our business relationship with RS is going to last much longer.
Posted by FcUK_A, 09-25-2009, 04:26 PM
Busiest night of the week for gaming and all the clans i host for are bolloxed !
Posted by nilocj-d, 09-25-2009, 04:27 PM
If I drove down there with a laptop, reckon they'd let me strip a couple of rented servers of data?
Posted by FcUK_A, 09-25-2009, 04:31 PM
You'll be lucky !!
Posted by DonTregartha, 09-25-2009, 04:32 PM
I'm an end user developer with a massive site which is stuck on this pile of Cr*p. My client has been incredibly patient considering, but RS's approach to redundancy and fail over beggars belief. Luckily I got a mirror of the data and we're off.
Posted by brycecohen, 09-25-2009, 04:34 PM
You have a realtime mirror of your database? You're lucky. Not me though. Without access to any of our RS boxes I only have a one day old snapshot on offsite backup. Driving down there with a laptop (and a gun perhaps) might be the best option.
Posted by serverFFS, 09-25-2009, 04:35 PM
A few days after they spammed all Poundhost customers to switch to them, they deserve it to be honest. Too bad I have 2 servers hosted with them before I discovered Poundhost.
Posted by nilocj-d, 09-25-2009, 04:36 PM
What if I took a bazooka Two of my servers just came back up. Another still down for 9 hours continuously though.
Posted by FcUK_A, 09-25-2009, 04:37 PM
A few off mine are back on but not all !!
Posted by kparks, 09-25-2009, 04:39 PM

Posted by daveharris, 09-25-2009, 04:39 PM
One Server back up but RS control Panel Still shows it as offline!
Posted by FcUK_A, 09-25-2009, 04:41 PM
This is a quick update to keep you informed of the situation. The CPU usage on the RSH North cluster is still at a high level which will result in degraded network performance. We are in permanent contact with Cisco engineers and are continuing to troubleshoot the issue. This work will continue to cause interruptions to connectivity. As soon as there is any further information we will provide this to you. Regards, RapidSwitch Support
Posted by tero, 09-25-2009, 04:43 PM
Oh joy, can't wait to see more downtime ...
Posted by daveharris, 09-25-2009, 04:44 PM
"interruptions to connectivity" So thats what you call four hours and still no connectivity!
Posted by yellowcar, 09-25-2009, 04:44 PM
What are they talking to the Cisco engineers with an empty bean can and some string!
Posted by brycecohen, 09-25-2009, 04:44 PM
"interruptions to connectivity" LOL Downtime now 5 hours for us...
Posted by FcUK_A, 09-25-2009, 04:47 PM
I'd say paying through the nose for out of hours vendor support they never paid for when they agreed an SLA and agreement with them !!!!
Posted by brycecohen, 09-25-2009, 04:47 PM
Get the Cisco guys onsite. I will pay. Yes I mean that. Our business would normally have done about £2000 of orders during this period of outage. I'm going to the pub now and if it's not backup when I get back then the Poundhost guys have offered us to transfer a few key boxes at 3am and I'll probably take them up on their offer.
Posted by blueskimonkey, 09-25-2009, 04:48 PM
Poundhost are offering to collect servers from RS for free, I'm about to sign up and get at least one of mine moved. Has anyone got any experience of PH?
Posted by Exoware, 09-25-2009, 04:51 PM
I'd recommend Poundhost completely.
Posted by brycecohen, 09-25-2009, 04:52 PM
Back up now!!!
Posted by FcUK_A, 09-25-2009, 04:52 PM
Will only take 5 mins from RS new building back to BlueSquare were RS we're and where PH are still !
Posted by FcUK_A, 09-25-2009, 04:55 PM
My other ones have come back online now as well !! How long for tho !!
Posted by sco1, 09-25-2009, 04:56 PM
2 of ours have risen from the dead, still waiting fro the rest !!
Posted by Mitsurugi, 09-25-2009, 05:01 PM
The fact is, they're going to continue to go up and down until they can get a grip of this situation. There will be no cause to rejoice until they release a message stating they've cracked it. Personally, I didn't think over 24 hours of major outages was possible for a data centre of this kind. Amazing.
Posted by nilocj-d, 09-25-2009, 05:06 PM
Got access to a couple of my servers and I'm pulling data off them at 12MB/s... hope I get it all off before they go down again (which I'm sure they will)... fair play to PH offering free collection from Spectrum House if RS (or is that BS?) really did spam their customers.
Posted by brycecohen, 09-25-2009, 05:06 PM
Yeah maybe good idea to get a box or two at poundhost now just in case this nightmare starts happening again and run a 5 minute rsync to it just so I can get data off. Worst cost would be a hundred quid or so but would have some peace of mind at least.
Posted by CretaForce, 09-25-2009, 05:08 PM
Is someone with servers still down?
Posted by sco1, 09-25-2009, 05:08 PM
I am not holding my breath .... I will be interested for the explanation and compensation to RS customers as pointed out in PTG message earlier on
Posted by serverFFS, 09-25-2009, 05:09 PM
Switching my other servers to Poundhost now, should have done this earlier.
Posted by Warpline, 09-25-2009, 05:09 PM
Placed my order with Poundhost
Posted by PCS-Jason, 09-25-2009, 05:10 PM
It needs to be AT LEAST a FULL months invoice
Posted by kparks, 09-25-2009, 05:11 PM
until they sound the all clear, I'm not holding my breath...
Posted by blueskimonkey, 09-25-2009, 05:12 PM
PH have let me have the discount for a 2U COLO which I will post down to them next week, hopefully RS gets better soon so I can manually migrate all my customers to the new server otherwise might have to get PH to pull them out. Very disappointed with RS at the moment.
Posted by kparks, 09-25-2009, 05:13 PM
Other than poundhost, who do people recommend? I would like to spread my risk.. ;-)
Posted by sco1, 09-25-2009, 05:13 PM
I would expect more than this, if not, I cut my losses and jump ship PH orders are probably up 1000% ;-)
Posted by Votii, 09-25-2009, 05:15 PM
Unlikely. I would hope for at least a 20% discount on next months invoice. After all Colo dont have any SLA.
Posted by PCS-Jason, 09-25-2009, 05:16 PM
Frankly 20% would be insulting after all this
Posted by sco1, 09-25-2009, 05:17 PM
Oh dear .... subnet down !
Posted by Votii, 09-25-2009, 05:17 PM
Server down again!
Posted by FcUK_A, 09-25-2009, 05:18 PM
Bollox And down again !!!!
Posted by slev, 09-25-2009, 05:19 PM
arhhh down again
Posted by richardwasf, 09-25-2009, 05:22 PM
i think no matter how much compensation RS pay out, or waive for this months bill, they will lose 30%+ of their business as people have no choice but to spread the servers they have with other people. Considering what i have with RS 80% of it will move over the next 6 months to other providers for no other reason that I do not want to take much more of the customer angst I have receiving ever again. R
Posted by FcUK_A, 09-25-2009, 05:22 PM
Will PH be flexible about Amps used, there 1u colo is with 0.4a amp but I’m running 0.6 - 0.8 and most of my machines .
Posted by blueskimonkey, 09-25-2009, 05:25 PM
This is the same for me, its such a shame as the past 3 years for me personally rapidswitch have been excellent however with the last 24 hours and current events it's too much for me to risk my business having all my eggs in one basket. I will be looking to spread my load across 3 data centres and may still keep RS for a small portion of my none critical servers. It seems a very costly mistake for them, i'm just shocked that they appear to only have one route into the north block with no redundancy at all.
Posted by FcUK_A, 09-25-2009, 05:25 PM
That’s the knock on effect, Your customers don’t give a **** about your provider they want you to sort it ASAP but you can’t cause it’s out of your hands !!
Posted by Oldboy, 09-25-2009, 05:25 PM
I've had a server with Uk2 for four years plus with only minutes of downtime in that whole period. Managed over 600 days of continuous uptime before a maintenance reboot and am now back to (only ) 355 days uptime... However I've outgrown the diskspace I have with them and although they offered a disk upgrade, I decided to check out what other deals were on the market. I'd almost decided to sign up with Rapidswitch when their site went down last night. I'm glad I dodged a bullet there. I've now jumped in on the Poundhost offer as that is such great value. Last edited by Oldboy; 09-25-2009 at 05:27 PM. Reason: spelling ::)
Posted by FcUK_A, 09-25-2009, 05:28 PM
Very amateur not to have any redundancy
Posted by sco1, 09-25-2009, 05:30 PM
I feel annoyed as we have only just jumped into the RS ship in the last couple of months. Looks like we need to paddle again. What joy!
Posted by serverFFS, 09-25-2009, 05:32 PM
The main providers are UK2, Poundhost and Rapidswitch at this moment. Seeing RapidSwitch is turning into failure and UK2 is rather pricy in comparison, Poundhost is my favorite.
Posted by blueskimonkey, 09-25-2009, 05:36 PM
down again
Posted by CretaForce, 09-25-2009, 05:37 PM
Both servers down again.
Posted by Votii, 09-25-2009, 05:39 PM
Back again for me! (78.129)
Posted by FcUK_A, 09-25-2009, 05:39 PM
Whores Draws comes to mind !!
Posted by Mitsurugi, 09-25-2009, 05:43 PM
I can't take watching this anymore, I'm going to bed. I've got this feeling they're not going to fix it. 24 hours and counting, they obviously haven't got a clue.
Posted by fredsinbed, 09-25-2009, 05:44 PM
How would you guys go about splitting your sites between hosts? Do you mean spreading your sites between hosts or having a single site hosted in multiple DCs? Would love to find a solution that protected us from this problem in future...!
Posted by sco1, 09-25-2009, 05:44 PM
Mine must be a rate dirty slapper then !! ooooo--- back up again Last edited by sco1; 09-25-2009 at 05:46 PM. Reason: RS desided to open the router up !!
Posted by Nisd, 09-25-2009, 05:47 PM
Ahhhhhh Just went from all red to all green again!
Posted by ValueVPS-Dave, 09-25-2009, 05:47 PM
We're down... we have 3 servers that have not been up at all today
Posted by brycecohen, 09-25-2009, 05:49 PM
Just signed up @ Poundhost. Luckily boxes came up long enough to pull all the data I needed off them. Then they went down again.
Posted by Nisd, 09-25-2009, 05:51 PM
Trying to get a offer from Poundhost too, using the "Rapid switch" offer. etc 2 month free
Posted by brightstation, 09-25-2009, 05:52 PM
This is costing me a lot of money and I am happy to pay a large premium for my primary dedicated servers currently hosted with RS in return for regular updates, incident manager etc. Looking at Rack-Space and Zen internet, any thoughts or recommendations?
Posted by fredsinbed, 09-25-2009, 05:52 PM
Are Poundhost able to get servers online overnight tonight / at the weekend or are you guys waiting til Monday?
Posted by ValueVPS-Dave, 09-25-2009, 05:53 PM
i can't do anything - RS won't answer my tickets about releasing my servers.
Posted by DATARTIM, 09-25-2009, 05:54 PM
Are you looking for managed ? Certainly Rackspace are but not sure they cover cPanel (if that's what you have). Well I saw they mentioned they had stock so hopefully so, but they must have quite a lot of orders !
Posted by techwacky, 09-25-2009, 05:58 PM
Yes mine got up (78.129) All ports are up now.
Posted by qlites, 09-25-2009, 06:07 PM
We ordered servers from poundhost at 1:00PM CST and they had them up at 3:25PM CST. Just got done transferring everything from rapidswitch and not looking back!
Posted by FcUK_A, 09-25-2009, 06:07 PM
After removing the VSS cluster technology from the RSH North zone routers and restoring the routers to a similar configuration to the ones which served RHC stably for over two years, considerable progress was made to restore connectivity for 90% of clients. However, service is not stable and clients will see differing levels of connectivity ranging from complete loss of service to severely degraded. Obviously we are continuing to work on this with Cisco. Please be assured that further updates will be provided as and when we have them. RapidSwitch
Posted by brycecohen, 09-25-2009, 06:08 PM
All servers back up for first time. But this is totally unacceptable experience so... moving all services to Poundhost as we speak They're nearby and cheap. Then will look at Rackspace too.
Posted by Votii, 09-25-2009, 06:09 PM
Seems servers are backup for now edit: (78.129)
Posted by ndn23, 09-25-2009, 06:13 PM
Same here dude. They put me off until Monday.
Posted by yello, 09-25-2009, 06:15 PM
Moving is not so easy if you have serious applications on RS servers depending on custom server configuration. You have to do all the server configuration from scratch. So i am among the ones who is going to stay and see. They could not became swindlers all of a sudden as i was completely satisfied during the previous years. I hope that this disaster will make them be more serious in the future.
Posted by Nisd, 09-25-2009, 06:15 PM
Poundhost here we go And because of the current deal they got, my server is cheaper there, and more powerfull, it is just Win Win
Posted by ValueVPS-Dave, 09-25-2009, 06:15 PM
we still have all servers in 87.117.231.x down
Posted by CretaForce, 09-25-2009, 06:18 PM
Does your servers became accessible today?
Posted by Soluslabs, 09-25-2009, 06:21 PM
New Update
Posted by FcUK_A, 09-25-2009, 06:22 PM
Put that on the last page
Posted by brycecohen, 09-25-2009, 06:22 PM
Yeah tell me about it. "rsync -e ssh -avr" is your friend though. We have about 10 major sites, 1 particular which is a bitch to configure using Java and PHP apache2 postfix etc etc. No way we can stay with Rapidswitch after this. Something was seriously seriously wrong with their setup.
Posted by Soluslabs, 09-25-2009, 06:23 PM
Maybe not that new ......
Posted by ValueVPS-Dave, 09-25-2009, 06:24 PM
Last time any of our servers in that range were available was 2:25pm (GMT) on 25/09/2009 - that is 9 hours ago!
Posted by NETlayer, 09-25-2009, 06:30 PM
Does PoundHost offer Plesk?
Posted by ValueVPS-Dave, 09-25-2009, 06:31 PM
Has anyone managed to get their colo'd servers out of RapidSwitch?
Posted by nilocj-d, 09-25-2009, 06:32 PM
I'm with you, although I'm having to check my rage! With their new control panel API and previously solid/fast network performance, I'm going to stick around as long as this is sorted by morning. A huge blow for us in terms of our customers confidence in our service, but it is the kick we needed to implement our long planned failover system. If you are completely relying on a server in a single datacentre for your absolutely critical services with no failover, then you've only yourself to blame. But I guess it's easier to move to PH
Posted by Steven, 09-25-2009, 06:39 PM
This is the exact reason people need to be smart when they configure thier servers. If it takes an excessive amount of time to 'custom' configure your server... then suffice it to say.. you are doing it wrong. You need to have a disaster recovery plan in place that enables you to rebuild at a moments notice if necessary, quickly. Anything can be happen... YOU... have to be prepared
Posted by Scott.Mc, 09-25-2009, 06:41 PM
For those that have not seen it yet, http://www.webhostingtalk.com/showthread.php?p=6411621 now is the time to switch since it ends in about an hour.
Posted by brycecohen, 09-25-2009, 06:41 PM
We have a whole load of servers at RS - all intricately configured with multiple redundancy and failovers. No good if RS decide to hose their entire network. The reason we are all with RS is their good history and great control panel. Sadly there is no excuse whatsoever for this massive disaster. We've lossed thousands of pounds of revenue. Never seen anything like this before. PH might well be not amazing but we need an urgent alternative in case this nightmare continues tomorrow and we all get fired. Will look at Rackspace managed hosting as fed up with these bargain services. It aint worth the hassle. Last edited by brycecohen; 09-25-2009 at 06:42 PM. Reason: typo
Posted by ichilton, 09-25-2009, 06:43 PM
Hi, I was considering a move from Poundhost to Rapidswitch as I could get it cheaper but thanks to this thread i've decided to stick with Poundhost. I've been with them since the start of the year and never had any known downtime/outages and no complains whatsoever. If anyone is looking to move away, they are offering a great "Rapidly Switching" deal on the Dedicated Servers offer forum! Ian
Posted by Steven, 09-25-2009, 06:45 PM
Cheaper rarely is better.
Posted by kparks, 09-25-2009, 06:47 PM
Downtime and outages do happen from time to time, but seeing how a company responds at those points is very telling. Redundant hardware obviously wasn't high on the priority with RS new center.
Posted by ichilton, 09-25-2009, 06:48 PM
Yeah yeah lesson learned :-) Defiantly sticking with Poundhost - reliability over price!! Ian
Posted by Steven, 09-25-2009, 06:49 PM
If you read previously, due to the nature of the initial bug redundancy could not be maintained in order to perform the upgrade. It was a downhill spiral after that.
Posted by ValueVPS-Dave, 09-25-2009, 06:50 PM
....and RapidSwitch is which?
Posted by Steven, 09-25-2009, 06:50 PM
Rapidswitch obviously was the cheaper offer. They were not better then pound host.
Posted by ichilton, 09-25-2009, 06:52 PM
Indeed - madness. Not only do they seem to have no redundancy but 24hrs on they are no further forward - they are still down and saying they are working with Cisco. I can't believe a company such as Rapidswitch dont have redundancy AND hot swap backups for such critial datacenter-wide equipment and a plan B to get things back online in minutes rather than hours should the redundancy and plan A fail. They've probably spent millions on diverse fibre and power and then they wipe the whole thing out for a day with a few router problems... Ian
Posted by ValueVPS-Dave, 09-25-2009, 06:55 PM
PH are obviously cashing in on the bad fortune(?) of RS so they have to be cheaper to make things even more attractive to us. RapidSwitch are as guilty of the same thing - i remember getting conned with the 10tb deal only a few years ago. There are a load of hidden costs with RS as well... Unless you live on their doorstep, having colo's with them is expensive. If you do live on their doorstep and need to sya, change a hdd, you have to book and pay for their build room... they never 'go the extra mile' for their clients.
Posted by ichilton, 09-25-2009, 06:56 PM
Yep, that's correct.
Posted by garrence, 09-25-2009, 06:57 PM
You can buy an extra 0.5A for £15 per month.
Posted by ichilton, 09-25-2009, 06:59 PM
Yeah, why not - unhappy clients leaving RS have to go somewhere - they are not doing anything untoward by offering a few well timed special offers... Ian
Posted by ValueVPS-Dave, 09-25-2009, 06:59 PM
I wonder if you can pool that... would be handy for me as we have some that use 0.3a and some that are 0.8a - i always find that RS measure well over - i measure at 0.5, i can guarantee it will be billed to me at 0.8
Posted by garrence, 09-25-2009, 06:59 PM
Maff posted to webhostchat.co.uk to say He's also said they are ending their dedicated server Rapidly Switching offer in 30 minutes time (00:30 UK) due to massive demand. The colo offer will continue and you'll only be billed from when you've installed your kit. They'll even collect your servers from RS for no extra cost. link Last edited by garrence; 09-25-2009 at 07:05 PM.
Posted by garrence, 09-25-2009, 07:00 PM
Sounds like you want a quarter cabinet or similar.
Posted by ValueVPS-Dave, 09-25-2009, 07:02 PM
wow! Hi, Sorry for the delay in getting back to you. Our policy for server removals is that you can remove your servers at any time as long as your account balance with us has been settled. You would also need to give 30 days notice for cancellation of service as per our terms and conditions. Regards, Shuja RapidSwitch
Posted by nilocj-d, 09-25-2009, 07:03 PM
That's exactly it.
Posted by UNIXy, 09-25-2009, 07:24 PM
A single facility cannot be trusted to have any kind of redundancy in place.until one accepts the fact that the facility is a one-legged entity and act accordinly, there's a very slim chance of business survival. Regards
Posted by serverFFS, 09-25-2009, 07:31 PM
I have the feeling RapidSwitch is gonna be very difficult and annoying when you try to cancel your server, trying to save whatever they can. However, I think I have the right to move my servers to people with brains after this huge gap of failure after failure.
Posted by nilocj-d, 09-25-2009, 07:50 PM
Check out the web host cat fight post on theregister.co.uk I'm going to demand a magnum.
Posted by Craig, 09-25-2009, 07:59 PM
It's made the register: http://www.theregister.co.uk/2009/09...host_catfight/ Last edited by writespeak; 09-25-2009 at 08:23 PM. Reason: By request
Posted by ValueVPS-Dave, 09-25-2009, 08:03 PM
lol pmsl - we are still the only losers here though :-(
Posted by brightstation, 09-25-2009, 08:11 PM
Anyone knows if things are back to normal at RS yet? servers seem to be up but for how long?
Posted by dazmanultra, 09-25-2009, 08:16 PM
I think what they've done is decoupled their VSS to turn it back into a pair of 6509s... I would imagine they've had a lot of work to do setting the configuration back up on each router.. I'm not a network guy though.
Posted by nilocj-d, 09-25-2009, 08:17 PM
Our 78.129.* servers are all back up and have been for a while (with 1 minute outages every now and again) but our servers in the 87.117.* range haven't been seen since before 2pm.
Posted by bluewizard, 09-25-2009, 08:18 PM
Mine are still down
Posted by StreamGBcom, 09-25-2009, 08:20 PM
mine are still yo yoing given up on checking them now.
Posted by PCS-Jason, 09-25-2009, 09:28 PM
We were provided a solution by Cisco TAC at approximately 01:40 UK time. This was the third solution provided and we are pleased to say that it seems to have been effective. Network capacity and performance is running at normal levels, and over 99% of clients are back online. Initial diagnosis shows that any servers not currently online are in their current state for a reason other than network. We are running at a reduced redundancy level as was necessitated to resolve the problem as quickly as possible. RapidSwitch
Posted by mlini, 09-25-2009, 09:53 PM
Post here, Post there, Reply left, Reply Right, First time down in 24 months I can't complain for 2 problem days they are the number one in UK and I will never switch with Pound Host. "very bad experience" "Pound Host = Garage Sale" I'm not here to comfort RapidSwich but if I need to vote from the 2 servers in UK I will give 90% Rapid Switch and 10% Pound Host for the bad experience. Let Them Work! Good Bless UK Last edited by mlini; 09-25-2009 at 10:00 PM.
Posted by Votii, 09-25-2009, 10:02 PM
Looks like this has been resolved... for now.
Posted by mlini, 09-25-2009, 10:16 PM
We don't need an update we already know
Posted by StreamGBcom, 09-25-2009, 10:43 PM
Well ive just sent out a stable message to all clients so if it goes down again it will be my fault lol
Posted by richardwasf, 09-25-2009, 11:32 PM
well i now have 1/2 up and 1/2 down. Been a while since the last update to say things were stable ... R
Posted by Nisd, 09-26-2009, 02:41 AM
New Update: Service continues not to be stable and clients will see differing levels of connectivity ranging from complete loss of service to severely degraded. We are continuing to work on this in conjunction with Cisco. Further updates will be provided as and when we have them. RapidSwitch
Posted by richardwasf, 09-26-2009, 03:00 AM
well the two companies I own, 1 is fine and up and running, the other more bandwidth intensive is still down. My engineering staff have just have a response from RS, which I wont post, but suffice to say it provides an opportunity for severe recompense It appears the all encompasing 'myservers' really does not monitor anything as if it did, surely it would be presenting alarms to the engineering staff. R
Posted by nilocj-d, 09-26-2009, 03:14 AM
Just got myself out of bed to find that one of my servers and its cisco firewall are STILL not available. I phoned and spoke to a very tired sounding guy who asked if I thought it was related to their network problem. Hrm. 'It might be back on today but can't promise anything' was the end result of the call. Anybody using 87.117.* back up and running? Last edited by nilocj-d; 09-26-2009 at 03:22 AM.
Posted by Soluslabs, 09-26-2009, 03:57 AM
Our private subnets are still dead 17hrs solid downtime so far
Posted by sco1, 09-26-2009, 04:13 AM
ours are back up from about 10ish (BST )last night. Everthing looks stables this morning and the hungry boxes are flowing ok. I thought ours being off line for 6 hours was long enough , but 17 hours is incredible!
Posted by richardwasf, 09-26-2009, 04:23 AM
Having been through the intial tickets opened with RS relating to the network and its performance we are now standing at 70 hours of problems and stems from changes made at 23rd of September at 23.00. That has to be a record, but of course it is not resolved yet... R
Posted by Soluslabs, 09-26-2009, 04:32 AM
By looking at there tos and responses we have had from them so far, legal action may be our only option.
Posted by blueskimonkey, 09-26-2009, 04:50 AM
My servers are showing a good 10 hours of being online since around 11pm UK Time last night. Currently debating if I should hold out until I can send my 2U server to Poundhost and split the load in RS, or to rent a dedi now with PH and work on a DR just in case.
Posted by richardwasf, 09-26-2009, 05:00 AM
Legal action, perhaps, the best way to respond to this is save the costs, time and emotional expense of legal pursuits and just move somewhere else. Voting with your feet is considerably more hurtful to a provider than paying out from their insurance. R
Posted by chrisbyrd, 09-26-2009, 05:12 AM
I've not seen them myself, but I suspect the agreements/terms you've entered into with RapidSwitch will be very tight in the areas of what liability they'll accept - I expect you would have signed away any possible 'legal' routes
Posted by ValueVPS-Dave, 09-26-2009, 05:40 AM
i can beat the 16 hours: Neptune (XEN Node) (87.117.231.xxx) is UP again at 26/09/2009 10:27:06AM, after 20h 2m of downtime. Mars (XEN Node) (87.117.231.xxx) is UP again at 26/09/2009 10:26:57AM, after 20h 3m of downtime. Eris (OVZ Node) (87.117.231.xxx) is UP again at 26/09/2009 10:26:35AM, after 20h 1m of downtime. ^^^ This is SOLID btw, no uptime at all for over 20 hrs! Last edited by ValueVPS-Dave; 09-26-2009 at 05:47 AM.
Posted by Nisd, 09-26-2009, 05:41 AM
Any one got a working BW Graph? Mine is showing zero usage.
Posted by richardwasf, 09-26-2009, 06:24 AM
So we now know what terms you offer. Perhaps not the best place to look for hosting then .... R
Posted by eatmyd0g, 09-26-2009, 06:33 AM
i wouldn't be surprised to see RS announce something quite special for all customers affected by this downtime so dont walk just yet. it is quite likely a one off, which by no means makes it acceptable, but does put things into perspective if they turn around to everyone tomorrow and say free for a month, then we'd all be happy. as it stands now, thats what i want to be hearing because 45 minutes is sort of acceptable for an emergency fix to a router (although why they didn't have a backup just sitting there waiting to take the load baffles me) but most of yesterday isn't and while we all agreed to what we agreed to when signing up to RS in the first place in my ten years doing this, i've never experienced this before and have moved for a lot less bottom line. RS would have learnt from this and plans will be written up to make sure this never ever happens again and they are the joke of the industry now. they have to turn that joke around and say sorry, give us a months free hosting and assure us it wont be happening again. do that quickly and they might keep all the customers .. but some will leave off the back of this if they do nothing
Posted by chicsystems, 09-26-2009, 06:46 AM
Heres my take on the problems and why my server has moved already and wont be going back EVER to rapidswitch. I have been in the biz for many years so these things happen. How companies deal with it define how we view them and whether we stay with them. I have a dedicated server with Starteck/ValueVPS and on thurs morning I was ready to move. I decided to stick with David at starteck as he has been excelent. But by friday morning I was spitting feathers, the site is a growing social network one and we do not need downtime especially at busy times and after a quiet summer when we are rebuilding. David's support kept me with him after I had started to read this thread. At about 9pm after effectively continuous downtime of 22hours we decided to move. David had already made a copy. He set up a new server, I did the DNS and by 1130 we were running and had users online. NOW!!! If rapidswitch had been more communicative then that would be fine, proper notice, regular proper news, an sincere apology, even a "we are screwed but trying" would have been good. But NOTHING From now on we will be watching which datacenters our hosting providers use and rapidswitch will NOT be on the acceptable list. We are now having to offer incentives to keep our paid users happy which will cost us. However the damage by sites being off cannot so easily be undone. SHAME on Rapidswitch for the way they have handled this. I will be recommending David for a Knighthood. And my sympathies to the other users whose sole living was on Rapidswitch and/or whose clients were screaming at them and will continue to do so for some time. Rapidswitch have a chance to redeem a little. Full details of the problem and what happened and why it went so wrong, a sincere apology and sensible compensation so we, the Clients of Rapid can compensate our clients so we dont lose out nor do the end users so bad. If Rapidswitch fail because of this then then can only blame themselves for the appalling way they treated us.
Posted by Watchur, 09-26-2009, 06:49 AM
Our Experience with RapidSwitch over last few days The whole incident lasted 52 hours starting at 11pm (UK) on Wednesday 23rd and finishing Saturday around 2:30am. Since the network changes on Wednesday (and the 6 hour odd outage that took) there was constant 30% packet loss with loss of connectivity lasting from a few milliseconds to a few seconds many times a minute. The low level of network service meant we had thousands of our end users complaining about issues. We reported this to RapidSwitch as a Ticket and got no response, despite providing all the evidence they demand before they'll look in to such issues. They didn't even acknowledge the ticket with a "Were sorry about the issues and we're looking in to it". No general notification was sent out to us about the ongoing issue until 6 or 7 hours had passed. Things then went from bad to worse when they tried to fix things on Thursday night with a second overnight outage. This outage lasted many hours longer than they had suggested. After the change, the network quality was even worse, with pings reaching up to 1000ms in spikes and increased packet loss and frequency. Around 4:30pm(UK) we lost all our servers for a 3rd time and this was without any warning. Although they claimed they sent emails, they didn't actually deliver them until after their mail server came back online in the evening several hours later. You couldn't get through to them on the phone (engaged). Their RapidSwitch website was offline for an extended period for the 3rd time in 2 days so you couldn't check status on that either. Our servers continued to be offline until the following morning around 2:30am. There were some periods where a few of our servers became visible, but even those had constant on/off outages. The combined total outages of our servers put together was overall around 90 hours. And even when they were "pingable" during the incident, the low level net connectivity meant we couldnt offer a reliable service to our clients until today. I understand to get things working again were now running in a non redundant configuration. This means there will be yet more network engineering to take place in the near future. I hear RapidSwitch were offering PoundHost clients 2 months free service to change providers to them. I hope RapidSwitch offer their EXISTING clients similar compensation just to stay with them after all these problems.
Posted by eatmyd0g, 09-26-2009, 06:52 AM
i've given all my customers a free month hosting. i dont have many but two were emailing in about their sites so rather than wait around to hear from rapdiswitch - i've had to act to keep my customers happy hence why i want a free month from rapdswitch for my 2 servers because frankly i think it is the least they can do and if they don't it will make the decision i have to make a little easier and i have no doubt that if i offer my next supplier three months in advance (i know who it will be) for four months, then i know i'll get it
Posted by llamasteve, 09-26-2009, 07:11 AM
We've still got one of our servers at RS down. It has been down continously for about 21 hours now. The other two servers are now back up but not without serious repercussions for clients. This is the second serious (i.e. > 8 hours continuous downtime) in the last couple of months. Last time it took them several hours to recognise there was even a problem. Then at least another 2 hours for the senior administator to be alerted and come back from lunch - when he plugged the cable into the right socket and everything started working again. We received a broadcast message at about 2am this morning that included: "Network capacity and performance is running at normal levels, and over 99% of clients are back online. Initial diagnosis shows that any servers not currently online are in their current state for a reason other than network." Absolute nonsense - that message was followed with one over 6 hours later: "It has come to our attention that some clients with private subnets are still being affected by the network issue that we have been experiencing. This was not picked up as the monitoring service was showing that everything was fine from our end." How could it take them over 6 hours to realise private subnets weren't working? Just what sort of monitoring do they have in there. That's what worries me - we all know things can go wrong with any host but there is no excuse for not being able to monitor your own network. The business with Poundhost has come back to bite them big time. I think RS deserve all they get from the way they behaved after the PH email. I don't think I want to do business with a company that carries on like that.
Posted by PaulTG, 09-26-2009, 07:36 AM
Hi everyone, We were working non-stop with Cisco to resolve this, and fixed the issue in the early hours of this morning. All of our clients got regular updates by email (if spam filters allowed) and through our online portal. We had a Cisco TAC priority 1 case open, and various sets of bizarre situations that we'll update our clients with in a full RFO. Cisco were sitting there saying it was the first time they'd seen any issue like this, and gave us three different solutions before we got one that solved things. The final solution was delivered by Cisco when they had engineers conferenced in from all over the world - I think there were 4 or 5 all in the Webex conference working on it at the same time to resolve. As we've said in our updates to clients, we have to take full responsibility for vendor and equipment selection. The Cisco VSS-1440 cluster is the "biggest and baddest" Catalyst switch you can get, and we made this equipment selection so ultimately it is our fault. Many thanks to Chris Byrd (GlobalAXS) and Alex Cruz-Farmer (Netsumo) for their offers of help - unfortunately the only people that could really help in this were Cisco. All our clients will get lots of information over the next few days and we are trying very hard to be as open and honest as possible. We want to give everyone the hard facts to what happened as we have always tried to do in the past. Regards, Paul Tacey-Green Operations Director RapidSwitch Ltd
Posted by richardwasf, 09-26-2009, 07:57 AM
I believe we have the root of the problem. Biggest and Baddest is probably not the best way forward. Many vendors sell the latest kit, however due to this it never works first release. Simplest and most stable would be preferred. I know many other clients would like to see it, so I'll ask for a copy in public. Can we see the test plan you performed on these routers before deploying it ? R
Posted by llamasteve, 09-26-2009, 08:01 AM
The issue may have been partially fixed in the early hours but we were still down until about half an hour ago. Which in my book means it was not fixed. We have received updates, but I would not agree that they were regular. Often these were many hours apart which is not really acceptable in my view.
Posted by techwacky, 09-26-2009, 08:02 AM
hmm.. so how much compensation should rs give. mine is up now. was down for 7-8 hrs
Posted by eatmyd0g, 09-26-2009, 08:04 AM
paul, thanks for coming on and telling us what you've told us. i've be a customer for about 6 months and now have 2 servers with you. today i've emailed my few customers apologising and informing them that they have a free month of hosting with me. basically, i had to do something to appease them and at the same time i didn't just want to pass the buck. i'm not saying you are passing the buck by blaming cisco and i have been a very happy customer the past six months but yesterday was, i think you'll agree, an unacceptable level of service i would basically like to have my next months invoices credited. what is the likelihood of this happening?
Posted by Watchur, 09-26-2009, 09:09 AM
RapidSwitch should stop trying to shun the blame on cisco. All hardware has problems time to time, and things don't always go to plan when making changes. The difference between the quality of companies is how they cope with those problems when they occur. Its standard practice that you have a backout plan to put things back to the last working configuration if things dont work as expected. You have this plan in place before you start making changes and you know in advance how long it would to take to roll back if you need to. As soon as you had problems, you should have been able to invoke your backout plan and notify people how long it would take - because you planned in advance. Its organising and planning around changes which is at fault here. Technology isn't a perfect science, and thats why planning, communication and backout plans are so important. Communication was very poor during the last 2 days and the duration of the issues unacceptable as a service level.
Posted by fwaggle, 09-26-2009, 10:28 AM
We're in the same boat as you, we weren't about to wait to see if compensation was coming from RS before trying to appease customers affected, so we went ahead and started crediting. I actually thought the communication was reasonable, particularly once I found the Twitter. The emails were pretty well garbage because they didn't seem to make it out during the periods of outages, so the Twitter was a godsend.
Posted by StreamGBcom, 09-26-2009, 11:03 AM
Dear Sirs, As you are aware we have had technical issues over the last two days which have now been resolved. We will put out a full technical explanation next week. We are deeply distressed that this has occurred and take full responsibility. The issue was primarily caused by the final elements of our migration of customer servers to our new state of the art data centre which we believe will serve our customers well for many years to come. It is unfortunate this occurred right at the end of a long, complex, but generally very successful migration project. We are examining our architecture closely to eliminate any outages in the future. We will continue to remain completely dedicated to ensuring that we have no single points of failure across our infrastructure. Our parent company has authorised the spend to maintain, and insisted on the commitment to, 100% uptime. We will learn and improve from this experience. In the meantime if any customer would like to speak to myself or a fellow director or indeed to our group CEO then we would be more than happy to accommodate all requests, please just email us at support@rapidswitch.com. By doing this, we would hope to provide you with the reassurance you are looking for in the strength of our infrastructure, services and offerings. We would very much like the opportunity to do this for you. By way of an apology, and to convey our continued customer focus we will be offering a free upgrade of RAM or hard drives over the next two weeks. We recognise that this is not the important issue in-hand but we hope that as many clients as possible will take us up on this apology. We will forward details as soon as practically possible. I would like to finally express that as MD I am personally committed to making sure that each and every one of our customers has peace of mind when it comes to the resilience of our infrastructure and we are all strongly committed to regaining the long-standing faith put into us by our clients in recent years. I will keep in touch over the coming days and weeks with further news. Yours faithfully, Ed Butler Managing Director
Posted by serverFFS, 09-26-2009, 11:05 AM
A free RAM or harddisk upgrade, damn we are getting spoiled. That compensation is absolutely pathetic and does not compensate for it by far.
Posted by StreamGBcom, 09-26-2009, 11:06 AM
would prefer a month free instead of taking the server down for upgrades then i can pass on the saving to my customers.
Posted by ValueVPS-Dave, 09-26-2009, 11:07 AM
Think yourself lucky - no mention of anything for colo'd people....
Posted by zibby, 09-26-2009, 11:10 AM
More space or RAM is pointless to me as I'm within the limits of my VPS so the offer is no compensation at all. I guess the colo people are screwed the most by it though.
Posted by StreamGBcom, 09-26-2009, 11:16 AM
I dont think customers want to see another maintenance window in the next year for server upgrades or not.
Posted by slev, 09-26-2009, 11:27 AM
They can shove their free upgrade that isnt going to pay for customer refunds. They should give everyone a free month, they were willing to give 6 months free going by the email that was sent out including a bottle of bubbly. Yet another insult.
Posted by Warpline, 09-26-2009, 12:27 PM
I agree with you completely. The chance to steal a client from another business is worth 6 free months, but the chance to keep your existing customer is only worth.... A free hard drive or memory upgrade. When we cancel, the memory goes back to the company and so does the hard drive. They're out nothing. Along with that, the responsible companies who need to issue actual refunds back to their own clients are now faced with: 1) Taking it out of their own pockets 2) Not offering a refund/credit at all Also, you all make very valid points. Who wants to be down again for a memory/hd "upgrade" when we've been down enough already? I'm not sure what they are thinking on this one in terms of PR. Downtime is quickly forgotten (look at how the thread has slowed since they're up) and a simple month free (not even a refund) would do wonders for a lot of people. Perhaps if their marketing department had a little say over their customer service department, we'd all be getting a month (or 6?) free.
Posted by datajack, 09-26-2009, 12:29 PM
What use is an upgrade to people who have properly specced servers?
Posted by Lewcy, 09-26-2009, 12:37 PM
Agreed, the last thing we will be doing is taking up this offer (suprisingly) as the last thing we want is our customers to face further downtime on what are already well spec'd servers. This clearly hasnt been thought through from a customer view point. The only view point is that memory/hdd goes on their balance sheet and doesnt come off their bottom line so it is win win for them.
Posted by gone-afk, 09-26-2009, 12:41 PM
I have colo and dedicated here, I've been very happy for the past 18 months or so but this outage really damaged our client base. Shocked that one month free hasn't been offered at the bare minimum. I can't imagine many people wanting to take down servers for an upgrade, my clients certainly won't put up with any more downtime.
Posted by ValueVPS-Dave, 09-26-2009, 12:54 PM
Well we guess exactly how this magnificent and generous offer was conceived: Ed: Pssst Paul, what crap have we got sitting in the stock room that we can't get shot of? Paul: well we got a whole crap load of RMA'd ram and HDD's Ed: That will do, any old shite to keep the feckers happy Paul: you reckon? Ed: Yeah, the plonkers will think they are getting value out of us - daft sods don't realised we will still own all that equipment when they have long gone Paul: yeah great idea - the girls in accounts will love that - we can give them a case of this bubbly we bought in for the ex Poundhost clients to celebrate us getting out of a corner in such a clever way... Ed: er Paul, why have we got so much of that bubbly left? Paul: Well i guess maybe they didn't read the email properly Ed: What email, i know nothing about that email...remember. We do nothing underhand at RapidSwitch - we are a 100% customer focused company ;-) Paul: shall i get that email drafted up ready for WatchDog? Ed: Nah, we'll just bury our heads in the sand if that moment ever comes....
Posted by Warpline, 09-26-2009, 12:57 PM
I think you have too much free time...
Posted by Tech-Hosts Team, 09-26-2009, 01:03 PM
I saw that one as well after reading the email - sent a ticket in offering an ultimatum. Probably won't get anywhere with it, but there is no harm in trying. I've asked for a quote from PH just in case
Posted by ValueVPS-Dave, 09-26-2009, 01:03 PM
I do now, i've answered all my support tickets ;-)
Posted by slev, 09-26-2009, 01:40 PM
Looks like support is just using stock replies now. I sent a ticket to Ed Butler as stated in his last global message he sent out offering the insulting upgrade instead of compensation. Did I get a reply from him or any director or the CEO as he states . Nope just a standard stock reply from support. What a joke.
Posted by Tech-Hosts Team, 09-26-2009, 01:42 PM
Yup I got the exact same one asking what colo clients get, trying to swindle some bandwidth upgrades. PH have replied to me with a very attractive offer, so I think I know where i'll be heading now - it annoys the hell out of me that they can't be bothered to reply to individual cases. They should have every member of staff they can get their hands on working on this. Not one guy buzzing on coffee wearing out the CTRL, C and V keys on his keyboard
Posted by garrence, 09-26-2009, 03:03 PM
Folks - there is one point to remember when telling us about your angry customers, lost revenue and compensation demands. RapidSwitch are cheap. You have chosen to put your server with the provider at the bottom end of the colo market who charge half the price of competitors. Cheap means naff Cisco routers instead of the much better and more expensive Juniper ones. If these servers are so important to you and your customers are so angry then perhaps you should be using proper server hardware (not white box rubbish) on a good network, paying sensible sums for it, and charging accordingly. Compete on quality not price.
Posted by ValueVPS-Dave, 09-26-2009, 03:13 PM
Wow - thats one way to start upsetting people. As it happens, we don't use 'whitebox' servers. Our colo is not cheap - by the time the extras are paid for it costs as much as most other places. RapidSwitch are still under an obligation to provide the service that we pay them for - they failed in that respect. It doesn't matter where in the market they are placed - cheap and cheerful or overpriced - they still have that obligation and so it is right for people a) complain, b) request a compensation Are you affected by the current RapidSwitch situation? If not, why are you even posting here other than trolling to get someone's back up?
Posted by garrence, 09-26-2009, 03:20 PM
Hi RapidSwitch do use a significant amount of whitebox servers for their dedis. You're right that a provider must offer what they claim. Compensation, however, is down to whatever you have contractually agreed. RS' standard SLA will only provide for a few days of compensation, not the month many are demanding. We do have a secondary box at RS but would not use them for anything mission critical. A look at their historical outages is enough to cut through the hype and see that multiple hours downtime is not uncommon. Last edited by garrence; 09-26-2009 at 03:22 PM. Reason: typo
Posted by jarimh1984, 09-26-2009, 03:35 PM
"1 day service credit for each hour when the service is not available." Reads in my 100% SLA contact for the two high end servers, maybe they should change it something like "1 extra gigabyte to hard drives for each hour when the service is not available."
Posted by Votii, 09-26-2009, 03:36 PM
I got the same manufactured message when I asked about compensation.
Posted by -Edward-, 09-26-2009, 03:37 PM
If you are wanting to get a call from the MD etc, it might be worth allowing them a few day's to get back to normal levels of communication as your likely to just get a predefined reply for a few days due to work loads.
Posted by garrence, 09-26-2009, 03:43 PM
That's a reasonably good SLA. I'd assume the upgrade is a goodwill gesture in addition to the contractual SLA payments. Check your paperwork because some providers insist that you initiate a SLA claim soon after an incident.
Posted by brightstation, 09-26-2009, 04:09 PM
I know you've upset some people but I am leaning to agreeing with you. Can you recommend a dedicated server provider (own DC/ 24Hr support) that I can use for mission critical servers. Despite this outage I will probably still us RS for the less critical servers but I am on the lookout as I can't afford for this to happen again. Many thanks
Posted by datajack, 09-26-2009, 04:20 PM
I am one of their smallest dedi customers - two of their cheapest servers, yet they explicitly claim that both are Dell PowerEdge servers commissioned at significantly different times. Now Dell are hardly the most expensive or reliable of boxes, but equally they are not white-box either. I cannot see why they would either lie to me (especially when it's easy to check) or give better kit to their low-profit customers than they do to their major customers. Aside from datacentre migration, the only outage I can remember is the UPS issue a couple of years back.
Posted by garrence, 09-26-2009, 04:23 PM
Brightstation, I'll send you a private message to tell you who we use because I don't want people to think I have a commercial interest. (EDIT: You have PMs turned off so I can't.... sod it... it's gyron.net) Posting on the Requests board at www.webhostchat.co.uk is a good place for offers too. Again in RS' defence - they and Poundhost are by far the best of the low cost companies. They both do an excellent job all things considered. You're just a bit more exposed to risk because a cheaper network is probably more likely to go belly up. Last edited by garrence; 09-26-2009 at 04:33 PM.
Posted by richardwasf, 09-26-2009, 04:23 PM
I have just been passed this extract from an SLA for RS in relation to colo, noticing alot of concern from colo people on this thread. I have to agree with another poster on here, regardless of the stock answer, worth opening a ticket so they have it logged that you are interested.
Posted by garrence, 09-26-2009, 04:24 PM
Datajack, Yes RS have a lot of Dells also. If their offer doesn't say it's a Dell then assume it's a whitebox. I seem to remember something like 8 hours downtime a few Decembers ago?
Posted by datajack, 09-26-2009, 04:47 PM
I'll take your word for that. It's not impossible that I just got lucky. That said, there's nothing inherently wrong with white-box if they are put together right. I don't have access to my historical monitoring data at the mo, but could that have been the ups failure?
Posted by Votii, 09-26-2009, 04:57 PM
Yes that was the UPS problem. Something wrong with the battery maintenance if I remember correctly.
Posted by garrence, 09-26-2009, 04:59 PM
I vaguely recall a failure to a cabinet of routing equipment? Could be mistaken. Wasn't with them at the time. I just remember making a mental note of "Great for cheap and cheerful mass market hosting that can tolerate a rare bit of downtime. Perhaps not best tailored for mission critical applications." But then who do you chose for mission critical? A customer uses a provider that pitches themselves to this market. They've got boxes spread across 2 of their datacentres for supposed resilience, yet have still been exposed to several hours of network instability this year.
Posted by datajack, 09-26-2009, 05:15 PM
No single supplier. It is not impossible for unforseeable circumstances (especially when it comes to software, not configuration, problems from respected companies) to arise. A major cluster collapse could have happened to any network operated by any company. In that regard, anyone running critical services should have redundancy between suppliers. If those who are saying that RS are trying to pin responsibility on Cisco don't have redundancy elsewhere, they are trying to do the same by trying to shift responsibility on RS. True redundancy does not rely on a single datacentre or supplier. Yes it's difficult and expensive but thems the breaks. Any company can have a major problem and the network is not everything. What happens if whatever host you move everything to implodes financially tomorrow? - That is something that in the current economic climate could happen to even the biggest suppliers.
Posted by Scott.Mc, 09-26-2009, 06:35 PM
I agree gyron are top notch.
Posted by Votii, 09-26-2009, 08:06 PM
What I don't like about hosting companies like gyron is their "corporate" website. Why not just give us a price list straight up rather than having to go through the rigmarole of contacting them for prices.
Posted by franky500, 09-27-2009, 01:21 PM
same here.. my single thought when i find something i like the look of but has no price attatche is "must be too expensive".. in reality it might be quite reasonable. but the lack of the prices being advertised leads me to believe things have OTT pricing. just my 2 penies on that. I use Rapidswitch at the moment. a ram upgrades no good to me and neither is a HD upgrade. so whats next...
Posted by admind, 09-27-2009, 02:32 PM
same thing here. I don't need more RAM or HD or more server downtime to up upgrade
Posted by mrakoni, 09-27-2009, 07:26 PM
hmm... nevermind Last edited by mrakoni; 09-27-2009 at 07:31 PM.
Posted by PaulTG, 09-28-2009, 04:01 AM
Just a quick note this morning incase any RapidSwitch clients would prefer not to take us up on the offer of a free upgrade to please get in contact with us through our ticket system in MyServers. We'd really like to hear from you!
Posted by openmind, 09-28-2009, 04:11 AM
We already have Paul and have a ticket waiting in the system...
Posted by ValueVPS-Dave, 09-28-2009, 04:20 AM
Morning Paul, I presume this is upgrade offer is for your dedicated server clients only - has any offer been made for colo clients yet? Regards, David
Posted by daveharris, 09-28-2009, 04:26 AM
Paul I share Davids comments and likewise have a ticket in your system. Dave Harris DHConsulting
Posted by dila, 09-28-2009, 06:58 AM
Has anyone had confirmed that colo clients have an SLA? I can't find any mention of one on the RS website.
Posted by suhailc, 09-28-2009, 07:46 AM
Actually they were offering 6 MONTHS FREE to PoundHost customers.
Posted by unix101, 09-28-2009, 10:36 AM
Service Credits Should the Client not be able to access RapidSwitch’s backbone, RapidSwitch shall credit Client according to the following table. The percentage applies to the contracted service. Colocated servers: Availability Minutes Downtime Service Credit 100---99.95%-------< 22-----------------0% 99.85---99.95%-------22 – 66------------20% 99.5%---99.85%------67 – 219-----------50% < 99.5%-------------> 219----------------100% Taken from my SLA agreement
Posted by Watchur, 09-28-2009, 04:55 PM
By the letter of the SLA, everyone will receive at least a month's rebate on their costs. Given that the combined outages from start to finish where our system's were not usable was multiples of "219 minutes" I suspect it will be much more, especially as it will be important to RapidSwitch to maintain credability that they value their customers. The stock answer if you make a post to RapidSwitch about any of this is the details of the full compensation package will be announced this week. I suspect everyone's waiting to hear the details. This will be separate to their minor 'good will gesture' of a memory/disk upgrade to the subset of people that would be of any interest to. We should give RapidSwitch the opportunity to announce their compensation plans before we pass judgement on how serious they are about their service levels and customer satisfaction.
Posted by Votii, 09-28-2009, 05:08 PM
I doubt we will hear anything until after this months invoice deadline (Wednesday)
Posted by openmind, 09-28-2009, 05:52 PM
Just wanted to add to this to say I have communicated with Paul at RapidSwitch about the issue of last week and we are happy to remain a client of theirs...
Posted by admind, 09-28-2009, 06:01 PM
good for you.... but why?
Posted by XFactorServers, 09-28-2009, 07:02 PM
Yea but why ? I sent Paul 2 emails and never heard back in 1 week+
Posted by openmind, 09-29-2009, 03:26 AM
Simply because after discussion with him, I feel confident that last weeks outage was a one off and they have learnt from a huge mistake. Don't get me wrong, I was seriously considering moving out and we have a lot more than one server but sometimes it's better to look at the bigger picture...
Posted by davidman, 10-01-2009, 09:26 AM
I guess that's lucky for you. We had yet another outage last night.
Posted by richardwasf, 10-01-2009, 09:31 AM
Following conversations with RS it appears they have little knowledge of their network. They have bought equipment and taken the specifications and expected it to work without a significant view of configuration control, utilisation and implementation. The information gleamed from RS indicates this is something they have learnt from but it does leave a massive gap in confidence going forward. R
Posted by Votii, 10-01-2009, 12:54 PM
So did we. Lasted about 25 minutes apparently it was the PSU on the rack where our servers sit.
Posted by bobarm, 10-01-2009, 02:08 PM
Invoice turned up today with no mention of service credits being offered, I realise the invoice system will be automated but how long does it really take to calculate the downtime, turn that into service credits + more if they want to keep customers and let people know?
Posted by Tech-Hosts Team, 10-01-2009, 02:16 PM
What happened to the email out early this week explaining how they are going to make it up to us? I've just paid my invoice to get it out the way, but if I don't get something then i'll be taking my quote with PH.
Posted by dila, 10-03-2009, 07:02 AM
guys, if you want a month free then open a support ticket and request it! rs won't do it automatically to everyone cause it would probably bankrupt them.
Posted by inCharge.co.uk, 10-05-2009, 04:11 AM
> if you want a month free then open a support ticket and request it! I can confirm that. That's what RapidSwitch meant when they said... > Just a quick note this morning in case any RapidSwitch clients > would prefer not to take us up on the offer of a free upgrade > to please get in contact with us through our ticket system in > MyServers. > PaulTG Obviously the single point of failure, combined with lack of on-hand test/replacement equipment was a big, regrettable mistake. However, RapidSwitch have said they are going to address that and I believe them because they'd be crazy not to. I have been using them for several years, and I've used a lot of other hosts, and I will be staying with them because I like their combination of low prices, efficiency, openness. RapidSwitch prices are not low due to cutting corners, as was said earlier on this thread. They are low because their servers are unmanaged, because RapidSwitch own their data centre and network to cut out the middle man, and because you can do everything through the control panel instead of you requesting support, because they run a very efficient operation. Also, even if you disagree with the above, you have to admit that they kept their cool in a crisis, unlike the hysterics on this thread. Driving down to Maidenhead in the middle of the night, demanding your servers back and taking them to another data centre! What kind of disaster recovery strategy is that?
Posted by Votii, 10-05-2009, 08:47 AM
Rapidswitch have told me that they are going to give 100% SLA to everyone as they are now confident about their architecture. They have also offered me a month free. There will also be a full RFO (reason for outage) and a document about what will be changing in the next few days.
Posted by Tech-Hosts Team, 10-06-2009, 04:43 AM
Paul over at rapidswitch is a very polite and considerate guy, especially under the pressure he must be under at the moment. They have given me a credit note for next months fees and that is more than acceptable for the outages. So i'll be sticking with them .
Posted by davidman, 10-06-2009, 05:24 AM
As much as I'm loathe to agree (we've lost new customers due to this), Paul has been very professional throughout and as Dan said - considering the pressure he must be under, he has done very well.
Posted by Tech-Hosts Team, 10-06-2009, 05:39 AM
I've seen it acceptable because I haven't started trading with my new business yet due to the economy, so I haven't lost any customers. Previously when I had my last business I lost about 25 clients bringing in around £600/m due to an outage in the datacentre i used a few years back.
Posted by TrippR, 10-13-2009, 05:36 AM
Just wondered if everyone got the same mail as me yesterday from RS? This is a message sent through MyServers. You can also read this message by logging into your account and going to the message section. We will be performing maintenance which will put your service in an at risk period. This will last about 60 minutes, and during this time, your service is running at a higher risk of interruption if a failure occurs. Date: 14 October 2009 Window: 16:00 for 2 Hours Duration: < 60 minutes This maintenance is to improve the network and is necessary under our programme for increasing the reliability and redundancy of the network. This will not have any impact on the service provided by RapidSwitch. We apologise for any inconvenience this may cause, please do not hesitate to contact us if you have any queries or questions regarding this maintenance window. Regards, The RapidSwitch Team
Posted by ServerSean, 10-13-2009, 05:38 AM
Yes, by this looks of it this only puts our service "at risk" rather than definately going down. The very short notice AND performing the maintenance during business hours is not acceptable. If it does go down I will be very angry.
Posted by ValueVPS-Dave, 10-13-2009, 05:57 AM
Yes, I got it - and i am not looking forward to sending out a mass mail to our clients in the next hour as i know it's going to cause another mass exodus. We are still dealing with issues related to the last major outage and a refund doesn't keep the customer sweet - they want decent service. It seems like RS like to treat their clients with contempt which is costing me money :-(
Posted by franky500, 10-13-2009, 06:17 AM
Actually 4PM works better for me then 9, 10, 11 PM, as the servers i run with them are used for gameserver hosting during normal business hours is better for me.
Posted by ValueVPS-Dave, 10-13-2009, 06:26 AM
I'm sure you've all heard this a million times before... I always get tickets from my clients whenever there is any kind of planned maintenance or unplanned outage (it really doesn't matter)... "This is happening during my busiest time" It's ALWAYS during someones businest time ;-) I fully appreciate that RS need to do this work, however, i think for the majority of us it would be better if this was performed around 2am GMT as i presume the majority of their customers are UK/Europe based.
Posted by Tech-Hosts Team, 10-13-2009, 06:35 AM
Let's just hope it's not the monkeys who were in charge on the last outage!
Posted by ServerSean, 10-13-2009, 07:24 AM
Our clients do not want to hear of any outages. Any more outages would scare most of our client base off, as it will with any other provider. Fingers crossed there is outage.
Posted by ServerSean, 10-13-2009, 09:27 AM
And so it continues....
Posted by ValueVPS-Dave, 10-13-2009, 09:33 AM
Oh Crap :-(
Posted by CretaForce, 10-13-2009, 03:18 PM
I haven't receive the latest message which I believe is good for me.
Posted by Tech-Hosts Team, 10-14-2009, 04:45 AM
Hi, This one looks even worse!!! We are going to be performing some maintenance soon, the details are as follows. Maintenance Type: Network Maintenance Expected effect on your service: No network access Expected duration: 30 seconds This will occur between: 23:00 on Tuesday 20th Oct 2009 and 05:00 on Wednesday 21st Oct 2009 (All times are UK) Area affected: RSH.North from our Maidenhead Datacentre The nature of the maintenance window is to reconfigure the core routers as a VSS-1440 cluster. This will provide an increase in redundancy for your service and will provide additional capacity in terms of port aggregation and network resilience. The migration has been carefully planned and we are using the multiple uplinks that each rack has to stage the process and minimise the impact to service. All of the racks in our Spectrum House data centre have two uplinks. During the 30 second window each rack will, in-turn, have one uplink migrated to the chassis that is configured as a cluster. The link connected to the existing configuration will then be shut down. Once all the racks are connected to the cluster chassis, the secondary chassis will be configured as part of the cluster. At this point the secondary ports will be brought back online and a fully redundant configuration will once again be achieved. The issues previously experienced with the Cisco systems have been completely ironed out and we are confident that this work will be the last major network maintenance for the foreseeable future. We have, however, prepared the process so that a complete rollback is quick and easy to achieve should it be required. We apologise for any inconvenience this may cause and we would like to reiterate at this time that the patience shown by all clients continues to be much-appreciated. Please do not hesitate to contact us if you have any queries or questions regarding this maintenance window. Regards, The RapidSwitch Team
Posted by ValueVPS-Dave, 10-14-2009, 04:58 AM
At least it's a more sensible time. Todays work is at 4pm!
Posted by Tech-Hosts Team, 10-14-2009, 05:10 AM
That is a fair point - but they are moving back over to this VSS cluster, which caused all the problems last time. Lets hope they really have ironed the problems out with it!
Posted by davidman, 10-14-2009, 09:46 AM
So in the next 5 days we have 2 more outages. I can only hope they do just last 30 seconds each.
Posted by aeris, 10-14-2009, 02:05 PM
No problems detected on my side. I'm still waiting for the RFO from the original outage, but I was satisfied with the overall response after it so I'm not planning to move. They offered a free month for all my 11 servers, but I asked for (and got) a free 10 TB/monthly upgrade for all the servers as well as a disk upgrade for a few of them instead.
Posted by techwacky, 10-15-2009, 03:58 AM
oh that great... i also just asked them yesterday for a free month. hope to get response from Paul T soon
Posted by Watchur, 12-30-2009, 08:24 PM
Rapid Switch down AGAIN? All servers unpingable and again their website is down, both their main sales site and their server portal. There's no way to contact their support. Anyone know what's going on this time?
Posted by suhailc, 12-30-2009, 08:29 PM
Their site is up for me.
Posted by JSHosts, 12-30-2009, 08:30 PM
They seem to be online here. Here's the number to call support if you need it: 020 7106 0730
Posted by fanatixx, 12-30-2009, 08:30 PM
It's back up for me!
Posted by RockyRocks, 12-30-2009, 08:30 PM
Back on for now, not sure whats wrong with them though 2009 wasn't good enough for them.
Posted by Watchur, 12-30-2009, 09:03 PM
If you check your rapidswitch server bandwidth monitor for your servers you'll see the outage there. (Assuming you had traffic to see it notice it drop to 0 during the outage period). They do appear up again. Not had any explanation as to the cause of this one yet. I expect they are still investigating.
Posted by PCS-Chris, 12-30-2009, 10:39 PM
Was due to a large scale DDoS attack. I dont think 13mins down is bad considering on-call staff had to go onsite.
Posted by gone-afk, 01-01-2010, 09:02 PM
Still no RFO, I don't see one in my account...
Posted by Tech-Hosts Team, 01-01-2010, 09:08 PM
They haven't even announced it happened have they? £5 says they are going to try and get away with it haha
Posted by gone-afk, 01-01-2010, 09:16 PM
Seems a bit odd, hopefully just a case of management being afk on vacation or something. The outages this year hurt a lot but their openness and communication is the reason I've stuck with them.
Posted by Tech-Hosts Team, 01-01-2010, 09:24 PM
Yeah hopefully - I go back to work on 4th, so maybe thats when their powers that be return to office. Lets wait and see! I paid my invoice this morning regardless
Posted by ThomasC, 01-02-2010, 09:00 AM
Your both correct. An RFO is due to be issued after management return from NY break. We can assume that we will get the RFO (DDOS) on Monday the 4th.
Posted by Watchur, 01-07-2010, 09:35 PM
All my servers down again. The rapid switch portal and their own website is also down. My own support line is ringing madly from my own client complaints and rapid switches phones are engaged so no info available. Been about 15 minutes total black out so far. Anyone know of another means to contact them? Somebody mentioned twitter or something on one of the previous outages.
Posted by RockyRocks, 01-07-2010, 09:45 PM
They are back on now -- Viraj Nevase
Posted by ThomasC, 01-07-2010, 09:47 PM
The line was not engaged at that time, it was also down because the PBX is not hosted externally..
Posted by Jon-RackSRV, 01-07-2010, 09:51 PM
The phrase 'don't put all your eggs in once basket' comes to mind You would have thought that after all the downtime of recent they would have set something up off site in one of iomart's other DC's?
Posted by ThomasC, 01-07-2010, 09:54 PM
That's exactly why it's so frustrating. They have iomarts resources to hand and they have not made any use of them. I wonder if any RFO (I imagine another DDOS attack) will be given to the customers.
Posted by ThomasC, 01-07-2010, 09:55 PM
Yes. I'm slightly taking the piss now
Posted by Watchur, 01-07-2010, 10:00 PM
What would be really good is they at least had their own website/support portal spread across multiple data centers so they don't also go down everytime there is a problem. Thats the very time their portal must stay working - to keep people informed.
Posted by ThomasC, 01-07-2010, 10:04 PM
Rapidswitch have confirmed a RFO will be issued to everyone shortly.

Add to Favourites Print this Article

Knowledgebase

rapidswitch down?

Our Services

Client Menu

Legal