Shared Hosting
Shared HostingKnowledgebase
Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > EuroVPS
EuroVPS
 |
 |
 |
Posted by Shardhost, 01-19-2013, 10:02 AM EuroVPS appears to be down. All of our services are offline and their website is down also
Posted by n!ghtmare, 01-19-2013, 10:13 AM Down here too
Posted by n!ghtmare, 01-19-2013, 10:14 AM And their phone number has no connection
Posted by Maak, 01-19-2013, 10:17 AM
http://www.just-ping.com/index.php?v....com&c=&s=ping!
Posted by target, 01-19-2013, 10:20 AM Weird, no news on twitter also announcing the problem.
Posted by n!ghtmare, 01-19-2013, 10:21 AM I am definitely moving after they come back up. Been with them for around three months and had 5+ downtimes.
Posted by a_system, 01-19-2013, 10:23 AM Down for me too
Posted by pedrojose, 01-19-2013, 10:27 AM also down from here.. no twitter updates and they should have their helpdesk on a different network for cases like this.
i have checked leaseweb status page and not problem there. it seems to be just eurovps
Posted by Shardhost, 01-19-2013, 10:33 AM I've managed to speak to livechat using googles webcache of their page. Support seemed unaware of a downtime. Downtime was confirmed after though and said hope to have it up shortly.
Posted by Maak, 01-19-2013, 10:38 AM
Posted by Maak, 01-19-2013, 10:40 AM OK ONline now!
Posted by pedrojose, 01-19-2013, 10:41 AM They have just now updated their twitter with:
Update on Downtime: Major inbound DDoS. Will update with developments asap.
Posted by pedrojose, 01-19-2013, 10:45 AM My monitoring tools reports that now its up again but it was a total downtime of 44 minutes.
Posted by Maak, 01-19-2013, 10:47 AM
http://www.just-ping.com/index.php?v...779&vhost=_&c=
Posted by a_system, 01-19-2013, 10:53 AM my vps back online - client area not yet - but I surely prefer tis than the opposite...
Posted by target, 01-19-2013, 11:48 AM
Posted by a_system, 01-19-2013, 12:22 PM I have been with EUROVPS with a VPS3 for week now and I am:
- very happy with performance
- very happy with 2-level disk setup
- OK with support and communication (got silver option - too early to have clear view - I see they respond fast but won't get very deep in things that they consider out of the scope of your support plan)
- not happy with the lazyness of greek ISPs DNS (truely awful in some cases but of course not related to eurovps)
So I really wish EUROVPS would prove to be a steady ground for me and I am very worried about this incident and especially the "seemed unaware of a downtime". I would imagine bells ringing and alarms screaming in such a situation, wouldn't you?
Which brings me to another weird thing: my UptimeRobot monitors that I did setup a few days ago show the same "unaware" behavior (see attachment). Why would that happening? Is UptimeRobot some free crap?
Posted by target, 01-19-2013, 12:36 PM
- very happy with performance
- very happy with 2-level disk setup
- OK with support and communication (got silver option - too early to have clear view - I see they respond fast but won't get very deep in things that they consider out of the scope of your support plan)
Posted by pedrojose, 01-20-2013, 07:55 AM Hi,
Does anyone received any kind of official information by email or other way from EuroVPS about what was the issue that caused the 44 minutes downtime yesterday?
And also the interesting part was when the user Shardhost said that when he talked with EuroVPS team it seemed that they where not aware of the problem and if we take the timeframe with attention they have only update their twitter few minutes Shardhost as talked with them by live chat.
Im a long time EuroVPS client and i think they are an excellent provider, but this lack of information and also the re-active way that it seem the problem was handle leaves me with a step back towards the future.
Posted by Shardhost, 01-20-2013, 09:47 AM
Does anyone received any kind of official information by email or other way from EuroVPS about what was the issue that caused the 44 minutes downtime yesterday?
And also the interesting part was when the user Shardhost said that when he talked with EuroVPS team it seemed that they where not aware of the problem and if we take the timeframe with attention they have only update their twitter few minutes Shardhost as talked with them by live chat.
Im a long time EuroVPS client and i think they are an excellent provider, but this lack of information and also the re-active way that it seem the problem was handle leaves me with a step back towards the future.
Since the downtime my sites have been quite slow. Some page loads taking 12 seconds
Posted by pedrojose, 01-20-2013, 10:00 AM
Since the downtime my sites have been quite slow. Some page loads taking 12 seconds
Also regarding the RFO in the past they have always send it in a very short time.. but until now nothing. It seems that they still must be searching and trying to fully understand what happen.
Posted by Shardhost, 01-20-2013, 10:13 AM
Also regarding the RFO in the past they have always send it in a very short time.. but until now nothing. It seems that they still must be searching and trying to fully understand what happen.
Me telling them was downtime
Then telling me there shouldn't be although there might be some maintenance going on
Me telling them that all services were down including their website and I was using the google cache to use the live chat
Posted by Time4VPS, 01-20-2013, 10:19 AM
Me telling them was downtime
Then telling me there shouldn't be although there might be some maintenance going on
Me telling them that all services were down including their website and I was using the google cache to use the live chat
Posted by pedrojose, 01-20-2013, 10:28 AM
They say and i quote: "We dont outsource our support to third parties" on their support info page at: http://www.eurovps.com/support
Posted by Shardhost, 01-20-2013, 10:35 AM Either way. DDOS happens and the downtime I can suffer. They have credited as per their SLA and I await an RFO.
Posted by EuroVPS/Dennis, 01-21-2013, 05:43 PM As promised, here is the reason for outage. Our initial tweet cited a DDoS attack as the cause of downtime. Although this is partly true, now that the dust has settled, we are now able to provide a more detailed uncensored explanation.
When we detected the attack, we dispatched one engineer to the datacenter to investigate the problem, due to heavy snow fall in Amsterdam it took approximately 23 minutes for the engineer to reach the datacenter. During this time, I was attempting to control the situation and inadvertently, due to a human error, created a network loop between our LAN and WAN network. This was definitely one of the most dreadful moments of my life.
Our administrative systems were affected due to this loop as they lost both paths to their iSCSI datastores, as both iSCSI networks were looped. We discovered that an earlier, undetected human error had resulted in a bridge between switches, and this became a cataclysmic issue when I was attempting to control the situation.
When the engineer arrived on site a few minutes later, he corrected the loop, and reloaded the routers. From this moment, internet traffic began to flow normally once again, however systems which were dependant on iSCSI (these are our own systems only, we do not use iSCSI for customer property) were severely affected.
As a result of this, EuroVPS has completed a full migration of it's corporate and administrative systems to Fibre Channel, and even in the case of any future network anomaly, loop, etc, such an event cannot reoccur.
We do not use iSCSI for any customer properties for various reasons, and as of now, we are also glad to have moved our own systems away from iSCSI. iSCSI is excellent if used properly, however if any human error is introduced and/or any networking is shared between it and other non-iSCSI systems, disaster can ensue.
We were aware of this deficiency and had already planned a maintenance window to migrate all of our systems, administration platforms, email, away from iSCSI and to Fibre Channel. This now, was completed during the last 36 hours.
Again, we do not use iSCSI for our customer servers, and no customers experienced any data loss or corruption due to this fault, however some customers were exposed to network downtime (public internet access) for up to 44 minutes.
The first racks started coming online 6 minutes after the reload, however the initial cause of the downtime was still ensuing, and therefore more work was needed to get other racks online. Within 40 minutes, all customer services were online.
Our administration systems however were offline for a slightly longer than our customers', which is why there were reports of not being able to login to the client area portal. This was because the migration towards Fibre Channel started immediately after our customers WAN & public VLANs were online.
We apologize for this very embarrassing mishap, and rest assured that we have taken corrective measures so this never happens again. Any clients that want to claim SLA compensation are of course welcome to request it by sending an email to administration@eurovps.com.
Best Regards,
Dennis M.
EuroVPS Sr. Network Engineer
Posted by pedrojose, 01-21-2013, 07:16 PM
When we detected the attack, we dispatched one engineer to the datacenter to investigate the problem, due to heavy snow fall in Amsterdam it took approximately 23 minutes for the engineer to reach the datacenter. During this time, I was attempting to control the situation and inadvertently, due to a human error, created a network loop between our LAN and WAN network. This was definitely one of the most dreadful moments of my life.
Our administrative systems were affected due to this loop as they lost both paths to their iSCSI datastores, as both iSCSI networks were looped. We discovered that an earlier, undetected human error had resulted in a bridge between switches, and this became a cataclysmic issue when I was attempting to control the situation.
When the engineer arrived on site a few minutes later, he corrected the loop, and reloaded the routers. From this moment, internet traffic began to flow normally once again, however systems which were dependant on iSCSI (these are our own systems only, we do not use iSCSI for customer property) were severely affected.
As a result of this, EuroVPS has completed a full migration of it's corporate and administrative systems to Fibre Channel, and even in the case of any future network anomaly, loop, etc, such an event cannot reoccur.
We do not use iSCSI for any customer properties for various reasons, and as of now, we are also glad to have moved our own systems away from iSCSI. iSCSI is excellent if used properly, however if any human error is introduced and/or any networking is shared between it and other non-iSCSI systems, disaster can ensue.
We were aware of this deficiency and had already planned a maintenance window to migrate all of our systems, administration platforms, email, away from iSCSI and to Fibre Channel. This now, was completed during the last 36 hours.
Again, we do not use iSCSI for our customer servers, and no customers experienced any data loss or corruption due to this fault, however some customers were exposed to network downtime (public internet access) for up to 44 minutes.
The first racks started coming online 6 minutes after the reload, however the initial cause of the downtime was still ensuing, and therefore more work was needed to get other racks online. Within 40 minutes, all customer services were online.
Our administration systems however were offline for a slightly longer than our customers', which is why there were reports of not being able to login to the client area portal. This was because the migration towards Fibre Channel started immediately after our customers WAN & public VLANs were online.
We apologize for this very embarrassing mishap, and rest assured that we have taken corrective measures so this never happens again. Any clients that want to claim SLA compensation are of course welcome to request it by sending an email to administration@eurovps.com.
Best Regards,
Dennis M.
EuroVPS Sr. Network Engineer
Its great that EuroVPS can be so clear about what happen.. but i think that before you posted to the public you should first send out the information to your clients (by Email / RFO) and only then to public announce.
Take me as an example, one of my clients read this before me and call me. I did know nothing at the moment...
Posted by pedrojose, 01-21-2013, 09:17 PM already received direct mail from EuroVPS with RFO. things things happen.
anyway i will still keep my services at eurovps, as until now their services as always been very reliable
Posted by kpmedia, 01-22-2013, 12:42 AM
_______
This was under an hour, the first time in years, over the weekend (down time for most sites; i.e. not rush hours), and honestly the looping the servers is funny. If you got to screw up, at least make it humorous! I'm going to cut them slack on this one. Even at their worst, they're better than 99% of the host (and "hosts") out there.
.
.
Posted by target, 01-22-2013, 03:34 AM At least they are upfront and not making up excuses. +1 for Eurovps.
Posted by layer0, 01-22-2013, 06:30 AM That's a pretty transparent and professional explanation for the outage, which wasn't even that long in duration in the grand scheme of things. Kudos to them. Downtime happens to everybody, what counts is how you handle it.
Add to Favourites 
Print this Article
Also Read
