100TB servers in London, down

Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > 100TB servers in London, down

Posted by elmister, 12-07-2013, 04:16 PM
I have multiple servers at 100TB.com facility in London an they are unreachable from multiple locations.
Posted by nero16, 12-07-2013, 04:22 PM
yep, just contacted support. they told me network issue at their london facility, been going on for 30 minute. no announcement anywhere, no update.

pretty bad for a well established company.

waiting for some form of official update from them somewhere.
Posted by hostlab, 12-07-2013, 04:24 PM
Down here too.

Interesting that uk2.net is still online?
Posted by TmzHosting, 12-07-2013, 04:24 PM
It doesn't seem like its a network issue on the whole London DC since only part of our nodes are down.

- Daniel
Posted by elmister, 12-07-2013, 04:28 PM
from tech support live chat

Quote:

Tyler C.: [2:22:04 PM] We are having networking issues in London at this time. We are actively working to resolve the issue.

Posted by nero16, 12-07-2013, 04:57 PM

Quote:

We do not have an ETA at this time, but we will provide more information when it is available.

seems like its going to take longer than first anticipated
bad timing on a saturday
Posted by eduardomaio, 12-07-2013, 05:13 PM
I received an email from Pingdom with the info that my server was down at 19:52 GMT. It's now 21:12 GMT

I was getting lots of traffic today because my website was featured on a renowned website, so not happy at all.
Posted by elmister, 12-07-2013, 05:29 PM
We're back online now
Posted by gooolll, 12-07-2013, 05:31 PM
It is back online.
bunch amateurs no announcement.
Posted by eduardomaio, 12-07-2013, 05:32 PM
Yes, back online
Posted by Dan_EZPZ, 12-07-2013, 05:32 PM
And down again...
Posted by elmister, 12-07-2013, 05:32 PM
and we are back offline again
Posted by media22, 12-07-2013, 05:34 PM
it's the up and down game =D
Posted by gooolll, 12-07-2013, 05:35 PM
Up and down up and down.
Posted by media22, 12-07-2013, 05:38 PM
Who peed on that cable?
Posted by gooolll, 12-07-2013, 05:41 PM
Online now
Posted by media22, 12-07-2013, 05:46 PM
Must a guy trying to switch the light on but too drunk an switching servers instead
Posted by TmzHosting, 12-07-2013, 05:50 PM
Seems like it's offline again.

- Daniel
Posted by media22, 12-07-2013, 05:55 PM
see Daniel, for once I haven't contacted you... thanks for that email
Posted by pedrojose, 12-07-2013, 05:57 PM
Hi,

Down again from here too...

Does anyone already get a clear and exact explanation from what is happening from 100tb support team? A "netwkr issue" can be a lot of things...

Hope they can get back online soon
Posted by TmzHosting, 12-07-2013, 05:58 PM

Quote:

Originally Posted by media22

see Daniel, for once I haven't contacted you... thanks for that email

No problem . Right when we found out, we sent the e-mail out. We had a bit trouble since we had a thousand clients trying to access solus at the same time. .

- Daniel
Posted by TmzHosting, 12-07-2013, 06:07 PM
To me it seems like a major carrier is down, and not 100TB specifically.

- Daniel
Posted by Kody, 12-07-2013, 06:07 PM
Hey guys,

We are aware of the network issues in the London Datacenter. This is primarily effecting 100TB / MidPhase dedicated customers.

Unfortunately, I don't have a whole lot of information on this yet from our network team. They believe they have isolated the problem, and they are all working to restore service (full hands on deck).

I will update you with more information as soon as I have it.

-k
Posted by pedrojose, 12-07-2013, 06:18 PM

Quote:

Originally Posted by Kody

Hey guys,

We are aware of the network issues in the London Datacenter. This is primarily effecting 100TB / MidPhase dedicated customers.

Unfortunately, I don't have a whole lot of information on this yet from our network team. They believe they have isolated the problem, and they are all working to restore service (full hands on deck).

I will update you with more information as soon as I have it.

-k

Hi Kody,

Do you work at 100TB or UK2Group? If yes, please keep us posted about this problem.

Thanks
Posted by Kody, 12-07-2013, 06:43 PM

Quote:

Originally Posted by pedrojose

Hi Kody,

Do you work at 100TB or UK2Group? If yes, please keep us posted about this problem.

Thanks

Hi Pedro,

Yes, I work for UK2 Group.

-k
Posted by elmister, 12-07-2013, 06:48 PM
20 minutes ago i got this answer

Quote:

There has been a hardware failure on one of the core switches in our London Datacenter. Cisco has been involved in helping us to resolve this as well. Connectivity should be restored in ~30 minutes.

Posted by QuickWeb-Roel, 12-07-2013, 06:50 PM
seems like no redundancy.
Posted by Kody, 12-07-2013, 06:59 PM
@elmister - I can confirm that at this time, we have isolated the issue to failed hardware.

@QuickWeb-Roel - There is redundancy in place, however, due to the way that this has failed the failover did not work as expected. Details about this should be released in the RFO.

Our network team is still working to resolve the issue at this time, and they are working in tandem with Cisco TAC.

I appreciate everyones patience during this outage, and as before as soon as I get more information I will update it here. On a side note, we will be sending a full detailed RFO to affected customers early next week.

-k
Posted by media22, 12-07-2013, 07:01 PM
seems online now
Posted by TmzHosting, 12-07-2013, 07:02 PM
Everything is back online now
Posted by Kody, 12-07-2013, 07:16 PM
Hello,

At this time I can confirm everything is back up and running as expected. As I said before, we will be releasing an RFO early next week.

Thank you for your patience!

-k
Posted by Thomas, 12-07-2013, 07:21 PM
Even though Kody has updated already, I thought I would address two points.

We do run redundant cisco nexus equipment but due to the nature of the failure it caused issues on the vpc peer-link (Cisco Nexus world) which in turn caused issues on links to the N5k switch fabric. It’s a very complex issue as you can imagine, we had all of our network team & Cisco working on this. Things appear to be back now. Obviously I'd like to apologise and we'll immediately as a team get together and see what else we can do here.

Next week we'll also release our status update code into our CP. Due to us having 9 locations, we'll want it so you can select + see all + pull maintenance info from all locations + from our internal teams + our SoftLayer locations etc. We'll make sure this is launched next week.
Posted by gooolll, 12-07-2013, 08:29 PM
No code has to be inserted here.
I was with your company and your support hosting is sucks
Posted by Thomas, 12-07-2013, 08:54 PM
Well, we have to release that feature immediately so we will. If you could send your support concerns to Kody - who is in this thread (you can PM him) - he will look at them immediately, he is a major part of the support team we have. Again, I apologise.

Tom
Posted by pedrojose, 12-07-2013, 10:29 PM

Quote:

Originally Posted by Thomas

Well, we have to release that feature immediately so we will.
Tom

What feature?
Posted by elmister, 12-08-2013, 06:35 AM

Quote:

Originally Posted by gooolll

No code has to be inserted here.
I was with your company and your support hosting is sucks

Are you looking at cp.100tb.com? or somewhere else? i can't find that in cp.100tb.com
Posted by AzaraT, 12-08-2013, 02:49 PM
Anyone else having issues again?
Posted by Dan_EZPZ, 12-08-2013, 02:51 PM
Yep, down again.
Posted by Thomas, 12-08-2013, 02:53 PM
Network team is looking at the issue in the UK now. I will update here as I get more info. I'll try get a temporary status page for London up here too for you to follow.
Posted by pedrojose, 12-08-2013, 03:12 PM
Hi,

Up and Down for the last 30 minutes. Seems network problem again.

Hope @Thomas can give us a explanation and ETA soon.

Thanks
Posted by elmister, 12-08-2013, 03:12 PM
down again, and have customers shouting at the phone
Posted by nero16, 12-08-2013, 03:14 PM
yep all down .. really bad!
Posted by eduardomaio, 12-08-2013, 03:21 PM
Down again...
Posted by Dan_EZPZ, 12-08-2013, 03:22 PM
The issue appears to be worse than yesterday, more servers are offline rather than just a few.

We've also just had a couple come back online while others are still down.
Posted by gooolll, 12-08-2013, 03:36 PM
Garbage is time to move.
Posted by media22, 12-08-2013, 03:51 PM
so, based on the fact that we pay for a service that is down, we should get discount on next bill innit?
Posted by EmptySD, 12-08-2013, 03:55 PM
It will be really nice that anyone write anything anywhere. Server just went down and no info anywhere.
At last they can send email to customers, twit something or update facebook page.
Posted by EmptySD, 12-08-2013, 03:57 PM

Quote:

Originally Posted by elmister

down again, and have customers shouting at the phone

Same here, skype is like Christmas tree.
Posted by Thomas, 12-08-2013, 04:00 PM
This looks related to the issue from yesterday. All the team + Cisco TAC are looking into it. Submit a ticket to billing who will come in tomorrow for credits.
Posted by 24x7group, 12-08-2013, 04:00 PM
I hope they can get everyone up&running again soon.
Does anybody already have an official comment on the cause of this?
Posted by gooolll, 12-08-2013, 04:00 PM
bunch amateurs
Who knows, maybe they not pay the bill.
Posted by Host Surf UK, 12-08-2013, 04:02 PM
I come across this: http://status.vps.net/

Quote:

Sunday, December 8, 2013 at 7:34 PM
London Packet Loss
We are currently experiencing a large volume of intermittent packet loss to the London datacenter. Our network infrastructure engineers are investigating the issue and we will have more information soon.

At the moment some of our servers are up and others not.
Posted by EmptySD, 12-08-2013, 04:04 PM

Quote:

Originally Posted by gooolll

bunch amateurs
Who knows, maybe they not pay the bill.

By my opinion, this is the BEST hosters that can be found. Professional, fast and excelent service and support.
So that's why this is little strange. For last 5 month there was no any problem at all. One problem with HDD and i was back online in ~20 minutes.
Posted by pedrojose, 12-08-2013, 04:05 PM
On ezpzhosting.co.uk host Status Website that hosts some of their servers with 100TB, they are saying and i quote:

"The datacenter is currently offline due to a suspected attack causing all VPC peer links to fail."

Please check at: http://status.ezpzhosting.com/services/kvm1

So i don't know if the problem is related with yesterdays problem.

Hope to get some official information from 100TB here on WHT.
Posted by EmptySD, 12-08-2013, 04:06 PM

Quote:

Originally Posted by pedrojose

On ezpzhosting.co.uk host Status Website that hosts some of their servers with 100TB, they are saying and i quote:

"The datacenter is currently offline due to a suspected attack causing all VPC peer links to fail."

Please check at: http://status.ezpzhosting.com/services/kvm1

So i don't know if the problem is related with yesterdays problem.

Hope to get some official information from 100TB here on WHT.

Just check few posts up.
Posted by gooolll, 12-08-2013, 04:08 PM
fast support? ...... did you receive any email or announcement ?
regarding this issue today?
Posted by EmptySD, 12-08-2013, 04:09 PM

Quote:

Originally Posted by gooolll

fast support? ...... did you receive any email or announcement ?
regarding this issue today?

Yes, today and yesterday via live chat.
But not via email and i don't like that part. Everyone can make mistake.
Posted by gooolll, 12-08-2013, 04:12 PM
WOW they contact you via Live Chat?
you must be special.
Posted by EmptySD, 12-08-2013, 04:13 PM

Quote:

Originally Posted by gooolll

WOW they contact you via Live Chat?
you must be special.

Looks like that
Posted by Thomas, 12-08-2013, 04:13 PM
So basically it looks like the VPC issue from yesterday which I gave information about in this thread and we think there is an attack related part to this. We've got Cisco TAC engineers involved at the moment.
Posted by TmzHosting, 12-08-2013, 04:16 PM
Yep, now all services we have with them are completely down.

- Daniel
Posted by NexD, 12-08-2013, 04:17 PM
Thanks for updating us. This is rather unfortunate to have happened AGAIN.
Posted by gooolll, 12-08-2013, 04:17 PM

Quote:

Originally Posted by EmptySD

Looks like that

Nice so I will contact you next time and you can update us if there is any issue.
Posted by EmptySD, 12-08-2013, 04:18 PM

Quote:

Originally Posted by gooolll

Nice so I will contact you next time and you can update us if there is any issue.

Sure, feel free to contact me, there is my skype
Posted by gooolll, 12-08-2013, 04:28 PM

Quote:

Originally Posted by EmptySD

Sure, feel free to contact me, there is my skype

Ok I will do, make sure to be online at the moment you are offline like the London DC.have a great day
Posted by QuickWeb-Roel, 12-08-2013, 04:55 PM
yup, failed redundancy is like no redundancy at all.
Posted by pedrojose, 12-08-2013, 05:41 PM
Hi,

At this moment my server seems to be back up. Does anyone has their servers back online?

Also does anyone has any updated information about the problem?

Thanks
Posted by RC-Martin, 12-08-2013, 05:53 PM
yes,it appears to be online.let's hope it stays that way
Posted by EmptySD, 12-08-2013, 05:56 PM
Online here too, also got answer from support that problem is solved.
@gooolll problem is solved, your server is online.
Posted by pedrojose, 12-08-2013, 06:10 PM
Hi,

From my end is up. Does anyone get an official explanation from 100TB company or UK2Group?

I need to know in detail what happen yesterday, today and what measures will be taken in order for this kind of problem not happen in the future.

I need this information to inform my clients. Hope some from 100TB or UK2 Group, can tell us here on WHT in detail a full and detailed explanation.

Thanks
Posted by Thomas, 12-08-2013, 06:26 PM
Hi,

So as you can see, London services are back up - we're working on an full RFO now - which will give you the information you require. I provided some basic info in the thread for you guys as an update but the RFO likely early tomorrow will provide most of the details. We'll make this available to the London clients so they can get full details.

Tom
Posted by media22, 12-08-2013, 06:34 PM
yup, back up here too
Posted by gooolll, 12-08-2013, 08:23 PM

Quote:

Originally Posted by pedrojose

Hi,

At this moment my server seems to be back up. Does anyone has their servers back online?

Also does anyone has any updated information about the problem?

Thanks

You can ask the user above : EmptySD ==>> 100tb contact only him when there is a issue on is personal live help maybe you can use his Skype he is always online it's a great support and if there is any problem on any DC he gets contacted first from all the clients 100TB host is a SPECIAL CLIENT don't miss him.
Posted by gooolll, 12-08-2013, 08:30 PM

Quote:

Originally Posted by EmptySD

Online here too, also got answer from support that problem is solved.
@gooolll problem is solved, your server is online.

To Late some one answer back before you Maybe your live Help was not working due to the server in London down sleep like a horse
Posted by EmptySD, Yesterday, 10:13 AM

Quote:

Originally Posted by gooolll

... is a SPECIAL CLIENT...

I am sorry to hear that you are jealous

Quote:

Originally Posted by gooolll

To Late some one answer back before you Maybe your live Help was not working due to the server in London down sleep like a horse

I must tell news to Subscribers firs, then i told to you

You servers are on now, so you can make little research about this Here

Add to Favourites Print this Article

Knowledgebase

100TB servers in London, down

Our Services

Client Menu

Legal