Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > BuyVM down
BuyVM down
| Posted by OakHosting_James, 06-26-2013, 05:19 PM |
It would seem that the whole of BuyVM's Las Vegas infrastructure has been down for the past 40 minutes.
Chat over at Low End Talk would suggest that the problem is wider, and affects most / all of the FiberHub facility.
Anyone know any more? |
| Posted by elrooted, 06-26-2013, 05:20 PM |
Yeah, power issue at Fiberhub.
Hot day in Vegas.
This isn't the first power issue at FH. |
| Posted by Loon, 06-26-2013, 05:24 PM |
|
Mine is up, so it's back or I'm just lucky. |
| Posted by OakHosting_James, 06-26-2013, 05:26 PM |
|
Do you know which node you're on? |
| Posted by 24x7group, 06-26-2013, 05:26 PM |
|
Their site is not loading here from Europe, so I do think it's down. |
| Posted by OakHosting_James, 06-26-2013, 05:28 PM |
I can now get into manage.buyvm, which shows... my LV containers are down!
I won't start a backlog of "Boot" commands  |
| Posted by Loon, 06-26-2013, 05:32 PM |
Quote:
|
Originally Posted by JamesOakley
Do you know which node you're on?
|
lv-node50
Oddly, manage.buyvm.net says it's down, but it's not. |
| Posted by elrooted, 06-26-2013, 05:33 PM |
BuyVM Server Status:
http://buyvmstatus.com/
Nothing definitive yet about the outage other than it is power related.
See nodes with BuyVM in Vegas that *appear* to be up according to that status page. They are single powered at last check, so that is odd.
24khost is down.
Quickpacket is in the same facility with their own network and they appear up.
My guess, partial electric failure inside the facility. |
| Posted by elrooted, 06-26-2013, 05:35 PM |
Quote:
|
Originally Posted by JamesOakley
I can now get into manage.buyvm, which shows... my LV containers are down!
I won't start a backlog of "Boot" commands
|
Good luck there. BuyVM's servers take a while to get booted and online. Noticed that last major outage in Vegas. |
| Posted by CloudComputingLV, 06-26-2013, 05:36 PM |
Looks like SonicVPS is up which from my understanding is a customer of BuyVM and running a Traceroute from FiberHub network shows that it's not a network issue at http://lg.sonicvps.com
1 209.141.63.129 (209.141.63.129) 0.300 ms 0.307 ms 0.133 ms
2 10.1.1.1 (10.1.1.1) 0.464 ms 0.526 ms 0.553 ms
3 te1-1.core2.las1.fiberhub.net (199.47.208.10) 0.460 ms 0.441 ms 0.590 ms
4 207.114.154.141 (207.114.154.141) 7.653 ms 7.626 ms 7.740 ms
5 x.255.129.64 (64.129.255.x) 8.563 ms 8.437 ms 8.346 m |
| Posted by OakHosting_James, 06-26-2013, 05:42 PM |
Quote:
|
Originally Posted by elrooted
Good luck there. BuyVM's servers take a while to get booted and online. Noticed that last major outage in Vegas.
|
I tried once, and it didn't work.
I think it annoys Aldryic if people sit there hitting "boot. ... boot. ... boot."
As for buyvmstatus, I've never quite sussed out who runs that page, but it's not BuyVM themselves. It seems to be accurate only so far as it goes.
Oh well, sit and wait for now. |
| Posted by Host4Geeks-Kushal, 06-26-2013, 05:44 PM |
|
I can confirm FiberHub is having some power issues. Our nodes at Vegas is down as well. Although a few seem to be up now. |
| Posted by elrooted, 06-26-2013, 05:47 PM |
Quote:
|
I think it annoys Aldryic if people sit there hitting "boot. ... boot. ... boot."
|
Ideally those get auto-ignored and kicked of of queuing.
The status page I am unsure also about. It tends to work fairly well. Well enough that I stopped checking to see if it was wrong.
BuyVM has been in the facility's ear about getting the electric upgrades in place for them to have A+B power redundancy to their rack(s). This downtime should put a big priority on that. |
| Posted by Host4Geeks-Kushal, 06-26-2013, 05:49 PM |
|
We are backup now. Our's was about 15 mins. |
| Posted by eva2000, 06-26-2013, 05:56 PM |
Logged into buyvm client area and this is the news
Quote:
Las Vegas Outage - 26June2013
Hi folks;
It seems FiberHub (our Vegas datacentre) suffered a power outage at approximately 1330h PST. We don't have many details yet about what happened, simply that (in FH's words) "There was a partial outage on the utility side. We have started the backup generators by hand, and the contracted electrician is currently en route to the site". More information, as well as their official RFO, will be released as soon as we have it.
We're currently spinning up all nodes to get everyone back online as quickly as possible. Thank you for your patience while we get everything operational again - we'll post another announcement as soon as we have more information to give.
UPDATE: [1440h PST] At this time most of our hardware is back online, and we are spinning up the VPSes as quickly as is safe to do so. If the StallionVM Control Panel (https://manage.buyvm.net/) still shows your VPS as offline, please do not attempt to manually start the VPS, as this only backfills the job queue and causes the panel to respond much slower. We will be manually starting all VMs, so once your VPS is ready for use again it will show as online in the panel.
There are currently a handful of nodes that did not power back on after the outage - we are currently waiting for these nodes to be powered up by the on-site techs. Once they're back online, we'll continue to get all VPSes back up an running as quickly as possible. Thank you for your patience.
|
Strange manage.buyvm.net reports my 3x LV VPSes as down including my primary load balancer. But failover load balancer in buyvm NY VPS which is now working is still serving out pages from 2 or of 3 LV VPSes. |
| Posted by Hoopla-Brad, 06-26-2013, 05:57 PM |
|
It looks like Fiberhub are still missing a number of routes. Perhaps a default route of the facility wouldn't go a astray? |
| Posted by elrooted, 06-26-2013, 06:00 PM |
Yowzers. Hopefully the HVAC wasn't impacted.
101F outside in Vegas now. 110-118F every day through this weekend. Nighttime low of 87-92F. |
| Posted by Aldryic C'boas, 06-26-2013, 06:35 PM |
Hey folks;
Really not much more to say than I've already put into the announcement. FiberHub has taken down their phone lines; and with their website still down, I can't harrass them for even a temporary RFO. From the description they gave, it sounds like they lost grid though.
We're still getting VM's up and running - at this point, the best way to check is to simply ping your IP. There are several hardware nodes that are physically powered off (didn't restart after the outage), so we're having to wait until we can actually contact FiberHub and get someone there to boot them up for us. |
| Posted by Steven, 06-26-2013, 06:47 PM |
Quote:
|
Originally Posted by Aldryic C'boas
Hey folks;
Really not much more to say than I've already put into the announcement. FiberHub has taken down their phone lines; and with their website still down, I can't harrass them for even a temporary RFO. From the description they gave, it sounds like they lost grid though.
We're still getting VM's up and running - at this point, the best way to check is to simply ping your IP. There are several hardware nodes that are physically powered off (didn't restart after the outage), so we're having to wait until we can actually contact FiberHub and get someone there to boot them up for us.
|
Ubersmith is up for me through our raw isp connection. Not accessible via our VPN since everything is dying over timewarner right now.
I opened a ticket 50 minutes ago reporting the routing issues and there has not been a reply yet. |
| Posted by Aldryic C'boas, 06-26-2013, 06:51 PM |
Quote:
|
Originally Posted by Steven
Ubersmith is up for me through our raw isp connection. Not accessible via our VPN since everything is dying over timewarner right now.
I opened a ticket 50 minutes ago reporting the routing issues and there has not been a reply yet.
|
Not seeing any open network related tickets for a 'Steven'. Got a ticket ID I can check? |
| Posted by Steven, 06-26-2013, 06:53 PM |
Quote:
|
Originally Posted by Aldryic C'boas
Not seeing any open network related tickets for a 'Steven'. Got a ticket ID I can check?
|
Sorry, I mean fiberhub. You mentioned they were down and you couldn't contact them. Over some routes, its actually accessible. |
| Posted by Aldryic C'boas, 06-26-2013, 06:55 PM |
|
Aah, you're doing better than us then if you have a ticket open. I still can't get to their site at all. |
| Posted by Hoopla-Brad, 06-26-2013, 06:58 PM |
Quote:
|
Originally Posted by Aldryic C'boas
Aah, you're doing better than us then if you have a ticket open. I still can't get to their site at all.
|
https://noc.fiberhub.com
Looks to be poor network design. At least 3/4 of the global routing table is missing from Fiberhubs network. |
| Posted by Rob T, 06-26-2013, 07:07 PM |
Hello,
Thank you for your patience. Our facility experienced a partial utility power failure this afternoon. Because the failure was only partial and not a complete loss of utility power, our ATS system failed to automatically switch us over from battery to generator power. We subsequently were able to locate the issue and transferred power to our backup generator manually, but unfortunately our UPS system had run out of capacity before we could complete manual transfer. Our electrical contractors are on site now and are working to determine when it will be safe for us to return to utility power. The loss of power also affected one of our core routers, which we are working on now.
We appreciate your patience, and are working diligently to restore 100% service. An RFO will be released as soon as available. |
| Posted by DeltaAnime, 06-26-2013, 07:17 PM |
Quote:
|
Originally Posted by Rob T
Hello,
Thank you for your patience. Our facility experienced a partial utility power failure this afternoon. Because the failure was only partial and not a complete loss of utility power, our ATS system failed to automatically switch us over from battery to generator power. We subsequently were able to locate the issue and transferred power to our backup generator manually, but unfortunately our UPS system had run out of capacity before we could complete manual transfer. Our electrical contractors are on site now and are working to determine when it will be safe for us to return to utility power. The loss of power also affected one of our core routers, which we are working on now.
We appreciate your patience, and are working diligently to restore 100% service. An RFO will be released as soon as available.
|
Thanks Rob.
We've had people scouring the Nevada news sites looking to see if it was a city outage or something just related to FH.
Francisco |
| Posted by Rob T, 06-26-2013, 07:46 PM |
Hello,
If any customers are still experiencing issues, please submit a support ticket. Our helpdesk system is online now.
Thanks. |
| Posted by DeltaAnime, 06-26-2013, 08:17 PM |
Quote:
|
Originally Posted by Rob T
Hello,
If any customers are still experiencing issues, please submit a support ticket. Our helpdesk system is online now.
Thanks.
|
It isn't working over here boss. It's network related since I was able to submit a ticket to the noc page via a VPN.
Francisco |
| Posted by Steven, 06-26-2013, 08:20 PM |
DeltaAnime,
Is it dns resolution related? Google resolvers are having a bit of an issue with the FH network right now. |
| Posted by CloudComputingLV, 06-26-2013, 08:26 PM |
|
I can confirm that there is no issues with TW Telecom. We have a TW Telecom circuit in Vegas with no issues. Other data centers were unaffected in Vegas this was strictly related to FH. |
| Posted by Hoopla-Brad, 06-26-2013, 08:28 PM |
Most likely an issue with their route optimizer? Most routes are back for us but Google DNS looks to be an issue at the moment.
Time for an internal resolver. |
| Posted by qps, 06-26-2013, 08:47 PM |
Quote:
|
Originally Posted by Rob T
If any customers are still experiencing issues, please submit a support ticket. Our helpdesk system is online now.
|
Any ETA for response? I've had a ticket open for over 2 hours. |
| Posted by DeltaAnime, 06-26-2013, 08:56 PM |
Quote:
|
Originally Posted by Steven
DeltaAnime,
Is it dns resolution related? Google resolvers are having a bit of an issue with the FH network right now.
|
I can't ping into most of FH's IP space from Shaw.
I'm thinking the issue is their cross connect to Seattle since FH has a link at SIX.
Francisco |
| Posted by Steven, 06-26-2013, 08:56 PM |
Quote:
|
Originally Posted by DeltaAnime
I can't ping into most of FH's IP space from Shaw.
I'm thinking the issue is their cross connect to Seattle since FH has a link at SIX.
Francisco
|
I was thinking the same thing, I just had some funky cogent routing dying off in SEA. |
| Posted by Steven, 06-26-2013, 09:04 PM |
Quote:
|
Originally Posted by Hoopla-Brad
Most likely an issue with their route optimizer? Most routes are back for us but Google DNS looks to be an issue at the moment.
Time for an internal resolver.
|
Looks like anything to gmail is timing out. |
| Posted by DeltaAnime, 06-26-2013, 09:08 PM |
Quote:
|
Originally Posted by Steven
I was thinking the same thing, I just had some funky cogent routing dying off in SEA.
|
I can ping google from our cage.
I'm having issues bringing up my HE tunnel and I think it might be related. I got a ticket pending right now to get some KVM nodes rebooted but I'm thinking Don is busy with the electricians, Rob's busy with the network, & Zach is likely handling tickets FIFO.
Francisco |
| Posted by CloudComputingLV, 06-26-2013, 09:15 PM |
Quote:
|
Originally Posted by DeltaAnime
I can ping google from our cage.
I'm having issues bringing up my HE tunnel and I think it might be related. I got a ticket pending right now to get some KVM nodes rebooted but I'm thinking Don is busy with the electricians, Rob's busy with the network, & Zach is likely handling tickets FIFO.
Francisco
|
This may help,.
traceroute to fiberhub.com (204.77.1.4), 30 hops max, 60 byte packets
1 x.25x.129.64.corexlayer.com (64.129.25x.x) 0.906 ms 0.874 ms 0.858 ms
2 64-129-238-186.static.twtelecom.net (64.129.238.186) 7.403 ms 7.459 ms 7.447 ms
3 10gigabitethernet1-3.core1.lax1.he.net (206.223.123.37) 7.552 ms 7.860 ms 7.850 ms
4 10gigabitethernet1-3.core1.lax2.he.net (72.52.92.122) 11.134 ms 11.309 ms 11.553 ms
5 10gigabitethernet3-1.core1.las1.he.net (184.105.222.162) 15.607 ms 15.868 ms 15.855 ms
6 vegasnap-llc.10gigabitethernet5-1.core1.lax2.he.net (64.62.249.90) 25.628 ms 25.386 ms 25.343 ms
7 204.77.7.66 (204.77.7.66) 3798.068 ms !H 3258.879 ms !H 3259.082 ms !H |
| Posted by qps, 06-26-2013, 09:29 PM |
|
FYI - e-mails from Fiberhub's ticket system do not appear to be going out. I had to log in and check the status of my ticket. |
| Posted by Rob T, 06-26-2013, 09:49 PM |
|
The majority of our network issues should now be resolved, and helpdesk ticket emails should be going out now. Please feel free to update your ticket if you are still noting any issues. |
| Posted by cloudrck, 06-26-2013, 10:32 PM |
Can't access our VM nor can I submit a ticket. Oddly I can login the Stallion CP and I just reset my VM. Still no luck
Quote:
5 ae-0-0.pr0.atl20.tbone.rr.com (66.109.6.171) 38.018 ms 107.14.19.11 (107.14.19.11) 35.257 ms 35.195 ms
6 * * *
7 te0-4-0-7.mpd21.atl01.atlas.cogentco.com (154.54.88.209) 36.352 ms te0-4-0-7.mpd22.atl01.atlas.cogentco.com (154.54.88.213) 35.869 ms te0-0-0-1.ccr22.atl01.atlas.cogentco.com (154.54.0.161) 35.501 ms
8 te0-0-0-6.ccr21.iah01.atlas.cogentco.com (154.54.29.2) 59.387 ms te0-2-0-1.ccr22.iah01.atlas.cogentco.com (154.54.7.238) 58.499 ms te0-0-0-1.ccr22.iah01.atlas.cogentco.com (154.54.5.94) 89.350 ms
9 te0-3-0-6.mpd22.lax01.atlas.cogentco.com (154.54.0.253) 124.343 ms te0-2-0-3.ccr21.lax01.atlas.cogentco.com (154.54.44.242) 124.530 ms te0-3-0-6.mpd22.lax01.atlas.cogentco.com (154.54.0.253) 124.729 ms
10 * * *
11 * * *
12 38.88.14.2 (38.88.14.2) 115.451 ms 115.524 ms 115.445 ms
13 10.1.1.2 (10.1.1.2) 114.597 ms 114.547 ms 114.614 ms
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
|
|
| Posted by DeltaAnime, 06-26-2013, 10:43 PM |
Quote:
|
Originally Posted by cloudrck
Can't access our VM nor can I submit a ticket. Oddly I can login the Stallion CP and I just reset my VM. Still no luck
|
Is it a KVM?
Currently there is ~6 KVM nodes down, the rest are fine. V6 is back online too.
Francisco |
| Posted by cloudrck, 06-26-2013, 10:55 PM |
Quote:
|
Originally Posted by DeltaAnime
Is it a KVM?
Currently there is ~6 KVM nodes down, the rest are fine. V6 is back online too.
Francisco
|
Yes sir it is. |
| Posted by DeltaAnime, 06-27-2013, 01:37 AM |
Quote:
|
Originally Posted by cloudrck
Yes sir it is.
|
All are up.
From what FH said, they had to reboot them to get them to boot the IPMI (they hung for whatever reason). Since the IPMI didn't complete booting they didn't restart either.
Sometime in August we're going to Vegas to do some hardware upgrades so i'll likely drop in ATS power units that have remote reboot ability too, just incase we ever have IPMI issues.
Francisco |
| Posted by eva2000, 06-27-2013, 02:08 AM |
|
Yeah looks like all my vps in LV are now.. thanks ! |
| Posted by Hoopla-Brad, 06-27-2013, 04:11 AM |
|
Nah, our rack is down again. |
| Posted by Hoopla-Brad, 06-27-2013, 04:19 AM |
|
Back up. Looks to be network, no power loss. |
| Posted by DeltaAnime, 06-27-2013, 04:22 AM |
Given the way the pings were replying FH had a massive massive flood.
I'm not seeing anything reported in our autonull so it must of been someone else.
Francisco |
| Posted by 24x7group, 06-27-2013, 04:50 AM |
I hear from other sources that there was a partial power outage due to the heat as the backup generator didn't kick in.
Which part is true? Is it the network or the power which is/was the cause of this? |
| Posted by cloudrck, 06-27-2013, 10:44 AM |
Quote:
|
Originally Posted by DeltaAnime
All are up.
From what FH said, they had to reboot them to get them to boot the IPMI (they hung for whatever reason). Since the IPMI didn't complete booting they didn't restart either.
Sometime in August we're going to Vegas to do some hardware upgrades so i'll likely drop in ATS power units that have remote reboot ability too, just incase we ever have IPMI issues.
Francisco
|
Still down. Unless my node is stuck in the matrix. |
| Posted by Aldryic C'boas, 06-27-2013, 10:48 AM |
Quote:
|
Originally Posted by cloudrck
Still down. Unless my node is stuck in the matrix.
|
I took care of a few "offline" KVMs this morning already - most common issues are either they booted to the ISO rather than the HDD (easy fix), or the IP Locks didn't reengage properly (which a reboot should fix). Got an IP or a ticket ID I can take a look at? I'll have a look, and get it squared away for ya.
(Are you still unable to reach Stallion or the ticket system, btw?) |
| Posted by cloudrck, 06-27-2013, 11:05 AM |
Quote:
|
Originally Posted by Aldryic C'boas
I took care of a few "offline" KVMs this morning already - most common issues are either they booted to the ISO rather than the HDD (easy fix), or the IP Locks didn't reengage properly (which a reboot should fix). Got an IP or a ticket ID I can take a look at? I'll have a look, and get it squared away for ya.
(Are you still unable to reach Stallion or the ticket system, btw?)
|
I'll PM you |
Add to Favourites 
Print this Article
Also Read
JodoHost Down? (Views: 1509)
Shared Hosting
Shared Hosting
