Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Softlayer DNS Down for +4 hours because of a Distributed Denial of Service (DDOS)
Softlayer DNS Down for +4 hours because of a Distributed Denial of Service (DDOS)
| Posted by nikkoid, 07-28-2009, 02:03 PM |
Softlayer is under a massive DDOS right now...
Since this morning 9am all our Softlayer dedicated servers are down (4 of them). I opened a ticket when their website was still working and after 2 hours they sent an official response saying that they are fighting an ongoing DDOS on their DNS authorities and that they are working on it, but they do not have any time or delay about when it will be solved.
I've never seen something that bad at Softlayer. Even their website is not working at the time I'm writing this post.
Anyone knows more about what is happening? |
| Posted by PersonalJ, 07-28-2009, 02:07 PM |
If their panel is down, perhaps you can get SLA credit  . It's been a long time since I've used SL. |
| Posted by nikkoid, 07-28-2009, 02:13 PM |
|
Yes I guess... and yes their panel is down, but in the mean time this will not compensate the loss of business... we have popular eCommerce websites! |
| Posted by PersonalJ, 07-28-2009, 02:18 PM |
Quote:
|
Originally Posted by nikkoid
Yes I guess... and yes their panel is down, but in the mean time this will not compensate the loss of business... we have popular eCommerce websites!
|
All the more reason for you to take advantage of the SLA credit. If you can connect to their private network via PPTP I would do that and then use Supermicro IPMI to manually reboot your servers if they are down. I assume their web based vpn portal is down right now. |
| Posted by nikkoid, 07-28-2009, 02:24 PM |
Nothing works. Their front page website is working randomly but when you log in, you have a Page Load Error...
It is getting worse, because this morning I was able to send a ticket. I would like to post their official announcement here.
These DDOS are a really big issues, a lot of companies and government agencies are targeted.
How many websites are hosted at Softlayer? +100.000? a Million? More?, what a huge mess! |
| Posted by nikkoid, 07-28-2009, 02:26 PM |
Ok I've now access to their backend, and here is the official announcement :
At 21:49 CDT SoftLayer engineers were alerted to widespread instability in authoritative dns resolution. It was quickly determined that the root cause of the instability was a distributed denial of service attack directed at the authorities dns infrastructure. The attack was mitigated and instability in authoritative dns resolution began to dissipate at about 23:05.
Unfortunately this has yet to completely clear up. Due to this you may see sites available from some locations and not from others. We are working to try and clear this up as soon as possible however we at this time can not provide an eta on when this will be cleared and normal operation restored completely. We will update as soon as we find a resolution, we appreciate your patience.
Thank you,
Romeo R.
SoftLayer CSA |
| Posted by hosteur, 07-28-2009, 02:26 PM |
|
The panel looks good at my end... |
| Posted by nikkoid, 07-28-2009, 02:28 PM |
Here is their official comment about the issue :
At 21:49 CDT SoftLayer engineers were alerted to widespread instability in authoritative dns resolution. It was quickly determined that the root cause of the instability was a distributed denial of service attack directed at the authorities dns infrastructure. The attack was mitigated and instability in authoritative dns resolution began to dissipate at about 23:05.
Unfortunately this has yet to completely clear up. Due to this you may see sites available from some locations and not from others. We are working to try and clear this up as soon as possible however we at this time can not provide an eta on when this will be cleared and normal operation restored completely. We will update as soon as we find a resolution, we appreciate your patience.
Thank you,
Romeo R.
SoftLayer CSA |
| Posted by natecarlson, 07-28-2009, 02:38 PM |
Quote:
|
Originally Posted by nikkoid
Softlayer is under a massive DDOS right now...
Since this morning 9am all our Softlayer dedicated servers are down (4 of them). I opened a ticket when their website was still working and after 2 hours they sent an official response saying that they are fighting an ongoing DDOS on their DNS authorities and that they are working on it, but they do not have any time or delay about when it will be solved.
|
Good reason to distribute your DNS slaves among multiple companies.  |
| Posted by FastServ, 07-28-2009, 02:48 PM |
This exact same thing happened to ThePlanet not too long ago. Seems to be a popular trend.
Move your DNS off-site, ASAP. There's no reason to be using your provider's DNS services unless you like having single point of failure for your entire business. |
| Posted by sirius, 07-28-2009, 03:07 PM |
Quote:
|
Originally Posted by FastServ
Move your DNS off-site, ASAP. There's no reason to be using your provider's DNS services unless you like having single point of failure for your entire business.
|
Bingo!
Sirius |
| Posted by nikkoid, 07-28-2009, 04:17 PM |
|
Yep that's what I'm doing right now... Softlayer's DNS are still down after 7 hours! |
| Posted by BigBison, 07-28-2009, 04:26 PM |
|
There's really no excuse for DNS failures. DNS, done properly, will never fail to resolve. Personally, I just use UltraDNS and have not needed to worry about such things for several years now. |
| Posted by coax, 07-28-2009, 04:52 PM |
If someone really went in for it, I'm sure they could take down a large portion of UltraDNS too..
Nobody is really completely safe from DDoS. |
| Posted by SA-ChrisM, 07-28-2009, 04:56 PM |
|
So much for a properly setup, anycasted DNS service. =P |
| Posted by FastServ, 07-28-2009, 05:40 PM |
|
Anycasting is not immune to DDoS. |
| Posted by cperciva, 07-28-2009, 05:43 PM |
Quote:
|
Originally Posted by nikkoid
At 21:49 CDT SoftLayer engineers were alerted to widespread instability in authoritative dns resolution. It was quickly determined that the root cause of the instability was a distributed denial of service attack directed at the authorities dns infrastructure.
|
DDoS attack, huh? Interesting that a DDoS attack should happen at the moment that a one-packet bind9-crashing exploit gets (accidentally) released to the public.
Makes me wonder if softlayer saw DNS servers falling over and assumed it was a DDoS attack without actually having any evidence for that. |
| Posted by dotHostel, 07-28-2009, 06:17 PM |
Quote:
|
Originally Posted by cperciva
DDoS attack, huh? Interesting that a DDoS attack should happen at the moment that a one-packet bind9-crashing exploit gets (accidentally) released to the public.
|
Could you please post (or PM) a link to the bind exploit? thanks |
| Posted by cperciva, 07-28-2009, 06:22 PM |
Quote:
|
Originally Posted by dotHostel
Could you please post (or PM) a link to the bind exploit? thanks
|
That would be irresponsible.
But I can give you a link to ISC's advisory: https://www.isc.org/node/474 |
| Posted by dotHostel, 07-28-2009, 06:27 PM |
Quote:
|
Originally Posted by cperciva
|
Thank you very much!  |
| Posted by TimothyH, 07-28-2009, 06:30 PM |
Quote:
|
Originally Posted by coax
If someone really went in for it, I'm sure they could take down a large portion of UltraDNS too..
Nobody is really completely safe from DDoS.
|
It seems its been done before: http://www.networkworld.com/news/200...-attacked.html |
| Posted by Acroplex, 07-28-2009, 06:53 PM |
|
Totally down. Can't even access the manage site. WTF. |
| Posted by BuffaloBill, 07-28-2009, 07:57 PM |
Quote:
|
Originally Posted by SA-ChrisM
So much for a properly setup, anycasted DNS service. =P
|
Eh... I've used UltraDNS and DNSMadeEasy for years and have never really had any problems like this. Give them a try. |
| Posted by MikeDVB, 07-28-2009, 08:28 PM |
Looks like it's just the DNS, I logged into a server and did a DNS lookup on the local private network DNS servers for manage.softlayer.com and got http://66.228.118.115/
It works |
| Posted by plumsauce, 07-28-2009, 09:28 PM |
Quote:
|
Originally Posted by BigBison
There's really no excuse for DNS failures. DNS, done properly, will never fail to resolve. Personally, I just use UltraDNS and have not needed to worry about such things for several years now.
|
UltraDNS had an extended outage caused by DDOS a few months ago.
It just takes more bots and more packets.
Mitigating DNS ddos is quite a different animal than HTTP ddos because it's almost trivial to do. |
| Posted by plumsauce, 07-28-2009, 09:30 PM |
Quote:
|
Originally Posted by cperciva
|
That's probably what he meant anyways  |
| Posted by plumsauce, 07-28-2009, 09:33 PM |
One small suggestion:
Always keep a list of the ip addresses of vital control panels or support sites handy in a hosts.txt file that you can use locally if it becomes necessary. |
| Posted by HD Fanatic, 07-29-2009, 01:16 AM |
|
I moved my domains to dnsmadeeasy now. Their prices cannot be beat. |
| Posted by UNIXy, 07-29-2009, 01:38 AM |
Quote:
|
Originally Posted by dotHostel
Could you please post (or PM) a link to the bind exploit? thanks
|
Just read the ISC note and learn how to use nsupdate. I ran the combination of the dynamic update payload but it didn't crash an updated CentOS 5.2/5.3. Can anyone else try it from their end and report?
Thanks |
| Posted by webgearhosting, 07-29-2009, 03:35 AM |
|
I have 1 server at Softlayer, i can acess my website via ip/~username , is there anything i can do for now to up all my client website? Thanks |
| Posted by MikeDVB, 07-29-2009, 03:46 AM |
Quote:
|
Originally Posted by plumsauce
One small suggestion:
Always keep a list of the ip addresses of vital control panels or support sites handy in a hosts.txt file that you can use locally if it becomes necessary.
|
I've started noting all of them down - and I set up the PPTP VPN access via IP instead of domain
Quote:
|
Originally Posted by webgearhosting
I have 1 server at Softlayer, i can acess my website via ip/~username , is there anything i can do for now to up all my client website? Thanks
|
You can use external dns. |
| Posted by webgearhosting, 07-29-2009, 03:49 AM |
Quote:
|
Originally Posted by MikeDVB
You can use external dns.
|
External DNS? can you guide me how to do it? I want my site up and running ASAP, i'm manage to login to WHM via ip by the way.
Thanks
Andy |
| Posted by plumsauce, 07-29-2009, 05:38 AM |
Quote:
|
Originally Posted by webgearhosting
External DNS? can you guide me how to do it? I want my site up and running ASAP, i'm manage to login to WHM via ip by the way.
Thanks
Andy
|
He means go and get an account at a dns provider service and load all your zones there. |
| Posted by plumsauce, 07-29-2009, 05:43 AM |
Quote:
|
Originally Posted by MikeDVB
I've started noting all of them down - and I set up the PPTP VPN access via IP instead of domain
|
You don't need to do that, just load the standby hosts file during an emergency. I have a batch file that does this with the commands:
dns local
dns server
all it does is copy one file or another to the name "hosts" which is required by the resolver.
That's because I am always switching back and forth between production and test servers that need to resolve with the production name during testing.
Just shut down all browser windows, run the right batch file and it's done. No reboot needed. |
| Posted by BuffaloBill, 07-29-2009, 09:25 AM |
Quote:
|
Originally Posted by plumsauce
You don't need to do that, just load the standby hosts file during an emergency. I have a batch file that does this with the commands:
|
You are not suggesting that all of their customers do this correct?
But yes, plumsauce is 100% correct. If you know the IP and you just need access then set it up in the hosts file.
Very funny posts found "from" Softlayer at:
http://dbcohen.com/blog/2007/07/19/s...a-ddos-attack/
They are claiming no customers were affected.... Hmm... Maybe they are trying to get out of their SLA? |
| Posted by BuffaloBill, 07-29-2009, 09:48 AM |
Quote:
|
Originally Posted by HD Fanatic
I moved my domains to dnsmadeeasy now. Their prices cannot be beat.
|
I have used DNSMadeEasy for years as well and have never had a problem. 8+ years of 100% uptime is really a good statistic that many other providers can not claim.
How has it been working so far for you? Did you try other services like UltraDNS as well? |
| Posted by linuxfan, 07-29-2009, 12:38 PM |
Soflayer dns means their name servers like ns1.softlayer.com etc ?
I dont see sense why people using external dns.I always using internal dns for each server and it works perfectly fine. |
| Posted by HD Fanatic, 07-29-2009, 04:27 PM |
Quote:
|
Originally Posted by BuffaloBill
I have used DNSMadeEasy for years as well and have never had a problem. 8+ years of 100% uptime is really a good statistic that many other providers can not claim.
How has it been working so far for you? Did you try other services like UltraDNS as well? |
This is the first time using a third party dns. It was confusing at first but I think I got it working fine now. Their control panel is pretty plain and ugly but it gets the job done. |
| Posted by stablehost, 07-29-2009, 05:42 PM |
|
DNSMadeEasy is very ugly, but it works |
| Posted by woods01, 07-29-2009, 07:47 PM |
Softlayer previously had their company dns done by NeuStar. I raised the question on their forum why a company of SL's size couldn't do it's own DNS, after all they have what 3 or more datacenters? I wouldn't consider multiple datacenters at different physical locations to be a single point of failure.
Not long after that SL started to do their own dns and we see where that got em. If the company can't handle their own dns, how can they service clients?
I think cperciva is on to something. We intentionally run instances of bind, pdns, and djbdns just in case something like this were to occur.
I can't say it would be a quick transition, but it wouldn't be hours of downtime. |
| Posted by BuffaloBill, 07-29-2009, 07:59 PM |
Quote:
|
Originally Posted by nerdie
DNSMadeEasy is very ugly, but it works
|
I agree!!! They claim that the system administrators that use their took do not want the graphics and they just want a simple page.... But yes... DNSMadeEasy does work... and I think it is easy... just does not look that good.  |
| Posted by HD Fanatic, 07-29-2009, 09:29 PM |
Quote:
|
Originally Posted by woods01
Softlayer previously had their company dns done by NeuStar. I raised the question on their forum why a company of SL's size couldn't do it's own DNS, after all they have what 3 or more datacenters? I wouldn't consider multiple datacenters at different physical locations to be a single point of failure.
Not long after that SL started to do their own dns and we see where that got em. If the company can't handle their own dns, how can they service clients?
I think cperciva is on to something. We intentionally run instances of bind, pdns, and djbdns just in case something like this were to occur.
I can't say it would be a quick transition, but it wouldn't be hours of downtime.
|
When did they start to offer ns1 and ns2.softlayer.com to their customers? |
| Posted by MikeDVB, 07-29-2009, 11:21 PM |
Quote:
|
Originally Posted by plumsauce
You don't need to do that, just load the standby hosts file during an emergency. I have a batch file that does this with the commands:
dns local
dns server
all it does is copy one file or another to the name "hosts" which is required by the resolver.
That's because I am always switching back and forth between production and test servers that need to resolve with the production name during testing.
Just shut down all browser windows, run the right batch file and it's done. No reboot needed.
|
I've just got them stored in a text file so I can paste them into hosts if I need.
Quote:
|
Originally Posted by BuffaloBill
They are claiming no customers were affected.... Hmm... Maybe they are trying to get out of their SLA?
|
Connections to all of our servers slowed a bit but there was no interruption of service so I don't see why they should give out SLA credits. As near as I remember their SLA doesn't cover DNS availability...
Quote:
|
Originally Posted by woods01
Softlayer previously had their company dns done by NeuStar. I raised the question on their forum why a company of SL's size couldn't do it's own DNS, after all they have what 3 or more datacenters? I wouldn't consider multiple datacenters at different physical locations to be a single point of failure.
Not long after that SL started to do their own dns and we see where that got em. If the company can't handle their own dns, how can they service clients?
|
Being that you don't know the details - I wouldn't make such bold statements about them personally... Also I don't think their DNS' ability to stay online under an extremely heavy DDoS attack says anything (good or bad) about their services in general.
Quote:
|
Originally Posted by HD Fanatic
When did they start to offer ns1 and ns2.softlayer.com to their customers?
|
I wasn't aware that they had but then again we don't use SoftLayer's DNS (only the rDNS). |
| Posted by layer0, 07-30-2009, 12:49 AM |
DNSMadeEasy is a good option to avoid being affected by issues like this, and as others mentioned it's really not expensive...
Quote:
|
Originally Posted by BuffaloBill
I agree!!! They claim that the system administrators that use their took do not want the graphics and they just want a simple page.... But yes... DNSMadeEasy does work... and I think it is easy... just does not look that good.
|
Honestly, I agree with their claim.
Yes it's not the best looking interface but it gets the job done quite well. |
| Posted by MikeDVB, 07-30-2009, 01:10 AM |
|
Ease of use is very important when it comes to the average consumer - I've not used DNSMadeEasy - even though it doesn't look great is it easy to use from the standpoint of the average consumer (assuming they understand DNS in the first place)? |
| Posted by subigo, 07-30-2009, 01:33 AM |
|
Another vote for DNSMadeEasy. We use them and have never had a problem. |
| Posted by quantumphysics, 07-30-2009, 10:38 AM |
Quote:
|
Originally Posted by BuffaloBill
You are not suggesting that all of their customers do this correct?
But yes, plumsauce is 100% correct. If you know the IP and you just need access then set it up in the hosts file.
Very funny posts found "from" Softlayer at: http://dbcohen.com/blog/2007/07/19/s...a-ddos-attack/
They are claiming no customers were affected.... Hmm... Maybe they are trying to get out of their SLA? |
This entry was written by dbcohen and posted on Thursday, July 19, 2007 at 10:22 AM and filed under Site. |
| Posted by UH-Bobby, 07-31-2009, 12:46 PM |
Quote:
|
Originally Posted by sirius
Bingo!
Sirius
|
Yep.
Not to mention, doing your own DNS is not costly. For a small host, this can be done with a few VPS servers. |
| Posted by plumsauce, 08-03-2009, 05:51 AM |
Quote:
|
Originally Posted by BuffaloBill
I agree!!! They claim that the system administrators that use their took do not want the graphics and they just want a simple page.... But yes... DNSMadeEasy does work... and I think it is easy... just does not look that good.
|
We get the same complaints.^H^H^H observations.
Functional is more important than pretty. The page has to work *every* time, after all it's your dns records.
Bet I could drive up the page count and time spent on site for sure though, and no complaints about ugly. Just post random nude pics on every page |
| Posted by BigBison, 08-04-2009, 11:17 PM |
Quote:
|
Originally Posted by plumsauce
UltraDNS had an extended outage caused by DDOS a few months ago...
|
No, they didn't. I've had zero interruption in my service for over six years now. Get your facts straight...
http://www.circleid.com/posts/200904...e_dyndns_heat/
...especially about your competitors. Interesting way of entering a thread, bashing a competitor by propagating an Internet myth to flash your sig offering a competing service...
UltraDNS customers were attacked, causing geo-local resolving problems for those customers only. You basically have to be UltraDNS in order to have customers worth attacking in such a massive and sophisticated fashion.
Internet crime has certainly gotten out of control these days, but I'll continue to trust my business to NeuStar (as perhaps SoftLayer should have done) because of how little effect that attack actually had on their services. I wouldn't feel comfortable moving to another provider even at significantly lesser cost, until they've proven they wouldn't be taken down entirely by the sort of attack UltraDNS ('s customers) faced.
The incident mentioned is as close as UltraDNS has come to actually being down for any customer, in its entire existence. The affected customers weren't taken down, only slowed down for a few hours, and not globally. Again, in over six years with UltraDNS, my uptime on name resolution has been 100%. |
| Posted by rougemarshal, 08-09-2009, 01:19 PM |
|
We are still having issues (with our sites being unavailable) on the east coast with our sites hosted in the Seattle DS. We too are considering moving to DNSmadeeasy. Support hasn't been able to offer any suggestions. |
| Posted by nikkoid, 08-10-2009, 11:06 AM |
I'm pissed, when their DNS were down, the back end was not working, so I' entitled to get SLA refund for the month. But their accounting think differently.
Conclusion, don't trust their SLA policy, it is BS!
Hello,
Thank you for your patience with this update. The events that transpired with our DNS servers have been extremely regrettable and we share in your frustration. With month to month contracts, it is always in our best interest to keep our customers happy at all times and we hope that we have done that up until this recent event.
Regrettably, DNS is not something that is covered under our SLA. Softlayer views this as an additional free service that is given to our customers at no additional cost. Furthermore, we do not require our customers to use our name servers nor do we assist in setting up their DNS. As a result, we cannot be held accountable for any resulting downtimes. Having said that, we do understand that all service interruptions are detrimental to our customers and at Softlayer and we are always looking at ways to improve on our internal DNS configuration. We have been meeting with several vendors and hope to have a new system out in a few weeks. On that same note, we would hope that you would also look into putting in place some redundancy and/or secondary DNS in order to insure something like this is not possible again moving towards the future.
Again, we apologize for any inconveniences this has caused. We appreciate your business and we are working diligently to insure nothing of this nature is to happen again.
Thank you very much. |
| Posted by HD Fanatic, 08-12-2009, 10:22 PM |
Quote:
|
Originally Posted by rougemarshal
We are still having issues (with our sites being unavailable) on the east coast with our sites hosted in the Seattle DS. We too are considering moving to DNSmadeeasy. Support hasn't been able to offer any suggestions.
|
Many have made the move to dnsmadeeasy. It will be a wise choice. |
Add to Favourites 
Print this Article
Also Read
Shared Hosting
Shared Hosting
