Portal Home > Knowledgebase > Articles Database > Trying to build a Disaster Recovery Plan
Trying to build a Disaster Recovery Plan
| Posted by kennethhood, 05-13-2008, 04:08 PM |
| There is so much information on disaster recovery and backing up one's server, that I'm getting glassy-eyed trying to take it in. Maybe if I became an actual case study, and get some "group think" help, this thread could benefit many others in a similar situation.
Current Situation:
1. I'm a small hosting company, 5 years in existence, with about 350 clients. www.mlhi.net
2. Dedicated Linux server, PLESK CP w/unlimited domains license, fully managed at HostNexus (great guys). It does not have a RAID array (used to have that at Rackspace) but it does have a backup drive that everything is backed up to with a cron job every night.
3) In addition I have a Linux Sys Admin on retainer, www.linuxbox.co.uk (he is better than excellent). Two years of excellent server maintenance and security on top of the managed service I get at HostNexus.
4) I just bought a VPS plan at JauguarPC.com after much research (a lot of it here at WHT) and as they say "so far so good" with the ease of dealing with them. I have not setup anything there yet- just got the VPS provisioned a few days ago.
Fears and Concerns:
1. Data center destroyed/ my server burns up (including backup drive) etc etc.
2. DDOS attack (which did hit this data center a few months ago and I was down for hours)
3. If I had to FTP everything back to another server from my local, at 18 GB, it's not too cool.
Want to do this:
1. I want my Sys Admin to run a backup copy (and incrementals every night) to an identically configured VPS server at JaguarPC. Both servers are now running identical PLESK 8.4.
2. I want the fastest recovery possible without spending a ton of money. I know this means I don't get an "instant" recovery, but recovery within 24 hours is more than OK. None of my customers are ecommerce... just brochureware sites.
My "I'm not an expert" plan:
1. If primary server goes bye-bye forever, I can login to my BulkRegister/Enom account and change the child nameserver IPs to the IP's of the VPS. In 24 hours or less, every request for the nameservers would then be routed to the new server.
2. I can create an A record on every domain like www2.johndoeinsurance.com that would point to the IP at the VPS, so I can ease my mind anytime I want to make sure everything is safe and sound on the second server, and ready to go in an emergency.
How do I configure the DNS?
I control dns at Enom for about two-thirds of my customers. I have ALL domains pointed to ns.mlhi.net and ns2.mlhi.net. Here are my options??
1. I create two more child nameservers... ns3 and ns4 and have then pointed to the IPs at the new server, then update all the domains I control. The rest of the customers I can email and ask them to add the additional nameservers. I know... good luck on them doing it.
2. I change the ns2 IP to go to the new server. And I make sure when I make edits on a website during the day that I FTP to both servers.
3. I don't have any nameservers assigned to the new server. I just change the IP on the existing nameservers in the event of an emergency.
I'm sure my ignorance shows... requesting suggestions.
|
| Posted by tepezcuintle, 05-14-2008, 01:54 PM |
| The thing with disaster plans is that a lot of people don't test them before the disaster happens.
You should do the following. On a friday night around 8:00 PM prepare the backup servers and make them ready to accept traffic.
Then at 12:00 midnight change your DNS to point to these servers. Saturday morning start testing those sites. Then make sure everything works. Around 10:00 PM on Saturday point back your DNS to your current servers and make sure everything works.
Then on Monday everything is back to normal
of course let your clients know of your plan so they know that on friday and saturday their sites will be going some maintenance.
I hope this made some sense.
Tepezcuintle
|
| Posted by kennethhood, 05-14-2008, 02:44 PM |
| Simple but great idea. Thanks. I also had posted in dnsstuff.com forum this same question, and a senior guy there said that the best way to protect myself and not spend a fortune on doing truly mirrored servers in realtime was to not use dns as the solution. In other words, I should just remap the existing nameservers to the to the new server in the event of an unrecoverable failure, and just accept the fact that dns caching by the ISPs could/would pose a stumbling block for a day or two. I can live with that.
Your suggestion to run a test (fire drill) over the weekend is great and would make me sleep better at night I think.
I should start posting and replying in WHT and start giving back a little... I have found so many answers here it's time to share what I *do* know with others.
|
| Posted by RelativeDesign-Jerret, 05-14-2008, 07:17 PM |
| Here's what I do:
1) Instead of backing up daily I setup rsync to run every fifteen minutes. I use rsync through an SSH tunnel and have it copy over all my hosted websites, apache configuration files and a user list. Part of my replication script also goes through and updates all the IP addresses to the backup server IP address.
2) I run mysql in master - master mode where all data is current between the two machines (assumes you need mysql replicated as well).
3) I do not use my own nameservers, I use DNSMadeEasy and use their auto failover service.
4) E-mail is a bit tougher so I don't host any e-mail - I use GoogleApps for all my customers and they've all been happy.
What does this give me? Well, if my primary server were to go down:
a) Their e-mail is still available as it's off site.
b) DNSMadeEasy will switch the A record's IP to my backup server and within ten minutes (generally) everyone will be routed to the new server. This is done by setting a low TTL on the records.
c) The backup server is VERY closely synced with the master server so your customers probably won't even know.
d) When the primary server comes back online DNSMadeEasy switches back the A record to the primary IP.
Regards,
Jerret
|
| Posted by RelativeDesign-Jerret, 05-14-2008, 07:19 PM |
| Oh yeah, and you can setup "vanity" nameservers at DNSMadeEasy so all you need to do is switch your host IPs for ns1 and ns2 when DNSMadeEasy is ready to accept your traffic.
|
| Posted by kennethhood, 05-14-2008, 07:56 PM |
| Thanks for the heads up... I will review that solution tomorrow.
|
Add to Favourites 
Print this Article
Also Read
High Load (Views: 678)
Shared Hosting
Shared Hosting
