Portal Home > Knowledgebase > Articles Database > correct way to set up linux software raid1 (md raid) ??
correct way to set up linux software raid1 (md raid) ??
| Posted by papi, 11-16-2014, 04:13 AM |
| I'm putting together some new servers, each will have 2 SSD's that I wish to run in md raid1.
In the past I've used adaptec HW controllers and that was always easy as - as far as linux was concerned it was just one HDD volume, the hw controller took care of the rest.
Anyway .. seems md raid is quite a bit more "involved" and there is a LOT of conflicting info and instructions regarding it on the net.
Most importantly - when setting up raid1 in anaconda (the gui setup of centos6) - after creating the raid partitions/devices, it asks where I want the bootloader to be installed ie. /dev/sda or there is also the option of /dev/md0
Now why is this important you ask? Well because from what I've read if I put grub into /dev/sda and /dev/sda dies, I won't be able to boot from /dev/sdb (so much for redundancy .. seems hw controllers are way superior in this regard)
Some people claim if using md raid1 you should pick the /dev/md0 option and that way if either drive fails, the other one will be bootable. Other people say no that won't work at all. Some say they can't even boot servers AT ALL if they install grub to /dev/md0
what the hell? Why is it so hard to find a real answer ??
I don't want to end up in the situation where /dev/sda dies and I can't boot from the 2nd drive (/dev/sdb) without driving to the DC and rebooting into rescue mode, then praying that some 'hack' like forcing installation of grub into the surviving raid member (/dev/sdb) - I've seen posts about this not always working either.
Then there are other issues ... like people saying upon reboot, the system can change the /dev/sda to /dev/sdb and vice-versa, so you really never know which drive is really dead. What the hell?
I am really reaaaaaallly starting to think that linux software raid is more of a headache and disaster waiting to happen than something that's supposed to allow me to sleep easier at night knowing that should a HDD fail, there's a mirror that will take over until the failed drive can be replaced. I mean what the hell is the point of md raid if it can't TRULY mirror a hdd INCLUDING Grub / bootloader ?
Anyway, I'd love to hear from those experienced with md raid .. not just general sw vs hw raid (I've read it all before)
|
| Posted by madaboutlinux, 11-16-2014, 06:37 AM |
| You can install bootloader on the first partition of both the drives and the server will boot if any one of the drive fails. You need not need to install it on /dev/md0.
After installing grub on say /dev/sda, you can use the 'device' command with /dev/sdb to make grub assume that /dev/sdb is actually /dev/sda and install the bootloader again which then install it on /dev/sdb.
Once the bootloader is installed successfully, check with the below command at the grub prompt to find which partiton contains the bootloader:
After you successfully complete the setup, simulate the failure of anyone drive using the mdadm command and see if things work as expected.
|
| Posted by papi, 11-16-2014, 07:08 AM |
| My understanding is that the bootloader (grub stage1) gets installed on the first block of a drive ...not a partition as such. So what do you mean by "first partition of each drive" ???
You yourself are referring to /dev/sda which is a drive, not a partition.
I am sorry I don't mean rude but your comments are so vague and convoluted I have no idea what I am supposed to take from them. I've found much clearer info already on the net and I am still beyond confused, so this does not help me at all.
I am paranoid about trying to install the md raid on my new SSD drives until I understand this issue (how to get grub to load regardless of which drive fails) BECAUSE once I install md raid, it will write the superblocks on the ssd drives which contain the mdraid info ...which from my understanding is not so easy to remove (eg. if I make a mess and reinstall centos6, the old raid info will STILL be on the drive(s) first block).
I am serioulsy just about ready to give up on the mess that md raid appears to be and get a couple of adaptec 8405's.. To be perfectly honest I think the whole mdraid is done arse-backwards. I mean you have to create partitions FIRST on each new drive, THEN the raids. With HW controllers you create the RAID first ...then partition any way you like - which is how it should be.IMO.
|
| Posted by madaboutlinux, 11-16-2014, 08:23 AM |
| That's OK. I was referring the drive in contest of grub installation on both drives. It will of-course be installed on the first partition of the drive i.e. where /boot generally is.
I feel you have enough knowledge to try things out practically instead of getting confused by reading lots of tutorials available on the net.
|
| Posted by papi, 11-16-2014, 09:31 AM |
| well I've never used it in the past.
I've run some tests on a virtualbox VM (centos6) ..and wasn't too impressed. I 'simulated' drive failures by shutting down the VM, removing one of the attached (virtual) drives and restarting VM. The results were ... how shall I put it, confusing. At first when I removed one of the VD's, it booted up normally in degraded mode. Then when I shut the VM down and re-attached old VD and started VM again, out of the 3 partitions / raid devices that I have (/boot / and swap) 2 were 'removed' and one was OK according to /proc/mdstat .. what that means I have no idea.
Anyway, testing in VM is clunky to say the least so I've just installed 2 new sata HDDs into the server and installed centos6 on md raid1. Even though I chose /dev/md0 as my bootloader target, when I first rebooted after centos installation - it didn't find anything in the boot sector so just got the standard 'insert valid disk blah blah'
So I remembered something I read earlier and booted back into rescue mode, chroot /mnt/sysimage and did a "grub-install /dev/md0" and rebooted. Now it boots. Why I don't know. The centos setup was supposed to do the same thing (ie. instal grub to /dev/md0)
So this leaves me with the impression that linux software raid is at best inconsistent at worst, a risky and dangerous toy to be used on production enterprise servers. That's my impression so far. It looks like a neat way to have raid1 for your local workstation and whatnot (ie. non critical machines) but it doesn't look like something that's meant to be used in production servers with the ****** documentation, inconsistent documentation and unexpected results .. at this point I'm 90% sure I'll just be ordering a couple of Adaptec 8405's within 48 hours unless further testing proves me wrong.
|
| Posted by papi, 11-17-2014, 08:45 PM |
| Went out and bought some cheap 64GB SSDs yesterday for testing purposes and installed them in the servers.
md raid was a lot better behaved and the results were better than expected during my few hours of testing (still more testing to do).
ie. I installed fresh copy of centos 6.6 and chose /dev/sda as bootloader destination and it appears that the bootloader was installed to both /dev/sda AND /dev/sdb (which is good). How do I know this? Well, I shut down the server, removed one drive each time ..booted up, it worked both times. So it was able to find the bootloader (grub) on either drive, which is good.
I've also used a 3rd (new) ssd drive as a replacement to simulate hdd replacement - ie. shutdown working server, removed one of the drives and inserted the new one. The new one obviously didn't have any bootloader info but I was able to copy the bootloader using the grub menu and commands that can be found on the net eg. http://www.anchor.com.au/hosting/sup...re_RAID_Repair under "Install GRUB" and it worked. Copying the partition info to new drive also worked without dramas and mdraid synced the drives.
So to sum up ... mdraid in a VM was/is a pain in the ass and seemed rather flaky. But it seems to be very predictable and works as advertised when used on dedicated server.
Could it be better? Hell yes. For one, I still think mdraid shouldn't be creating multiple raids - it should just create a single raid volume and let you partition AFTER that (ie. like HW raid controllers do) and there should be a simple and verifiable way of finding out whether GRUB bootloader is correctly installed in each drive (in case of failure ..last thing you want is to be unable to boot).
|
| Posted by TQ Mark, 11-18-2014, 09:14 AM |
| Thanks for posting your results from testing!
That is good, but really weird that the centos installer GUI gives no indication it is installing grub to both drives. Usually I make /dev/md0 as the first RAID device for /boot, and then install GRUB to /dev/md0 (first partition of each disk)
true, but that would involve more complexity with what is sitting on top of the RAID device. You can minimize the # of RAID devices you need by using LVM - i.e., just make two RAID devices, the first for /boot and the second for LVM in which you create logical volumes for swap, root, and more if you want separate filesystems for things like /home
|
| Posted by RRWH, 11-18-2014, 07:59 PM |
| a reasonable guide can be found at http://www.howtoforge.com/how-to-cre...hat-6.0-system
It is not perfect, but if you have never done it - a good starting point.
Whenever we build systems with mdadm (soft raid) We prefer to do the build on a single disk then set up the appropriate raid (1 or 10) and then manually create the groups. Main reason is that the few times we have just used the installer, it was right royally messed up when it came time to do our fail tests before moving the server to prod and had to start again anyway!
When setting up soft raid, we usually go with a /boot and an LVM PV soft raid arrays, and install Grub onto all the physical disks and test by failing 1 at a time all disks in the array and ensure we can boot up.
And yes, it is a good idea to grab a handful of disks, put them into a test server and try things for yourself.
|
Add to Favourites 
Print this Article
Also Read
Gigenet down? (Views: 804)
Shared Hosting
Shared Hosting
