Portal Home > Knowledgebase > Articles Database > Someone did rm -rf / on my server
Someone did rm -rf / on my server
| Posted by wtim, 03-04-2008, 09:01 PM |
| I was logged in under the root ...and someone in my office did run this command..
rm -rf /
Now.i can't ftp to the server nor enter the shell panel..it's a cpanel server..
Please guide what needs to be done.. i have asked to reboot the server and don't know what should be done here..
I have 40-50 sites on this server and none seems to be loading right now..
Any help is appreciated..
|
| Posted by Patrick, 03-04-2008, 09:03 PM |
| 1) Do not reboot.
2) You do realize what that command does, right?
3) Does the command ls work?
|
| Posted by (Stephen), 03-04-2008, 09:04 PM |
| rm -rf /, just wiped out your entire server, time to pull out the backups!
|
| Posted by wtim, 03-04-2008, 09:07 PM |
| The server has been rebooted..However we had a backup server..but will that bring everything..??
I mean the mysql and the email data..
|
| Posted by (Stephen), 03-04-2008, 09:08 PM |
| It depends, do your backups contain the mysql and email partitions/files?
|
| Posted by wtim, 03-04-2008, 09:12 PM |
| I I don't know..we had a second drive configured on that server for weekly backups...does that backup mysql as well??
|
| Posted by Patrick, 03-04-2008, 09:14 PM |
| You should hope so, along with hoping the rm -rf / didn't erase the backup drive if it was mounted too.
|
| Posted by wtim, 03-04-2008, 09:24 PM |
| Yes..it was mounted..but I don't know as I am not being able to login to the shell prompt..
I can feel I am having a bad day..my entire work of 2-3 years has gone in vain if there isn't anything much to be done..
I have a 3rd party backup as well..but I only have www files over there..I am just scared of the mysql databases..because there really isn't any way to backthem up..
|
| Posted by Patrick, 03-04-2008, 09:31 PM |
| All you can do now is ask your data center to check the backup hard drive to see if the content is intact.
If the content is no longer available, you could have the drive sent to a data recovery company, that is assuming your data center will release the hard drive to you.
|
| Posted by wtim, 03-04-2008, 09:36 PM |
| Our server is on layeredtech ..and they are a self managed company..I heard from the technician that they don't do restore stuff..however they can reload the OS and asking me to restore from the secondary drive...( considering if the data is still there )..
|
| Posted by wtim, 03-04-2008, 09:55 PM |
| Can someone tell me where does the .sql file resides in a CPanel server for every account created..
let's say..i have abc.com on a cpanel server..and have 3 db's ..can you tell where can i find the sql files for them in a server??
|
| Posted by wtim, 03-04-2008, 11:29 PM |
| Just recieved a message from the datacenter that they can't enter into any run levels even from the single user mode..
The only option they suggest is to reload the primary drive OS and see if the data still exists in secondary drive and could be restored..
What do you guys think..
a) Will the data exists on the secondary drive.
b) Reloading the OS on the primary drive effect the secondary drive.
I am just thinking..if the secondary drive data also got deleted..then there is no option i believe..??
|
| Posted by Patrick, 03-04-2008, 11:52 PM |
| If the secondary hard drive was mounted and the rm -rf / command was left to execute, then there's a good chance the data has been erased. Hopefully by some fluke, it's still there...
Reloading the OS on the primary hard drive should have no effect on the secondary drive. The biggest concern would be a data center technician not paying attention and accidentally formatting the drive, or installing the OS on it... make sure they are well aware to not touch the secondary drive.
|
| Posted by wtim, 03-05-2008, 12:07 AM |
| They said..they would reloading the OS on a seperate drive..keeping the two primary and secondary drive untouched...
Let's keep the fingers crossed..and thanks PatH for the responses..it;s you guys who keep the noobs alive by these response..really much appreicated..
|
| Posted by -OY-, 03-05-2008, 01:09 AM |
| Who would do such thing in your office? I'll need to be careful too after hearing this, somebody might come over and just do it for the fun of it.
|
| Posted by subzer0, 03-05-2008, 01:34 AM |
| yeah I was wondering the same....
I guess you must have stepped away and left your computer unlocked? If you wiped out a fortune 500 company's data like that, chances are you might go to jail. So whoever did this to you, you need to try and find them and have them face the penalty. If you know who it might be, report them to your manager (assuming by office you meant your day job). I would even consider a lawsuit against the offender. That's BS!
Last edited by subzer0; 03-05-2008 at 01:38 AM.
|
| Posted by @Matt, 03-05-2008, 01:40 AM |
| I'm so sorry to hear that someone would do something like this. I will be a little more cautious when logged in as root.
|
| Posted by CretaForce, 03-05-2008, 04:09 AM |
| I haven't test what happens when rm -fr / deletes the /bin/rm
Does deleting of files still continue or it stops? I think it continues because the command loaded on the memory.
|
| Posted by Extreme43, 03-05-2008, 06:03 AM |
| would be handy to be able to setup a password when removing files from particular directories, suppose this is where not using root directly comes in.
If you've run "rm -rf /" than i would say it would remove all data in your mounts. Your only option is to have the hard disk sent to you or a 3rd party company for data recovery, it may even be possible to pay layeredtech do do such a thing but i'm thinking this is your only option.
|
| Posted by wtim, 03-05-2008, 06:32 AM |
| This is what scaring me the most..I have all the data files backed up..but not the database..on a 3rd party server...
I can easily restore all the php., html files from the 3rd disk..but it won't function until the whole DB is not there...
|
| Posted by Extreme43, 03-05-2008, 06:50 AM |
| How old is the backup and what directories are you backing up? If you have the files backed up than the mysql files should still be intact (depending on your backup scheme).
Checkout /var/lib/mysql/ on your backup, if you have backed this up you will most likely find the files within this directory. If not, do a search for *.MYD and *.MYI (MySQL)files and hopefully it will come up with something.
I would say the last resort would be to contact your customers, offer them a refund for the full or partial period with a free future month or two thrown in if they wish to continue your services - ask them if they have made there own backups, you will find often that customers wont trust hosting services and make there own regular backups.
Than you would be on your steps to recovering your business, if your customers want something - throw it in free. After all, you were responsible for there websites and you have failed (no intentional offense). If it were me i would probably make the person responsible for running that command take on the bulk of the work (if they are your employee that is).
|
| Posted by wtim, 03-05-2008, 06:58 AM |
| Yeah..I am all at this..I have already informed the clients and offer them a solution..they are happy with it..
Unfortunately..I just backed up the home partition and nothing else.....which was the problem..
We haven't recieved any notice from the datacenter about the second drive..which is what I am waiting for at the moment...If the second drive data is also deleted..then the only solution left for me is to start fresh with copying just the files from the 3rd server..
Now.. Is there any way to transfer the files from one FTP server to another server.using a ftp software.i.e 2 remote servers..as it would be impossible for me to copy 1 files each from the linux server..or download them on a local machine and then re-upload on the main machine..
|
| Posted by Extreme43, 03-05-2008, 07:07 AM |
| ok, glad to see your customers are understanding with the situation. I don't see why it would be impossible to copy the files, you could tar the home directory - move it to the webserver and wget it into your other server. Than simply untar the files.
You could use a site-to-site transfer (FXP i think it is) but it would take forever for the fact there are would be so many individual files.
I still think you should look into a data recovery service, it is not as difficult as it sounds. To break it down, when you delete a file on the disk the sector/block is simply set to "Allow data to overwrite" and your data is NOT destroyed. I know this applies to NTFS but am unsure about others - wouldn't see a reason not too.
Furthermore your customers will be impressed to see this.
|
| Posted by wtim, 03-05-2008, 07:14 AM |
| Yeah...I would request the LT guys to check with data recovery on the primary drive which will be kept on a pending que..but I am not sure how much it might cost ....and since they are a self managed company..I doubt they would escalate the process of data recovery on the primary drive..
Yeah..I would try the TAR option ..just wanted to check if there is an easier process..However..since the server is a CPANEL server..does the mail folder in the /home/x123/mail/ would still contain all the mails??
|
| Posted by 040Hosting, 03-05-2008, 07:46 AM |
| OMG, i hope you find out who did this, these are no jokes. Can't imagine anyone would do this to any server. Lesson learned : Off-site backups no mounts.
|
| Posted by Scott.Mc, 03-05-2008, 08:14 AM |
| That is the importance of offsite backups, if your drive was mounted chances are it was wiped too.
"rm -rf /" itself will not do anything, it will just return an error so I assume you mean "rm -rf /*", when it was done you should still be able to browse as once it gets to /bin most of the utils will be gone but you can still browse around with the built in shell commands such as "echo *"
|
| Posted by wtim, 03-05-2008, 08:20 AM |
| Yes..offsite backups of every server is must..which we had..the only thing i feel sad about is the mysql wasn't backed up..and until/unless the datacenter confirms..i am still holding on with the second drive..which was mounted yes..I will confirm here if the files were deleted...
"rm -rf /" was the command since the employee was trying to delete a directory of a user account but accidentally hit enter after the slash and didn't notice..
However..since the server is a CPANEL server..does the mail folder in the /home/x123/mail/ would still contain all the mails and if we create new accounts in the server..would copy/paste the mail folder on the new drive would bring back all the mails??
|
| Posted by Ashley Merrick, 03-05-2008, 01:48 PM |
| So was the command found running and Cntrl + C, or was it found out after the server just went down?
I really think the rm command should ask you for the root password again when you run "rm -rf /*"
Amount of people/servers I have seen thats something along them lines has happened.
|
| Posted by amex, 03-05-2008, 02:06 PM |
| Is there anyway to block the rm -rf / command?
|
| Posted by euselect, 03-05-2008, 02:35 PM |
| I feel sorry for you, every good admin learned his or her lessons.
However, there are good ways to backup most things, including mysql.
There is no excuse for that, but you can learn some lessons.
mysqldump is included in your setup for backing up mysql
backups should always be copied to another place, so if the server fails completely you can restore it again.
rm -fr / is serious, but recoverable if you dont mind paying an expert or you are experienced in hard disk recovery.
Best of luck,
Neil
|
| Posted by derek.bodner, 03-05-2008, 04:15 PM |
| It will continue. Try it once. Install a vps (vmware, xen, your choice of software), and do it in a vm. It's actually entertaining (when you're not doing it on a real server with real data!).
Last edited by derek.bodner; 03-05-2008 at 04:18 PM.
|
| Posted by PrezKennedy, 03-05-2008, 06:09 PM |
| I disagree. It's large security risk to leave your computer unlocked for anything that takes you out of sight of it. The person who gets blamed for deleting company files is going to be the one who left their computer unlocked. You'd need verifiable proof that it was someone else, since all the logging evidence would point to your username.
This was a terrible lapse in judgement, especially since you were logged in as root.
Where I work, leaving one's computer unlocked is an invitation for anyone who notices it to screw with it. It's only a nuisance, but it's enough so that everyone locks their screen now.
I'm sorry you had to go through that in order to learn your lesson though. The person who did that to you was particularly cruel.
|
| Posted by Georgecooldude, 03-05-2008, 06:40 PM |
| If that were someone in my office they'd be escorted off the premises immediately and then a police complain filed for criminal damages.
What is the data worth to you?
less than $750 - take the loss don't pay recovery firm, this will be cheaper than their fees.
$750-2000 - Seriously consider paying for data recovery
$2000+ - Pretty much a no brainer. Go with the data recovery.
|
| Posted by MaB, 03-05-2008, 07:18 PM |
| Hi There,
I'm sorry to hear about what happened
1) the rm command will not stop even after the rm binary is deleted because a copy of the executable is made when the program runs
2) an rm -r -f / is very difficult to recover from if you're running ext3. EXT3 wipes out the inode block #s and so its virtually impossible to reconstruct very large files. If the datacenter will ship you the drive, put it in your computer, purchase a copy of WINHEX and search for key phrases in important files. If you're looking for MySQL data, search for key MySQL phrases or use the MySQL docs to search for the hex codes that can identify a MySQL file. If you're lucky, large amounts of data from MySQL files will be continuous on the drive
The best thing to come out of this will be learning some lessons:
1) Backups on a 3rd harddrive mounted as /backup are NOT sufficient even when combined with RAID. Off-site/off-server backups are MUST
2) Be sure to backup not only /home but also use MySQL dump to backup MySQL files and vital files in /var/cpanel and /etc
3) Write a wrapper to prevent rm -r -f / from running under ANY circumstances! But this gets messy if any os updates try to update the rm binary.
|
| Posted by jamesmoey, 03-05-2008, 07:31 PM |
| I agree offsite backup is a must, always backup your entire disk (not just the /home directory) less the package that can be reinstall from rpm and/or deb repository. Lot of configure and data file are located else where. Also use rdiff-backup, you get incremental backup and uses less bandwidth.
|
| Posted by subzer0, 03-06-2008, 12:20 AM |
| I agree. or at least "are you sure". There should be more safety precautions built into the command when you execute it with those parameters.
|
| Posted by jamesmoey, 03-06-2008, 12:23 AM |
| by default, it does.
But the problem is you put in the option of -f, -f mean that you say yes to everything without prompt. Since you are root, no permission denied there. So basically, all is GONE.
|
| Posted by subzer0, 03-06-2008, 12:29 AM |
| Well duh, the point I was making was if it was an act of sabotage and you knew exactly who did it. Anyway, in the case of OP, the rm appears to have been carried out accidentally by one of his employees.
|
| Posted by subzer0, 03-06-2008, 12:31 AM |
| Then rm should be programmed to ignore -f when path = "/"
|
| Posted by wtim, 03-06-2008, 08:42 AM |
| Just an update..:
The second drive backup got deleted too! Probably because it was mounted too!
|
| Posted by Techbrace, 03-06-2008, 10:00 AM |
| wtim,
I feel sorry for you! With regards to the backup, it's good to have an offsite backup as it is highly reliable, though it's got it's own demerits. You could also keep your local backup drive unmounted unless the data is being backed up or restored. This is highly recommended for those who have got only local backups.
And request all the customers, who have got their own backup, to re-upload the data to come out of this crisis situation and get as many sites online asap. Also proceed with the data recovery, if it's feasible for you.
|
| Posted by Virtuoso Host, 03-06-2008, 02:18 PM |
| The only way left with you is backups , and you should be aware of such people doing these things
|
| Posted by TVLm, 03-06-2008, 05:24 PM |
| If LT manage to install an OS on a new drive, I'd ask them to plug your old 2 in as slaves temporarily. You could attempt to use TestDisk(http://www.cgsecurity.org/wiki/TestDisk_Step_By_Step), PhotoRec(http://www.cgsecurity.org/wiki/PhotoRec) maybe http://foremost.sourceforge.net/ (not used it myself)
I've used PhotoRec to recover data from a shot ext3 raid array, but it take a lot of time sifting through all the recovered files! My advice attempt to recover your backup if they're in tar archive and maybe DB .MYD/MYI files.
Or find out who in your office preformed the rm -f and get them to pay for professional data recovery. Apparently good data recovery companies can recover data overwritten 12/13 times
|
| Posted by MaB, 03-06-2008, 05:29 PM |
| TVLm,
Using those tools to recover a ext3 raid array that got corrupt is likely to work a lot better than using them on an ext3 system that was rm'd because the linux rm command intentionally deletes very crucial block data in the inodes while a corrupt array may have the vast majority of that data still intact.
But, it's always worth a shot depending on how valuable that data is...
|
| Posted by jamesmoey, 03-07-2008, 05:41 AM |
| Newbies should not use -f option.This is clearly human mistake, not bad design of the program.
|
| Posted by superprogram, 03-07-2008, 07:41 AM |
| Really got scarred after reading this thread...
|
| Posted by biggdogg285, 03-07-2008, 08:19 PM |
| I had a similar problem with LT. Primary went bad, so I had layeredtech slave the secondary. The DC tech put in someone elses drive. Nobody knew where my drive went. 3 days it took them to find my backups and put my server back together so I could start restoring.
I had to do a rm -f on my layeredtech account after that.
ThePlanet's taking real good care of me now.
Good luck to you friend!
|
| Posted by omega36, 03-07-2008, 09:24 PM |
| I hope you find out who did this to you @ the office...
|
Add to Favourites 
Print this Article
Also Read
Shared Hosting
Shared Hosting
