The whole server READ-ONLY

Portal Home > Knowledgebase > Articles Database > The whole server READ-ONLY

Posted by Markovic, 08-27-2010, 03:35 PM
Hey, /tmp is chmoded to 777 but it appears to be read only. root@helena [/tmp]# touch test touch: cannot touch `test': Read-only file system How to make it normal again? Thanks
Posted by Markovic, 08-27-2010, 03:39 PM
cat /proc/mounts rootfs / rootfs rw 0 0 /dev/root / ext3 rw,data=ordered,usrquota 0 0 /dev /dev tmpfs rw 0 0 /proc /proc proc rw 0 0 /sys /sys sysfs rw 0 0 /proc/bus/usb /proc/bus/usb usbfs rw 0 0 devpts /dev/pts devpts rw 0 0 /dev/sda1 /boot ext3 rw,data=ordered 0 0 tmpfs /dev/shm tmpfs rw 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 /dev/loop0 /tmp ext3 ro,nosuid,noexec,data=ordered 0 0 /dev/loop0 /var/tmp ext3 ro,nosuid,noexec,data=ordered 0 0 # cat /etc/fstab /dev/VolGroup00/LogVol00 / ext3 defaults,usrquota 1 1 LABEL=/boot /boot ext3 defaults 1 2 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 /dev/VolGroup00/LogVol01 swap swap defaults 0 0 /usr/tmpDSK /tmp ext3 defaults,noauto 0 0
Posted by Markovic, 08-27-2010, 03:42 PM
mount /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw,usrquota) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/sda1 on /boot type ext3 (rw) tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) /usr/tmpDSK on /tmp type ext3 (rw,noexec,nosuid,loop=/dev/loop0) /tmp on /var/tmp type none (rw,noexec,nosuid,bind)
Posted by Cymbiant, 08-27-2010, 03:56 PM
Anything in /var/log/messages ?
Posted by DeltaAnime, 08-27-2010, 04:08 PM
Try doing that on another location, namely /home If a filesystem is kicking over into RO, it means it ran into a badsector/error and is trying to protect further issues. Francisco
Posted by M Bacon, 08-27-2010, 04:38 PM
Try chomdding it to 1777.
Posted by aeris, 08-27-2010, 04:55 PM
You'll probably have to fsck it.
Posted by lynxus, 08-27-2010, 04:57 PM
AFTER! taking a backup of all your data!
Posted by UnderHost, 08-27-2010, 04:58 PM
Yup, then if the server reboot after FSCK and still not readable you will be probably need to change your hard drive due to badsector on it
Posted by Steven, 08-27-2010, 06:04 PM
Its a pseudo filesystem. It happens from time to time. Probably a aborted journal. from his output: /usr/tmpDSK on /tmp type ext3 (rw,noexec,nosuid,loop=/dev/loop0) Run this should bring you back online
Posted by Markovic, 08-28-2010, 03:30 AM
hey, My whole server gone READ-ONLY and it caused problems for everyone. root@server [~]# touch test touch: cannot touch `test': Read-only file system Here is what /etc/log/messages says Aug 28 02:01:53 server kernel: EXT3-fs error (device dm-0): ext3_lookup: unlinked inode 46894565 in dir #46893904 Tried to run fsck but I'm getting an error so I don't know is it smart to continue. WARNING!!! Running e2fsck on a mounted filesystem may cause SEVERE filesystem damage. Please help
Posted by Markovic, 08-28-2010, 03:32 AM
Now the whole server gone READ-ONLY. <> This issue I solved by deleting /usr/TMPDSK and recreating it. Last edited by bear; 08-28-2010 at 07:21 AM.
Posted by tanfwc, 08-28-2010, 03:45 AM
Ask your provider to run fsck when the system boot up. You can't run fsck because your main file system is mounted, unless you are doing fsck on a secondary disk
Posted by lynxus, 08-28-2010, 04:02 AM
Have you tried restarting it? ( I assume you have? ) Sounds like the disk is failing.... tried a mount -a ?
Posted by Markovic, 08-28-2010, 04:13 AM
The server didin't come back after a reboot. Is it possible I lost all thed ata? I can't believe
Posted by lynxus, 08-28-2010, 04:18 AM
Its unlikely, The best thing to do is either get your host to check its vga output ( see what's on the screen, It may be its waiting for some kind of disk check ) If its showing a terrible failure, Maybe get a new disk put in, rebuild the server and have the broken disk plugged in to the secondary port? Maybe you will be able to mount the drive as readonly and pull what data you can from it... I suppose this shows the problem of not having a raided redundant array.. Wish i could help further at this point.
Posted by gone-afk, 08-28-2010, 04:41 AM
It's probably running a fsck, and/or hung on one of the errors. Contact your host and have them step through the errors/fix them.
Posted by mugo, 08-28-2010, 04:46 AM
Also, if it found errors and forced a check on reboot, that may take quite a while to run. You may see it come up after the scan completes if no bad issues are found. I've had them come up 1-2 hours or more after the reboot, just depending on hdd size, data, amount of errors, etc. Best thing is to do as lynxus said, get your host to put eyes on it and see what's happening. You don't want them to just power cycle it if it's in the middle of a file check. That could make things worse.
Posted by madaboutlinux, 08-28-2010, 04:51 AM
The system might be performing a File System Check while booting and another reboot may make matter worse. It is best to ask your host to take care of the issue OR if they can attach a KVM unit to the server, you can also monitor and perform FSCK on all the partitions by booting the server in the single-user mode.
Posted by Cymbiant, 08-28-2010, 06:49 AM
Get someone to put a monitor on the server as it may be waiting for user input after the fsck. Have seen it before where the server sits at a 'press enter to continue' type prompt after performing a check. Also is the server raided or single disk? and are your backups upto date (Just in case)
Posted by Markovic, 08-28-2010, 01:34 PM
The datacenter says there is a file system corrupted. Whats to blame? How could I recover the data? Would a livecd + fsck help? Your help would be MUCH appreciated. Thanks
Posted by lynxus, 08-28-2010, 01:39 PM
Did they not give you any more info? I suppose a liveCD could help, You might be able to mount the drive / check disk.. Im surprised on boot it didnt ask to check the disk. it may just be that the drive has died. It happens
Posted by Markovic, 08-28-2010, 02:43 PM
alright. I got KVM access. Please suggest me what to do next. I don't want to lost the data. Can I do fsck via KVM?
Posted by HostingBig, 08-28-2010, 02:52 PM
it is probably waiting on a manual fsck fsck -y /dev/yourhd or forced manual fsck -f or third option you have a corrupt superblock you will need to switch it with a backup superblock what does it say at the prompt?
Posted by Ronald_Craft, 08-28-2010, 03:06 PM
Oh joy. We just dealt with a similar issue on a server earlier in the week. Luckily, our servers all run in RAID so we just had the DC replace the bad harddrive and were back up and running in about an hour. I highly recommend you invest into a RAID setup in the future. This might seem a bit more costly, but imagine the money you'll save by avoiding situations like this. As recommend already, get into single user mode and run a manual fsck. That will give you an idea of where you're at. Normally, the FSCK will complete, the system will boot and then you're good to go. However, if your filesystem really is seriously corrupt then your data may not be recoverable. At that point you can only hope that you have backups of your data.
Posted by HostingBig, 08-28-2010, 03:08 PM
Raid does not protect against file corruption but yes Raid arrays are the best
Posted by Markovic, 08-28-2010, 03:25 PM
KVM console says: EXT3-fs error(device dm-0): ext3_lookup unlinked inode 35193133 in #dir 35192883 exec of init (/sbi/initd) failed!!!: No such file of directory Kernel panic - not syncing: Attempted to kill init! And I can't get the system to boot. How could I recompile the kernel and do fsck when the system can't even boot?
Posted by HostingBig, 08-28-2010, 03:31 PM
you will likely have to get a live CD or rescue disk installed and fsck it from there
Posted by Ronald_Craft, 08-28-2010, 03:35 PM
It's time to get the datacenter involved. There's really not much else you can do from KVM at this point.
Posted by HostingBig, 08-28-2010, 03:43 PM
Agreed they should have done it from the start unless your a unmanaged colo customer, if you were our customer we would have saw the issue when the server from the console and all ready took the appropriate action and all ready FSCKED your drive and you would likely been on line by now no sense letting the customer fumble around not knowing what to do.
Posted by Markovic, 08-28-2010, 03:53 PM
Thing is, nobody is in the datacenter on weekends and sending someone in the DC with livecd assisting me in this matter prices more than server, per one hour. Although we know what to do, me or my staff can't to anything as we don't have console access Thanks for your help
Posted by HostingBig, 08-28-2010, 04:07 PM
who installed the KVM ? you could try booting to a older kernel from the grub menu
Posted by Ronald_Craft, 08-28-2010, 04:11 PM
What the heck kind of a DC is this? Servers don't care if it's the weekend. They crash whenever they feel like. Any good DC has staff available 24/7 to assist with this kind of issue.
Posted by HostingBig, 08-28-2010, 04:23 PM
probably to save money the risk you take to save money but I find that servers never fail 9:00 - 5:00 as a mater of fact I don think a server of ours has ever failed at that time is always late P.M or early A.M

Add to Favourites Print this Article

Knowledgebase

The whole server READ-ONLY

Our Services

Client Menu

Legal