On Friday 16 Jul 2010  this dedicated server from Godaddy shown some signs of file system errors, basically whenever I did a search through SSH command it kept saying that some paths could not be searched and it recommended to run a file system check.

I did some quick research and found that one of the commands to check file system is the FSCK command, so I typed it trying to scan from root directory (“/”).

Server then says that you should be carefull and you should not run FSCK command on a mounted file system, or something like that.

WARNING!!!  Running e2fsck on a mounted filesystem may cause SEVERE filesystem damage.
Do you really want to continue (y/n)? yes

Right after I pressed “y” the server started to behave abnormally, sites shown header session errors, files in site appeared unavailable, etc, then server went offline completely and it was not available by any means (plesk panel access, SSH, etc), server crashed. Obviously I did a mistake by running the command like that.

This server has 30 websites on it (including this blog), many databases, blogs, user uploads, important data gathered over around 3 years. I had no recent backups, this is because I have some more important websites and because I am trying to manage around 40 websites alone, regularly backing up the data can be a pain.

So I contacted the GoDaddy support team about the server crash, I must say I was not expecting much help because I knew I didn’t purchase the server with “assistance plan”, basically if you get the server without assistance option, then you are on your own and you are responsible for your own actions. They had a look, and they said it looks bad, the HDD data was unreadable from what I know. I had the option to let them run a FSCK file system check again but this would most probably not recover the data and it might make things worse, OR pay a fee and have the HDD drive in an USB enclosure next to the freshly installed server.

At this point I already got a new hosting plan from them and started to upload sites from my computer (mostly php scripts but with no content or database), I was 95% sure I lost all data from that dedicated server and all work for over 3 years.

While searching for Linux data recovery after FSCK, I found a data recovery company, I explained the problem to them, they were happy to help if I send them the actual hard drive and approximate cost would be $3000-$4000 where $1000 is paid up front, the rest depends on total GB, type of data, etc. So I decided to let the GoDaddy support team try a recovery by another FSCK command.

After a few hours I got the good news

Our data center technicians have run a file system check on your server’s hard drive and it appears to have resolved the issue. Your server is back online and is currently accessible via SSH and Plesk.
Please respond to this email if you have any further questions or issues.

All this stress took around 2 very loooong days while I thought I lost all my data, and you can lose all your data just by one command, so be very careful, it looks like running a file system check on Linux is not as easy as on windows

I think Godaddy did a good job even though it was not really their fault or their responsibility, basically they fixed mistakes that I did; even though I was on a dedicated server without assistance plan.

Tags: command, crash, file, fsck, godaddy, linux, server, ssh, system

This entry was posted on Monday, July 19th, 2010 at 1:15 am and is filed under Web Development. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.