Elv13
Guru
Joined: 13 Nov 2005
Posts: 388
Location: Socialist land of North America
|
 Posted: Mon Aug 08, 2011 10:31 pm Post subject: HOWTO fix&back "input/output error", on a moun |
 |
|
First, if it happen to you, your disk is dieing, change it. But if you need to fix the problem to do a backup, or simply want to fix the problem for now and tag the bad block so the disk is "safe" for a few more months, it worth trying.
I just had the problem, /usr/portage was broken and I could not use portage. It's quite bad, I will replace the disk soon, but for now, I just wanted it to work again, I needed portage. So, what could I do?
| Code: | ┌─(root@lepagee-desktop)─────────────────────────────────────────────────────────────────────────────────(/home/lepagee)─┐
└─: rm /usr/portage/x11-themes/gtk-engines-nodoka -rf ──(Mon 17:49)─┘
rm: cannot remove `/usr/portage/x11-themes/gtk-engines-nodoka/Manifest': Input/output error
rm: cannot remove `/usr/portage/x11-themes/gtk-engines-nodoka/gtk-engines-nodoka-0.7.5.ebuild': Input/output error
rm: cannot remove `/usr/portage/x11-themes/gtk-engines-nodoka/ChangeLog': Input/output error
|
Nope, too easy
fsck? no, it never found the problem.
After a little googling, I heard about debugfs for some other unrelated purpose, but I tried it. Bingo! This tool come with Linux and offer a wide range of tools to fix file system. See it as a manual fsck.
| Code: |
Available debugfs requests:
show_debugfs_params, params
Show debugfs parameters
open_filesys, open Open a filesystem
close_filesys, close Close the filesystem
feature, features Set/print superblock features
dirty_filesys, dirty Mark the filesystem as dirty
init_filesys Initialize a filesystem (DESTROYS DATA)
show_super_stats, stats Show superblock statistics
ncheck Do inode->name translation
icheck Do block->inode translation
change_root_directory, chroot
Change root directory
change_working_directory, cd
Change working directory
list_directory, ls List directory
show_inode_info, stat Show inode information
dump_extents, extents, ex
Dump extents information
link, ln Create directory link
unlink Delete a directory link
mkdir Create a directory
rmdir Remove a directory
rm Remove a file (unlink and kill_file, if appropriate)
kill_file Deallocate an inode and its blocks
clri Clear an inode's contents
freei Clear an inode's in-use flag
seti Set an inode's in-use flag
testi Test an inode's in-use flag
freeb Clear a block's in-use flag
setb Set a block's in-use flag
testb Test a block's in-use flag
modify_inode, mi Modify an inode by structure
find_free_block, ffb Find free block(s)
find_free_inode, ffi Find free inode(s)
print_working_directory, pwd
Print current working directory
expand_dir, expand Expand directory
mknod Create a special file
list_deleted_inodes, lsdel
List deleted inodes
undelete, undel Undelete file
write Copy a file from your native filesystem
dump_inode, dump Dump an inode out to a file
cat Dump an inode out to stdout
lcd Change the current directory on your native filesystem
rdump Recursively dump a directory to the native filesystem
set_super_value, ssv Set superblock value
set_inode_field, sif Set inode field
set_block_group, set_bg Set block group descriptor field
logdump Dump the contents of the journal
htree_dump, htree Dump a hash-indexed directory
dx_hash, hash Calculate the directory hash of a filename
dirsearch Search a directory for a particular filename
bmap Calculate the logical->physical block mapping for an inode
imap Calculate the location of an inode
dump_unused Dump unused blocks
set_current_time Set current time to use when setting filesystme fields
supported_features Print features supported by this version of e2fsprogs
help Display info on command or topic.
list_requests, lr, ? List available commands.
quit, q Leave the subsystem.
|
So, now I had everything I needed not only to fix the file system, but the information about the physical sector to ban too (bmap).
| Code: | debugfs: rm /usr/portage/x11-themes/gtk-engines-nodoka/ChangeLog
debugfs: rm /usr/portage/x11-themes/gtk-engines-nodoka/Manifest
debugfs: rm /usr/portage/x11-themes/gtk-engines-nodoka/gtk-engines-nodoka-0.7.5.ebuild |
and
| Code: | | rmdir /usr/portage/x11-themes/gtk-engines-nodoka/ |
and then
| Code: | | rm -rf /usr/portage/x11-themes/gtk-engines-nodoka/ #In bash (not debugfs) |
everything went back to a working state. A few S.M.A.R.T command later, the disk is now safe to use again. It's quite cool to know when you need to fix thing now. A nice thing to know for every plumber/sysadmin out there.
-THIS IS A TEMPORARY SOLUTION- but it still cool to have
I hope it can save your day or help you laugh at in the face of the Murphy law in a critical moment of your career 
Update: As "planned" the disk became unusable a few days later. To backup your failed disk offline, use
| Code: | | cp -a /mnt/old/* /mnt/new/ |
This will properly copy everything and keep the symlink. Dont use DD as you will probably produce a corrupted file system.
Or, if you really want no downtime, online
| Code: | | ls / | grep -vE "(dev|proc|sys)" | xargs -i cp -a /{} /mnt/new/ ; mkdir /mnt/new/{dev,proc,sys} |
To see what you lost, run this command:
| Code: | | for FILE in `find /mnt/old/ | sed "s/\/mnt\/old//"`; do find /mnt/new/ | sed "s/\/mnt\/new//" | grep $FILE > /dev/null || echo $FILE | tee /mnt/new/lost_file_list.txt ;done |
As root, it will take a while (hours) |
|
Documentation, Tips & Tricks
