HOWTO fix&back "input/output error", on a mounted filesystem

[Elv13]

First, if it happen to you, your disk is dieing, change it. But if you need to fix the problem to do a backup, or simply want to fix the problem for now and tag the bad block so the disk is "safe" for a few more months, it worth trying.

I just had the problem, /usr/portage was broken and I could not use portage. It's quite bad, I will replace the disk soon, but for now, I just wanted it to work again, I needed portage. So, what could I do?

Code: Select all

┌─(root@lepagee-desktop)─────────────────────────────────────────────────────────────────────────────────(/home/lepagee)─┐ 
└─: rm /usr/portage/x11-themes/gtk-engines-nodoka -rf                                                      ──(Mon 17:49)─┘
rm: cannot remove `/usr/portage/x11-themes/gtk-engines-nodoka/Manifest': Input/output error
rm: cannot remove `/usr/portage/x11-themes/gtk-engines-nodoka/gtk-engines-nodoka-0.7.5.ebuild': Input/output error
rm: cannot remove `/usr/portage/x11-themes/gtk-engines-nodoka/ChangeLog': Input/output error

Nope, too easy

fsck? no, it never found the problem.

After a little googling, I heard about debugfs for some other unrelated purpose, but I tried it. Bingo! This tool come with Linux and offer a wide range of tools to fix file system. See it as a manual fsck.

Code: Select all

Available debugfs requests:

show_debugfs_params, params
                         Show debugfs parameters
open_filesys, open       Open a filesystem
close_filesys, close     Close the filesystem
feature, features        Set/print superblock features
dirty_filesys, dirty     Mark the filesystem as dirty
init_filesys             Initialize a filesystem (DESTROYS DATA)
show_super_stats, stats  Show superblock statistics
ncheck                   Do inode->name translation
icheck                   Do block->inode translation
change_root_directory, chroot
                         Change root directory
change_working_directory, cd
                         Change working directory
list_directory, ls       List directory
show_inode_info, stat    Show inode information 
dump_extents, extents, ex
                         Dump extents information 
link, ln                 Create directory link
unlink                   Delete a directory link
mkdir                    Create a directory
rmdir                    Remove a directory
rm                       Remove a file (unlink and kill_file, if appropriate)
kill_file                Deallocate an inode and its blocks
clri                     Clear an inode's contents
freei                    Clear an inode's in-use flag
seti                     Set an inode's in-use flag
testi                    Test an inode's in-use flag
freeb                    Clear a block's in-use flag
setb                     Set a block's in-use flag
testb                    Test a block's in-use flag
modify_inode, mi         Modify an inode by structure
find_free_block, ffb     Find free block(s)
find_free_inode, ffi     Find free inode(s)
print_working_directory, pwd
                         Print current working directory
expand_dir, expand       Expand directory
mknod                    Create a special file
list_deleted_inodes, lsdel
                         List deleted inodes
undelete, undel          Undelete file
write                    Copy a file from your native filesystem
dump_inode, dump         Dump an inode out to a file
cat                      Dump an inode out to stdout
lcd                      Change the current directory on your native filesystem
rdump                    Recursively dump a directory to the native filesystem
set_super_value, ssv     Set superblock value
set_inode_field, sif     Set inode field
set_block_group, set_bg  Set block group descriptor field
logdump                  Dump the contents of the journal
htree_dump, htree        Dump a hash-indexed directory
dx_hash, hash            Calculate the directory hash of a filename
dirsearch                Search a directory for a particular filename
bmap                     Calculate the logical->physical block mapping for an inode
imap                     Calculate the location of an inode
dump_unused              Dump unused blocks
set_current_time         Set current time to use when setting filesystme fields
supported_features       Print features supported by this version of e2fsprogs
help                     Display info on command or topic.
list_requests, lr, ?     List available commands.
quit, q                  Leave the subsystem.

So, now I had everything I needed not only to fix the file system, but the information about the physical sector to ban too (bmap).

Code: Select all

debugfs:  rm /usr/portage/x11-themes/gtk-engines-nodoka/ChangeLog 
debugfs:  rm /usr/portage/x11-themes/gtk-engines-nodoka/Manifest 
debugfs:  rm /usr/portage/x11-themes/gtk-engines-nodoka/gtk-engines-nodoka-0.7.5.ebuild

and

Code: Select all

rmdir /usr/portage/x11-themes/gtk-engines-nodoka/

and then

Code: Select all

rm -rf /usr/portage/x11-themes/gtk-engines-nodoka/ #In bash (not debugfs)

everything went back to a working state. A few S.M.A.R.T command later, the disk is now safe to use again. It's quite cool to know when you need to fix thing now. A nice thing to know for every plumber/sysadmin out there.

-THIS IS A TEMPORARY SOLUTION- but it still cool to have

I hope it can save your day or help you laugh at in the face of the Murphy law in a critical moment of your career


Update: As "planned" the disk became unusable a few days later. To backup your failed disk offline, use

Code: Select all

cp -a /mnt/old/* /mnt/new/

This will properly copy everything and keep the symlink. Dont use DD as you will probably produce a corrupted file system.

Or, if you really want no downtime, online

Code: Select all

ls / | grep -vE "(dev|proc|sys)" | xargs -i cp -a /{} /mnt/new/ ; mkdir /mnt/new/{dev,proc,sys}

To see what you lost, run this command:

Code: Select all

for FILE in `find /mnt/old/ | sed "s/\/mnt\/old//"`; do find /mnt/new/  | sed "s/\/mnt\/new//" | grep $FILE > /dev/null || echo $FILE | tee /mnt/new/lost_file_list.txt ;done

As root, it will take a while (hours)