View previous topic :: View next topic |
Author |
Message |
DaggyStyle Watchman
Joined: 22 Mar 2006 Posts: 5910
|
Posted: Sun Aug 24, 2014 3:40 pm Post subject: ebuilds get cut before eof |
|
|
greetings,
I have a very strange issue, I'm running 3.16.1 amd64 on a system that uses 4 1tb raid5.
my portage tree files are in a specific partition formatted in reiserfs.
in the last few days I've noticed that some ebuilds are failing SHA256 verification.
looking at the file shows that the files are partial.
I've tried both normal sync and webrsync, checked the partition for errors (none found) and even reformatted the partition but the issue still persists.
looking at basic stats of the raid shows it is ok, I have other partitions on the raid with reiserfs and the seems to be ok.
before I switch fs, any ideas what that can be? _________________ Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein |
|
Back to top |
|
|
DaggyStyle Watchman
Joined: 22 Mar 2006 Posts: 5910
|
Posted: Sun Aug 24, 2014 3:45 pm Post subject: |
|
|
this is extremlly strange, as I run emerge -puD world, more and more files are get fucked up!
any ideas? _________________ Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein |
|
Back to top |
|
|
mv Watchman
Joined: 20 Apr 2005 Posts: 6749
|
Posted: Sun Aug 24, 2014 3:51 pm Post subject: |
|
|
Typical symptom of file system corruption. DId you already run fsck? |
|
Back to top |
|
|
046 Apprentice
Joined: 21 Jul 2004 Posts: 231 Location: Yaroslavl, Russia
|
|
Back to top |
|
|
Rion Guru
Joined: 12 Oct 2006 Posts: 383 Location: Minsk, Belarus
|
Posted: Tue Aug 26, 2014 8:37 am Post subject: |
|
|
oh now I understand what's wrong with my portage partition... _________________ rion-overlay |
|
Back to top |
|
|
DaggyStyle Watchman
Joined: 22 Mar 2006 Posts: 5910
|
Posted: Tue Aug 26, 2014 8:51 am Post subject: |
|
|
mv wrote: | Typical symptom of file system corruption. DId you already run fsck? |
ofcourse
Rion wrote: | oh now I understand what's wrong with my portage partition... |
fuck!!!! my entire system is reiserfs based...
I guess that the only way to go is to reinstalled the os (root is reiserfs) unless I can be sure that it affects only the portage files, any ideas how to verify? _________________ Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein |
|
Back to top |
|
|
mv Watchman
Joined: 20 Apr 2005 Posts: 6749
|
Posted: Tue Aug 26, 2014 9:05 am Post subject: |
|
|
DaggyStyle wrote: | I guess that the only way to go is to reinstalled the os (root is reiserfs) unless I can be sure that it affects only the portage files, any ideas how to verify?
| app-portage/portage-utils:
Since reiser packs "short" files together it seems likely that the bug is in that part of the code. So maybe "most" of your main partition is still ok: The "short" files are mainly in /etc which you have likely changed anyway (i.e. the report of qcheck on these files is likely to be a false positive). |
|
Back to top |
|
|
DaggyStyle Watchman
Joined: 22 Mar 2006 Posts: 5910
|
Posted: Tue Aug 26, 2014 9:08 am Post subject: |
|
|
mv wrote: | DaggyStyle wrote: | I guess that the only way to go is to reinstalled the os (root is reiserfs) unless I can be sure that it affects only the portage files, any ideas how to verify?
| app-portage/portage-utils:
Since reiser packs "short" files together it seems likely that the bug is in that part of the code. So maybe "most" of your main partition is still ok: The "short" files are mainly in /etc which you have likely changed anyway (i.e. the report of qcheck on these files is likely to be a false positive). |
will check, thanks. _________________ Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein |
|
Back to top |
|
|
046 Apprentice
Joined: 21 Jul 2004 Posts: 231 Location: Yaroslavl, Russia
|
|
Back to top |
|
|
miket Guru
Joined: 28 Apr 2007 Posts: 488 Location: Gainesville, FL, USA
|
Posted: Wed Aug 27, 2014 3:22 am Post subject: |
|
|
I was on SuSE before I came to Gentoo in 2006. I was on Reiser back then, but when I ditched SuSE, I ditched Reiser too.
The big problem I had from that point was that ReiserFS suffered from, well, arrested development.
I wonder if booting with an older kernel might give you better success in recovering data from your partitions. I've got the sneak suspicion that the 3.16 kernel has a regression in the Reiser driver and that it has suffered from neglect.
Sorry to hear that you got caught in it on a production machine. |
|
Back to top |
|
|
DaggyStyle Watchman
Joined: 22 Mar 2006 Posts: 5910
|
Posted: Wed Aug 27, 2014 6:48 am Post subject: |
|
|
this is what I get after running qcheck -a http://pastebin.com/FZhTSybW
should I reemerge then after I boot my system with a safe kernel?
also this doesn't solves the issue of my private files.
I guess I need to fire up the backups if I encounter a bad file... _________________ Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein |
|
Back to top |
|
|
miket Guru
Joined: 28 Apr 2007 Posts: 488 Location: Gainesville, FL, USA
|
Posted: Wed Aug 27, 2014 7:37 am Post subject: |
|
|
DaggyStyle wrote: | should I reemerge then after I boot my system with a safe kernel?
also this doesn't solves the issue of my private files. |
Ow! I'd be afraid of all kinds of trouble. I'd boot with a live CD, mount the partitions read-only, and try to get recent backups for as many of the files as I could. I would not be brimming over with confidence in the integrity of the file systems. If you want to try to repair in place, you could--but I'd want to get recent changed backed up first!
It might be that 3.16 kernel did not do enough damage to the Reiser filesystems to put them beyond repair, and I suppose that in later revisions in the 3.16 series they'll fix the Reiser-corruption problems (if indeed that's what did it). All the same, I'd plan to move away from Reiser. Too bad for you that btrfs is not really stable yet. |
|
Back to top |
|
|
pappy_mcfae Watchman
Joined: 27 Dec 2007 Posts: 5999 Location: Pomona, California.
|
Posted: Wed Aug 27, 2014 7:51 am Post subject: |
|
|
Portage was DOA on my main 64 bit system. I had to create binaries for python-2.7.8 and portage using my other 64 bit machine (which isn't running 3.16.x) so I could manually reinstall python and portage. I then had to pluck out individual corrupted .pyc files in various subdirectories until I got portage back to working. I no sooner get that working than something sneezes all over /etc. That results in a no boot.
Wow! It was intense! After some intense mental and keyboard gymnastics, copying a lot of files, and smoking copious, I've finally gotten things to calm a bit. I'm currently recompiling everything under 3.15.2. Once that's done, it's back up time. It's running under it's own steam, and seems to be doing just fine. Whew!
I am so glad I didn't play, "whoops, I dropped it," with the drive I thought was dead. That would have been a bad thing. I'll buy a housing for it and make it external. I was thinking about that, anyway.
Cheers,
Pappy _________________ This space left intentionally blank, except for these ASCII symbols. |
|
Back to top |
|
|
pappy_mcfae Watchman
Joined: 27 Dec 2007 Posts: 5999 Location: Pomona, California.
|
Posted: Wed Aug 27, 2014 8:18 am Post subject: |
|
|
I made a bug report, for those interested.
Cheers,
Pappy _________________ This space left intentionally blank, except for these ASCII symbols. |
|
Back to top |
|
|
046 Apprentice
Joined: 21 Jul 2004 Posts: 231 Location: Yaroslavl, Russia
|
|
Back to top |
|
|
DaggyStyle Watchman
Joined: 22 Mar 2006 Posts: 5910
|
Posted: Wed Aug 27, 2014 9:05 am Post subject: |
|
|
miket wrote: | DaggyStyle wrote: | should I reemerge then after I boot my system with a safe kernel?
also this doesn't solves the issue of my private files. |
Ow! I'd be afraid of all kinds of trouble. I'd boot with a live CD, mount the partitions read-only, and try to get recent backups for as many of the files as I could. I would not be brimming over with confidence in the integrity of the file systems. If you want to try to repair in place, you could--but I'd want to get recent changed backed up first!
It might be that 3.16 kernel did not do enough damage to the Reiser filesystems to put them beyond repair, and I suppose that in later revisions in the 3.16 series they'll fix the Reiser-corruption problems (if indeed that's what did it). All the same, I'd plan to move away from Reiser. Too bad for you that btrfs is not really stable yet. |
this won't cause me to move away, stuff like this happens, I use ext4 that a few years ago made me lost data on my hd due to a bug. althought my main fs is reiserfs (root, home, gentoo trees and var) my other partitions are not (xfs, ext4 and ext2). imho it is wise to use various fs to prevent such issues.
actually I'm using linux for 13 years, 10 of them gentoo and reiserfs and that is the first time I had any bugs with it.
from what I can see (and as stated above) this affects only small (ebuid like in size) files, so most of the data should be intact. _________________ Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein |
|
Back to top |
|
|
DaggyStyle Watchman
Joined: 22 Mar 2006 Posts: 5910
|
Posted: Wed Aug 27, 2014 9:09 am Post subject: |
|
|
pappy_mcfae wrote: | Portage was DOA on my main 64 bit system. I had to create binaries for python-2.7.8 and portage using my other 64 bit machine (which isn't running 3.16.x) so I could manually reinstall python and portage. I then had to pluck out individual corrupted .pyc files in various subdirectories until I got portage back to working. I no sooner get that working than something sneezes all over /etc. That results in a no boot.
Wow! It was intense! After some intense mental and keyboard gymnastics, copying a lot of files, and smoking copious, I've finally gotten things to calm a bit. I'm currently recompiling everything under 3.15.2. Once that's done, it's back up time. It's running under it's own steam, and seems to be doing just fine. Whew!
I am so glad I didn't play, "whoops, I dropped it," with the drive I thought was dead. That would have been a bad thing. I'll buy a housing for it and make it external. I was thinking about that, anyway.
Cheers,
Pappy |
I can confirm that 3.10 doesn't exhibit this issue (using latest sysrescuecd), can you confirm that 3.15.x is safe? _________________ Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein |
|
Back to top |
|
|
Anon-E-moose Watchman
Joined: 23 May 2008 Posts: 6103 Location: Dallas area
|
Posted: Wed Aug 27, 2014 9:38 am Post subject: |
|
|
I run 3.15.9, reiser on several partitions (one is for portage), and haven't seen any problems.
Code: | /dev/sdc2 on /usr/src type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdc5 on /usr/portage type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdc1 on /var type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdc3 on /x type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdb1 on /n type reiserfs (rw,noatime,nodiratime,notail) |
_________________ PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland |
|
Back to top |
|
|
DaggyStyle Watchman
Joined: 22 Mar 2006 Posts: 5910
|
Posted: Wed Aug 27, 2014 10:10 am Post subject: |
|
|
Anon-E-moose wrote: | I run 3.15.9, reiser on several partitions (one is for portage), and haven't seen any problems.
Code: | /dev/sdc2 on /usr/src type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdc5 on /usr/portage type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdc1 on /var type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdc3 on /x type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdb1 on /n type reiserfs (rw,noatime,nodiratime,notail) |
|
how does the following params noatime,nodiratime,notail affect the preformence? _________________ Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein |
|
Back to top |
|
|
046 Apprentice
Joined: 21 Jul 2004 Posts: 231 Location: Yaroslavl, Russia
|
Posted: Wed Aug 27, 2014 10:19 am Post subject: |
|
|
DaggyStyle wrote: | can you confirm that 3.15.x is safe? |
3.15.10 run without problems.
3.16-rc7 is ok too. bisect in progress ... |
|
Back to top |
|
|
046 Apprentice
Joined: 21 Jul 2004 Posts: 231 Location: Yaroslavl, Russia
|
Posted: Wed Aug 27, 2014 10:24 am Post subject: |
|
|
DaggyStyle wrote: | how does the following params noatime,nodiratime,notail affect the preformence? | negligible |
|
Back to top |
|
|
Anon-E-moose Watchman
Joined: 23 May 2008 Posts: 6103 Location: Dallas area
|
Posted: Wed Aug 27, 2014 12:25 pm Post subject: |
|
|
046 wrote: | DaggyStyle wrote: | how does the following params noatime,nodiratime,notail affect the preformence? | negligible |
Agreed,
noatime/nodiratime just keeps reiserfs from trying to update access times for files and dirs
notail is recommended when using reiserfs
From the man page
Code: | By default, reiserfs stores small files and `file tails' directly into its tree. This confuses some
utilities such as LILO(8). This option is used to disable packing of files into the tree. |
_________________ PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland |
|
Back to top |
|
|
046 Apprentice
Joined: 21 Jul 2004 Posts: 231 Location: Yaroslavl, Russia
|
Posted: Wed Aug 27, 2014 1:09 pm Post subject: |
|
|
Anon-E-moose wrote: | notail is recommended when using reiserfs
From the man page
Code: | By default, reiserfs stores small files and `file tails' directly into its tree. This confuses some
utilities such as LILO(8). This option is used to disable packing of files into the tree. |
|
notail should be used in boot partition and old kernel loader like LILO.
But notail cause internal fragmentation, data on disk use bigger space than user data length. With tail optimisation almost all partition size can be used by user data. Do not use reiserfs in boot partition, and save many space by tail optimisation. |
|
Back to top |
|
|
Kaorukun n00b
Joined: 27 Aug 2014 Posts: 5
|
Posted: Wed Aug 27, 2014 7:13 pm Post subject: |
|
|
I have the same issue, it started happening only a week or so ago.
Assuming a possible filesystem corruption I reformatted the image where /usr/portage resides and filled it fresh with a portage snapshot - after the first emerge --sync I got corrupted files again.
I'm also using reiserfs for /usr/portage - it resides not on a partition but on a 1G image file.
I'm guessing it might have started after I updated the kernel from 3.15.7 to 3.16.1
Code: |
/dev/loop0 /usr/portage reiserfs rw,noatime 0 0
|
|
|
Back to top |
|
|
pappy_mcfae Watchman
Joined: 27 Dec 2007 Posts: 5999 Location: Pomona, California.
|
Posted: Wed Aug 27, 2014 7:54 pm Post subject: |
|
|
I hadn't noticed the issue before 3.16.x. Until that time, I wasn't having anything blowing holes in not only /usr/portage, but /usr/lib64/python-2.7/*, and a few other places.
While this lappy is running funtoo, it was also effected, but not to the degree of the 64 bit machine. Under 3.16.0, I couldn't update portage without some ebuild or other file associated with portage being gone or corrupted beyond use. I rebooted with 3.15.2, deleted /usr/portage, and reinstalled it. Since then, this machine has not shown any issues with portage or associated files.
As far as verifying that the 3.15.x family seems unaffected, the 64 bit machine I used to make the fixes for the 64 bit machine with issues is running 3.15.2. It has no issues whatsoever with file corruption. It runs 24/7, and would surely show me if there were issues.
I would have posted this last night, but the Internet wasn't operating all that well. That's the fault of the cable modem, not a kernel issue.
Cheers,
Pappy _________________ This space left intentionally blank, except for these ASCII symbols. |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|