ebuilds get cut before eof

Message

DaggyStyle · Post by **DaggyStyle** » Sun Aug 24, 2014 3:40 pm

greetings,

I have a very strange issue, I'm running 3.16.1 amd64 on a system that uses 4 1tb raid5.
my portage tree files are in a specific partition formatted in reiserfs.
in the last few days I've noticed that some ebuilds are failing SHA256 verification.
looking at the file shows that the files are partial.

I've tried both normal sync and webrsync, checked the partition for errors (none found) and even reformatted the partition but the issue still persists.
looking at basic stats of the raid shows it is ok, I have other partitions on the raid with reiserfs and the seems to be ok.

before I switch fs, any ideas what that can be?

DaggyStyle · Post by **DaggyStyle** » Sun Aug 24, 2014 3:45 pm

this is extremlly strange, as I run emerge -puD world, more and more files are get fucked up!
any ideas?

mv · Post by mv » Sun Aug 24, 2014 3:51 pm

Typical symptom of file system corruption. DId you already run fsck?

046 · Post by **046** » Sun Aug 24, 2014 7:11 pm

Same problem.

3.16.1 data corruption in reiserfs. Files looks like truncated.
3.16.0 affected too ((

http://forums.gentoo.org/viewtopic-t-998318.html

Rion · Post by **Rion** » Tue Aug 26, 2014 8:37 am

oh now I understand what's wrong with my portage partition...

DaggyStyle · Post by **DaggyStyle** » Tue Aug 26, 2014 8:51 am

mv wrote:Typical symptom of file system corruption. DId you already run fsck?

ofcourse

046 wrote:Same problem.

3.16.1 data corruption in reiserfs. Files looks like truncated.
3.16.0 affected too ((

http://forums.gentoo.org/viewtopic-t-998318.html

Rion wrote:oh now I understand what's wrong with my portage partition...

fuck!!!! my entire system is reiserfs based...

I guess that the only way to go is to reinstalled the os (root is reiserfs) unless I can be sure that it affects only the portage files, any ideas how to verify?

mv · Post by mv » Tue Aug 26, 2014 9:05 am

DaggyStyle wrote:I guess that the only way to go is to reinstalled the os (root is reiserfs) unless I can be sure that it affects only the portage files, any ideas how to verify?

app-portage/portage-utils:

Code: Select all

qcheck -a

Since reiser packs "short" files together it seems likely that the bug is in that part of the code. So maybe "most" of your main partition is still ok: The "short" files are mainly in /etc which you have likely changed anyway (i.e. the report of qcheck on these files is likely to be a false positive).

DaggyStyle · Post by **DaggyStyle** » Tue Aug 26, 2014 9:08 am

mv wrote:
DaggyStyle wrote:I guess that the only way to go is to reinstalled the os (root is reiserfs) unless I can be sure that it affects only the portage files, any ideas how to verify?
app-portage/portage-utils:
Code: Select all
qcheck -a
Since reiser packs "short" files together it seems likely that the bug is in that part of the code. So maybe "most" of your main partition is still ok: The "short" files are mainly in /etc which you have likely changed anyway (i.e. the report of qcheck on these files is likely to be a false positive).

will check, thanks.

046 · Post by **046** » Tue Aug 26, 2014 10:00 pm

data corrupted in batch created files in one directory.
/usr/share/mime - truncate some files in update, like portage files ))

Nice reiserfs cleanup

https://git.kernel.org/cgit/linux/kerne ... 49d9395050

miket · Post by **miket** » Wed Aug 27, 2014 3:22 am

I was on SuSE before I came to Gentoo in 2006. I was on Reiser back then, but when I ditched SuSE, I ditched Reiser too.

The big problem I had from that point was that ReiserFS suffered from, well, arrested development.

I wonder if booting with an older kernel might give you better success in recovering data from your partitions. I've got the sneak suspicion that the 3.16 kernel has a regression in the Reiser driver and that it has suffered from neglect.

Sorry to hear that you got caught in it on a production machine.

DaggyStyle · Post by **DaggyStyle** » Wed Aug 27, 2014 6:48 am

this is what I get after running qcheck -a http://pastebin.com/FZhTSybW

should I reemerge then after I boot my system with a safe kernel?

also this doesn't solves the issue of my private files.

I guess I need to fire up the backups if I encounter a bad file...

miket · Post by **miket** » Wed Aug 27, 2014 7:37 am

DaggyStyle wrote:should I reemerge then after I boot my system with a safe kernel?

also this doesn't solves the issue of my private files.

Ow! I'd be afraid of all kinds of trouble. I'd boot with a live CD, mount the partitions read-only, and try to get recent backups for as many of the files as I could. I would not be brimming over with confidence in the integrity of the file systems. If you want to try to repair in place, you could--but I'd want to get recent changed backed up first!

It might be that 3.16 kernel did not do enough damage to the Reiser filesystems to put them beyond repair, and I suppose that in later revisions in the 3.16 series they'll fix the Reiser-corruption problems (if indeed that's what did it). All the same, I'd plan to move away from Reiser. Too bad for you that btrfs is not really stable yet.

pappy_mcfae · Post by **pappy_mcfae** » Wed Aug 27, 2014 7:51 am

Portage was DOA on my main 64 bit system. I had to create binaries for python-2.7.8 and portage using my other 64 bit machine (which isn't running 3.16.x) so I could manually reinstall python and portage. I then had to pluck out individual corrupted .pyc files in various subdirectories until I got portage back to working. I no sooner get that working than something sneezes all over /etc. That results in a no boot.

Wow! It was intense! After some intense mental and keyboard gymnastics, copying a lot of files, and smoking copious, I've finally gotten things to calm a bit. I'm currently recompiling everything under 3.15.2. Once that's done, it's back up time. It's running under it's own steam, and seems to be doing just fine. Whew!

I am so glad I didn't play, "whoops, I dropped it," with the drive I thought was dead. That would have been a bad thing. I'll buy a housing for it and make it external. I was thinking about that, anyway.

Cheers,
Pappy

pappy_mcfae · Post by **pappy_mcfae** » Wed Aug 27, 2014 8:18 am

I made a bug report, for those interested.

Cheers,
Pappy

046 · Post by **046** » Wed Aug 27, 2014 8:25 am

pappy_mcfae wrote:I made a bug report

duplicate https://bugzilla.kernel.org/show_bug.cgi?id=83121

DaggyStyle · Post by **DaggyStyle** » Wed Aug 27, 2014 9:05 am

miket wrote:
DaggyStyle wrote:should I reemerge then after I boot my system with a safe kernel?

also this doesn't solves the issue of my private files.
Ow! I'd be afraid of all kinds of trouble. I'd boot with a live CD, mount the partitions read-only, and try to get recent backups for as many of the files as I could. I would not be brimming over with confidence in the integrity of the file systems. If you want to try to repair in place, you could--but I'd want to get recent changed backed up first!

It might be that 3.16 kernel did not do enough damage to the Reiser filesystems to put them beyond repair, and I suppose that in later revisions in the 3.16 series they'll fix the Reiser-corruption problems (if indeed that's what did it). All the same, I'd plan to move away from Reiser. Too bad for you that btrfs is not really stable yet.

this won't cause me to move away, stuff like this happens, I use ext4 that a few years ago made me lost data on my hd due to a bug. althought my main fs is reiserfs (root, home, gentoo trees and var) my other partitions are not (xfs, ext4 and ext2). imho it is wise to use various fs to prevent such issues.

actually I'm using linux for 13 years, 10 of them gentoo and reiserfs and that is the first time I had any bugs with it.

from what I can see (and as stated above) this affects only small (ebuid like in size) files, so most of the data should be intact.

DaggyStyle · Post by **DaggyStyle** » Wed Aug 27, 2014 9:09 am

pappy_mcfae wrote:Portage was DOA on my main 64 bit system. I had to create binaries for python-2.7.8 and portage using my other 64 bit machine (which isn't running 3.16.x) so I could manually reinstall python and portage. I then had to pluck out individual corrupted .pyc files in various subdirectories until I got portage back to working. I no sooner get that working than something sneezes all over /etc. That results in a no boot.

Wow! It was intense! After some intense mental and keyboard gymnastics, copying a lot of files, and smoking copious, I've finally gotten things to calm a bit. I'm currently recompiling everything under 3.15.2. Once that's done, it's back up time. It's running under it's own steam, and seems to be doing just fine. Whew!

I am so glad I didn't play, "whoops, I dropped it," with the drive I thought was dead. That would have been a bad thing. I'll buy a housing for it and make it external. I was thinking about that, anyway.

Cheers,
Pappy

I can confirm that 3.10 doesn't exhibit this issue (using latest sysrescuecd), can you confirm that 3.15.x is safe?

Anon-E-moose · Post by **Anon-E-moose** » Wed Aug 27, 2014 9:38 am

I run 3.15.9, reiser on several partitions (one is for portage), and haven't seen any problems.

Code: Select all

/dev/sdc2 on /usr/src type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdc5 on /usr/portage type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdc1 on /var type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdc3 on /x type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdb1 on /n type reiserfs (rw,noatime,nodiratime,notail)

DaggyStyle · Post by **DaggyStyle** » Wed Aug 27, 2014 10:10 am

Anon-E-moose wrote:I run 3.15.9, reiser on several partitions (one is for portage), and haven't seen any problems.

Code: Select all

/dev/sdc2 on /usr/src type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdc5 on /usr/portage type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdc1 on /var type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdc3 on /x type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdb1 on /n type reiserfs (rw,noatime,nodiratime,notail)

how does the following params noatime,nodiratime,notail affect the preformence?

046 · Post by **046** » Wed Aug 27, 2014 10:19 am

DaggyStyle wrote:can you confirm that 3.15.x is safe?

3.15.10 run without problems.

3.16-rc7 is ok too. bisect in progress ...

046 · Post by **046** » Wed Aug 27, 2014 10:24 am

DaggyStyle wrote:how does the following params noatime,nodiratime,notail affect the preformence?

negligible

Anon-E-moose · Post by **Anon-E-moose** » Wed Aug 27, 2014 12:25 pm

046 wrote:
DaggyStyle wrote:how does the following params noatime,nodiratime,notail affect the preformence?
negligible

Agreed,

noatime/nodiratime just keeps reiserfs from trying to update access times for files and dirs

notail is recommended when using reiserfs

From the man page

Code: Select all

By default, reiserfs stores small files and `file tails' directly into its  tree.  This  confuses  some
utilities such as LILO(8).  This option is used to disable packing of files into the tree.

046 · Post by **046** » Wed Aug 27, 2014 1:09 pm

Anon-E-moose wrote:notail is recommended when using reiserfs

From the man page
Code: Select all
By default, reiserfs stores small files and `file tails' directly into its  tree.  This  confuses  some
utilities such as LILO(8).  This option is used to disable packing of files into the tree.

notail should be used in boot partition and old kernel loader like LILO.

But notail cause internal fragmentation, data on disk use bigger space than user data length. With tail optimisation almost all partition size can be used by user data. Do not use reiserfs in boot partition, and save many space by tail optimisation.

Kaorukun · Post by **Kaorukun** » Wed Aug 27, 2014 7:13 pm

I have the same issue, it started happening only a week or so ago.
Assuming a possible filesystem corruption I reformatted the image where /usr/portage resides and filled it fresh with a portage snapshot - after the first emerge --sync I got corrupted files again.

I'm also using reiserfs for /usr/portage - it resides not on a partition but on a 1G image file.

I'm guessing it might have started after I updated the kernel from 3.15.7 to 3.16.1

Code: Select all

/dev/loop0 /usr/portage reiserfs rw,noatime 0 0

pappy_mcfae · Post by **pappy_mcfae** » Wed Aug 27, 2014 7:54 pm

I hadn't noticed the issue before 3.16.x. Until that time, I wasn't having anything blowing holes in not only /usr/portage, but /usr/lib64/python-2.7/*, and a few other places.

While this lappy is running funtoo, it was also effected, but not to the degree of the 64 bit machine. Under 3.16.0, I couldn't update portage without some ebuild or other file associated with portage being gone or corrupted beyond use. I rebooted with 3.15.2, deleted /usr/portage, and reinstalled it. Since then, this machine has not shown any issues with portage or associated files.

As far as verifying that the 3.15.x family seems unaffected, the 64 bit machine I used to make the fixes for the 64 bit machine with issues is running 3.15.2. It has no issues whatsoever with file corruption. It runs 24/7, and would surely show me if there were issues.

I would have posted this last night, but the Internet wasn't operating all that well. That's the fault of the cable modem, not a kernel issue.

Cheers,
Pappy