Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
ebuilds get cut before eof
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5909

PostPosted: Sun Aug 24, 2014 3:40 pm    Post subject: ebuilds get cut before eof Reply with quote

greetings,

I have a very strange issue, I'm running 3.16.1 amd64 on a system that uses 4 1tb raid5.
my portage tree files are in a specific partition formatted in reiserfs.
in the last few days I've noticed that some ebuilds are failing SHA256 verification.
looking at the file shows that the files are partial.

I've tried both normal sync and webrsync, checked the partition for errors (none found) and even reformatted the partition but the issue still persists.
looking at basic stats of the raid shows it is ok, I have other partitions on the raid with reiserfs and the seems to be ok.

before I switch fs, any ideas what that can be?
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
Back to top
View user's profile Send private message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5909

PostPosted: Sun Aug 24, 2014 3:45 pm    Post subject: Reply with quote

this is extremlly strange, as I run emerge -puD world, more and more files are get fucked up!
any ideas?
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sun Aug 24, 2014 3:51 pm    Post subject: Reply with quote

Typical symptom of file system corruption. DId you already run fsck?
Back to top
View user's profile Send private message
046
Apprentice
Apprentice


Joined: 21 Jul 2004
Posts: 231
Location: Yaroslavl, Russia

PostPosted: Sun Aug 24, 2014 7:11 pm    Post subject: Reply with quote

Same problem.

3.16.1 data corruption in reiserfs. Files looks like truncated.
3.16.0 affected too ((

https://forums.gentoo.org/viewtopic-t-998318.html
Back to top
View user's profile Send private message
Rion
Guru
Guru


Joined: 12 Oct 2006
Posts: 382
Location: Minsk, Belarus

PostPosted: Tue Aug 26, 2014 8:37 am    Post subject: Reply with quote

oh now I understand what's wrong with my portage partition...
_________________
rion-overlay
Back to top
View user's profile Send private message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5909

PostPosted: Tue Aug 26, 2014 8:51 am    Post subject: Reply with quote

mv wrote:
Typical symptom of file system corruption. DId you already run fsck?


ofcourse

046 wrote:
Same problem.

3.16.1 data corruption in reiserfs. Files looks like truncated.
3.16.0 affected too ((

https://forums.gentoo.org/viewtopic-t-998318.html


Rion wrote:
oh now I understand what's wrong with my portage partition...


fuck!!!! my entire system is reiserfs based...

I guess that the only way to go is to reinstalled the os (root is reiserfs) unless I can be sure that it affects only the portage files, any ideas how to verify?
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Tue Aug 26, 2014 9:05 am    Post subject: Reply with quote

DaggyStyle wrote:
I guess that the only way to go is to reinstalled the os (root is reiserfs) unless I can be sure that it affects only the portage files, any ideas how to verify?
app-portage/portage-utils:
Code:
qcheck -a

Since reiser packs "short" files together it seems likely that the bug is in that part of the code. So maybe "most" of your main partition is still ok: The "short" files are mainly in /etc which you have likely changed anyway (i.e. the report of qcheck on these files is likely to be a false positive).
Back to top
View user's profile Send private message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5909

PostPosted: Tue Aug 26, 2014 9:08 am    Post subject: Reply with quote

mv wrote:
DaggyStyle wrote:
I guess that the only way to go is to reinstalled the os (root is reiserfs) unless I can be sure that it affects only the portage files, any ideas how to verify?
app-portage/portage-utils:
Code:
qcheck -a

Since reiser packs "short" files together it seems likely that the bug is in that part of the code. So maybe "most" of your main partition is still ok: The "short" files are mainly in /etc which you have likely changed anyway (i.e. the report of qcheck on these files is likely to be a false positive).

will check, thanks.
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
Back to top
View user's profile Send private message
046
Apprentice
Apprentice


Joined: 21 Jul 2004
Posts: 231
Location: Yaroslavl, Russia

PostPosted: Tue Aug 26, 2014 10:00 pm    Post subject: Reply with quote

data corrupted in batch created files in one directory.
/usr/share/mime - truncate some files in update, like portage files ))

Nice reiserfs cleanup :D
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?h=linux-3.16.y&id=2840c566e95599cd60c7143762ca8b49d9395050
Back to top
View user's profile Send private message
miket
Guru
Guru


Joined: 28 Apr 2007
Posts: 483
Location: Gainesville, FL, USA

PostPosted: Wed Aug 27, 2014 3:22 am    Post subject: Reply with quote

I was on SuSE before I came to Gentoo in 2006. I was on Reiser back then, but when I ditched SuSE, I ditched Reiser too.

The big problem I had from that point was that ReiserFS suffered from, well, arrested development.

I wonder if booting with an older kernel might give you better success in recovering data from your partitions. I've got the sneak suspicion that the 3.16 kernel has a regression in the Reiser driver and that it has suffered from neglect.

Sorry to hear that you got caught in it on a production machine.
Back to top
View user's profile Send private message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5909

PostPosted: Wed Aug 27, 2014 6:48 am    Post subject: Reply with quote

this is what I get after running qcheck -a http://pastebin.com/FZhTSybW

should I reemerge then after I boot my system with a safe kernel?

also this doesn't solves the issue of my private files.

I guess I need to fire up the backups if I encounter a bad file...
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
Back to top
View user's profile Send private message
miket
Guru
Guru


Joined: 28 Apr 2007
Posts: 483
Location: Gainesville, FL, USA

PostPosted: Wed Aug 27, 2014 7:37 am    Post subject: Reply with quote

DaggyStyle wrote:
should I reemerge then after I boot my system with a safe kernel?

also this doesn't solves the issue of my private files.

Ow! I'd be afraid of all kinds of trouble. I'd boot with a live CD, mount the partitions read-only, and try to get recent backups for as many of the files as I could. I would not be brimming over with confidence in the integrity of the file systems. If you want to try to repair in place, you could--but I'd want to get recent changed backed up first!

It might be that 3.16 kernel did not do enough damage to the Reiser filesystems to put them beyond repair, and I suppose that in later revisions in the 3.16 series they'll fix the Reiser-corruption problems (if indeed that's what did it). All the same, I'd plan to move away from Reiser. Too bad for you that btrfs is not really stable yet.
Back to top
View user's profile Send private message
pappy_mcfae
Watchman
Watchman


Joined: 27 Dec 2007
Posts: 5999
Location: Pomona, California.

PostPosted: Wed Aug 27, 2014 7:51 am    Post subject: Reply with quote

Portage was DOA on my main 64 bit system. I had to create binaries for python-2.7.8 and portage using my other 64 bit machine (which isn't running 3.16.x) so I could manually reinstall python and portage. I then had to pluck out individual corrupted .pyc files in various subdirectories until I got portage back to working. I no sooner get that working than something sneezes all over /etc. That results in a no boot.

Wow! It was intense! After some intense mental and keyboard gymnastics, copying a lot of files, and smoking copious, I've finally gotten things to calm a bit. I'm currently recompiling everything under 3.15.2. Once that's done, it's back up time. It's running under it's own steam, and seems to be doing just fine. Whew!

I am so glad I didn't play, "whoops, I dropped it," with the drive I thought was dead. That would have been a bad thing. I'll buy a housing for it and make it external. I was thinking about that, anyway.

Cheers,
Pappy
_________________
This space left intentionally blank, except for these ASCII symbols.
Back to top
View user's profile Send private message
pappy_mcfae
Watchman
Watchman


Joined: 27 Dec 2007
Posts: 5999
Location: Pomona, California.

PostPosted: Wed Aug 27, 2014 8:18 am    Post subject: Reply with quote

I made a bug report, for those interested.

Cheers,
Pappy
_________________
This space left intentionally blank, except for these ASCII symbols.
Back to top
View user's profile Send private message
046
Apprentice
Apprentice


Joined: 21 Jul 2004
Posts: 231
Location: Yaroslavl, Russia

PostPosted: Wed Aug 27, 2014 8:25 am    Post subject: Reply with quote

pappy_mcfae wrote:
I made a bug report

duplicate https://bugzilla.kernel.org/show_bug.cgi?id=83121
Back to top
View user's profile Send private message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5909

PostPosted: Wed Aug 27, 2014 9:05 am    Post subject: Reply with quote

miket wrote:
DaggyStyle wrote:
should I reemerge then after I boot my system with a safe kernel?

also this doesn't solves the issue of my private files.

Ow! I'd be afraid of all kinds of trouble. I'd boot with a live CD, mount the partitions read-only, and try to get recent backups for as many of the files as I could. I would not be brimming over with confidence in the integrity of the file systems. If you want to try to repair in place, you could--but I'd want to get recent changed backed up first!

It might be that 3.16 kernel did not do enough damage to the Reiser filesystems to put them beyond repair, and I suppose that in later revisions in the 3.16 series they'll fix the Reiser-corruption problems (if indeed that's what did it). All the same, I'd plan to move away from Reiser. Too bad for you that btrfs is not really stable yet.


this won't cause me to move away, stuff like this happens, I use ext4 that a few years ago made me lost data on my hd due to a bug. althought my main fs is reiserfs (root, home, gentoo trees and var) my other partitions are not (xfs, ext4 and ext2). imho it is wise to use various fs to prevent such issues.

actually I'm using linux for 13 years, 10 of them gentoo and reiserfs and that is the first time I had any bugs with it.

from what I can see (and as stated above) this affects only small (ebuid like in size) files, so most of the data should be intact.
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
Back to top
View user's profile Send private message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5909

PostPosted: Wed Aug 27, 2014 9:09 am    Post subject: Reply with quote

pappy_mcfae wrote:
Portage was DOA on my main 64 bit system. I had to create binaries for python-2.7.8 and portage using my other 64 bit machine (which isn't running 3.16.x) so I could manually reinstall python and portage. I then had to pluck out individual corrupted .pyc files in various subdirectories until I got portage back to working. I no sooner get that working than something sneezes all over /etc. That results in a no boot.

Wow! It was intense! After some intense mental and keyboard gymnastics, copying a lot of files, and smoking copious, I've finally gotten things to calm a bit. I'm currently recompiling everything under 3.15.2. Once that's done, it's back up time. It's running under it's own steam, and seems to be doing just fine. Whew!

I am so glad I didn't play, "whoops, I dropped it," with the drive I thought was dead. That would have been a bad thing. I'll buy a housing for it and make it external. I was thinking about that, anyway.

Cheers,
Pappy


I can confirm that 3.10 doesn't exhibit this issue (using latest sysrescuecd), can you confirm that 3.15.x is safe?
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 6095
Location: Dallas area

PostPosted: Wed Aug 27, 2014 9:38 am    Post subject: Reply with quote

I run 3.15.9, reiser on several partitions (one is for portage), and haven't seen any problems.

Code:
/dev/sdc2 on /usr/src type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdc5 on /usr/portage type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdc1 on /var type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdc3 on /x type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdb1 on /n type reiserfs (rw,noatime,nodiratime,notail)

_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland
Back to top
View user's profile Send private message
DaggyStyle
Watchman
Watchman


Joined: 22 Mar 2006
Posts: 5909

PostPosted: Wed Aug 27, 2014 10:10 am    Post subject: Reply with quote

Anon-E-moose wrote:
I run 3.15.9, reiser on several partitions (one is for portage), and haven't seen any problems.

Code:
/dev/sdc2 on /usr/src type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdc5 on /usr/portage type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdc1 on /var type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdc3 on /x type reiserfs (rw,noatime,nodiratime,notail)
/dev/sdb1 on /n type reiserfs (rw,noatime,nodiratime,notail)


how does the following params noatime,nodiratime,notail affect the preformence?
_________________
Only two things are infinite, the universe and human stupidity and I'm not sure about the former - Albert Einstein
Back to top
View user's profile Send private message
046
Apprentice
Apprentice


Joined: 21 Jul 2004
Posts: 231
Location: Yaroslavl, Russia

PostPosted: Wed Aug 27, 2014 10:19 am    Post subject: Reply with quote

DaggyStyle wrote:
can you confirm that 3.15.x is safe?

3.15.10 run without problems.

3.16-rc7 is ok too. bisect in progress ...
Back to top
View user's profile Send private message
046
Apprentice
Apprentice


Joined: 21 Jul 2004
Posts: 231
Location: Yaroslavl, Russia

PostPosted: Wed Aug 27, 2014 10:24 am    Post subject: Reply with quote

DaggyStyle wrote:
how does the following params noatime,nodiratime,notail affect the preformence?
negligible
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 6095
Location: Dallas area

PostPosted: Wed Aug 27, 2014 12:25 pm    Post subject: Reply with quote

046 wrote:
DaggyStyle wrote:
how does the following params noatime,nodiratime,notail affect the preformence?
negligible


Agreed,

noatime/nodiratime just keeps reiserfs from trying to update access times for files and dirs


notail is recommended when using reiserfs

From the man page
Code:
By default, reiserfs stores small files and `file tails' directly into its  tree.  This  confuses  some
utilities such as LILO(8).  This option is used to disable packing of files into the tree.

_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland
Back to top
View user's profile Send private message
046
Apprentice
Apprentice


Joined: 21 Jul 2004
Posts: 231
Location: Yaroslavl, Russia

PostPosted: Wed Aug 27, 2014 1:09 pm    Post subject: Reply with quote

Anon-E-moose wrote:
notail is recommended when using reiserfs

From the man page
Code:
By default, reiserfs stores small files and `file tails' directly into its  tree.  This  confuses  some
utilities such as LILO(8).  This option is used to disable packing of files into the tree.


notail should be used in boot partition and old kernel loader like LILO.

But notail cause internal fragmentation, data on disk use bigger space than user data length. With tail optimisation almost all partition size can be used by user data. Do not use reiserfs in boot partition, and save many space by tail optimisation.
Back to top
View user's profile Send private message
Kaorukun
n00b
n00b


Joined: 27 Aug 2014
Posts: 5

PostPosted: Wed Aug 27, 2014 7:13 pm    Post subject: Reply with quote

I have the same issue, it started happening only a week or so ago.
Assuming a possible filesystem corruption I reformatted the image where /usr/portage resides and filled it fresh with a portage snapshot - after the first emerge --sync I got corrupted files again.

I'm also using reiserfs for /usr/portage - it resides not on a partition but on a 1G image file.

I'm guessing it might have started after I updated the kernel from 3.15.7 to 3.16.1

Code:

/dev/loop0 /usr/portage reiserfs rw,noatime 0 0
Back to top
View user's profile Send private message
pappy_mcfae
Watchman
Watchman


Joined: 27 Dec 2007
Posts: 5999
Location: Pomona, California.

PostPosted: Wed Aug 27, 2014 7:54 pm    Post subject: Reply with quote

I hadn't noticed the issue before 3.16.x. Until that time, I wasn't having anything blowing holes in not only /usr/portage, but /usr/lib64/python-2.7/*, and a few other places.

While this lappy is running funtoo, it was also effected, but not to the degree of the 64 bit machine. Under 3.16.0, I couldn't update portage without some ebuild or other file associated with portage being gone or corrupted beyond use. I rebooted with 3.15.2, deleted /usr/portage, and reinstalled it. Since then, this machine has not shown any issues with portage or associated files.

As far as verifying that the 3.15.x family seems unaffected, the 64 bit machine I used to make the fixes for the 64 bit machine with issues is running 3.15.2. It has no issues whatsoever with file corruption. It runs 24/7, and would surely show me if there were issues.

I would have posted this last night, but the Internet wasn't operating all that well. That's the fault of the cable modem, not a kernel issue.

Cheers,
Pappy
_________________
This space left intentionally blank, except for these ASCII symbols.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum