View previous topic :: View next topic |
Author |
Message |
iTux Guru
Joined: 07 Sep 2004 Posts: 586 Location: Toronto
|
Posted: Sat Apr 16, 2005 8:41 pm Post subject: Portage does more security checks? Digest verification fails |
|
|
I did an emerge sync. I did not upgrade portage or other system utilities and I did not modified my config.
I noticed that portage does more checking that it used before:
Code: |
ibou sablevm-svn # USE="sablejit -gtk" nice ebuild sablevm-svn-20050414.ebuild compile
!!! Security Violation: A file exists that is not in the manifest.
!!! File: files/makefile-common.patch
|
Never got that before and I often forget to run ebuild package digest...
Code: |
ibou sablevm-svn # USE="sablejit -gtk" nice ebuild sablevm-svn-20050414.ebuild compile
>>> md5 files ;-) sablevm-svn.tar.gz
>>> md5 files ;-) README
>>> md5 files ;-) sablevm-svn-20050414.ebuild
>>> md5 files ;-) uploadfiles
>>> md5 files ;-) sablevm-svn-20050205-r5.ebuild
>>> md5 files ;-) files/digest-sablevm-svn-20050205-r5
>>> md5 files ;-) files/makefile-sablejit.patch
>>> md5 files ;-) files/autogen.patch
>>> md5 files ;-) files/ltmain.patch
>>> md5 files ;-) files/version.patch
>>> md5 files ;-) files/makefile-config.patch
>>> md5 files ;-) files/sablevm-svn-20050205
>>> md5 files ;-) files/digest-sablevm-svn-20050414
>>> md5 files ;-) files/sablevm-svn-20050414
>>> md5 files ;-) files/makefile-common.patch
|
Portage used to md5check only files externally downlowded i.e. in distfiles...
Why do I get these extra integrity checks today? Remember that I did not upgrade portage or change config. I am puzzled. Or some default setting changed? If yes, which one?
iTux
mod edit: sticky and expanded subject to address the "digest verification failure" posts that will come up in the near future. --Earthwings |
|
Back to top |
|
|
iTux Guru
Joined: 07 Sep 2004 Posts: 586 Location: Toronto
|
Posted: Sat Apr 16, 2005 8:49 pm Post subject: |
|
|
Seems to be due to FEATURES=strict that is enabled by default. I guess it was disabled by default before?
iTux |
|
Back to top |
|
|
Earthwings Bodhisattva
Joined: 14 Apr 2003 Posts: 7753 Location: Germany
|
Posted: Sun Apr 24, 2005 9:59 am Post subject: |
|
|
Yes, it was changed recently. http://www.gentoo.org/cgi-bin/viewcvs.cgi/profiles/base/make.defaults?r1=1.4&r2=1.5
Here's a short summary what changed and what can be done against "Digest verification failure".
FEATURES=strict (which is now enabled on your system if you ran emerge --sync in the last days and don't have FEATURES=-strict in make.conf) tells emerge to check the digest (md5 sum) not only for the files in SRC_URI, but also for ebuilds. This is more secure and therefore a good thing, but you may experience "digest verification failure" more often. Here is how to handle this situation.
First, determine whether the digest failed for a file from SRC_URI (most often a .gz or .bz2 or .rpm). If this is the case try the following:
- delete the file from /usr/portage/distfiles and restart emerge. It will be downloaded again from the mirror.
- delete the file from /usr/portage/distfiles and download in manually from another mirror, cause the mirror may contain a broken file.
- run emerge --sync and try again. The problem might be already fixed
- search forums.gentoo.org and bugs.gentoo.org to see if other people experience the same problem and the recorded md5 sum is wrong
- write a bug report if none exists or contact a developer in irc at irc.freenode.net
- If you are absolutely sure that the file you downloaded is ok, you can generate a new digest with ebuild /path/to/ebuild digest.
If you get a digest failure for a file from the Manifest (typically a .ebuild), do the following:
- run emerge --sync to see if the problem is already fixed.
- search forums.gentoo.org and bugs.gentoo.org to see if other people experience the same problem and the recorded md5 sum is wrong
- write a bug report if none exists or contact a developer in irc at irc.freenode.net
- if you verified the file is ok (have a look at it), you can generate a new digest with ebuild /path/to/ebuild digest. _________________ KDE |
|
Back to top |
|
|
Bob P Advocate
Joined: 20 Oct 2004 Posts: 3355 Location: Jackass! Development Labs
|
Posted: Sun Apr 24, 2005 12:05 pm Post subject: |
|
|
ARGH!
i'm performing an emerge -e system to rebuild the toolkit on a fleet of systems (every possible x86 architecture) after downloading the most recent portage snapshot. would anyone care to guess how often this error pops up:
Code: | !!! Security Violation: A file exists that is not in the manifest. |
its hit me on two out of three ebuilds so far, and i've got 91 of them to deal with in the emerge. this has to be some sort of sick belated April Fool's joke. _________________ .
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks |
|
Back to top |
|
|
Earthwings Bodhisattva
Joined: 14 Apr 2003 Posts: 7753 Location: Germany
|
Posted: Sun Apr 24, 2005 12:56 pm Post subject: |
|
|
I think I would disable strict in emerge -e until everyone uses repoman for checking things in. FEATURES=-strict emerge -e system _________________ KDE |
|
Back to top |
|
|
kimchi_sg Advocate
Joined: 26 Nov 2004 Posts: 2968
|
Posted: Sun Apr 24, 2005 12:59 pm Post subject: |
|
|
In a perfect world, this would be a Good Thing. |
|
Back to top |
|
|
Bob P Advocate
Joined: 20 Oct 2004 Posts: 3355 Location: Jackass! Development Labs
|
Posted: Sun Apr 24, 2005 1:03 pm Post subject: |
|
|
Earthwings wrote: | I think I would disable strict in emerge -e until everyone uses repoman for checking things in. FEATURES=-strict emerge -e system |
thanks. i have already configured portage to work the old way by just adding "-strict" to my FEATURES in make.conf. i'm running with -strict on all of the boxes except one to get the builds finished. i'm going to do the good thing -- on the one system that's running with +strict, i'm posting bug reports for all of the errors that come up ... that is ... until i just can't take it anymore. i have 3 reports so far, with 84 total packages to go... _________________ .
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks |
|
Back to top |
|
|
Bob P Advocate
Joined: 20 Oct 2004 Posts: 3355 Location: Jackass! Development Labs
|
Posted: Sun Apr 24, 2005 7:06 pm Post subject: |
|
|
well, i have to admit, i'm giving up on the Boy Scout approach of doing it the hard way and reporting all of the bugs to Bugzilla. unfortunately, its impossible to perform an "emerge --resume --skipfirst" following a critical stop during an "emerge -e system". because there's no way to recover, and because its pointless to keep starting the whole emerge over again, i'm just proceeding with "-strict" in make.conf. other people will have to report bugs as they encounter them. _________________ .
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks |
|
Back to top |
|
|
rhill Retired Dev
Joined: 22 Oct 2004 Posts: 1629 Location: sk.ca
|
|
Back to top |
|
|
moocha Watchman
Joined: 21 Oct 2003 Posts: 5722
|
Posted: Mon Apr 25, 2005 10:13 pm Post subject: |
|
|
Most of the Code: | !!! Security Violation: A file exists that is not in the manifest. | errors can simply be solved by deleting the portage tree (move distfiles and packages away first), untarring a recent portage snapshot, and re-syncing. (I recommend untarring a snapshot because syncing with an empty tree would do the same for the most part, but would create more traffic since bzip2 is better than gzip, and hitting the rsync servers that much needlessly is just cruel ). _________________ Military Commissions Act of 2006: http://tinyurl.com/jrcto
"Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety."
-- attributed to Benjamin Franklin |
|
Back to top |
|
|
Bob P Advocate
Joined: 20 Oct 2004 Posts: 3355 Location: Jackass! Development Labs
|
Posted: Mon Apr 25, 2005 10:37 pm Post subject: |
|
|
in my case, i was rebuilding freshly extracted tarballs, rebuilding their toolkits for an entire array of architectures, using the most current portage snapshot. there was no old portage tree to replace, and i was already using the most recent portage snapshot.
the idea of syncing is well advised. but in my case, i had the most current snapshots and i was trying to build binary packages for distribution. in a situation like that, syncing doesn't help, as the users have to be given a snapshot that matches their binaries. turning off that horrible "strict" feature was the only way around the problem. otherwise, our entire Jackass! production effort would have been torpedoed by the introduction of strict checking as a "feature" in portage.
i can see where strict checking would be an asset as default portage behavior ... that is, if ebuild maintainers were willing to take the effort to fix their ebuilds in time to have them ready for the "strict" rollout. but that wasn't the case -- strict got implemented, ebuilds weren't ready, and plenty of crappy ebuilds broke as a result. one of the ebuilds i found had contained a stray file that was 2 years old. that's just sloppy programming, and it shows that there wasn't much effort to synchronize actions on the part of portage developers and ebuild maintainers.
i honestly think that the implementation of strict checking would have been better received if there had been a heads-up that it was coming. it seemed to come out of nowhere, and hit everybody like it was a sucker punch. _________________ .
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks |
|
Back to top |
|
|
Bob P Advocate
Joined: 20 Oct 2004 Posts: 3355 Location: Jackass! Development Labs
|
|
Back to top |
|
|
Genone Retired Dev
Joined: 14 Mar 2003 Posts: 9522 Location: beyond the rim
|
Posted: Tue Apr 26, 2005 1:35 am Post subject: |
|
|
Hmm, reminds me to put it in make.globals (it should never have been in make.defaults IMO). |
|
Back to top |
|
|
opentaka l33t
Joined: 18 Feb 2005 Posts: 840 Location: Japan
|
Posted: Tue Apr 26, 2005 7:29 am Post subject: |
|
|
cool, but i saw something saying "Gentoo Portage Backdoor" thing somewhere in the web.. I think it fixed by now _________________ "Being defeated is often a temporary condition. Giving up is what makes it permanent" - Marilyn vos Savant
|
|
Back to top |
|
|
vonhelmet l33t
Joined: 06 Apr 2004 Posts: 770 Location: Somewhere in a school
|
Posted: Fri May 06, 2005 11:11 am Post subject: |
|
|
I've just run into this problem with files not listed in the manifest.
I don't have a net connection at home, so I download snapshots and distfiles at work. I "sync" by untarring the whole tree over the /usr/portage directory. This isn't a real sync, as old files aren't deleted, which then causes problems because there's loads of old ebuilds and stuff that aren't in the manifest.
Anyone got any suggestions on how I can "sync" without running into this manifest problem? For the time being I've been deleting all the files in portage (except distfiles and a couple of others) and untarring the new tarball in the nearly empty directory. I'm thinking I might try to put together a perl script that will "sync" with a tarball offline in some way, but that sounds a bit complex to me. _________________ My blog
nvtuner software - enhance your AGP Geforce 6800 or 6200! |
|
Back to top |
|
|
Ciaran n00b
Joined: 10 May 2005 Posts: 2 Location: England, UK
|
Posted: Tue May 10, 2005 11:55 am Post subject: |
|
|
FWIW, I've never had any problem with the new strict checking... _________________ Please note: I'm not ciaranm. This is an entirely different Ciaran. |
|
Back to top |
|
|
nexus780 Apprentice
Joined: 17 Sep 2004 Posts: 206 Location: Manchester
|
Posted: Tue May 10, 2005 1:44 pm Post subject: |
|
|
Ciaran wrote: | FWIW, I've never had any problem with the new strict checking... |
Strangely enough, but same here. I've been running a manual emerge -e world since may 4th and I'm quite certain I haven't come around one of those yet. Got 15 or something (out of about 250) that fail on multilib-strict and/or test, but I activated those with the purpose of finding and getting rid off packages that don't do things properly. |
|
Back to top |
|
|
mikenerone n00b
Joined: 11 Feb 2004 Posts: 22 Location: San Antonio, TX
|
Posted: Tue May 10, 2005 11:04 pm Post subject: |
|
|
vonhelmet wrote: | Anyone got any suggestions on how I can "sync" without running into this manifest problem? For the time being I've been deleting all the files in portage (except distfiles and a couple of others) and untarring the new tarball in the nearly empty directory. I'm thinking I might try to put together a perl script that will "sync" with a tarball offline in some way, but that sounds a bit complex to me. |
I just threw together a quick script that I call "shotsync" (hehe...originally snapsync, but I edited the post because I realized that was taken by a popular piece of software) to help you out with that:
Code: | #!/bin/bash
# shotsync-0.1.2
# Ripped almost line for line from the sync_local() function of emerge-webrsync.
# It's purpose is to provide an easy way to sync your Gentoo portage tree to a snapshot file.
# Usage example: "shotsync /path/to/portage-20050507.tar.bz2" (relative path is fine, too)
# There is no warranty of any kind associated with the use of this script.
# Trivially created by Mike Nerone
SNAPSHOT="$1"
PORTDIR="$(/usr/lib/portage/bin/portageq portdir)"
PORTDIR="${PORTDIR%%/}"
TMPDIR="$(/usr/lib/portage/bin/portageq envvar PORTAGE_TMPDIR)"
TMPDIR="${TMPDIR%%/}/shotsync"
if [ -e "$TMPDIR" ]; then
echo Cleaning out shotsync\'s tmpdir...
rm -rf "$TMPDIR"
fi
mkdir -p "$TMPDIR"
echo Extracting snapshot file "$SNAPSHOT"...
if ! tar -C "$TMPDIR" -jxf "$SNAPSHOT"; then
echo "Tar failed to extract the image. Please review the output."
echo "Executed command: tar -C \"$TMPDIR\" -jxf \"$SNAPSHOT\""
exit 1
fi
# Uncomment the next line if you'd like the snapshot file you provided
# to be deleted automatically after it is extracted.
#rm -f "$SNAPSHOT"
# Make sure user and group file ownership is root
chown -R 0:0 "$TMPDIR/portage/"
echo Syncing local portage tree to extracted snapshot...
rsync -av --progress --stats --delete --delete-after \
--exclude='/distfiles' --exclude='/packages' \
--exclude='/local' "$TMPDIR/portage/" "$PORTDIR"
echo "Cleaning up..."
rm -rf "$TMPDIR"
echo "Updating portage cache..."
emerge metadata |
Incidentally, I've had strict in my FEATURES for at least a year or so, and I, too, have never experienced the errors you guys describe.
Last edited by mikenerone on Wed Jul 14, 2010 7:30 am; edited 1 time in total |
|
Back to top |
|
|
paultorbens n00b
Joined: 12 May 2005 Posts: 2
|
Posted: Fri May 13, 2005 12:33 am Post subject: |
|
|
thanks for the script - it worked great for me ... |
|
Back to top |
|
|
vonhelmet l33t
Joined: 06 Apr 2004 Posts: 770 Location: Somewhere in a school
|
|
Back to top |
|
|
dundas Guru
Joined: 16 Dec 2004 Posts: 317 Location: China, Earth
|
Posted: Fri Jun 24, 2005 6:46 am Post subject: |
|
|
I had this prob after emerge --sync
adding
FEATURES=-strict
in make.conf
worked fine for me, thx buddy. |
|
Back to top |
|
|
sinewalker n00b
Joined: 27 Jun 2005 Posts: 1 Location: Blue Mts, Australia
|
Posted: Mon Jun 27, 2005 5:08 am Post subject: Updated Gentoo Wiki TIP for low-bandwidth |
|
|
I stumbled across this issue when adding new packages for the first time on my bandwidth-impared home PC. I have updated the wiki to use mikenerone's shell script after downloading a snapshot but before doing emerge. It works wonders.
So now I can emerge using work's bandwidth and still be "strict".
I am really impressed by the Gentoo comunity -- so many answers solved quickly! |
|
Back to top |
|
|
dundas Guru
Joined: 16 Dec 2004 Posts: 317 Location: China, Earth
|
Posted: Sat Aug 13, 2005 9:01 am Post subject: Files listed in the manifest do not exist |
|
|
Dear all:
Here is the problem while I updating world today,
[NOTE] I dont have features="strict" in my make.conf
Code: | # emerge -uvND world
Calculating world dependencies ...done!
>>> emerge (1 of 7) app-editors/vim-core-6.3.084 to /
*** Adjusting cvs-src permissions for portage user...
>>> Downloading http://distfiles.gentoo.org/distfiles/vim-6.3.068-netrw.tar.bz2
.
.
.
.
Downloaded 106.1 kilobytes in 1 second. (76.37 KB/s)
>>> Downloading http://distfiles.gentoo.org/distfiles/vim-6.3.084-gentoo-patches.tar.bz2
Initializing download: http://distfiles.gentoo.org/distfiles/vim-6.3.084-gentoo-patches.tar.bz2
File size: 14269 bytes
Opening output file /usr/portage/distfiles/vim-6.3.084-gentoo-patches.tar.bz2
Starting download
Connection 0 finished
Connection 3 finished ]
Connection 2 finished ]
[100%] [..................................................] [ 7.2KB/s] [00:00]
Downloaded 13.9 kilobytes in 1 second. (7.20 KB/s)
!!! Files listed in the manifest do not exist!
vim-core-6.3.084-r1.ebuild
files/digest-vim-core-6.3.084-r1
|
[TRIED] As I googled for the solution, some post suggested emerge --sync again would fix it, however, I'm not able to emerge --sync again as it says:
Code: | >>>
>>> Timestamps on the server and in the local repository are the same.
>>> Cancelling all further sync action. You are already up to date.
>>> |
Any ways I can force emerge --sync at this moment?
[TRIED2] Code: | # equery which vim-core
/usr/portage/app-editors/vim-core/vim-core-7.0_alpha20050809.ebuild |
which is different from the required one vim-core-6.3.084-r1.ebuild....weird...
and I can't find vim-core-6.3.084-r1.ebuild using
Code: | find / -name vim-core-6.3.084-r1.ebuild |
Any ideas are appreciated!
mod edit: Merged to this thread. --Earthwings _________________ Appreciate Gentoo: Best Devs, Best Forums. YOU could help too: Help Answer |
|
Back to top |
|
|
dundas Guru
Joined: 16 Dec 2004 Posts: 317 Location: China, Earth
|
Posted: Sat Aug 13, 2005 9:29 am Post subject: |
|
|
even if I rm -rf /var/tmp/portage/*, rm -rf /usr/portage/distfiles/* and emerge --sync again, I couldn't get it working, still same errors
[SOLUTION]
The only thing worked for me is to put the "-strict" in my features
p.s. I did use -strict before but after some while, I tried to use strict, but now....guess I'm back to the old way again....feeling dizzy....I need security....
anyways, thx for merging my post. _________________ Appreciate Gentoo: Best Devs, Best Forums. YOU could help too: Help Answer |
|
Back to top |
|
|
Bob P Advocate
Joined: 20 Oct 2004 Posts: 3355 Location: Jackass! Development Labs
|
Posted: Mon Aug 15, 2005 2:11 am Post subject: |
|
|
i'm going to rant a little about this problem, which hasn't gone away after a period of months. the long and short of it is that ebuild maintainers are just doing a half-baked job at maintaining their ebuilds. unfortunately, there are ALOT of ebuilds that are gaining status as offenders in this regard, and the problem seems to be getting worse rather than better. such wouldn't be the case if the people maintaining ebuilds bothered to check their work before releasing it into the portage tree.
<rant>
a conscientious ebuild maintainer would actually test their ebuild before publishing it, by performing a test emerge after finishing the build. if such testing were performed, a light bulb would go on over their heads as they emerged their ebuild and saw the bright red letters warning about files existing that are not in the manifest, and the emerge stopped with a critical error. the fact that these mistakes are slipping through means that error checks aren't being performed. how could this happen?
well, for one, it appears that the offending ebuild maintainers have to be using FEATURES=-strict settings on their development boxes. this effectively suppresses the error warnings and the critical stops so that they never see them. in contrast, conscientious users who try to protect their systems by not using the FEATURES=-strict settings get jammed when they try to emerge the ebuilds.
the fact that these problems are continuously popping up in the ebuilds means that there are quite a few ebuild maintainers who just don't bother to perform QA. sheesh. this happened to me four times tonight trying to perform a basic install, and included ebuilds such as grub, hotplug or coldplug, acpid and gentoo-sources-2.6.12-r6.
this is a very simple problem to fix. the ebuild maintainers should STOP using "FEATURES=-strict" on their development boxes. that will result in the error not slipping by them, and then we'll stop having the problem. until EVERY ebuild maintainer does this, the problem will never go away.
</rant> _________________ .
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks |
|
Back to top |
|
|
|