Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Portage does more security checks? Digest verification fails
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
iTux
Guru
Guru


Joined: 07 Sep 2004
Posts: 586
Location: Toronto

PostPosted: Sat Apr 16, 2005 8:41 pm    Post subject: Portage does more security checks? Digest verification fails Reply with quote

I did an emerge sync. I did not upgrade portage or other system utilities and I did not modified my config.

I noticed that portage does more checking that it used before:
Code:

ibou sablevm-svn # USE="sablejit -gtk" nice ebuild sablevm-svn-20050414.ebuild compile
!!! Security Violation: A file exists that is not in the manifest.
!!! File: files/makefile-common.patch


Never got that before and I often forget to run ebuild package digest...

Code:

ibou sablevm-svn # USE="sablejit -gtk" nice ebuild sablevm-svn-20050414.ebuild compile
>>> md5 files   ;-) sablevm-svn.tar.gz
>>> md5 files   ;-) README
>>> md5 files   ;-) sablevm-svn-20050414.ebuild
>>> md5 files   ;-) uploadfiles
>>> md5 files   ;-) sablevm-svn-20050205-r5.ebuild
>>> md5 files   ;-) files/digest-sablevm-svn-20050205-r5
>>> md5 files   ;-) files/makefile-sablejit.patch
>>> md5 files   ;-) files/autogen.patch
>>> md5 files   ;-) files/ltmain.patch
>>> md5 files   ;-) files/version.patch
>>> md5 files   ;-) files/makefile-config.patch
>>> md5 files   ;-) files/sablevm-svn-20050205
>>> md5 files   ;-) files/digest-sablevm-svn-20050414
>>> md5 files   ;-) files/sablevm-svn-20050414
>>> md5 files   ;-) files/makefile-common.patch


Portage used to md5check only files externally downlowded i.e. in distfiles...

Why do I get these extra integrity checks today? Remember that I did not upgrade portage or change config. I am puzzled. Or some default setting changed? If yes, which one?

iTux

mod edit: sticky and expanded subject to address the "digest verification failure" posts that will come up in the near future. --Earthwings
Back to top
View user's profile Send private message
iTux
Guru
Guru


Joined: 07 Sep 2004
Posts: 586
Location: Toronto

PostPosted: Sat Apr 16, 2005 8:49 pm    Post subject: Reply with quote

Seems to be due to FEATURES=strict that is enabled by default. I guess it was disabled by default before?

iTux
Back to top
View user's profile Send private message
Earthwings
Administrator
Administrator


Joined: 14 Apr 2003
Posts: 7751
Location: Karlsruhe, Germany

PostPosted: Sun Apr 24, 2005 9:59 am    Post subject: Reply with quote

Yes, it was changed recently. http://www.gentoo.org/cgi-bin/viewcvs.cgi/profiles/base/make.defaults?r1=1.4&r2=1.5

Here's a short summary what changed and what can be done against "Digest verification failure".

FEATURES=strict (which is now enabled on your system if you ran emerge --sync in the last days and don't have FEATURES=-strict in make.conf) tells emerge to check the digest (md5 sum) not only for the files in SRC_URI, but also for ebuilds. This is more secure and therefore a good thing, but you may experience "digest verification failure" more often. Here is how to handle this situation.

First, determine whether the digest failed for a file from SRC_URI (most often a .gz or .bz2 or .rpm). If this is the case try the following:
- delete the file from /usr/portage/distfiles and restart emerge. It will be downloaded again from the mirror.
- delete the file from /usr/portage/distfiles and download in manually from another mirror, cause the mirror may contain a broken file.
- run emerge --sync and try again. The problem might be already fixed
- search forums.gentoo.org and bugs.gentoo.org to see if other people experience the same problem and the recorded md5 sum is wrong
- write a bug report if none exists or contact a developer in irc at irc.freenode.net
- If you are absolutely sure that the file you downloaded is ok, you can generate a new digest with ebuild /path/to/ebuild digest.

If you get a digest failure for a file from the Manifest (typically a .ebuild), do the following:
- run emerge --sync to see if the problem is already fixed.
- search forums.gentoo.org and bugs.gentoo.org to see if other people experience the same problem and the recorded md5 sum is wrong
- write a bug report if none exists or contact a developer in irc at irc.freenode.net
- if you verified the file is ok (have a look at it), you can generate a new digest with ebuild /path/to/ebuild digest.
_________________
KDE 4.13 - Get It While It's Hot!
Back to top
View user's profile Send private message
Bob P
Advocate
Advocate


Joined: 20 Oct 2004
Posts: 3355
Location: Jackass! Development Labs

PostPosted: Sun Apr 24, 2005 12:05 pm    Post subject: Reply with quote

ARGH! :evil:

i'm performing an emerge -e system to rebuild the toolkit on a fleet of systems (every possible x86 architecture) after downloading the most recent portage snapshot. would anyone care to guess how often this error pops up:

Code:
!!! Security Violation: A file exists that is not in the manifest.


its hit me on two out of three ebuilds so far, and i've got 91 of them to deal with in the emerge. this has to be some sort of sick belated April Fool's joke. :evil:
_________________
.
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks
Back to top
View user's profile Send private message
Earthwings
Administrator
Administrator


Joined: 14 Apr 2003
Posts: 7751
Location: Karlsruhe, Germany

PostPosted: Sun Apr 24, 2005 12:56 pm    Post subject: Reply with quote

I think I would disable strict in emerge -e until everyone uses repoman for checking things in. FEATURES=-strict emerge -e system
_________________
KDE 4.13 - Get It While It's Hot!
Back to top
View user's profile Send private message
kimchi_sg
Advocate
Advocate


Joined: 26 Nov 2004
Posts: 2915
Location: Singapore

PostPosted: Sun Apr 24, 2005 12:59 pm    Post subject: Reply with quote

In a perfect world, this would be a Good Thing.
_________________
Murphy's Law of Gentoo installation: If a compile can fail, it will.

MacGillicuddy's Corollary: At the most inopportune time.

Please search and read the FAQs before posting.
Back to top
View user's profile Send private message
Bob P
Advocate
Advocate


Joined: 20 Oct 2004
Posts: 3355
Location: Jackass! Development Labs

PostPosted: Sun Apr 24, 2005 1:03 pm    Post subject: Reply with quote

Earthwings wrote:
I think I would disable strict in emerge -e until everyone uses repoman for checking things in. FEATURES=-strict emerge -e system


thanks. i have already configured portage to work the old way by just adding "-strict" to my FEATURES in make.conf. i'm running with -strict on all of the boxes except one to get the builds finished. i'm going to do the good thing -- on the one system that's running with +strict, i'm posting bug reports for all of the errors that come up ... that is ... until i just can't take it anymore. i have 3 reports so far, with 84 total packages to go...
_________________
.
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks
Back to top
View user's profile Send private message
Bob P
Advocate
Advocate


Joined: 20 Oct 2004
Posts: 3355
Location: Jackass! Development Labs

PostPosted: Sun Apr 24, 2005 7:06 pm    Post subject: Reply with quote

well, i have to admit, i'm giving up on the Boy Scout approach of doing it the hard way and reporting all of the bugs to Bugzilla. unfortunately, its impossible to perform an "emerge --resume --skipfirst" following a critical stop during an "emerge -e system". because there's no way to recover, and because its pointless to keep starting the whole emerge over again, i'm just proceeding with "-strict" in make.conf. other people will have to report bugs as they encounter them. :?
_________________
.
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks
Back to top
View user's profile Send private message
rhill
Developer
Developer


Joined: 22 Oct 2004
Posts: 1629
Location: sk.ca

PostPosted: Mon Apr 25, 2005 5:23 am    Post subject: Reply with quote

heh.

https://bugs.gentoo.org/show_bug.cgi?id=89388
_________________
by design, by neglect
for a fact or just for effect
Back to top
View user's profile Send private message
moocha
Watchman
Watchman


Joined: 21 Oct 2003
Posts: 5722
Location: Cluj-Napoca, Romania

PostPosted: Mon Apr 25, 2005 10:13 pm    Post subject: Reply with quote

Most of the
Code:
!!! Security Violation: A file exists that is not in the manifest.
errors can simply be solved by deleting the portage tree (move distfiles and packages away first), untarring a recent portage snapshot, and re-syncing. (I recommend untarring a snapshot because syncing with an empty tree would do the same for the most part, but would create more traffic since bzip2 is better than gzip, and hitting the rsync servers that much needlessly is just cruel :D).
_________________
Military Commissions Act of 2006: http://tinyurl.com/jrcto

"Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety."
-- attributed to Benjamin Franklin
Back to top
View user's profile Send private message
Bob P
Advocate
Advocate


Joined: 20 Oct 2004
Posts: 3355
Location: Jackass! Development Labs

PostPosted: Mon Apr 25, 2005 10:37 pm    Post subject: Reply with quote

in my case, i was rebuilding freshly extracted tarballs, rebuilding their toolkits for an entire array of architectures, using the most current portage snapshot. there was no old portage tree to replace, and i was already using the most recent portage snapshot.

the idea of syncing is well advised. but in my case, i had the most current snapshots and i was trying to build binary packages for distribution. in a situation like that, syncing doesn't help, as the users have to be given a snapshot that matches their binaries. turning off that horrible "strict" feature was the only way around the problem. :? otherwise, our entire Jackass! production effort would have been torpedoed by the introduction of strict checking as a "feature" in portage.

i can see where strict checking would be an asset as default portage behavior ... that is, if ebuild maintainers were willing to take the effort to fix their ebuilds in time to have them ready for the "strict" rollout. but that wasn't the case -- strict got implemented, ebuilds weren't ready, and plenty of crappy ebuilds broke as a result. one of the ebuilds i found had contained a stray file that was 2 years old. that's just sloppy programming, and it shows that there wasn't much effort to synchronize actions on the part of portage developers and ebuild maintainers.

i honestly think that the implementation of strict checking would have been better received if there had been a heads-up that it was coming. it seemed to come out of nowhere, and hit everybody like it was a sucker punch. :cry:
_________________
.
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks
Back to top
View user's profile Send private message
Bob P
Advocate
Advocate


Joined: 20 Oct 2004
Posts: 3355
Location: Jackass! Development Labs

PostPosted: Mon Apr 25, 2005 10:44 pm    Post subject: Reply with quote

dirtyepic wrote:
heh.

https://bugs.gentoo.org/show_bug.cgi?id=89388


indeed! :idea:
_________________
.
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks
Back to top
View user's profile Send private message
Genone
Retired Dev
Retired Dev


Joined: 14 Mar 2003
Posts: 9011
Location: beyond the rim

PostPosted: Tue Apr 26, 2005 1:35 am    Post subject: Reply with quote

Hmm, reminds me to put it in make.globals (it should never have been in make.defaults IMO).
Back to top
View user's profile Send private message
opentaka
l33t
l33t


Joined: 18 Feb 2005
Posts: 840
Location: Japan

PostPosted: Tue Apr 26, 2005 7:29 am    Post subject: Reply with quote

cool, but i saw something saying "Gentoo Portage Backdoor" thing somewhere in the web.. I think it fixed by now
_________________
"Being defeated is often a temporary condition. Giving up is what makes it permanent" - Marilyn vos Savant
Back to top
View user's profile Send private message
vonhelmet
l33t
l33t


Joined: 06 Apr 2004
Posts: 770
Location: Somewhere in a school

PostPosted: Fri May 06, 2005 11:11 am    Post subject: Reply with quote

I've just run into this problem with files not listed in the manifest.

I don't have a net connection at home, so I download snapshots and distfiles at work. I "sync" by untarring the whole tree over the /usr/portage directory. This isn't a real sync, as old files aren't deleted, which then causes problems because there's loads of old ebuilds and stuff that aren't in the manifest.

Anyone got any suggestions on how I can "sync" without running into this manifest problem? For the time being I've been deleting all the files in portage (except distfiles and a couple of others) and untarring the new tarball in the nearly empty directory. I'm thinking I might try to put together a perl script that will "sync" with a tarball offline in some way, but that sounds a bit complex to me.
_________________
My blog
nvtuner software - enhance your AGP Geforce 6800 or 6200!
Back to top
View user's profile Send private message
Ciaran
n00b
n00b


Joined: 10 May 2005
Posts: 2
Location: England, UK

PostPosted: Tue May 10, 2005 11:55 am    Post subject: Reply with quote

FWIW, I've never had any problem with the new strict checking...
_________________
Please note: I'm not ciaranm. This is an entirely different Ciaran. ;-)
Back to top
View user's profile Send private message
nexus780
Apprentice
Apprentice


Joined: 17 Sep 2004
Posts: 206
Location: Manchester

PostPosted: Tue May 10, 2005 1:44 pm    Post subject: Reply with quote

Ciaran wrote:
FWIW, I've never had any problem with the new strict checking...

Strangely enough, but same here. I've been running a manual emerge -e world since may 4th and I'm quite certain I haven't come around one of those yet. Got 15 or something (out of about 250) that fail on multilib-strict and/or test, but I activated those with the purpose of finding and getting rid off packages that don't do things properly.
Back to top
View user's profile Send private message
mikenerone
n00b
n00b


Joined: 11 Feb 2004
Posts: 21
Location: San Antonio, TX

PostPosted: Tue May 10, 2005 11:04 pm    Post subject: Reply with quote

vonhelmet wrote:
Anyone got any suggestions on how I can "sync" without running into this manifest problem? For the time being I've been deleting all the files in portage (except distfiles and a couple of others) and untarring the new tarball in the nearly empty directory. I'm thinking I might try to put together a perl script that will "sync" with a tarball offline in some way, but that sounds a bit complex to me.

I just threw together a quick script that I call "shotsync" (hehe...originally snapsync, but I edited the post because I realized that was taken by a popular piece of software) to help you out with that:

Code:
#!/bin/bash
# shotsync-0.1.2
#   Ripped almost line for line from the sync_local() function of emerge-webrsync.
#   It's purpose is to provide an easy way to sync your Gentoo portage tree to a snapshot file.
#   Usage example: "shotsync /path/to/portage-20050507.tar.bz2" (relative path is fine, too)
#   There is no warranty of any kind associated with the use of this script.
#   Trivially created by Mike Nerone

SNAPSHOT="$1"
PORTDIR="$(/usr/lib/portage/bin/portageq portdir)"
PORTDIR="${PORTDIR%%/}"
TMPDIR="$(/usr/lib/portage/bin/portageq envvar PORTAGE_TMPDIR)"
TMPDIR="${TMPDIR%%/}/shotsync"

if [ -e "$TMPDIR" ]; then
  echo Cleaning out shotsync\'s tmpdir...
  rm -rf "$TMPDIR"
fi
mkdir -p "$TMPDIR"

echo Extracting snapshot file "$SNAPSHOT"...
if ! tar -C "$TMPDIR" -jxf "$SNAPSHOT"; then
  echo "Tar failed to extract the image. Please review the output."
  echo "Executed command: tar -C \"$TMPDIR\" -jxf \"$SNAPSHOT\""
  exit 1
fi

# Uncomment the next line if you'd like the snapshot file you provided
#   to be deleted automatically after it is extracted.
#rm -f "$SNAPSHOT"

# Make sure user and group file ownership is root
chown -R 0:0 "$TMPDIR/portage/"

echo Syncing local portage tree to extracted snapshot...
rsync -av --progress --stats --delete --delete-after \
--exclude='/distfiles' --exclude='/packages' \
--exclude='/local' "$TMPDIR/portage/" "$PORTDIR"
echo "Cleaning up..."
rm -rf "$TMPDIR"
echo "Updating portage cache..."
emerge metadata

Incidentally, I've had strict in my FEATURES for at least a year or so, and I, too, have never experienced the errors you guys describe.


Last edited by mikenerone on Wed Jul 14, 2010 7:30 am; edited 1 time in total
Back to top
View user's profile Send private message
paultorbens
n00b
n00b


Joined: 12 May 2005
Posts: 1

PostPosted: Fri May 13, 2005 12:33 am    Post subject: Reply with quote

thanks for the script - it worked great for me ...
Back to top
View user's profile Send private message
vonhelmet
l33t
l33t


Joined: 06 Apr 2004
Posts: 770
Location: Somewhere in a school

PostPosted: Mon May 16, 2005 7:56 am    Post subject: Reply with quote

Yeah, cheers, that did the job and made sync'ing offline a lot easier.
_________________
My blog
nvtuner software - enhance your AGP Geforce 6800 or 6200!
Back to top
View user's profile Send private message
dundas
Guru
Guru


Joined: 16 Dec 2004
Posts: 317
Location: China, Earth

PostPosted: Fri Jun 24, 2005 6:46 am    Post subject: Reply with quote

I had this prob after emerge --sync

adding
FEATURES=-strict
in make.conf


worked fine for me, thx buddy.
Back to top
View user's profile Send private message
sinewalker
n00b
n00b


Joined: 27 Jun 2005
Posts: 1
Location: Blue Mts, Australia

PostPosted: Mon Jun 27, 2005 5:08 am    Post subject: Updated Gentoo Wiki TIP for low-bandwidth Reply with quote

I stumbled across this issue when adding new packages for the first time on my bandwidth-impared home PC. I have updated the wiki to use mikenerone's shell script after downloading a snapshot but before doing emerge. It works wonders.

So now I can emerge using work's bandwidth and still be "strict".

I am really impressed by the Gentoo comunity -- so many answers solved quickly!
Back to top
View user's profile Send private message
dundas
Guru
Guru


Joined: 16 Dec 2004
Posts: 317
Location: China, Earth

PostPosted: Sat Aug 13, 2005 9:01 am    Post subject: Files listed in the manifest do not exist Reply with quote

Dear all:

Here is the problem while I updating world today,

[NOTE] I dont have features="strict" in my make.conf

Code:
# emerge -uvND world
Calculating world dependencies ...done!
>>> emerge (1 of 7) app-editors/vim-core-6.3.084 to /
*** Adjusting cvs-src permissions for portage user...
>>> Downloading http://distfiles.gentoo.org/distfiles/vim-6.3.068-netrw.tar.bz2

.
.
.
.

Downloaded 106.1 kilobytes in 1 second. (76.37 KB/s)
>>> Downloading http://distfiles.gentoo.org/distfiles/vim-6.3.084-gentoo-patches.tar.bz2
Initializing download: http://distfiles.gentoo.org/distfiles/vim-6.3.084-gentoo-patches.tar.bz2
File size: 14269 bytes
Opening output file /usr/portage/distfiles/vim-6.3.084-gentoo-patches.tar.bz2
Starting download

Connection 0 finished
Connection 3 finished                                                          ]
Connection 2 finished                                                          ]
[100%] [..................................................] [   7.2KB/s] [00:00]

Downloaded 13.9 kilobytes in 1 second. (7.20 KB/s)
!!! Files listed in the manifest do not exist!
vim-core-6.3.084-r1.ebuild
files/digest-vim-core-6.3.084-r1



[TRIED] As I googled for the solution, some post suggested emerge --sync again would fix it, however, I'm not able to emerge --sync again as it says:

Code:
>>>
>>> Timestamps on the server and in the local repository are the same.
>>> Cancelling all further sync action. You are already up to date.
>>>


Any ways I can force emerge --sync at this moment?

[TRIED2]
Code:
# equery which vim-core
/usr/portage/app-editors/vim-core/vim-core-7.0_alpha20050809.ebuild

which is different from the required one vim-core-6.3.084-r1.ebuild....weird...
and I can't find vim-core-6.3.084-r1.ebuild using
Code:
find / -name vim-core-6.3.084-r1.ebuild



Any ideas are appreciated!

mod edit: Merged to this thread. --Earthwings
_________________
Appreciate Gentoo: Best Devs, Best Forums. YOU could help too: Help Answer
Back to top
View user's profile Send private message
dundas
Guru
Guru


Joined: 16 Dec 2004
Posts: 317
Location: China, Earth

PostPosted: Sat Aug 13, 2005 9:29 am    Post subject: Reply with quote

even if I rm -rf /var/tmp/portage/*, rm -rf /usr/portage/distfiles/* and emerge --sync again, I couldn't get it working, still same errors

[SOLUTION]
The only thing worked for me is to put the "-strict" in my features

p.s. I did use -strict before but after some while, I tried to use strict, but now....guess I'm back to the old way again....feeling dizzy....I need security....

anyways, thx for merging my post.
_________________
Appreciate Gentoo: Best Devs, Best Forums. YOU could help too: Help Answer
Back to top
View user's profile Send private message
Bob P
Advocate
Advocate


Joined: 20 Oct 2004
Posts: 3355
Location: Jackass! Development Labs

PostPosted: Mon Aug 15, 2005 2:11 am    Post subject: Reply with quote

i'm going to rant a little about this problem, which hasn't gone away after a period of months. the long and short of it is that ebuild maintainers are just doing a half-baked job at maintaining their ebuilds. unfortunately, there are ALOT of ebuilds that are gaining status as offenders in this regard, and the problem seems to be getting worse rather than better. such wouldn't be the case if the people maintaining ebuilds bothered to check their work before releasing it into the portage tree.

<rant>
a conscientious ebuild maintainer would actually test their ebuild before publishing it, by performing a test emerge after finishing the build. if such testing were performed, a light bulb would go on over their heads as they emerged their ebuild and saw the bright red letters warning about files existing that are not in the manifest, and the emerge stopped with a critical error. the fact that these mistakes are slipping through means that error checks aren't being performed. how could this happen?

well, for one, it appears that the offending ebuild maintainers have to be using FEATURES=-strict settings on their development boxes. this effectively suppresses the error warnings and the critical stops so that they never see them. in contrast, conscientious users who try to protect their systems by not using the FEATURES=-strict settings get jammed when they try to emerge the ebuilds.

the fact that these problems are continuously popping up in the ebuilds means that there are quite a few ebuild maintainers who just don't bother to perform QA. sheesh. this happened to me four times tonight trying to perform a basic install, and included ebuilds such as grub, hotplug or coldplug, acpid and gentoo-sources-2.6.12-r6.

this is a very simple problem to fix. the ebuild maintainers should STOP using "FEATURES=-strict" on their development boxes. that will result in the error not slipping by them, and then we'll stop having the problem. until EVERY ebuild maintainer does this, the problem will never go away.

</rant>
_________________
.
Stage 1/3 | Jackass! | Rockhopper! | Thanks | Google Sucks
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum