portage - !!! Digest verification failed: [FIXED] updated

[dufeu]

see: final edit - November 15 ~ 8:00PM EST

Lately, I've been seeing a lot of these where I used to never see them:

Code: Select all

!!! Digest verification failed:
!!! /usr/portage/media-video/ffmpeg/ChangeLog
!!! Reason: Filesize does not match recorded size
!!! Got: 5765
!!! Expected: 5295

Code: Select all

!!! Digest verification failed:
!!! /usr/portage/kde-plasma/polkit-kde-agent/ChangeLog
!!! Reason: Filesize does not match recorded size
!!! Got: 3888
!!! Expected: 3690

Code: Select all

!!! Digest verification failed:
!!! /usr/portage/media-libs/libvpx/ChangeLog
!!! Reason: Filesize does not match recorded size
!!! Got: 3643
!!! Expected: 3420

Most of them get 'solved' by either waiting awhile and then re-running

Code: Select all

emaint --auto sync

or by manually running

Code: Select all

emerge -f package

In other words, it looks like some of the portage tree information is being updated asynchronously and the problem can be resolved over time.

This one has me stumped:

Code: Select all

# emerge -1f =sys-apps/busybox-1.24.1
Calculating dependencies... done!

>>> Fetching (1 of 1) sys-apps/busybox-1.24.1::gentoo

!!! Digest verification failed:
!!! /usr/portage/sys-apps/busybox/busybox-9999.ebuild
!!! Reason: Filesize does not match recorded size
!!! Got: 8493
!!! Expected: 8580

Waiting and re-pulling the portage tree hasn't resolved the problem and trying to manually fetch the package hasn't solved the problem. {I may not have waited long enough in this case}

Moreover, if I'm manually fetrching a specific version of a package I want, why is it checking the 9999 version? Though I suspect that last may be the size of the actual 9999 ebuild, not size of the git pulled 9999 package.

I can manually force create an ebuild manifest, but this is not something I want to be doing on a regular basis.


Anyone else having similar problems? or can shed some light on this?

edit -- 8 hours after initial post

From other people's comments, I'm actually happy to see I'm not the only one experiencing this issue.

Several points I gleaned from other peoples comments:

1. The main problem (Changelog) is caused by asynchronous updates to package changelogs (process takes longer than portage tree sync interval) as per

   There are currently problems, beacuse the initial updating of ChangeLogs takes longer than the sync interval.
   This will presumably no longer be an issue in a few days.

   comment below.
2. There appears to be a secondary problem of the same type [asynchronous tree updates] involving upstream trunk {9999} type ebuilds with 'busybox' being the prime example.
3. There may be a third similar problem but I can't find the example I thought I had. For now, I will assume there are only two of these types of issues with the portage tree but suggest people keep their eyes open for similar failures but not 'Changelog' nor '9999' ebuilds.
4. You can work around both problems by forcing your local package digest for the problem package to be re-created. This can have serious secutiry implications and I do not recommend doing this if at all possible. Since the more common 'Changelog' problem can be solved simply by re-pulling the tree, this represents the more secure work-around.

There are several different ways of doing creating new package digests included in other people's comments below. I'm pretty old school so I usually do {using busybox as an example}

Code: Select all

ebuild /usr/portage/sys-apps/busybox/busybox-9999.ebuild manifest

This is the original way to reset a given package digest.

I'd like to repeat and warn. There are security implications to overriding a package digest. The digest is there to assure you that no one has substituted a malicious package in place of the expected/desired package. Do not, repeat, NOT override package digests without having an excellent and well understood reason for doing so. This is because all files for that package in the portage tree will recieve new digests. This means all versions of the package ebuilds, any included patches and all other ancillary files for that package.

Executing my example above:

Code: Select all

# emerge -f =sys-apps/busybox-1.24.1
Calculating dependencies... done!

>>> Fetching (1 of 1) sys-apps/busybox-1.24.1::gentoo

!!! Digest verification failed:
!!! /usr/portage/sys-apps/busybox/busybox-9999.ebuild
!!! Reason: Filesize does not match recorded size
!!! Got: 8493
!!! Expected: 8580

>>> Failed to emerge sys-apps/busybox-1.24.1

Code: Select all

# ebuild /usr/portage/sys-apps/busybox/busybox-9999.ebuild manifest
>>> Creating Manifest for /usr/portage/sys-apps/busybox

Code: Select all

# emerge -f =sys-apps/busybox-1.24.1
Calculating dependencies... done!

>>> Fetching (1 of 1) sys-apps/busybox-1.24.1::gentoo
>>> Downloading 'http://distfiles.gentoo.org/distfiles/busybox-1.24.1.tar.bz2'
--2015-11-13 22:41:25--  http://distfiles.gentoo.org/distfiles/busybox-1.24.1.tar.bz2
Resolving distfiles.gentoo.org... 137.226.34.42, 216.165.129.135, 64.50.236.52, ...
Connecting to distfiles.gentoo.org|137.226.34.42|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2068803 (2.0M) [application/octet-stream]
Saving to: ‘/usr/portage/distfiles/busybox-1.24.1.tar.bz2’

/usr/portage/distfiles/busybox-1.24.1. 100%[=============================================================================>]   1.97M  1.46MB/s   in 1.3s   

2015-11-13 22:41:27 (1.46 MB/s) - ‘/usr/portage/distfiles/busybox-1.24.1.tar.bz2’ saved [2068803/2068803]

 * busybox-1.24.1.tar.bz2 SHA256 SHA512 WHIRLPOOL size ;-) ...                                                                                      [ ok ]

edit - November 15 ~ 830AM EST

While the 'changelog' issue seems to be resolved, the '-9999' bad manifest for sys-apps/busybox remains.

Until this is fixed, I'm running my --sync command like this:

Code: Select all

date && time emaint --auto sync && time eix-update && date && ebuild /usr/portage/sys-apps/busybox/busybox-9999.ebuild manifest

This is my personal choice for a work-around. If you read through the thread, you see other work-arounds and pertinent {security related} comments.

edit - November 15 ~ 530PM EST

The gentoo bug to follow is 565694

As I read the bug and comments, my interpretation is that the problem is in the automated script{s} used to update the public servers and the problem is not security related.

Disclaimer: I'm not a developer. I'm not related to or part of any gentoo team. I am not a gentoo developer. I could easily be wrong.

Reminder: The manifest is part of Gentoo's security scheme to ensure that no one has inserted a {possibly malicious} package from unknown sources into the protage tree. It is up to you and it is your responsibility to decide if any of the work arounds mentioned here can be performed on your system{s}.

final edit - November 15 ~ 8:00PM EST

As per the above bug, the request was to re-sync to the portage tree after 0:00 UTC Nov 16 and see if the problem was resolved. I re-pulled the portage tree ~ 1:00 UTC {8:00pm EST Nov 15} and can verify that the problem seems resolved. Thank you infr-structure team!

If you think you still have a problem please be sure to perform a --sync now and check again:

Code: Select all

emaint --auto sync

This was 'fixed' for me with the public servers I hit. Because there are quite a few public servers, this may or may not be 'fixed' for you depending on when the public server you actually access gets synced with the official master server.

If this is the case for you {still not 'fixed'} you may want to wait one day before trying again. If it's still not fixed for you then I suggest you read the above bug, check the information the devs are asking for and post that in a comment for them. They're basically asking for --sync timestamp information and copies of the actual manifest record{s} you ended up with.

But really wait one day first to give your "local" public server a chance to be synced with the master server.

[ruisantos1976]

I have this exact same problem too.

I've also deleted /usr/portage/sys-apps/busybox directory and performed an emaint sync -a, but the problem still remains.

I assume you haven't sorted that out (it's been only 14 minutes since you've posted this )...

If you, or anyone for that matter has a solution for this, please do let me/us know.

Thanks,
Rui

[C5ace]

I have this exact same problem too.

I've also deleted /usr/portage/sys-apps/busybox directory and performed an emaint sync -a, but the problem still remains.

I assume you haven't sorted that out (it's been only 14 minutes since you've posted this )...

If you, or anyone for that matter has a solution for this, please do let me/us know.

Thanks,
Rui

I get exactly the same error.

[slize12]

I have this exact same problem too.

I've also deleted /usr/portage/sys-apps/busybox directory and performed an emaint sync -a, but the problem still remains.

I assume you haven't sorted that out (it's been only 14 minutes since you've posted this )...

+1 here. Exactly the same problem stops me of compiling anything. I also tried the solution by removing the folder. It didn't work. :-\

[gerard27]

Same here.
I'll try again later.
Gerard.

[mv]

There are currently problems, beacuse the initial updating of ChangeLogs takes longer than the sync interval.
This will presumably no longer be an issue in a few days.

[Xapuyc]

Try
emerge --digest busybox.

To mask verification failure

Though, I'm not sure if it's safe enough =)

[Atmmac]

Yeah definitely a bit frustrating. Just going to have to wait it out and do another sync later on.

[kikko]

Hi all
to work around this issue, you can run "repoman manifest" in the ebuild directory:

Code: Select all

~ # cd /usr/portage/sys-apps/busybox
busybox # repoman manifest
>>> Creating Manifest for /usr/portage/sys-apps/busybox

it recalculates the Manifest file digests and allow you to emerge the package correcty, without wait for the upstream fix (this solved the busybox issue for me, btw)

Regards

[yaclo]

copied from https://github.com/gentoo/gentoo/tree/m ... ps/busybox files to portage tree than now works well

[NeddySeagoon]

Its a known bug - The Gentoo infra team have it in hand.

kikko,

Building the manifests yourself is not a secure thing to do. The manifests are built against whatever tarballs and tree you have installed.
When emerge checks them later, they will match because they were made to match.

When you get manifests from the sync server and tarballs from elsewhere you have a degree of confidence that you got what Gentoo intended if the manifests match.

As long as you are aware of the dangers of making your own manifests, you are good.

[dufeu]

Building the manifests yourself is not a secure thing to do. ...

I updated my original post trying to re-inforce that message

If there is anything I said wrong in my updated edit, please correct it and I'll change it in my post as well.

Thanks.

[mv]

Building the manifests yourself is not a secure thing to do. The manifests are built against whatever tarballs and tree you have installed.

This is not correct. Unless you do use --force, the existing tarballl checksums are taken from the manifests. Only modifications within the tree (ebuild, patches, ini-files, ...) will get rebuild.
And if the tree is compromised, you are currently screwed up, anyway, with rsync.

[NeddySeagoon]

mv,

Thank you.

[brendlefly62]

I haven't experienced the changelog symptom described above, but I do have the other:

!!! Digest verification failed:
!!! /usr/portage/sys-apps/busybox/busybox-9999.ebuild
!!! Reason: Filesize does not match recorded size
!!! Got: 8493
!!! Expected: 8580

-which is curious indeed, since I have no interest in that version (I too am after only the 1.24.1). Since the workaround is to locally regenerate the digest for the -9999,

Code: Select all

ebuild /usr/portage/sys-apps/busybox/busybox-9999.ebuild manifest

(which we have no intent to actually merge), is it reasonable to conclude that there is little risk in doing so? or would it be yet better to further mitigate risk by locally masking? like this:

Code: Select all

nano /etc/portage/package.mask
=sys-apps/busybox-9999

[NeddySeagoon]

brendlefly62,

Thats fine. Once the bug is fixed, normality will return and any manifests you have made in the portage tree will be validated or replaced at your next --sync.

That also means that if the tree still has an incorrect manifest, you manifest will be replaced with the incorrect one.

[quad]

Another workaround for now is to exclude that specific package in a larger update.

Code: Select all

emerge -vauND --exclude busybox @world

[patrix_neo]

I've gotten this for a couple of days now...

>>> Verifying ebuild manifests

!!! Digest verification failed:
!!! /usr/local/portage/sys-apps/busybox/busybox-1.24.1.ebuild
!!! Reason: Filesize does not match recorded size
!!! Got: 8577
!!! Expected: 8579

I usually make a refresh of the portage tree every second days. This one problem does not go away. What is happening here?

emerge --info
Portage 2.2.20.1 (python 3.4.3-final-0, default/linux/amd64/13.0/no-multilib, gcc-4.9.3, glibc-2.21-r1, 4.1.12-gentoo x86_64)
=================================================================
System uname: Linux-4.1.12-gentoo-x86_64-Intel-R-_Core-TM-_i5_CPU_650_@_3.20GHz-with-gentoo-2.2
KiB Mem: 8176304 total, 351432 free
KiB Swap: 0 total, 0 free
Timestamp of repository gentoo: Sun, 15 Nov 2015 01:00:02 +0000
sh bash 4.3_p39
ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1
app-shells/bash: 4.3_p39::gentoo
dev-lang/perl: 5.20.2::gentoo
dev-lang/python: 2.7.10::gentoo, 3.3.5-r1::gentoo, 3.4.3::gentoo
dev-util/cmake: 3.3.1-r1::gentoo
dev-util/pkgconfig: 0.28-r2::gentoo
sys-apps/baselayout: 2.2::gentoo
sys-apps/openrc: 0.17::gentoo
sys-apps/sandbox: 2.6-r1::gentoo
sys-devel/autoconf: 2.13::gentoo, 2.69::gentoo
sys-devel/automake: 1.11.6-r1::gentoo, 1.14.1::gentoo, 1.15::gentoo
sys-devel/binutils: 2.25.1-r1::gentoo
sys-devel/gcc: 4.9.3::gentoo
sys-devel/gcc-config: 1.7.3::gentoo
sys-devel/libtool: 2.4.6::gentoo
sys-devel/make: 4.1-r1::gentoo
sys-kernel/linux-headers: 3.18::gentoo (virtual/os-headers)
sys-libs/glibc: 2.21-r1::gentoo
Repositories:

gentoo
location: /usr/local/portage
sync-type: rsync
sync-uri: rsync://rsync.gentoo.org/gentoo-portage
masters: gentoo
priority: -1000

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/easy-rsa /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/games/angband/edit/ /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.6/ext-active/ /etc/php/cgi-php5.6/ext-active/ /etc/php/cli-php5.6/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://mirror.mdfnet.se/gentoo http://ftp.df.lth.se/pub/gentoo/ http://trumpetti.atm.tut.fi/gentoo/"
LANG="sv_SE.UTF8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j5"
PKGDIR="/usr/local/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/home/portage"
USE="X aac accessibility acl acpi alsa amd64 apache2 apng authlib auto-hinter autoipd avahi berkdb bzip2 cairo cdda cddb cgi chappa clamav clamd clamdtop cleartype cli consolekit cracklib crypt cups curl cxx dbus devhelp device-mapper directfb dri dvd dvdr encode exif expat extensions extras fat ffmpeg fontconfig fontforge fortran g3dvl gcj gd gdbm gdu gif gimp gles2 gnutls gtk gtk2 gudev hwdb iconv icu imagemagic imap inifile inotify ipv6 jack joystick jpeg jpeg2k keymap lastfm latin1 libextractor libmpeg2 libsamplerate lm_sensors lock lzo mad maildir matroska mbox mdadm mdnsresponder-compat mikmod milter minizip mjpeg mmx mmxext mod modules mp3 mpeg mplayer mysql mysqli ncurses new-login nls nptl nsplugin offensive ogg openal opengl openmp oss pam pcre pdf pdo perl png policykit postscript python qt3 qt4 quicktime razor readline resolvconf rpc sdl seccomp session smp snmp sockets spamassassin sqlite sqlite3 sse sse2 ssl stream subtitles taglib tcpd theora thumbnail thunar tiff truetype udev udisks unicode upower userlocales vcd vdpau vorbis vpx wayland x264 xattr xcomposite xfs xinerama xmlreader xscreensaver xvid xvmc zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="actions alias auth_basic auth_digest authn_anon authn_dbd authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock dbd deflate dir disk_cache env expires ext_filter file_cache filter headers ident imagemap include info log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif so speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev joystick" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="sv sv_SE fi en en_GB" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6" PYTHON_SINGLE_TARGET="python3_4" PYTHON_TARGETS="python2_7 python3_3 python3_4" RUBY_TARGETS="ruby20" USERLAND="GNU" VIDEO_CARDS="dummy nouveau nvidia v4l vesa vga" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
USE_PYTHON="3.4"
Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

patrix_neo,

Merged here as the busybox issue is due to the same problem that this thread is all about.
--
Neddyseagoon

[khayyam]

patrix_neo ...

there is already a thread open on this subject ... see [post=7841234]here[/post]

best ... khay

[yzg]

Same here

Code: Select all

!!! Digest verification failed:
!!! /usr/portage/sys-apps/busybox/busybox-9999.ebuild
!!! Reason: Filesize does not match recorded size
!!! Got: 8493
!!! Expected: 8580

[symon1980]

I did what quad suggested by excluding busybox in the upgrade and all is fine....

[StupidBunny]

I'm having the same problem with the same package. I'll try --exclude for the time being (I tried remaking the manifest as above, followed by a --sync, and that didn't work.)

[chr0nix]

Problem persists and is reproducible always:

Code: Select all

# /usr/bin/emerge -vauND @world

Code: Select all

>>> Verifying ebuild manifests

!!! Digest verification failed:
!!! /usr/portage/sys-apps/busybox/busybox-9999.ebuild
!!! Reason: Filesize does not match recorded size
!!! Got: 8493
!!! Expected: 8580

Workaround provided by quad confirmed working:

Code: Select all

emerge -vauND --exclude busybox @world

[MMMMM]

But should this ebuild file not corrected in portage tree anyway?

[dufeu]

I'm having the same problem with the same package. I'll try --exclude for the time being (I tried remaking the manifest as above, followed by a --sync, and that didn't work.)

--sync re-pulls the manifests in the portage tree.

What you did was correct the manifest locally - and then overwrite your correction by re-pulling the bad manifest from the portage tree.

The problem is that the manifest in the tree is wrong. The infrastructure team is aware of the problem and making a permanent solution to what ever caused the problem to begin with. The related 'changelog' problem appears to be resolved.

Baiscally, for now when you want to do an @world update - you need to do --sync and then either skip the package:

Code: Select all

emerge --exclude="sys-apps/buxybox" @world

or do your --sync and force correct your manifest locally then perform your @world update:

Code: Select all

ebuild /usr/portage/sys-apps/busybox/busybox-9999.ebuild manifest
emerge @world

You'll need to do this until the infrastructure team completes their fix.

I'd just go ahead with the force local manifest fix the first time so you can get the upgrade to busybox and then use the --exclude option until the portage tree is fixed.

And have patience. I'm certain the infrastructure team is as eager for a permanenet, correct fix so that the problem doesn't recur as everyone else.

But should this ebuild file not corrected in portage tree anyway?

Yes.

But since the problem shouldn't have occurred to begin with, the infrastructure team also needs to make sure the original cause can't recur.

I can't speak for the infrastucture team nor am I a gentoo developer or any other kind of developer. The team may have good reason not to correct the bad manifest record yet. I know they'll do it and I'm certain they want to do it right.

In the meantime, either of the work arounds above are pretty painless. It's just a bit annoying to remember to do them after every --sync. For now, I set up my --sync command like so:

Code: Select all

date && time emaint --auto sync && time eix-update && date && ebuild /usr/portage/sys-apps/busybox/busybox-9999.ebuild manifest

It's painless and I'm monitoring the situation unltil the issue is fixed. Then I'll drop rebuilding the busybox manifest after each --sync..