[SOLVED] Slow to Verify Portage Unverified-download-quaranti

Kellerkalt · n00b Joined: 22 Dec 2020 Posts: 36

I've noticed that portage takes a very long to verify the unverified download quarantine after an "emerge --sync" on my i7-5820K @ 3.30GHz computer, but portage performs the same task much much faster on an old laptop with a Core2 Duo CPU T7250 @ 2.00GHz. I haven't timed it, but it is very clear that the old laptop is more than twice as fast as the i7 at this task.

eccerr0r · Posted: Sun Apr 18, 2021 7:04 pm Post subject:

Is the process CPU (R) or I/O (D) bound?
Perhaps fragmentation is bad if you have rust spinners?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

mike155 · Posted: Sun Apr 18, 2021 7:45 pm Post subject:

Is /var/db/repos/gentoo on a local disk? Or is it on an NFS or SAMBA share?

Kellerkalt · n00b Joined: 22 Dec 2020 Posts: 36

eccerr0r - Neither the process CPU or I/O are bound or adjusted in any way at all.

mike155 - /var/db/repos/gentoo is on the / (root) partition for each computer.

It's worth noting that I also run a Gentoo lab KVM VM on the i7 computer and the process takes longer in that VM on the i7 host than it does on the Core2 Duo as well and it's been allocated 6 cores as well. I'm not running "emerge --sync" at the same time on the i7 host and the VM at the same time though.

eccerr0r · Posted: Sun Apr 18, 2021 11:57 pm Post subject:

I didn't ask whether they were throttled, rather was the process using CPU at 100% ("cpu bound") or waiting for disk ("i/o bound").

Also the check is single threaded last I checked, and thus core counts don't matter. Need raw CPU single thread performance.

KVM incurs CPU penalty on i/o operations.

So you have a Core2 Duo with 6 cores? If you do, you don't have a core2 duo. If you don't, this is why English sucks.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Kellerkalt · n00b Joined: 22 Dec 2020 Posts: 36

Sorry eccerr0r, I misunderstood. The i7 isn't doing too much else at the same time and I/O overhead isn't very heavy, but I didn't check to see if that particular thread is hitting 100% but I guess that is possible for the core being used for the single core check. The Core2 Duo is just a standard old 2 core T7250 CPU.

eccerr0r · Posted: Mon Apr 19, 2021 2:34 am Post subject:

T7250 is 2GHz 2C2T for reference.

Yes this should be slower for all intents and purposes unless you were running instruction level (versus processor virtualization) emulation on the i7. I don't even think instruction optimization would make much of a difference either. However disk i/o through the emulator could be very expensive and that may explain the difference.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Kellerkalt · n00b Joined: 22 Dec 2020 Posts: 36

Thanks for the insight eccerr0r. I guess I'll chalk it up to a win at the finish line for the old Core2 Duo!

Juippisi · Developer Joined: 30 Sep 2005 Posts: 727 Location: /home

Swith to git syncing,

eccerr0r · Posted: Mon Apr 19, 2021 7:03 am Post subject:

hold on, before giving the c2d a win, do some disk i/o benchmarks on both systems. if the i7 still comes out on top, then there's still something wrong...
I don't have much experience with vm's running on my i7, but definitely notice a bit of disk speed penalty on the VM's running on my core2 machine. I still don't think it should totally explain why it's horribly slow, however...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

Kellerkalt · n00b Joined: 22 Dec 2020 Posts: 36

Juippisi - I'm rsyncing only one machine on the network externally and then the rest of them are rsycning from that machine as the local mirror. I get added to the ban list If I don't do this. Thanks for the git mirror though, it's nice to have another external mirror to point to and I wasn't aware of this one.

eccerr0r - I'll test the disk i/o, but I failed to mention that the Core2Duo laptop has an SSD and the i7 is a standard platter hard drive (and only 7200 RPM) at that. Maybe that would explain the difference in speed?

mike155 · Posted: Mon Apr 19, 2021 2:24 pm Post subject:

You could turn off tree verification. That's what I do to speed up "emerge --sync":

See: https://forums.gentoo.org/viewtopic-p-8358476.html#8358476

gabrielg · Tux's lil' helper Joined: 16 Nov 2012 Posts: 134

As it was suggested, I would try to defragment /var/db/repos and see if performance improves.
While by no means I am recommending to rethink your fs, I can tell you that my verification speed went down considerably when I migrated from a RAID 10 md ext4 to a RAID 10 btrfs one, perhaps partly because of lack of fragmentation and some speed gains due to compression (so, more data for less IO).
Ultimately, if this bothers you too much, you could switch to git/turn off verification, as it was suggested as well.

jesnow · l33t Joined: 26 Apr 2006 Posts: 857

Is git syncing that much faster than rsync?

mike155 · Posted: Mon Apr 19, 2021 4:15 pm Post subject:

eccerr0r · Posted: Mon Apr 19, 2021 4:31 pm Post subject:

Hu · Administrator Joined: 06 Mar 2007 Posts: 21703

Although it would add complexity, if you have the free memory for it, you might get a major boost by changing your sync process to be:

Copy /usr/portage into a tmpfs.
Sync and perform tree verification with that tmpfs as your $PORTDIR.
On successful sync, rsync that tmpfs back to your persistent storage.

This will allow all the hardlinking to be done in a tmpfs, which should be quite cheap.

Goverp · Advocate Joined: 07 Mar 2007 Posts: 2014

Another approach: use the portage squashfs snapshots. There's only one file with a single sha512sum, and a single GPG signature to verify. Conversely, only one update a day, and it's not part of emerge --sync / emaint sync.
_________________
Greybeard