| View previous topic :: View next topic |
| Author |
Message |
yzg
Guru
Joined: 18 Jun 2005
Posts: 493
|
 Posted: Thu Jul 19, 2018 10:30 am Post subject: what is tmp-unverified-download-quarantine directory? |
 |
|
I noticed that emerge --sync command is doing a new behavior. It stops in the middle of updating and does the verification on /usr/portage/.tmp-unverified-download-quarantine and then continue.
The directory name has the two scariest words: unverified and quarantine inside portage directory.
| Code: |
total size is 218.90M speedup is 19.96
* Manifest timestamp: 2018-07-19 09:38:44 UTC
* Valid OpenPGP signature found:
* - primary key: DCD05B71EAB94199527F44ACDB6B8C1F96D8BF6D
* - subkey: E1D6ABB63BFCFB4BA02FDF1CEC590EEAC9189250
* - timestamp: 2018-07-19 09:38:44 UTC
* Verifying /usr/portage/.tmp-unverified-download-quarantine ... [ ok ]
sending incremental file list
Manifest
Manifest.files.gz
app-admin/Manifest.gz
|
Is this a new change? |
|
| Back to top |
|
 |
kurly
Apprentice
Joined: 02 Apr 2012
Posts: 260
|
| Posted: Sun Jul 22, 2018 12:34 am Post subject: |
|
|
| Yes, and there is a news item. It's nothing to be concerned about. |
|
| Back to top |
|
|
ryszardzonk
Apprentice
Joined: 18 Dec 2003
Posts: 225
Location: Rzeszów, POLAND
|
| Posted: Wed Jul 25, 2018 11:40 am Post subject: |
|
|
How about if it keeps saying Operation not permitted?
rsync: link "/mnt/4tb/portage/.tmp-unverified-download-quarantine/metadata/md5-cache/dev-ruby/rspec-expectations-3.7.0" => /mnt/4tb/portage/metadata/md5-cache/dev-ruby/rspec-expectations-3.7.0 failed: Operation not permitted (1)
Does not seem like the feature is properly used...
_________________
Sky is not the limit... |
|
| Back to top |
|
|
Chiitoo
Administrator
Joined: 28 Feb 2010
Posts: 2571
Location: Here and Away Again
|
|
| Back to top |
|
|
ryszardzonk
Apprentice
Joined: 18 Dec 2003
Posts: 225
Location: Rzeszów, POLAND
|
| Posted: Wed Jul 25, 2018 1:42 pm Post subject: |
|
|
@Chiitoo
I am already at portage-2.3.43-r1 and still receive this error. Maybe it has to do with "-rsync-verify" flag I set some time ago when sync verification was first introduced. I'll check and if I find any think I sure will post it in in the bug report.
_________________
Sky is not the limit... |
|
| Back to top |
|
|
Chiitoo
Administrator
Joined: 28 Feb 2010
Posts: 2571
Location: Here and Away Again
|
| Posted: Wed Jul 25, 2018 2:32 pm Post subject: |
|
|
Yeah, the bug is still open, only referenced by the commits there, so probably not fully fixed. :]
_________________
Kindest of regardses. |
|
| Back to top |
|
|
Dragonlord
Guru
Joined: 22 Aug 2004
Posts: 446
Location: Switzerland
|
| Posted: Mon Dec 31, 2018 2:37 am Post subject: |
|
|
What about this process taking hours and "NOT" finishing? Emerge is broken with this new action in place. Can it be disabled somehow?
_________________
DragonDreams: Leader and Head Programmer |
|
| Back to top |
|
|
bunder
Bodhisattva
Joined: 10 Apr 2004
Posts: 5934
|
| Posted: Mon Dec 31, 2018 3:09 am Post subject: |
|
|
https://www.gentoo.org/support/news-items/2018-01-30-portage-rsync-verification.html
i was adversed to this at first as well, but i turned it back on recently and haven't noticed much issues past the initial sync... i just synced now and it only took a minute give or take.
_________________
| Neddyseagoon wrote: | | The problem with leaving is that you can only do it once and it reduces your influence. |
banned from #gentoo since sept 2017 |
|
| Back to top |
|
|
Dragonlord
Guru
Joined: 22 Aug 2004
Posts: 446
Location: Switzerland
|
| Posted: Fri Jan 04, 2019 2:04 pm Post subject: |
|
|
This feature should be disabled by default. Sitting up to 2 hours through an emerge sync is NOT acceptable by any stretch of the imagination.
_________________
DragonDreams: Leader and Head Programmer |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54219
Location: 56N 3W
|
| Posted: Fri Jan 04, 2019 2:20 pm Post subject: |
|
|
Dragonlord,
Hours?
Even a Raspberry Pi doesn't take hours.
This suggests you are using low end hardware or there is something wrong with your setup.
Your sync goes into /usr/portage/.tmp-unverified-download-quarantine until the cryptographic signatures have been verified.
This intermediate step ensures a broken sync does not leave you with an inconsistent tree, as used to happen.
Its a good thing. There is a USE flag to turn off this behaviour if you really want to but better to fix your issue.
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Dragonlord
Guru
Joined: 22 Aug 2004
Posts: 446
Location: Switzerland
|
| Posted: Fri Jan 04, 2019 3:37 pm Post subject: |
|
|
| NeddySeagoon wrote: | Dragonlord,
Hours?
Even a Raspberry Pi doesn't take hours.
This suggests you are using low end hardware or there is something wrong with your setup.
Your sync goes into /usr/portage/.tmp-unverified-download-quarantine until the cryptographic signatures have been verified.
This intermediate step ensures a broken sync does not leave you with an inconsistent tree, as used to happen.
Its a good thing. There is a USE flag to turn off this behaviour if you really want to but better to fix your issue. |
I'm doing high-end game deving. I'm certain my hardware is not low-tier. Without this feature enabled a sync take less than a minute. With this feature enabled it runs for ages.
_________________
DragonDreams: Leader and Head Programmer |
|
| Back to top |
|
|
tholin
Apprentice
Joined: 04 Oct 2008
Posts: 203
|
| Posted: Fri Jan 04, 2019 5:38 pm Post subject: |
|
|
| If rsync-verify gets stuck it is probably trying to refresh the keys and fail for some reason. Assuming the portage repo is at /usr/portage try running "gemato verify -K /usr/share/openpgp-keys/gentoo-release.asc /usr/portage" and check the output for where it gets stuck. |
|
| Back to top |
|
|
Dragonlord
Guru
Joined: 22 Aug 2004
Posts: 446
Location: Switzerland
|
| Posted: Sat Jan 05, 2019 3:25 pm Post subject: |
|
|
I did some updates recently (which included a forced portage update) and the timing did improve but is in my opinion still too high. Here the command from you run today after the latest update:
| Code: | gemato verify -K /usr/share/openpgp-keys/gentoo-release.asc /usr/portage 2>&1 | ts '[%H:%M:%S]'
[16:07:03] INFO:root:Refreshing keys...
[16:07:04] INFO:root:Keys refreshed.
[16:07:04] INFO:root:Manifest timestamp: 2019-01-04 13:38:40 UTC
[16:07:04] INFO:root:Valid OpenPGP signature found:
[16:07:04] INFO:root:- primary key: DCD05B71EAB94199527F44ACDB6B8C1F96D8BF6D
[16:07:04] INFO:root:- subkey: E1D6ABB63BFCFB4BA02FDF1CEC590EEAC9189250
[16:07:04] INFO:root:- timestamp: 2019-01-04 13:38:40 UTC
[16:07:04] INFO:root:Verifying /usr/portage...
[16:21:12] INFO:root:/usr/portage verified in 847.86 seconds |
That's nearly a quarter hour. I will keep this feature disabled as this is still too high.
_________________
DragonDreams: Leader and Head Programmer |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54219
Location: 56N 3W
|
| Posted: Sat Jan 05, 2019 3:33 pm Post subject: |
|
|
Dragonlord,
That command verified the entire tree.
During updates, its only the subset that is fetched that is checked.
You don't have the portage repo in tmpfs I suppose, so the whole thing is fetched every time?
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Dragonlord
Guru
Joined: 22 Aug 2004
Posts: 446
Location: Switzerland
|
| Posted: Sat Jan 05, 2019 9:38 pm Post subject: |
|
|
| NeddySeagoon wrote: | Dragonlord,
That command verified the entire tree.
During updates, its only the subset that is fetched that is checked.
You don't have the portage repo in tmpfs I suppose, so the whole thing is fetched every time? |
Portage never had been in tmpfs as far as I know. The tree is updated with rsync so that does already partial updates. And no, the test time is always the same if I run the command of yours multiple times in a row without touching the repository.
_________________
DragonDreams: Leader and Head Programmer |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54219
Location: 56N 3W
|
| Posted: Sat Jan 05, 2019 10:30 pm Post subject: |
|
|
Dragonlord,
The | Code: | | gemato verify -K /usr/share/openpgp-keys/gentoo-release.asc /usr/portage 2>&1 | ts '[%H:%M:%S]' |
command checks the entire repository, every time.
To be able to reproduce this, I had to install moreutils, to get the ts command, so I don't think I gave you the command above.
On my Gen 7 HP Microserver, with a Neo N36L Dual-Core Processor, which is fairly low end hardware, and tree timestamp: 2018-11-23 02:08:47 UTC I get.
| Code: | Eccles_2 ~ # gemato verify -K /usr/share/openpgp-keys/gentoo-release.asc /usr/portage 2>&1 | ts '[%H:%M:%S]'
[22:13:01] INFO:root:Refreshing keys...
[22:13:04] INFO:root:Keys refreshed.
[22:13:04] INFO:root:Manifest timestamp: 2018-11-23 02:08:47 UTC
[22:13:04] INFO:root:Valid OpenPGP signature found:
[22:13:04] INFO:root:- primary key: DCD05B71EAB94199527F44ACDB6B8C1F96D8BF6D
[22:13:04] INFO:root:- subkey: E1D6ABB63BFCFB4BA02FDF1CEC590EEAC9189250
[22:13:04] INFO:root:- timestamp: 2018-11-23 02:08:47 UTC
[22:13:04] INFO:root:Verifying /usr/portage...
[22:17:11] INFO:root:/usr/portage verified in 247.06 seconds |
With emerge --sync
| Code: | [22:19:23] sent 513.39K bytes received 39.79M bytes 848.50K bytes/sec
[22:19:23] total size is 220.29M speedup is 5.47
[22:19:23] * Manifest timestamp: 2019-01-05 02:08:38 UTC
[22:19:23] * Valid OpenPGP signature found:
[22:19:23] * - primary key: DCD05B71EAB94199527F44ACDB6B8C1F96D8BF6D
[22:19:23] * - subkey: E1D6ABB63BFCFB4BA02FDF1CEC590EEAC9189250
[22:19:23] * - timestamp: 2019-01-05 02:08:38 UTC
[22:20:52] * Verifying /usr/portage/.tmp-unverified-download-quarantine ... [ ok ]
[22:21:23] === Sync completed for gentoo
[22:21:23] q: Updating ebuild cache for /usr/portage ...
[22:21:23] q: Finished 36211 entries in 0.468768 seconds
|
The verify time was 31 seconds for the delta (about 6 week) vs 247.06 seconds for the entire (old) tree.
What else is your system doing at the same time?
What CPU governor do you do have in use?
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
asaparov
n00b
Joined: 04 Nov 2016
Posts: 7
|
| Posted: Fri May 24, 2019 11:10 pm Post subject: |
|
|
I've noticed emerge --sync on one of my machines takes a lot longer than the others, and almost all of the time is spent in Verifying /usr/portage/.tmp-unverified-download-quarantine.
It takes about a couple minutes to finish this step, whereas running | Code: | | gemato verify -K /usr/share/openpgp-keys/gentoo-release.asc /usr/portage |
finishes in 26 seconds. Shouldn't emerge --sync only verify a small subset of the tree? If so, why would it be taking so much longer? My other machines go through this step fairly quickly.
I'm using the performance governor.
emerge --info: https://pastebin.com/WaavzvzK |
|
| Back to top |
|
|
j.hauser
n00b
Joined: 21 Aug 2019
Posts: 3
|
| Posted: Wed Aug 21, 2019 6:24 pm Post subject: |
|
|
I'm having the same problem as reported by asaparov: Portage verification via emerge --sync is very slow, calling gemato directly is much faster (roughly 14 minutes vs. 1 minute). I've noticed that emerge --sync causes almost no CPU load during verification while iotop reports a DISK READ around 500-700 k/s caused by emerge --sync (no other processes causing any noticeable DISK READ). On the other hand,
| Code: | | gemato verify -K /usr/share/openpgp-keys/gentoo-release.asc /usr/portage |
causes not DISK READ and a high CPU load on one core.
I've explicitly set
| Code: | sync-allow-hardlinks = yes
sync-rsync-verify-metamanifest = yes |
in the file /etc/portage/repos.conf/gentoo.conf.
I assume that there must be something wrong with my/our setup(s). |
|
| Back to top |
|
|
mike155
Advocate
Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany
|
| Posted: Wed Aug 21, 2019 6:43 pm Post subject: |
|
|
| Quote: | | I assume that there must be something wrong with my/our setup(s). |
No! Most probably, nothing is wrong with your setup!
There is something wrong with the developers and the algorithms they choose.
'emerge --sync' creates a subdirectory '/usr/portage/.tmp-unverified-download-quarantine'. After that, it creates 140.000 (one-hundred-forty-thousand) links in it. And everybody wonders why "emerge --sync" gets slower and slower... On my desktop machine (/usr/portage is mounted via NFS), that slows down 'emerge -sync' from 30 seconds to over 8 minutes!!!
That's ridiculous! |
|
| Back to top |
|
|
Anon-E-moose
Watchman
Joined: 23 May 2008
Posts: 6097
Location: Dallas area
|
| Posted: Wed Aug 21, 2019 7:35 pm Post subject: |
|
|
set sync-allow-hardlinks = no to do away with .tmp-unverified-download-quarantine nonsense
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland |
|
| Back to top |
|
|
skellr
l33t
Joined: 18 Jun 2005
Posts: 975
Location: The Village, Portmeirion
|
| Posted: Wed Aug 21, 2019 7:55 pm Post subject: |
|
|
Les go back to where rsync would partially update the repository and not throw an error. You could just be free to update whatever and not be inconvenienced. No need to find out what went wrong.
Yeah, ignorance is bliss. |
|
| Back to top |
|
|
Anon-E-moose
Watchman
Joined: 23 May 2008
Posts: 6097
Location: Dallas area
|
| Posted: Wed Aug 21, 2019 8:24 pm Post subject: |
|
|
sync-allow-hardlinks = no doesn't do away with verification, it simply stops the .tmp-unverified-download-quarantine dir and all the hardlinks within it, from being created, in other words it behaves the way emerge/rsync used to behave
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland |
|
| Back to top |
|
|
skellr
l33t
Joined: 18 Jun 2005
Posts: 975
Location: The Village, Portmeirion
|
| Posted: Wed Aug 21, 2019 8:38 pm Post subject: |
|
|
| Anon-E-moose wrote: | | sync-allow-hardlinks = no doesn't do away with verification, it simply stops the .tmp-unverified-download-quarantine dir from being created, in other words it behaves the way emerge/rsync used to behave |
It was just a comment on the thread as a whole. I did post right after you but, it wasn't meant explicitly for you. Seems like people were complaining about an inconvenience without understanding why it was happening, or why the implementation was implemented. Just a bunch of whining going on that gave the urge to slap someone.  |
|
| Back to top |
|
|
mike155
Advocate
Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany
|
| Posted: Wed Aug 21, 2019 8:48 pm Post subject: |
|
|
| Quote: | | Seems like people were complaining about an inconvenience without understanding why it was happening, |
I do understand what developers tried to achieve.
There are better ways to do this - and there's no need to create 140.000 hard links in /usr/portage/.
Developers just chose the wrong tools and algorithms. It would be much better and faster to do this in memory or on a tmpfs. |
|
| Back to top |
|
|
skellr
l33t
Joined: 18 Jun 2005
Posts: 975
Location: The Village, Portmeirion
|
| Posted: Wed Aug 21, 2019 9:07 pm Post subject: |
|
|
| mike155 wrote: | | Quote: | | Seems like people were complaining about an inconvenience without understanding why it was happening, |
I do understand what developers tried to achieve.
There are better ways to do this - and there's no need to create 140.000 hard links in /usr/portage/.
Developers just chose the wrong tools and algorithms. It would be much better and faster to do this in memory or on a tmpfs. |
Not everyone had that much to spare, some old pentium desktop, rasbery pi, or whatever embeded device... Not that easy. |
|
| Back to top |
|
|
|
Portage & Programming
