Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
what is tmp-unverified-download-quarantine directory?
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
yzg
Guru
Guru


Joined: 18 Jun 2005
Posts: 497

PostPosted: Thu Jul 19, 2018 10:30 am    Post subject: what is tmp-unverified-download-quarantine directory? Reply with quote

I noticed that emerge --sync command is doing a new behavior. It stops in the middle of updating and does the verification on /usr/portage/.tmp-unverified-download-quarantine and then continue.
The directory name has the two scariest words: unverified and quarantine inside portage directory.

Code:

total size is 218.90M  speedup is 19.96
 * Manifest timestamp: 2018-07-19 09:38:44 UTC
 * Valid OpenPGP signature found:
 * - primary key: DCD05B71EAB94199527F44ACDB6B8C1F96D8BF6D
 * - subkey: E1D6ABB63BFCFB4BA02FDF1CEC590EEAC9189250
 * - timestamp: 2018-07-19 09:38:44 UTC
 * Verifying /usr/portage/.tmp-unverified-download-quarantine ...                                            [ ok ]
sending incremental file list
Manifest
Manifest.files.gz
app-admin/Manifest.gz


Is this a new change?
Back to top
View user's profile Send private message
kurly
Apprentice
Apprentice


Joined: 02 Apr 2012
Posts: 260

PostPosted: Sun Jul 22, 2018 12:34 am    Post subject: Reply with quote

Yes, and there is a news item. It's nothing to be concerned about.
Back to top
View user's profile Send private message
ryszardzonk
Apprentice
Apprentice


Joined: 18 Dec 2003
Posts: 225
Location: Rzeszów, POLAND

PostPosted: Wed Jul 25, 2018 11:40 am    Post subject: Reply with quote

How about if it keeps saying Operation not permitted?

rsync: link "/mnt/4tb/portage/.tmp-unverified-download-quarantine/metadata/md5-cache/dev-ruby/rspec-expectations-3.7.0" => /mnt/4tb/portage/metadata/md5-cache/dev-ruby/rspec-expectations-3.7.0 failed: Operation not permitted (1)

Does not seem like the feature is properly used...
_________________
Sky is not the limit...
Back to top
View user's profile Send private message
Chiitoo
Administrator
Administrator


Joined: 28 Feb 2010
Posts: 2627
Location: Here and Away Again

PostPosted: Wed Jul 25, 2018 12:13 pm    Post subject: Reply with quote

ryszardzonk,

I got that too.

Also found this: Bug 661834 - =sys-apps/portage-2.3.43: synced files owned by root
_________________
Kindest of regardses.
Back to top
View user's profile Send private message
ryszardzonk
Apprentice
Apprentice


Joined: 18 Dec 2003
Posts: 225
Location: Rzeszów, POLAND

PostPosted: Wed Jul 25, 2018 1:42 pm    Post subject: Reply with quote

@Chiitoo

I am already at portage-2.3.43-r1 and still receive this error. Maybe it has to do with "-rsync-verify" flag I set some time ago when sync verification was first introduced. I'll check and if I find any think I sure will post it in in the bug report.
_________________
Sky is not the limit...
Back to top
View user's profile Send private message
Chiitoo
Administrator
Administrator


Joined: 28 Feb 2010
Posts: 2627
Location: Here and Away Again

PostPosted: Wed Jul 25, 2018 2:32 pm    Post subject: Reply with quote

Yeah, the bug is still open, only referenced by the commits there, so probably not fully fixed. :]
_________________
Kindest of regardses.
Back to top
View user's profile Send private message
Dragonlord
Guru
Guru


Joined: 22 Aug 2004
Posts: 446
Location: Switzerland

PostPosted: Mon Dec 31, 2018 2:37 am    Post subject: Reply with quote

What about this process taking hours and "NOT" finishing? Emerge is broken with this new action in place. Can it be disabled somehow?
_________________
DragonDreams: Leader and Head Programmer
Back to top
View user's profile Send private message
bunder
Bodhisattva
Bodhisattva


Joined: 10 Apr 2004
Posts: 5937

PostPosted: Mon Dec 31, 2018 3:09 am    Post subject: Reply with quote

https://www.gentoo.org/support/news-items/2018-01-30-portage-rsync-verification.html

i was adversed to this at first as well, but i turned it back on recently and haven't noticed much issues past the initial sync... i just synced now and it only took a minute give or take.
_________________
Neddyseagoon wrote:
The problem with leaving is that you can only do it once and it reduces your influence.

banned from #gentoo since sept 2017
Back to top
View user's profile Send private message
Dragonlord
Guru
Guru


Joined: 22 Aug 2004
Posts: 446
Location: Switzerland

PostPosted: Fri Jan 04, 2019 2:04 pm    Post subject: Reply with quote

This feature should be disabled by default. Sitting up to 2 hours through an emerge sync is NOT acceptable by any stretch of the imagination.
_________________
DragonDreams: Leader and Head Programmer
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54507
Location: 56N 3W

PostPosted: Fri Jan 04, 2019 2:20 pm    Post subject: Reply with quote

Dragonlord,

Hours?

Even a Raspberry Pi doesn't take hours.
This suggests you are using low end hardware or there is something wrong with your setup.

Your sync goes into /usr/portage/.tmp-unverified-download-quarantine until the cryptographic signatures have been verified.
This intermediate step ensures a broken sync does not leave you with an inconsistent tree, as used to happen.
Its a good thing. There is a USE flag to turn off this behaviour if you really want to but better to fix your issue.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Dragonlord
Guru
Guru


Joined: 22 Aug 2004
Posts: 446
Location: Switzerland

PostPosted: Fri Jan 04, 2019 3:37 pm    Post subject: Reply with quote

NeddySeagoon wrote:
Dragonlord,

Hours?

Even a Raspberry Pi doesn't take hours.
This suggests you are using low end hardware or there is something wrong with your setup.

Your sync goes into /usr/portage/.tmp-unverified-download-quarantine until the cryptographic signatures have been verified.
This intermediate step ensures a broken sync does not leave you with an inconsistent tree, as used to happen.
Its a good thing. There is a USE flag to turn off this behaviour if you really want to but better to fix your issue.

I'm doing high-end game deving. I'm certain my hardware is not low-tier. Without this feature enabled a sync take less than a minute. With this feature enabled it runs for ages.
_________________
DragonDreams: Leader and Head Programmer
Back to top
View user's profile Send private message
tholin
Apprentice
Apprentice


Joined: 04 Oct 2008
Posts: 204

PostPosted: Fri Jan 04, 2019 5:38 pm    Post subject: Reply with quote

If rsync-verify gets stuck it is probably trying to refresh the keys and fail for some reason. Assuming the portage repo is at /usr/portage try running "gemato verify -K /usr/share/openpgp-keys/gentoo-release.asc /usr/portage" and check the output for where it gets stuck.
Back to top
View user's profile Send private message
Dragonlord
Guru
Guru


Joined: 22 Aug 2004
Posts: 446
Location: Switzerland

PostPosted: Sat Jan 05, 2019 3:25 pm    Post subject: Reply with quote

I did some updates recently (which included a forced portage update) and the timing did improve but is in my opinion still too high. Here the command from you run today after the latest update:
Code:
gemato verify -K /usr/share/openpgp-keys/gentoo-release.asc /usr/portage  2>&1 | ts '[%H:%M:%S]'
[16:07:03] INFO:root:Refreshing keys...
[16:07:04] INFO:root:Keys refreshed.
[16:07:04] INFO:root:Manifest timestamp: 2019-01-04 13:38:40 UTC
[16:07:04] INFO:root:Valid OpenPGP signature found:
[16:07:04] INFO:root:- primary key: DCD05B71EAB94199527F44ACDB6B8C1F96D8BF6D
[16:07:04] INFO:root:- subkey: E1D6ABB63BFCFB4BA02FDF1CEC590EEAC9189250
[16:07:04] INFO:root:- timestamp: 2019-01-04 13:38:40 UTC
[16:07:04] INFO:root:Verifying /usr/portage...
[16:21:12] INFO:root:/usr/portage verified in 847.86 seconds


That's nearly a quarter hour. I will keep this feature disabled as this is still too high.
_________________
DragonDreams: Leader and Head Programmer
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54507
Location: 56N 3W

PostPosted: Sat Jan 05, 2019 3:33 pm    Post subject: Reply with quote

Dragonlord,

That command verified the entire tree.
During updates, its only the subset that is fetched that is checked.

You don't have the portage repo in tmpfs I suppose, so the whole thing is fetched every time?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Dragonlord
Guru
Guru


Joined: 22 Aug 2004
Posts: 446
Location: Switzerland

PostPosted: Sat Jan 05, 2019 9:38 pm    Post subject: Reply with quote

NeddySeagoon wrote:
Dragonlord,

That command verified the entire tree.
During updates, its only the subset that is fetched that is checked.

You don't have the portage repo in tmpfs I suppose, so the whole thing is fetched every time?

Portage never had been in tmpfs as far as I know. The tree is updated with rsync so that does already partial updates. And no, the test time is always the same if I run the command of yours multiple times in a row without touching the repository.
_________________
DragonDreams: Leader and Head Programmer
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54507
Location: 56N 3W

PostPosted: Sat Jan 05, 2019 10:30 pm    Post subject: Reply with quote

Dragonlord,

The
Code:
gemato verify -K /usr/share/openpgp-keys/gentoo-release.asc /usr/portage  2>&1 | ts '[%H:%M:%S]'
command checks the entire repository, every time.
To be able to reproduce this, I had to install moreutils, to get the ts command, so I don't think I gave you the command above.

On my Gen 7 HP Microserver, with a Neo N36L Dual-Core Processor, which is fairly low end hardware, and tree timestamp: 2018-11-23 02:08:47 UTC I get.
Code:
Eccles_2 ~ # gemato verify -K /usr/share/openpgp-keys/gentoo-release.asc /usr/portage  2>&1 | ts '[%H:%M:%S]'
[22:13:01] INFO:root:Refreshing keys...
[22:13:04] INFO:root:Keys refreshed.
[22:13:04] INFO:root:Manifest timestamp: 2018-11-23 02:08:47 UTC
[22:13:04] INFO:root:Valid OpenPGP signature found:
[22:13:04] INFO:root:- primary key: DCD05B71EAB94199527F44ACDB6B8C1F96D8BF6D
[22:13:04] INFO:root:- subkey: E1D6ABB63BFCFB4BA02FDF1CEC590EEAC9189250
[22:13:04] INFO:root:- timestamp: 2018-11-23 02:08:47 UTC
[22:13:04] INFO:root:Verifying /usr/portage...
[22:17:11] INFO:root:/usr/portage verified in 247.06 seconds


With emerge --sync
Code:
[22:19:23] sent 513.39K bytes  received 39.79M bytes  848.50K bytes/sec
[22:19:23] total size is 220.29M  speedup is 5.47
[22:19:23]  * Manifest timestamp: 2019-01-05 02:08:38 UTC
[22:19:23]  * Valid OpenPGP signature found:
[22:19:23]  * - primary key: DCD05B71EAB94199527F44ACDB6B8C1F96D8BF6D
[22:19:23]  * - subkey: E1D6ABB63BFCFB4BA02FDF1CEC590EEAC9189250
[22:19:23]  * - timestamp: 2019-01-05 02:08:38 UTC
[22:20:52]  * Verifying /usr/portage/.tmp-unverified-download-quarantine ...        [ ok ]
[22:21:23] === Sync completed for gentoo
[22:21:23] q: Updating ebuild cache for /usr/portage ...
[22:21:23] q: Finished 36211 entries in 0.468768 seconds

The verify time was 31 seconds for the delta (about 6 week) vs 247.06 seconds for the entire (old) tree.

What else is your system doing at the same time?
What CPU governor do you do have in use?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
asaparov
n00b
n00b


Joined: 04 Nov 2016
Posts: 7

PostPosted: Fri May 24, 2019 11:10 pm    Post subject: Reply with quote

I've noticed emerge --sync on one of my machines takes a lot longer than the others, and almost all of the time is spent in Verifying /usr/portage/.tmp-unverified-download-quarantine.
It takes about a couple minutes to finish this step, whereas running
Code:
gemato verify -K /usr/share/openpgp-keys/gentoo-release.asc /usr/portage
finishes in 26 seconds. Shouldn't emerge --sync only verify a small subset of the tree? If so, why would it be taking so much longer? My other machines go through this step fairly quickly.

I'm using the performance governor.

emerge --info: https://pastebin.com/WaavzvzK
Back to top
View user's profile Send private message
j.hauser
n00b
n00b


Joined: 21 Aug 2019
Posts: 3

PostPosted: Wed Aug 21, 2019 6:24 pm    Post subject: Reply with quote

I'm having the same problem as reported by asaparov: Portage verification via emerge --sync is very slow, calling gemato directly is much faster (roughly 14 minutes vs. 1 minute). I've noticed that emerge --sync causes almost no CPU load during verification while iotop reports a DISK READ around 500-700 k/s caused by emerge --sync (no other processes causing any noticeable DISK READ). On the other hand,
Code:
gemato verify -K /usr/share/openpgp-keys/gentoo-release.asc /usr/portage

causes not DISK READ and a high CPU load on one core.

I've explicitly set
Code:
sync-allow-hardlinks = yes
sync-rsync-verify-metamanifest = yes

in the file /etc/portage/repos.conf/gentoo.conf.

I assume that there must be something wrong with my/our setup(s).
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany

PostPosted: Wed Aug 21, 2019 6:43 pm    Post subject: Reply with quote

Quote:
I assume that there must be something wrong with my/our setup(s).

No! Most probably, nothing is wrong with your setup!

There is something wrong with the developers and the algorithms they choose.

'emerge --sync' creates a subdirectory '/usr/portage/.tmp-unverified-download-quarantine'. After that, it creates 140.000 (one-hundred-forty-thousand) links in it. And everybody wonders why "emerge --sync" gets slower and slower... On my desktop machine (/usr/portage is mounted via NFS), that slows down 'emerge -sync' from 30 seconds to over 8 minutes!!!

That's ridiculous!
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 6135
Location: Dallas area

PostPosted: Wed Aug 21, 2019 7:35 pm    Post subject: Reply with quote

set sync-allow-hardlinks = no to do away with .tmp-unverified-download-quarantine nonsense
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland
Back to top
View user's profile Send private message
skellr
l33t
l33t


Joined: 18 Jun 2005
Posts: 977
Location: The Village, Portmeirion

PostPosted: Wed Aug 21, 2019 7:55 pm    Post subject: Reply with quote

Les go back to where rsync would partially update the repository and not throw an error. You could just be free to update whatever and not be inconvenienced. No need to find out what went wrong.

Yeah, ignorance is bliss.
Back to top
View user's profile Send private message
Anon-E-moose
Watchman
Watchman


Joined: 23 May 2008
Posts: 6135
Location: Dallas area

PostPosted: Wed Aug 21, 2019 8:24 pm    Post subject: Reply with quote

sync-allow-hardlinks = no doesn't do away with verification, it simply stops the .tmp-unverified-download-quarantine dir and all the hardlinks within it, from being created, in other words it behaves the way emerge/rsync used to behave
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland
Back to top
View user's profile Send private message
skellr
l33t
l33t


Joined: 18 Jun 2005
Posts: 977
Location: The Village, Portmeirion

PostPosted: Wed Aug 21, 2019 8:38 pm    Post subject: Reply with quote

Anon-E-moose wrote:
sync-allow-hardlinks = no doesn't do away with verification, it simply stops the .tmp-unverified-download-quarantine dir from being created, in other words it behaves the way emerge/rsync used to behave

It was just a comment on the thread as a whole. I did post right after you but, it wasn't meant explicitly for you. Seems like people were complaining about an inconvenience without understanding why it was happening, or why the implementation was implemented. Just a bunch of whining going on that gave the urge to slap someone. :)
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany

PostPosted: Wed Aug 21, 2019 8:48 pm    Post subject: Reply with quote

Quote:
Seems like people were complaining about an inconvenience without understanding why it was happening,

I do understand what developers tried to achieve.

There are better ways to do this - and there's no need to create 140.000 hard links in /usr/portage/.

Developers just chose the wrong tools and algorithms. It would be much better and faster to do this in memory or on a tmpfs.
Back to top
View user's profile Send private message
skellr
l33t
l33t


Joined: 18 Jun 2005
Posts: 977
Location: The Village, Portmeirion

PostPosted: Wed Aug 21, 2019 9:07 pm    Post subject: Reply with quote

mike155 wrote:
Quote:
Seems like people were complaining about an inconvenience without understanding why it was happening,

I do understand what developers tried to achieve.

There are better ways to do this - and there's no need to create 140.000 hard links in /usr/portage/.

Developers just chose the wrong tools and algorithms. It would be much better and faster to do this in memory or on a tmpfs.

Not everyone had that much to spare, some old pentium desktop, rasbery pi, or whatever embeded device... Not that easy.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum