Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
What happened to "emerge --sync" (slooooow)?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
curmudgeon
Veteran
Veteran


Joined: 08 Aug 2003
Posts: 1543

PostPosted: Wed Jan 16, 2019 2:05 am    Post subject: What happened to "emerge --sync" (slooooow)? Reply with quote

I did search for this, and it seems strange that I can't find anything. Up until a few weeks ago, whenever I did "emerge --sync", the machine would quickly get started. I sync just one machine to the gentoo infrastructure, and all of the other machines to the local repository on that machine. The other machines would frequently sync completely in less than ten seconds.

Now whenever I "emerge --sync" (even on the other machines), the initial few lines pop up right away (looks like a sync of the repository timestamp to see if there is a need to sync the rest of the tree), and then there is a pause of about a minute before the list of files updated begins.

Unless there are a lot of new under the surface complexities, it appears that this is a deliberate pause (to combat network abuse?). Can anyone explain this, and is there someway to lessen or remove it (particularly on the local machines)?


Last edited by curmudgeon on Thu Jan 24, 2019 10:09 pm; edited 2 times in total
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7130
Location: almost Mile High in the USA

PostPosted: Wed Jan 16, 2019 3:22 am    Post subject: Reply with quote

You can stop updating the gpg key by adding
Code:
sync-rsync-verify-metamanifest = no

in your repos.conf ... if that's what it's stopping on. This of course will no longer let portage check the authenticity of the portage tree (which should be OK if you checked your main copy).

I should submit a different topic, but I find that if I try to sync over my VPN, it will fail to ever get the updated key for gpg... and emerge --sync always fails because it can't get the key...
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
curmudgeon
Veteran
Veteran


Joined: 08 Aug 2003
Posts: 1543

PostPosted: Wed Jan 16, 2019 4:50 am    Post subject: Reply with quote

Thanks for the explanation. I guess I can leave that on the the machine that syncs to the gentoo infrastructure, but it is complete overkill for the internal network (if the repository on the internal server gets altered after the sync to the outside world, I have bigger problems than the check was designed to detect).
Back to top
View user's profile Send private message
curmudgeon
Veteran
Veteran


Joined: 08 Aug 2003
Posts: 1543

PostPosted: Thu Jan 24, 2019 10:09 pm    Post subject: Reply with quote

Sorry, that is not the problem. I added that line to /etc/portage/repos.conf/gentoo.conf, and it made no difference. One of the machines just paused 150 seconds between the first part of the sync ("127 bytes transferred") and the second part.

Do you have any other possible cause?
Back to top
View user's profile Send private message
mike155
Veteran
Veteran


Joined: 17 Sep 2010
Posts: 1294
Location: Frankfurt, Germany

PostPosted: Thu Jan 24, 2019 10:46 pm    Post subject: Reply with quote

Below is the execution of 'emerge --sync' on my NFS server as well as on my NFS client:

Code:
+--------------------------------------+-------------+-------------+
|                                      |  NFS server |  NFS client |
+--------------------------------------+-------------+-------------+
| execution time of 'emerge --sync'    |  13 seconds | 598 seconds |
+--------------------------------------+-------------+-------------+

Why is 'emerge --sync' so much slower on my NFS client than on my NFS server??? I analyzed 'emerge --sync' and saw that 'emerge --sync' calls 'rsync' to create and fill a directory '/usr/portage/.tmp-unverified-download-quarantine':

Code:
rsync -a \
    --link-dest /usr/portage \
    --exclude=/distfiles \
    --exclude=/local \
    --exclude=/lost+found \
    --exclude=/packages \
    --exclude /.tmp-unverified-download-quarantine \
    /usr/portage/ \
    /usr/portage/.tmp-unverified-download-quarantine/

That rsync call takes 3 seconds on my NFS server, but more than 5 minutes (!) on my NFS client. Below is a table with my observations / measurements:

Code:
+------------------------------------------+--------------+--------------+
|                                          |   NFS server |   NFS client |
+------------------------------------------+--------------+--------------+
| Execution time of rsync-Statement        |    3 seconds |  344 seconds |
| .tmp-unverified-download-quarantine                                    |
| - size                                   |       111 MB |       111 MB |
| - directories                            |       27.000 |       27.000 |
| - files (hard links)                     |      136.000 |      136.000 |
| Execution time of rm .tmp-unv...         |  1.5 seconds |  113 seconds |
+------------------------------------------+--------------+--------------+

'emerge --sync' wastes nearly 8 minutes (!) on my NFS client with creation and removal of .tmp-unverified-download-quarantine. That's annoying.
Furthermore, 'emergy --sync' writes more than 110MB to my SSD just to create and fill a temporary directory - every time I run it. I really don't like that.
Back to top
View user's profile Send private message
C5ace
Apprentice
Apprentice


Joined: 23 Dec 2013
Posts: 277
Location: Brisbane, Australia

PostPosted: Fri Jan 25, 2019 10:45 am    Post subject: Reply with quote

My local server runs once a day 'emerge --sync' as cron job.

To upgrade my local clients connect their '/usr/portage' by NFS to the server's '/usr/portage'. Then run 'emerge -av --update --deep --with-bdeps=y --newuse --backtrack=300 --changed-deps=y --keep-going=y @world'. When done without error, emerge --depclean and revdep-rebuild.

This works 99% of the time if there are no USE changes, blockers or kernel upgrades.
_________________
Observation after 30 years working with computers:
All software has known and unknown bugs and vulnerabilities. Especially software written in complex, unstable and object oriented languages such as python, perl, C++, C#, Rust and the likes.
Back to top
View user's profile Send private message
dmpogo
Advocate
Advocate


Joined: 02 Sep 2004
Posts: 2512
Location: Canada

PostPosted: Sat Jan 26, 2019 4:57 am    Post subject: Re: What happened to "emerge --sync" (slooooow)? Reply with quote

curmudgeon wrote:
I did search for this, and it seems strange that I can't find anything. Up until a few weeks ago, whenever I did "emerge --sync", the machine would quickly get started. I sync just one machine to the gentoo infrastructure, and all of the other machines to the local repository on that machine. The other machines would frequently sync completely in less than ten seconds.

Now whenever I "emerge --sync" (even on the other machines), the initial few lines pop up right away (looks like a sync of the repository timestamp to see if there is a need to sync the rest of the tree), and then there is a pause of about a minute before the list of files updated begins.

Unless there are a lot of new under the surface complexities, it appears that this is a deliberate pause (to combat network abuse?). Can anyone explain this, and is there someway to lessen or remove it (particularly on the local machines)?


I got the same symptoms on one of my machines out of three. Although in my case then it will also hang for a minute or more at the end of the sync processes, after it already printed amount of data transferred information. What I noticed about the second part of the hang, is that the process that is waiting completion (without utilizing much of CPU) is 'rm'.
One difference of this machine from the other two, is that it still uses reiserfs v3 filesystem on the partition where all portage resides and the partition is on soft raid 1.
And the hangs started month or so ago when I also updated the kernel (though it was a minor update withing 4.4 family). So in head I am blaming disk operations, but have not yet investigated it furthe
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7130
Location: almost Mile High in the USA

PostPosted: Sat Jan 26, 2019 5:01 pm    Post subject: Reply with quote

Yeah this is not the healthiest thing to do for SSDs but it's abut the best that can be done to have consistent portage trees that are signed.

When signature checks are enabled, rsync apparently copies into a separate directory '/usr/portage/.tmp-unverified-download-quarantine' which is a complete copy of the portage tree plus the new stuff that's being rsynced in. The signature is checked on that directory before it gets copied back into the main directory. It's the typical copy then move back scheme so that things are kept consistent, except it uses a lot more disk space and needs to clean up the second copy. The second copy was kind of screwy, once there was some corruption in that quarantine and emerge --sync would constantly fail for me until I manually deleted the quarantine.

In any case, need to figure out what's stopping the sync, I don't know...
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
Anon-E-moose
Advocate
Advocate


Joined: 23 May 2008
Posts: 3929
Location: Dallas area

PostPosted: Sat Jan 26, 2019 6:08 pm    Post subject: Reply with quote

eccerr0r wrote:
Yeah this is not the healthiest thing to do for SSDs but it's abut the best that can be done to have consistent portage trees that are signed.

When signature checks are enabled, rsync apparently copies into a separate directory '/usr/portage/.tmp-unverified-download-quarantine' which is a complete copy of the portage tree plus the new stuff that's being rsynced in. The signature is checked on that directory before it gets copied back into the main directory. It's the typical copy then move back scheme so that things are kept consistent, except it uses a lot more disk space and needs to clean up the second copy. The second copy was kind of screwy, once there was some corruption in that quarantine and emerge --sync would constantly fail for me until I manually deleted the quarantine.

In any case, need to figure out what's stopping the sync, I don't know...


To get rid of the .tmp* stuff this is what I added
Code:
cat /etc/portage/repos.conf/gentoo.conf
[DEFAULT]
main-repo = gentoo
sync-allow-hardlinks = no


The sync-allow-hardlinks is the key.
Yes it might allow an unverified file through (usually caused by syncing while the sending end hasn't gotten a complete file yet) but that gets fixed with a later sync AND you don't wind up with a .tmp* directory and associated copying that might hang around for days/weeks.
I was getting a verification error on the metadata stuff, but I finally changed the time I synced (cron job) by 15 minutes and haven't had a problem in a while.


Edit to add: this is what I used to see occasionally
Code:
>>> Starting rsync with rsync://91.186.30.235/gentoo-portage...
>>> Checking server timestamp ...
 * Manifest timestamp: 2018-09-19 06:08:41 UTC
 * Valid OpenPGP signature found:
 * - primary key: DCD05B71EAB94199527F44ACDB6B8C1F96D8BF6D
 * - subkey: E1D6ABB63BFCFB4BA02FDF1CEC590EEAC9189250
 * - timestamp: 2018-09-19 06:08:41 UTC
 * Verifying /usr/portage/.tmp-unverified-download-quarantine ...!!! Manifest verification failed:
Manifest mismatch for metadata/Manifest.gz
  __size__: expected: 1981, have: 1980


I do notice that even with the .tmp* stuff not activated, there's a pause during verification at the end of the sync but it lasts for less than a minute, in my case.
_________________
Asus m5a99fx, FX 8320 - nouveau, oss4, rx550 for qemu passthrough
Acer laptop E5-575, i3-7100u - i965, alsa
---both---
5.0.13 zen kernel, profile 17.0 (no-pie) amd64-no-multilib
gcc 8.2.0, eudev, openrc, openbox, palemoon
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7130
Location: almost Mile High in the USA

PostPosted: Sat Jan 26, 2019 7:10 pm    Post subject: Reply with quote

The last emerge --sync I did, it roughly took 45 seconds to do signature verification. This was on a SSD. I haven't checked my other boxes, specifically the HDD machines, for relative sync times...
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
dmpogo
Advocate
Advocate


Joined: 02 Sep 2004
Posts: 2512
Location: Canada

PostPosted: Mon Jan 28, 2019 5:09 pm    Post subject: Reply with quote

So what is the conclusion, what is the proper (or best) way to disable .tmp-unverified-download-quarantine game ?
Back to top
View user's profile Send private message
mike155
Veteran
Veteran


Joined: 17 Sep 2010
Posts: 1294
Location: Frankfurt, Germany

PostPosted: Thu Jan 31, 2019 9:40 pm    Post subject: Reply with quote

Anon-E-moose wrote:
sync-allow-hardlinks = no

Thanks, Anon-E-moose!

That's exactly what I was looking for. The option above reduces execution time of 'emerge --sync' on my NFS client from 598 seconds down to 101 seconds. Pretty good!

Code:
+----------------------------------------+-------------+-------------+
|                                        |  NFS server |  NFS client |
+----------------------------------------+-------------+-------------+
| execution time of 'emerge --sync'      |  13 seconds | 598 seconds |
| - default options                      |             |             |
+----------------------------------------+-------------+-------------+
| execution time of 'emerge --sync'      |   7 seconds | 101 seconds |
| - sync-rsync-verify-metamanifest = no  |             |             |
| - sync-allow-hardlinks = no            |             |             |
+----------------------------------------+-------------+-------------+
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum