Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
WARNING: DO NOT SYNC! (it emptied my portage tree)
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
equaeghe
Guru
Guru


Joined: 22 Feb 2005
Posts: 434

PostPosted: Tue Sep 18, 2018 12:10 pm    Post subject: WARNING: DO NOT SYNC! (it emptied my portage tree) Reply with quote

I just sync'ed my portage tree, but something failed in a bad way and now my portage tree has been emptied, apparently.
Code:
directory has vanished: "/usr/portage/.tmp-unverified-download-quarantine/x11-libs"                                                                                                                                                [42/1821]
directory has vanished: "/usr/portage/.tmp-unverified-download-quarantine/x11-misc"
directory has vanished: "/usr/portage/.tmp-unverified-download-quarantine/x11-plugins"
directory has vanished: "/usr/portage/.tmp-unverified-download-quarantine/x11-terms"
directory has vanished: "/usr/portage/.tmp-unverified-download-quarantine/x11-themes"
directory has vanished: "/usr/portage/.tmp-unverified-download-quarantine/x11-wm"
directory has vanished: "/usr/portage/.tmp-unverified-download-quarantine/xfce-base"
directory has vanished: "/usr/portage/.tmp-unverified-download-quarantine/xfce-extra"
file has vanished: "/usr/portage/.tmp-unverified-download-quarantine/app-admin/Manifest.gz"

Number of files: 1,179 (reg: 691, dir: 488)
Number of created files: 0
Number of deleted files: 162,764 (reg: 135,255, dir: 27,509)
Number of regular files transferred: 3
Total file size: 977.91K bytes
Total transferred file size: 65.33K bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 32.20K
Total bytes received: 12.82M

sent 32.20K bytes  received 12.82M bytes  2.86M bytes/sec
total size is 977.91K  speedup is 0.08
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1189) [sender=3.1.3]
q: Updating ebuild cache in /usr/portage ...
q: Finished 227 entries in 0.169312 seconds

Action: sync for repo: gentoo, returned code = 24

 * An update to portage is available. It is _highly_ recommended
 * that you update portage now, before any other packages are updated.

 * To update portage, run 'emerge --oneshot portage' now.



 * emerge --sync failed
 * Time statistics:
   662 seconds for syncing
   664 seconds total


In fact, I knew I had to emerge @preserved-rebuild (for rust), but decided to sync my tree first. So this now gives
Code:
# emerge -qva @preserved-rebuild


!!! /etc/portage/make.profile is not a symlink and will probably prevent most merges.
!!! It should point into a profile within /usr/portage/profiles/
!!! (You can safely ignore this message when syncing. It's harmless.)


!!! Your current profile is invalid. If you have just changed your profile
!!! configuration, you should revert back to the previous configuration.
!!! Allowed actions are limited to --help, --info, --search, --sync, and
!!! --version.


And so I now have no profile file anymore!

A question mark is also listed after Package Database on https://infra-status.gentoo.org/.

What is going on?

I guess I best wait until this is fixed and then try and re-sync. Please let me know when that is once you find out. Thanks.
Back to top
View user's profile Send private message
bunder
Bodhisattva
Bodhisattva


Joined: 10 Apr 2004
Posts: 5692

PostPosted: Tue Sep 18, 2018 12:25 pm    Post subject: Reply with quote

package database is https://packages.gentoo.org, its been broken for a while. it http 500's all over the place.

edit: looks okay here https://gitweb.gentoo.org/repo/gentoo.git/tree/... wanna try syncing again?
_________________
overlay | patches
Neddyseagoon wrote:
The problem with leaving is that you can only do it once and it reduces your influence.
Back to top
View user's profile Send private message
equaeghe
Guru
Guru


Joined: 22 Feb 2005
Posts: 434

PostPosted: Tue Sep 18, 2018 12:33 pm    Post subject: Reply with quote

bunder wrote:
edit: looks okay here https://gitweb.gentoo.org/repo/gentoo.git/tree/... wanna try syncing again?

Tried. The same problem as before. (I sync using the classical rsync method.)
Back to top
View user's profile Send private message
bunder
Bodhisattva
Bodhisattva


Joined: 10 Apr 2004
Posts: 5692

PostPosted: Tue Sep 18, 2018 12:38 pm    Post subject: Reply with quote

Could be a bad rsync server maybe? https://www.gentoo.org/support/rsync-mirrors/
_________________
overlay | patches
Neddyseagoon wrote:
The problem with leaving is that you can only do it once and it reduces your influence.
Back to top
View user's profile Send private message
Anon-E-moose
Advocate
Advocate


Joined: 23 May 2008
Posts: 3354
Location: Dallas area

PostPosted: Tue Sep 18, 2018 12:41 pm    Post subject: Reply with quote

I had no trouble as of 3 AM this morning.

Code:
GENTOO_MIRRORS="rsync://rsync.gtlib.gatech.edu/gentoo ftp://ftp.ussg.iu.edu/pub/linux/gentoo http://gentoo.osuosl.org/ http://ftp.ucsb.edu/pub/mirrors/linux/gentoo/ rsync://gentoo.cs.uni.edu/gentoo-distfiles http://mirror.usu.edu/mirrors/gentoo/"

_________________
Asus m5a99fx, FX 8320 - nouveau & radeon, oss4
Acer laptop E5-575, i3-7100u - i965, alsa
---both---
4.14.62 kernel, profile 17.0 (no-pie) amd64-no-multilib
gcc 4.9.4 & 7.3.0, eudev, openrc, openbox, palemoon
Back to top
View user's profile Send private message
equaeghe
Guru
Guru


Joined: 22 Feb 2005
Posts: 434

PostPosted: Tue Sep 18, 2018 12:42 pm    Post subject: Reply with quote

bunder wrote:
Could be a bad rsync server maybe? https://www.gentoo.org/support/rsync-mirrors/

Then it is more than just one mirror:
Quote:
1537270442: Started emerge on: sep 18, 2018 13:34:01
1537270442: *** emerge --misspell-suggestions=n --with-bdeps=y --sync
1537270442: === sync
1537270442: >>> Syncing repository 'gentoo' into '/usr/portage'...
1537270446: >>> Starting rsync with rsync://193.136.164.6/gentoo-portage
1537270714: *** terminating.
1537270818: Started emerge on: sep 18, 2018 13:40:18
1537270818: *** emerge --misspell-suggestions=n --with-bdeps=y --sync
1537270818: === sync
1537270818: >>> Syncing repository 'gentoo' into '/usr/portage'...
1537270823: >>> Starting rsync with rsync://178.210.128.3/gentoo-portage
1537271480: *** terminating.
1537273891: Started emerge on: sep 18, 2018 14:31:31
1537273891: *** emerge --misspell-suggestions=n --with-bdeps=y --sync
1537273891: === sync
1537273891: >>> Syncing repository 'gentoo' into '/usr/portage'...
1537273896: >>> Starting rsync with rsync://92.60.51.128/gentoo-portage
1537273946: *** terminating.
Back to top
View user's profile Send private message
bunder
Bodhisattva
Bodhisattva


Joined: 10 Apr 2004
Posts: 5692

PostPosted: Tue Sep 18, 2018 12:47 pm    Post subject: Reply with quote

I just checked my usual mirror, rsync://rsync1.ca.gentoo.org/gentoo-portage/ all good here

edit: are you using overlayfs? I wonder if this might be related https://www.gentoo.org/support/news-items/2018-07-11-portage-sync-allow-hardlinks.html
_________________
overlay | patches
Neddyseagoon wrote:
The problem with leaving is that you can only do it once and it reduces your influence.
Back to top
View user's profile Send private message
Anon-E-moose
Advocate
Advocate


Joined: 23 May 2008
Posts: 3354
Location: Dallas area

PostPosted: Tue Sep 18, 2018 1:30 pm    Post subject: Reply with quote

I doubt it's a whole lot of different servers, more likely something on your end/your system.

Code:
1537254002: Started emerge on: Sep 18, 2018 02:00:01
1537254002:  *** emerge --with-bdeps=y --quiet-build=y --autounmask=n --sync
1537254002:  === sync
1537254002: >>> Syncing repository 'gentoo' into '/usr/portage'...
1537254015: >>> Starting rsync with rsync://91.186.30.235/gentoo-portage
1537254094: === Sync completed for gentoo

That was this morning.
and it corresponds to rsync://rsync.gtlib.gatech.edu/gentoo

Looking at the time stamp they seem to be active for ~50 seconds which is a little short, mine from this morning was 1 min and 20 seconds.

Edit to add: what does "cat /etc/portage/repos.conf/gentoo.conf" return

ETA2: is /usr/portage on the root filesystem, or a mounted filesystem?
What have you changed on your system lately, fstab, etc?

If you do "touch /usr/portage/test.file" is there a file called "test.file" in /usr/portage?
_________________
Asus m5a99fx, FX 8320 - nouveau & radeon, oss4
Acer laptop E5-575, i3-7100u - i965, alsa
---both---
4.14.62 kernel, profile 17.0 (no-pie) amd64-no-multilib
gcc 4.9.4 & 7.3.0, eudev, openrc, openbox, palemoon
Back to top
View user's profile Send private message
equaeghe
Guru
Guru


Joined: 22 Feb 2005
Posts: 434

PostPosted: Tue Sep 18, 2018 1:41 pm    Post subject: Reply with quote

bunder wrote:
are you using overlayfs? I wonder if this might be related https://www.gentoo.org/support/news-items/2018-07-11-portage-sync-allow-hardlinks.html

No.
Back to top
View user's profile Send private message
equaeghe
Guru
Guru


Joined: 22 Feb 2005
Posts: 434

PostPosted: Tue Sep 18, 2018 1:49 pm    Post subject: Reply with quote

Anon-E-moose wrote:
… and it corresponds to rsync://rsync.gtlib.gatech.edu/gentoo

I've changed my rsync mirror to the US one from the European one and am now syncing using the gatech mirror (goes very slowly!). Will report back once it's done.

Anon-E-moose wrote:
what does "cat /etc/portage/repos.conf/gentoo.conf" return

Code:
$ cat /etc/portage/repos.conf/gentoo.conf
[DEFAULT]
main-repo = gentoo

[gentoo]
location = /usr/portage
sync-type = rsync
sync-uri = rsync://rsync.us.gentoo.org/gentoo-portage
#sync-uri = rsync://rsync.europe.gentoo.org/gentoo-portage


Anon-E-moose wrote:
is /usr/portage on the root filesystem, or a mounted filesystem?

root file system

Anon-E-moose wrote:
What have you changed on your system lately, fstab, etc?

nothing since the previous, successful sync

Anon-E-moose wrote:
If you do "touch /usr/portage/test.file" is there a file called "test.file" in /usr/portage?

yes
Back to top
View user's profile Send private message
equaeghe
Guru
Guru


Joined: 22 Feb 2005
Posts: 434

PostPosted: Tue Sep 18, 2018 2:51 pm    Post subject: Reply with quote

equaeghe wrote:
I've changed my rsync mirror to the US one from the European one and am now syncing using the gatech mirror (goes very slowly!). Will report back once it's done.

My tree is still empty, but now the error was different:
Code:
sent 2.71M bytes  received 228.14M bytes  33.38K bytes/sec
total size is 218.82M  speedup is 0.95
 * Manifest timestamp: 2018-09-18 12:08:42 UTC
 * Valid OpenPGP signature found:
 * - primary key: DCD05B71EAB94199527F44ACDB6B8C1F96D8BF6D
 * - subkey: E1D6ABB63BFCFB4BA02FDF1CEC590EEAC9189250
 * - timestamp: 2018-09-18 12:08:42 UTC
 * Verifying /usr/portage/.tmp-unverified-download-quarantine ...!!! Manifest verification failed:
Manifest mismatch for metadata/Manifest.gz
  __size__: expected: 1982, have: 1979
q: Updating ebuild cache in /usr/portage ...
q: Finished 227 entries in 0.005788 seconds

Action: sync for repo: gentoo, returned code = 1

 * An update to portage is available. It is _highly_ recommended
 * that you update portage now, before any other packages are updated.

 * To update portage, run 'emerge --oneshot portage' now.



 * emerge --sync failed
 * Time statistics:
  6922 seconds for syncing
  6924 seconds total
Back to top
View user's profile Send private message
Anon-E-moose
Advocate
Advocate


Joined: 23 May 2008
Posts: 3354
Location: Dallas area

PostPosted: Tue Sep 18, 2018 2:59 pm    Post subject: Reply with quote

What is your portage version "emerge --version"

and what does "ls -la /usr/portage" return
_________________
Asus m5a99fx, FX 8320 - nouveau & radeon, oss4
Acer laptop E5-575, i3-7100u - i965, alsa
---both---
4.14.62 kernel, profile 17.0 (no-pie) amd64-no-multilib
gcc 4.9.4 & 7.3.0, eudev, openrc, openbox, palemoon
Back to top
View user's profile Send private message
equaeghe
Guru
Guru


Joined: 22 Feb 2005
Posts: 434

PostPosted: Tue Sep 18, 2018 3:26 pm    Post subject: Reply with quote

Anon-E-moose wrote:
What is your portage version "emerge --version"

Code:
# emerge --version
Portage 2.3.49 (python 3.6.5-final-0, unavailable, gcc-7.3.0, glibc-2.26-r7, 4.14.65-gentoo x86_64)


Anon-E-moose wrote:
what does "ls -la /usr/portage" return

output of exa -laT -L 2 --group-directories-first /usr/portage/: https://pastebin.com/raw/wR427Li1

So the tree is not entirely empty, as some app-a… categories have some content. The rest is stuffed into .tmp-unverified-download-quarantine for some reason.
Back to top
View user's profile Send private message
Anon-E-moose
Advocate
Advocate


Joined: 23 May 2008
Posts: 3354
Location: Dallas area

PostPosted: Tue Sep 18, 2018 4:01 pm    Post subject: Reply with quote

I would delete everything under /usr/portage, that includes .tmp-unverified-download-quarantine and stuff under it.
Do an "ls -la /usr/portage" and make sure there are absolutely no files there including the hidden ones.

Then run emerge-webrsync and see if that doesn't give you a complete /usr/portage
_________________
Asus m5a99fx, FX 8320 - nouveau & radeon, oss4
Acer laptop E5-575, i3-7100u - i965, alsa
---both---
4.14.62 kernel, profile 17.0 (no-pie) amd64-no-multilib
gcc 4.9.4 & 7.3.0, eudev, openrc, openbox, palemoon
Back to top
View user's profile Send private message
equaeghe
Guru
Guru


Joined: 22 Feb 2005
Posts: 434

PostPosted: Tue Sep 18, 2018 4:26 pm    Post subject: Reply with quote

Anon-E-moose wrote:
I would delete everything under /usr/portage, that includes .tmp-unverified-download-quarantine and stuff under it.
Do an "ls -la /usr/portage" and make sure there are absolutely no files there including the hidden ones.

Then run emerge-webrsync and see if that doesn't give you a complete /usr/portage

It does. But when I then sync again, the original issue reappears.
Back to top
View user's profile Send private message
Anon-E-moose
Advocate
Advocate


Joined: 23 May 2008
Posts: 3354
Location: Dallas area

PostPosted: Tue Sep 18, 2018 5:03 pm    Post subject: Reply with quote

equaeghe wrote:
Anon-E-moose wrote:
I would delete everything under /usr/portage, that includes .tmp-unverified-download-quarantine and stuff under it.
Do an "ls -la /usr/portage" and make sure there are absolutely no files there including the hidden ones.

Then run emerge-webrsync and see if that doesn't give you a complete /usr/portage

It does. But when I then sync again, the original issue reappears.


Don't do a sync, doing the webrsync should do the same thing.
After you do the webrsync I would do "emerge portage" then "emerge @world" and see if things don't start working properly.

If that doesn't work then you've done something horribly wrong to your portage environment and I can't help past this point.
_________________
Asus m5a99fx, FX 8320 - nouveau & radeon, oss4
Acer laptop E5-575, i3-7100u - i965, alsa
---both---
4.14.62 kernel, profile 17.0 (no-pie) amd64-no-multilib
gcc 4.9.4 & 7.3.0, eudev, openrc, openbox, palemoon
Back to top
View user's profile Send private message
equaeghe
Guru
Guru


Joined: 22 Feb 2005
Posts: 434

PostPosted: Tue Sep 18, 2018 8:36 pm    Post subject: Reply with quote

Anon-E-moose wrote:
Don't do a sync, doing the webrsync should do the same thing.

I did the webrsync and then subsequently a sync, it's the second one that messes things up.

Anon-E-moose wrote:
After you do the webrsync I would do "emerge portage" then "emerge @world" and see if things don't start working properly.

I've done the portage rebuild, but I'm refraining from rebuilding my whole world, as I don't see what that would change. I'm waiting a while to see if someone else knows more. The .tmp-unverified-download-quarantine thing must be related somehow.

Anon-E-moose wrote:
If that doesn't work then you've done something horribly wrong to your portage environment and I can't help past this point.

I can understand. You've already assisted me very kindly, thanks! (I do want to state that I did not mess up my system: I synced yesterday without issues, did nothing as a superuser in between and synced again today with the bad result.)
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6212
Location: Room 101

PostPosted: Tue Sep 18, 2018 10:59 pm    Post subject: Reply with quote

equaeghe wrote:
The .tmp-unverified-download-quarantine thing must be related somehow.

equaeghe ... this is the result of 'sync-allow-hardlinks = yes' (which was introduced and is the default), to disable it set 'sync-allow-hardlinks = no' in repos.conf (see the above linked, or 'eselect news', 2018-07-11-portage-sync-allow-hardlinks).

best ... khay
Back to top
View user's profile Send private message
Anon-E-moose
Advocate
Advocate


Joined: 23 May 2008
Posts: 3354
Location: Dallas area

PostPosted: Wed Sep 19, 2018 9:45 am    Post subject: Reply with quote

equaeghe wrote:
Anon-E-moose wrote:
If that doesn't work then you've done something horribly wrong to your portage environment and I can't help past this point.

I can understand. You've already assisted me very kindly, thanks! (I do want to state that I did not mess up my system: I synced yesterday without issues, did nothing as a superuser in between and synced again today with the bad result.)


I owe you an apology on that, as what happened to you happened to me last night,
and there's nothing wrong with hard links on my system, but I set the no flag in repos.conf since.

It seems the devs are bound and determined to destroy gentoo one way or another, "hey lets screw with whats been working forever" :roll:
_________________
Asus m5a99fx, FX 8320 - nouveau & radeon, oss4
Acer laptop E5-575, i3-7100u - i965, alsa
---both---
4.14.62 kernel, profile 17.0 (no-pie) amd64-no-multilib
gcc 4.9.4 & 7.3.0, eudev, openrc, openbox, palemoon
Back to top
View user's profile Send private message
Fitzcarraldo
Veteran
Veteran


Joined: 30 Aug 2008
Posts: 1540
Location: United Kingdom

PostPosted: Wed Sep 19, 2018 11:15 am    Post subject: Reply with quote

I'm trying to understand this thread before I sync my two Gentoo installations this week.

Anon-E-moose and equaeghe, are you saying you do not have an overlay filesystem (yet you nevertheless have a problem syncing)?

Excerpt from https://www.gentoo.org/support/news-items/2018-07-11-portage-sync-allow-hardlinks.html :

Zac Medico wrote:
The new behavior may conflict with configurations that restrict the use of hardlinks, such as overlay filesystems. Therefore, users will have to set "sync-allow-hardlinks = no" in repos.conf if they have a configuration that restricts the use of hardlinks, but this should not be very common

_________________
Clevo W230SS: amd64 OpenRC elogind nvidia-drivers & xf86-video-intel.
Compal NBLB2: ~amd64 OpenRC elogind xf86-video-ati. Dual boot Win 7 Pro 64-bit.
KDE on both.

Fitzcarraldo's blog
Back to top
View user's profile Send private message
equaeghe
Guru
Guru


Joined: 22 Feb 2005
Posts: 434

PostPosted: Wed Sep 19, 2018 11:53 am    Post subject: Reply with quote

Fitzcarraldo wrote:
are you saying you do not have an overlay filesystem (yet you nevertheless have a problem syncing)?

Yes. I will file a bug and report its number here.

khayyam wrote:
set 'sync-allow-hardlinks = no' in repos.conf

I tried this, and it works. Thanks! (Of course this is just a workaround; I guess I'll move to git syncing, where all the attention is nowadays.)
Back to top
View user's profile Send private message
Anon-E-moose
Advocate
Advocate


Joined: 23 May 2008
Posts: 3354
Location: Dallas area

PostPosted: Wed Sep 19, 2018 11:56 am    Post subject: Reply with quote

Fitzcarraldo, I don't use overlay at all, it's not even set in the kernel.

Code:
 * Verifying /usr/portage/.tmp-unverified-download-quarantine ...!!! Manifest verification failed:
Manifest mismatch for metadata/Manifest.gz
  __size__: expected: 1981, have: 1980


Day before yesterday it was fine, this morning's sync showed the problem.
so I set hardlinks no in repos.conf and resync'd, showed clean then.

Whichever dev came up with this idea and claimed that most wouldn't be affected as it's not common is a little wrong.

Edit to add: The problem is that any error will trigger off unwanted behavior, things left in .tmp-unverified-download-quarantine, main /usr/portage/* left in a weird and possibly unstable state. As is shown above it was simply a one byte off mismatch that triggered my problem.
_________________
Asus m5a99fx, FX 8320 - nouveau & radeon, oss4
Acer laptop E5-575, i3-7100u - i965, alsa
---both---
4.14.62 kernel, profile 17.0 (no-pie) amd64-no-multilib
gcc 4.9.4 & 7.3.0, eudev, openrc, openbox, palemoon


Last edited by Anon-E-moose on Wed Sep 19, 2018 9:00 pm; edited 2 times in total
Back to top
View user's profile Send private message
equaeghe
Guru
Guru


Joined: 22 Feb 2005
Posts: 434

PostPosted: Wed Sep 19, 2018 12:13 pm    Post subject: Reply with quote

equaeghe wrote:
I will file a bug and report its number here.

https://bugs.gentoo.org/666554
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum