Broken portage? What?

[Perfect Gentleman]

ct85711, how did you sync git-tree ?

[ct85711]

Here's the repos.conf I've been using for a while now (though I suspect that I will have to change it).

Code: Select all

ct85711@Oate ~ $ cat /etc/portage/repos.conf/gentoo.conf 
[DEFAULT]
main-repo = gentoo

[gentoo]
location = /usr/portage
#sync-type = rsync
sync-type = git
auto-sync = yes
#sync-uri = rsync://rsync3.us.gentoo.org/gentoo-portage
sync-uri = https://github.com/gentoo/gentoo-portage-rsync-mirror

From every check I've done, it's been staying in sync to the main tree so I haven't had any issues with it. Most likely we will (hopefuly) soon get a official announcement letting us know the official address to use for git syncing.

[Perfect Gentleman]

ct85711, use the one i've posted

[freke]

I synced yesterday and again today - it seems it pulls all Manifest-files at each sync?

I'm receiving some 60+ Mbyte on each sync - don't remember it doing that normally?
(I just tried another sync and again seems to pull all Manifest-files - total bytes received 66.38M)

That's using rsync

Code: Select all

[DEFAULT]
main-repo = gentoo

[gentoo]
location = /usr/portage
sync-type = rsync
sync-uri = rsync://rsync.gentoo.org/gentoo-portage
auto-sync = yes

[ganthore]

I'm seeing the same behaviour as freke when using good ol' rsync.

[Vulgar]

If using emerge-webrsync for gpg signature checking, which downloads the portage snapshots, the last snapshot is 20150808 on the mirrors. Looks like there still working on what ever it is they are trying to do, as there is generally a new snapshot everyday. 4 days is a long stretch.

[BT]

The Manifest issue is known and is still being worked on.

Edit: The issue is now fixed.

[hasufell]

Although some developers wrongly still recommend plain rsync-update method, you should never do that, since it is fundamentally insecure (emerge-webrsync is signed by a machine key, but still better).

You should instead start to sync via git (best do that from github, unless you want to break our infra server).

However, that repository does not have any metadata cache, so you have to generate that yourself or fetch pre-generated cache from somewhere else. Or just combine both, so this probably makes sense:

Code: Select all

# clone the main tree with history stripped to the last 50 commits
git clone --depth=50 git@github.com:gentoo/gentoo.git
cd gentoo
# clone a tree mirror which holds pre-generated cache (it might not be 100% in sync with the real tree though, we update the cache later)
git clone --depth=1 git@github.com:gentoo-mirror/gentoo.git metadata-mirror
# move the cache to the right place
mv metadata-mirror/metadata/md5-cache metadata/
# remove remaining files
rm -rf metadata-mirror

Now make sure this tree is in the correct place and with the correct access rights, so your PM can use it. After that, just run:

Code: Select all

egencache --jobs=8 --repo=gentoo --update --update-use-local-desc

So... if you later want to update the tree, you just cd into it and do:

Code: Select all

git pull --ff origin master
# this does NOT take long, because it just updates the existing cache
egencache --jobs=8 --repo=gentoo --update --update-use-local-desc

Now you have a git tree which is fully gpg signed (every commit on the leftmost branch at least) by gentoo devs and you can use git methods to verifiy, e.g. verify via:

Code: Select all

[ -z "$(git show -q --pretty="format:%G?" $(git rev-list --first-parent master) | grep -v G)" ] && echo "verification success" || echo "verification failure"

If the verification failed, you can check which commits did:

Code: Select all

git show -q --pretty="format:%h %an %G?" $(git rev-list --first-parent master) | grep '.* [NBU]$'

Mind that you need all gpg keys from all developers for this to work. Unfortunately, I don't think we have a full keyring for download yet, have we?

Note: at some point in the future, mgorny might fix the git@github.com:gentoo-mirror/gentoo.git mirror to have the true history PLUS the metadata-cache. That's currently not there yet, but if it is... you can just clone and update from there without regenerating the cache anymore.

[Tony0945]

I have to ask two questions:

1. Why wasn't there a news item alerting all users to this substantial change? At least when the environmental variables changed to repos.conf, the change was automatic.

2. What about local overlays? Do we have to run a git server now to have a local overlay?


Synced yesterday and it took twenty minutes. Then I emerged two ebuilds that had updated. That was twenty minutes from my local server that syncs at night while I'm sleeping. I'll have to check the log to see how long that took. I'll bet it was hours.

I suppose I can switch to updating once a month or once a quarter, or never, just reinstall from a portage tarball, it's faster.

[ct85711]

well, I can say on #2, that portage will work like it has if you use rsync or git syncing. The only that is changing, is how you are fetching the tree. From using git syncing for a while, I can say that the tree fetched via git is still the same from rsync, so you shouldn't see if any difference on if you have a local overlay or using layman. Just a note, if you use the old rsync feature (where it will remove files that wasn't sync, will no longer work with git).

As for #1, I'm taking a guess, but I suspect the reason an announcement hasn't been done yet, is because they are not finished getting everything organized. I would think, they are doing it this way, to give more people a chance to help test their work out to make sure everything is working properly and help iron out the proper method to do the transition (for documentation). Think of it this way, do you really think a regular user is going to understand what all those git commands do (hell, I don't fully understand them myself, beyond what I looked up and still don't feel comfortable on them). It is also just like, it allows us testers to find the easier ways on syncing the tree; to identify like the part hasufell posted on updating the tree to find out you don't need to do git pull and egencache when emerge --sync works just fine...

Note: To the devs and hasufell, I want to do some additional checks; with the updated info before we update that spot.

[asturm]

1. Why wasn't there a news item alerting all users to this substantial change? At least when the environmental variables changed to repos.conf, the change was automatic.

There is no change, because rsync just continues to work for users as it has always been...

Using git instead is an additional step that might be documented at a later point.

[hasufell]

to identify like the part hasufell posted on updating the tree to find out you don't need to do git pull and egencache when emerge --sync works just fine...

Portage supports git as a sync type, see https://wiki.gentoo.org/wiki/Project:Po ... #Migration

So you don't really have to manually do "git pull". That means you do your regular emerge --sync, but generate the cache on top of it (as said: that problem will be solved with another mirror that has pregenerated cache).

My steps just showed how you fetch that pre-generated cache into the real repo. Instead, you can probably just do (although I cannot test, since I don't use portage) in /etc/portage/repos.conf/gentoo.conf

Code: Select all

location = /usr/portage
sync-type = git
sync-uri = git@github.com:gentoo/gentoo.git
auto-sync = yes

Then do

Code: Select all

emerge --sync
egencache --jobs=8  --repo=gentoo --update --update-use-local-desc

not much of a difference there, really

[mrbassie]

My steps just showed how you fetch that pre-generated cache into the real repo. Instead, you can probably just do (although I cannot test, since I don't use portage) in /etc/portage/repos.conf/gentoo.conf

Code: Select all

location = /usr/portage
sync-type = git
sync-uri = git@github.com:gentoo/gentoo.git
auto-sync = yes

Then do

Code: Select all

emerge --sync
egencache --jobs=8  --repo=gentoo --update --update-use-local-desc

not much of a difference there, really

Returns:

Code: Select all

~$ sudo emerge --sync
>>> Syncing repository 'gentoo' into '/usr/portage'...
/usr/bin/git clone --depth 1 git@github.com:gentoo/gentoo.git .
fatal: destination path '.' already exists and is not an empty directory.
!!! git clone error in /usr/portage

Can't be bothered to troubleshoot/learn git as I just got home from work, just seeing what happened. I'll stick with web-rsync for the timebeing.

Similar for eix-sync which suggest you use the -a switch which does work but you still get the error.

[freke]

Shouldn't portage have a git use-flag to pull in git?

[yngwin]

Code: Select all

egencache --jobs=8  --repo=gentoo --update --update-use-local-desc

That's what repo.postsync.d is for.

[hasufell]

Shouldn't portage have a git use-flag to pull in git?

No. USE flags are not for just pulling in optional runtime dependencies.

[freke]

to identify like the part hasufell posted on updating the tree to find out you don't need to do git pull and egencache when emerge --sync works just fine...

Portage supports git as a sync type, see https://wiki.gentoo.org/wiki/Project:Po ... #Migration

So you don't really have to manually do "git pull". That means you do your regular emerge --sync, but generate the cache on top of it (as said: that problem will be solved with another mirror that has pregenerated cache).

My steps just showed how you fetch that pre-generated cache into the real repo. Instead, you can probably just do (although I cannot test, since I don't use portage) in /etc/portage/repos.conf/gentoo.conf

Code: Select all

location = /usr/portage
sync-type = git
sync-uri = git@github.com:gentoo/gentoo.git
auto-sync = yes

Then do

Code: Select all

emerge --sync
egencache --jobs=8  --repo=gentoo --update --update-use-local-desc

not much of a difference there, really

Code: Select all

>>> Syncing repository 'gentoo' into '/usr/portage'...
/usr/bin/git clone --depth 1 git@github.com:gentoo/gentoo.git .
Cloning into '.'...
Warning: Permanently added the RSA host key for IP address '192.30.252.131' to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
!!! git clone error in /usr/portage

[yngwin]

You will want to use git://anongit.gentoo.org/repo/gentoo.git

[hasufell]

You will want to use git://anongit.gentoo.org/repo/gentoo.git

No, please don't. That might cause trouble for the infra server if every user out there pulls the whole tree directly from infra.

If you don't have a github account, just use https://github.com/gentoo/gentoo.git. They can handle the load.

[Kollin]

You will want to use git://anongit.gentoo.org/repo/gentoo.git

No, please don't. That might cause trouble for the infra server if every user out there pulls the whole tree directly from infra.

If you don't have a github account, just use https://github.com/gentoo/gentoo.git. They can handle the load.

If I have a github account what link should I use?
Thank you!

[hasufell]

You will want to use git://anongit.gentoo.org/repo/gentoo.git

No, please don't. That might cause trouble for the infra server if every user out there pulls the whole tree directly from infra.

If you don't have a github account, just use https://github.com/gentoo/gentoo.git. They can handle the load.

If I have a github account what link should I use?

Then you can either use the https link or the ssh clone url git@github.com:gentoo/gentoo.git