With git headers removed from portage, how to track changes?

srcshelton · n00b Joined: 10 Apr 2004 Posts: 22 Location: Cambridge

With git '$Id$' headers removed from every file in the portage tree, how are overlay maintainers intended to track changes to ebuilds and dependent files?

Historically, this has been easy - pull the portage tree from https://github.com/gentoo/gentoo.git with a top-level '.gitattributes' containing:

srcshelton · n00b Joined: 10 Apr 2004 Posts: 22 Location: Cambridge

... except that a whole load of ebuilds (e.g. https://github.com/gentoo/gentoo/tree/master/x11-libs/gdk-pixbuf-loader-webp at random, but it looks as if 528 packages are potentially affected as-of 61b861a) are missing Manifest files entirely in the main tree - so digests would have to be generated on the client side, slowing the process and increasing the chance that either changes are missed or false-positives occur.

szatox · Advocate Joined: 27 Aug 2013 Posts: 3135

Just a shot in the dark: perhaps it would be a good idea to use git's branches feature?
Run a full clone rather than shallow one, create a local branch, modify ebuilds as needed and then use that local branch as an overlay?
You can then stay on sync by merging gentoo-maintained branch into your local branch after pulling it.

You will want to use a dedicated partition for this (tuned for small files) or even squash it. The latter looks particularly promising, since git repo tends to be append-only so you can gradually grow that squashfs without ever again rebuilding it from scratch to trim old versions - this is assuming you only squash .git directory and ignore working directory, as invoking checkout will populate it anyway.

I don't know if it fits your case, but I am successfully using this bifurcated approach to git-based deployments (branch provided by developers + deployment branch with local settings and top secret passwords :lol:

), and having played with squashfs a bit I expect those 2 tools to go along pretty well.

srcshelton · n00b Joined: 10 Apr 2004 Posts: 22 Location: Cambridge

I also push my overlay to overlays.gentoo.org, so pushing the entire portage tree with some modifications isn't an option: it must be possible to push the changes only. I'm sure that this could be achieved with some sequence which involves a complex git-filter command, but this appears to be a significantly non-trivial option.

(I'm also not entirely sure how well a branch which is regularly merged with the 'master' branch of the Gentoo repo would work, given that history re-writing regularly occurs)

srcshelton · n00b Joined: 10 Apr 2004 Posts: 22 Location: Cambridge

Okay - I hadn't realised that Manifest files in the main repo only contain 'DIST' hashes, whilst overlay repos also contain hashes for every ebuild and file (amongst others) too. Firstly that means that the 528 main-repo ebuilds without a Manifest aren't incorrect, they're simply ebuilds which don't rely on additional files or sources. Secondly, since even running 'ebuild ... digest' will only create 'DIST' entries, then Manifest files cannot be used to tell whether ebuild contents have changed. This is unfortunate, since it means that there now appears to be no method using any element of Gentoo-provided infrastructure directly to keep overlays in sync with the main repo

ct85711 · Veteran Joined: 27 Sep 2005 Posts: 1791

One part you are forgetting is that git tracks changes from all commits by default. Each commit keeps track of all changes (modifications, moving, deletion, new files, and renaming) for that commit. Now there may be a command to look up the commits on your copy of the repo, but I do not know what it is right now. If you use your browser and view the git log for the package in question, it shows you all of the commits and the commit message. Clicking on the commit title (same line the author is on) will bring up the full commit message and the diff contents showing exactly what changed when it was accepted into the tree.

Note, portage's default setting when you sync is to only grab the most current commit, not the entire history (to save hd space, and amount downloaded). If you wish want to change that, you need to change clone-depth (used to be sync-depth) to change amount of commit history you download (default is 1, 0 means all).

Hu · Moderator Joined: 06 Mar 2007 Posts: 21631

srcshelton: either I do not understand your use case or you are seriously underestimating Git's capabilities. As ct85711 says, Git is designed for tracking changes. Given a repository with full history, you can find all commits which touch a file, path, or set of files/paths. You can filter commits based on what was written in the log message. You can, with some limitations, search for commits that add/remove particular strings in the managed files. You can filter commits to a particular range of history, such as "from the last time I merged with this branch to now."

You twice mentioned history being rewritten. Generally, published history should not be rewritten specifically because it makes it difficult for downstream users to use Git to track that history. Rewriting unpublished history is fine. If users see only an exported tree representing the finished product, they never know whether you rewrote history on the way.

Pushing your overlay to another server could be expensive the first time if the repository into which you push has never seen portage.git history before. After that, it will be cheap, because Git pushes only the blobs that the peer lacks. All the historical blobs will still be on the peer from your prior push and will not be resent. You might have meant here that you want your overlay not to include any files which are unmodified relative to Gentoo Portage. If so, disregard this paragraph.

Based on your last paragraph in the opening post, it looks like your requirements are:

Publish only files changed by the overlay maintainer, not files which are identical to those in Gentoo Portage
Quickly locate the changed files for publication
(Maybe) Publish full overlay history, so readers can see what the overlay maintainer changed over time

Is this correct and complete? Do you have other requirements I did not recognize? Solving for the first two is easy. Including the third one is a bit messier.

ct85711 · Veteran Joined: 27 Sep 2005 Posts: 1791

One other thing I will put out, the commit ID's is an hash value, so in general no 2 commits should ever collide (ie a collision). More often, people use the shorthand on dealing with the commit ID because it's easier than writing the full thing out. This site explains more about git's commit ID https://www.perforce.com/blog/140723/working-git-commit-ids.

As for branches, it is intended so that you can do some work on the git tree independent from the master tree's changes. When you merge the branch back into the master tree, all of the commits done in the branch is added to the master tree (you often need to rebase your branch before merging, as the branch started off at the point you made the branch and does not include any changes in the master tree since then). Branches are great when you want to work on some new time and not risk screwing up every one else's work. Once you done, you merge and your changes gets added onto the master tree, moving it's location (and other people will rebase as they need to, to incorporate anything new on their work).

srcshelton · n00b Joined: 10 Apr 2004 Posts: 22 Location: Cambridge

srcshelton · n00b Joined: 10 Apr 2004 Posts: 22 Location: Cambridge

ct85711 · Veteran Joined: 27 Sep 2005 Posts: 1791

I will admit that I know no knowledge around layman, as I don't use any others repos outside the main tree. As such, I don't know if the clone-depth will apply in layman or not; you may need to look through its documentation to see if it has it's own independent git configuration.

From doing a quick search, it appears changing the clone-depth is retroactive, meaning it will grab the old commit history you don't have too.

One thing to remember is that each git tree is independent of each other. This means the repos you pull from layman is independent of portage's main tree (which should be easy to see, considering those repos are stored in a different place than gentoo's tree)

szatox · Advocate Joined: 27 Aug 2013 Posts: 3135

srcshelton · n00b Joined: 10 Apr 2004 Posts: 22 Location: Cambridge

srcshelton · n00b Joined: 10 Apr 2004 Posts: 22 Location: Cambridge

... although '/usr/bin/egencache --repo gentoo --write-timestamp --update' takes over 13 hours to complete on the system from which it runs, so can really only be run weekly at most

On the other hand, taking an md5sum of every file in the overlay repo takes... just under 20 seconds.

ct85711 · Veteran Joined: 27 Sep 2005 Posts: 1791

All file changes in the repo are logged in the commit, this includes the files directory and it's contents.

Git is tracking everything in the repo tree, minus the directories that is marked to exclude. So any changes you do, git is going to complain about until you commit them into your tree. This will stop git from syncing as it will see those changes that out has no clue about.

szatox · Advocate Joined: 27 Aug 2013 Posts: 3135

Hu · Moderator Joined: 06 Mar 2007 Posts: 21631

Since Git tracks the digest of every version of every file it manages, you can get that digest for any point in history, without needing to checkout those points and run an external tool. I think the flow you want is:

git clone $gentoo_portage
cp $gentoo_portage/category/package/package-1.0.ebuild $overlay/category/package/
cd $overlay
git init .
git add .
git commit -m 'Import package-1.0.ebuild from Gentoo Portage commit abc123' # Useful, but not absolutely necessary, to identify the HEAD of $gentoo_portage in this message

Time passes. One or both sides make changes to package-1.0.ebuild. For completeness, we will pretend that nobody (except Git itself) records which side(s) changed, so now we need to derive that information. To see local changes, git log -- category/package/package-1.0.ebuild. To see remote changes, we need to know the most recent common ancestor. Since this exercise intentionally does not have an actual common ancestor commit, we must fake it. In the good case, someone recorded the most recent Gentoo Portage commit ID in a Git log message. If so, then look up that ID, cd to $gentoo_portage, and run git log $recorded_id.. -- category/package/package-1.0.ebuild. If not, then we need to derive that by examining history. Run git log --raw --no-abbrev -- category/package/package-1.0.ebuild in both repositories. Find the most recent commit on the Gentoo Portage side where the shown hash matches one from the overlay side. That is the commit in Gentoo Portage when the overlay's ebuild matched the one in main Portage, either because Gentoo incorporated all changes from the overlay or because the overlay was synchronized from Gentoo. That commit is $recorded_id, which can then be used to see what Gentoo has done since that point.

srcshelton · n00b Joined: 10 Apr 2004 Posts: 22 Location: Cambridge

Last night, my checkout of the Portage mirror was up to date and in sync. Overnight, cron invokes 'emaint sync'. This morning I see:

ct85711 · Veteran Joined: 27 Sep 2005 Posts: 1791

srcshelton · n00b Joined: 10 Apr 2004 Posts: 22 Location: Cambridge

To be clear, none of the above changes were of my doing - they are the result of divergence in the repo history which occurred overnight last night and demonstrate that commit hashes are, in this case, unstable and therefore not useful in determining when changes have occurred.

(Or, at least, a non-matching hash certainly indicates a potential change, but there will be a whole host of false-positives in addition)

In any case, metadata would still need to be stored in order to know what the original state of the file was, so I've decided to simply MD5 each file in each package directory which also exists in the overlay, and then write this to a new, per-package 'files/Manifest.gentoo' file in the overlay, which can be updated after each successful upstream pull and used to detect changes irrespective of the git index state. If anyone's interested, I can post a gist of the script.

szatox · Advocate Joined: 27 Aug 2013 Posts: 3135

You have silently dismissed several good hints already and you're not even willing share anything more of your point of view than purely emotional "portage in git sucks".
Fine, you are a free man. This means you can just as well suffer from your ignorance.

ct85711 · Veteran Joined: 27 Sep 2005 Posts: 1791

Here's the thing, if there is an issue about broken/divergent history in git; this would be a very significant issue. Considering I'm not finding any recent or old bug reports or comments from anyone else about this kind of issue on the web, it leads me to believe it's something on your side. From what I was seeing on searching, the usual breaks is due to conversion from one version control software to another or due to someone resetting (which is effectively rollback some changes). Considering several companies use git for their software development, including the linux kernel, would surely have by now encounter an issue like this.

Now the least likely cause, could potentially be your hd, but I'm assuming you'd have noticed something like that... The next least option I can see, is maybe an incomplete download of the histories on your previous sync, but I'd consider than an unlikely situation. So, the only other options I know of is something with your sync scripts or some other package/program messing around in portage's tree.

Hu · Moderator Joined: 06 Mar 2007 Posts: 21631

Ant P. · Watchman Joined: 18 Apr 2009 Posts: 6920

pjp · Administrator Joined: 16 Apr 2002 Posts: 20067

Split off portage git pull: up to date (not recognizing changes).
_________________
Quis separabit? Quo animo?