Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Package / EBuild / Git Development Workflow
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 9549
Location: Somewhere over Atlanta, Georgia

PostPosted: Fri Oct 20, 2017 3:22 pm    Post subject: Package / EBuild / Git Development Workflow Reply with quote

In the privacy of my own cubicle, I'm working on several packages and their associated ebuilds. For now, they're all "live" ebuilds using Git (through the git-r3 eclass) to fetch the source from a company-run GitLab repository. The ebuilds react to version tags and also to the Gentoo standard convention that version 9999 means HEAD of the master branch.

One other kind of cool thing I've done in addition to the above is that version 9998 is mapped to the HEAD of the feature branch I'm currently developing on. Of course, I have to edit the ebuild whenever I change feature branches but that doesn't happen too often.

This got me thinking, can I have a reasonable development workflow that uses Portage exclusively for builds? So the workflow I'm trying out is as follows:
  1. Checkout / create the feature branch.
  2. Develop a little.
  3. Commit & push.
  4. emerge --oneshot =my-package-9998
  5. Test a little.
  6. Repeat 2 - 5 as necessary.
  7. Eventually merge my feature branch with master and tag a new version.
However that means that code to produce ephemeral debug chatter and the occasional dumb mistake get's made more or less permanent because it's pushed to upstream. I'd like the opportunity to squash / sanitize the commits in my local repository to eliminate ephemera before the commits are pushed to origin without losing the ability to do all my builds with Portage. So what I'd like to be able to do is to have the ebuild reference a local Git repository feature branch, only fetching changes if the local repository is out of date with respect to origin. At first (and second) glance, the git-r3 eclass doesn't appear to support this workflow.

Does this seem like a reasonable thing to do? What would you do differently?

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 11312

PostPosted: Sat Oct 21, 2017 12:45 am    Post subject: Reply with quote

That seems like a reasonable workflow. As an aside, you could rewrite history on the branch just before merging it to master, so as to eliminate the temporary / noise commits, but still enjoy the incidental benefits of storing your feature branch on the centrally managed (and centrally archived) company server. This would also provide access to the in-progress branch for peers to learn or do code review, let you checkout from other systems, etc. When you are ready to merge to master, use git rebase --interactive to reorder history, dropping/combining commits as needed, then merge the resulting branch to master. After the merge, delete the topic branch from the server or reset it to master. It won't (necessarily) merge cleanly with master after an altered version of it is merged.

Assuming you want to try your workflow of tracking features locally until finished: As I understand the Portage eclasses for working with Git, they sync from their chosen upstream (unless told not to) on every run. So it seems like all you should need to do is arrange for the -9998 ebuild to point to a Git clone that is private to your work, while the -9999 ebuild continues to point to the company IT server. Set EGIT_REPO_URI="file:///home/jrg/work/git" (assuming that ~/work/git is the local clone of the project, and that the path is readable by Portage). If you need more precise advice, would you mind posting appropriately anonymized ebuilds for your projects?
Back to top
View user's profile Send private message
miket
Guru
Guru


Joined: 28 Apr 2007
Posts: 371
Location: Gainesville, FL, USA

PostPosted: Sat Oct 21, 2017 5:58 am    Post subject: Reply with quote

I have been wanting to tinker with packages and find your suggestions to be incredibly helpful. It means not having to keep track of installed files or worrying how the files would get into place--or how to be able to revert cleanly to the base installation of the package.

It would appear this would be a good way to work for tweaking a package at a particular version. It seems that for the purposes of one's own development of such a package the "live" version would not have to be -9998 but something like, say, -3.1-r9998 (assuming the idea is to produce a new revision of version 3.1). The private ebuild would specify the head of of whatever branch it is for the version of interest and point to the local checkout where the work is being done. Normal emerge commands would do the build and install the binaries.

When the changes are ready, it would then be a simple matter of making a patch file from a git diff relative to the start of the changes, which is the tag which identifies the released version. This allows for the normal Gentoo pattern of applying patches to the expanded contents of existing upstream tarballs. A complicating factor at this point is that the existing ebuild might specify Gentoo patches to the tarball. I would think that the best way to deal with this would be to start by committing that patch (or those patches) to new branch and then modifying the ebuild not to do patches. Then, at the end, you could choose whether to meld the existing Gentoo patches into the new changes (by taking the diff from the tag at the beginning of the branch) or to retain the original patch file(s) by taking the diff from the revision that did the patch-file commit.
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 9549
Location: Somewhere over Atlanta, Georgia

PostPosted: Sat Oct 21, 2017 1:16 pm    Post subject: Reply with quote

Hu wrote:
... As an aside, you could rewrite history on the branch just before merging it to master, so as to eliminate the temporary / noise commits, but still enjoy the incidental benefits of storing your feature branch on the centrally managed (and centrally archived) company server. ...
Clearly I need to learn more about the nuances of merging; I'll do some studying. The second point had occurred to me as many of my development efforts are collaborative. Furthermore, although for my own benefit I follow a "commit often" paradigm, I'm not really trying to hide all the commit chatter from my colleagues short term, but just to eliminate ex post facto cruft that doesn't have long term benefit.

Hu wrote:
... As I understand the Portage eclasses for working with Git, they sync from their chosen upstream (unless told not to) on every run. So it seems like all you should need to do is arrange for the -9998 ebuild to point to a Git clone that is private to your work, while the -9999 ebuild continues to point to the company IT server. Set EGIT_REPO_URI="file:///home/jrg/work/git" (assuming that ~/work/git is the local clone of the project, and that the path is readable by Portage). If you need more precise advice, would you mind posting appropriately anonymized ebuilds for your projects?
I had tried some variation of this before the need to make progress against schedule became acute, without success, but I may have made a mistake. I'll look at it again. With regards to posting the ebuild, these almost never contain any proprietary information. I think I can post it verbatim. I almost posted it on Friday; will do so as soon as I can, especially since miket has expressed an interest.

Thanks for the continued advice.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.


Last edited by John R. Graham on Sat Oct 21, 2017 2:02 pm; edited 1 time in total
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 9549
Location: Somewhere over Atlanta, Georgia

PostPosted: Sat Oct 21, 2017 1:26 pm    Post subject: Reply with quote

miket wrote:
... It means not having to keep track of installed files or worrying how the files would get into place--or how to be able to revert cleanly to the base installation of the package.
Yes, that was my main goal: to have Portage track everything I install, period.

miket wrote:
... It would appear this would be a good way to work for tweaking a package at a particular version. It seems that for the purposes of one's own development of such a package the "live" version would not have to be -9998 but something like, say, -3.1-r9998 (assuming the idea is to produce a new revision of version 3.1). The private ebuild would specify the head of of whatever branch it is for the version of interest and point to the local checkout where the work is being done. Normal emerge commands would do the build and install the binaries.
Right. :) I don't know that 9998 is my final convention; I'm still thinking about that. But I do agree that meaningful variations on the theme are a good idea to differentiate feature branches.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 11312

PostPosted: Sat Oct 21, 2017 9:46 pm    Post subject: Reply with quote

Actually, for this part, you should be studying cherry-pick/rebase, not merge. See man git-rebase for full details. In particular, read about git rebase --interactive and the whole section INTERACTIVE MODE. In short, your workflow would be:
  • Decide that the work is ready to merge (but requires cleanup).
  • Checkout the tip of your topic branch. You may wish to use git checkout --detach or branch your topic to another (temporary) name so that mistakes are easier to undo. If you find that your rebase has made such a mess that you want to discard it and start over, you may use git rebase --abort if and only if you have not yet finished the rebase. If you finish the rebase, then discover you want to discard the rebase and start over, you need to return to the initial commit at the tip of your dirty feature branch. If you created a temporary branch or used a detached HEAD, you can git reset --hard to the dirty feature branch by name. If you did neither of those safety measures, then the dirty feature branch's name now refers to the commit you want to abandon. You will need some other way to identify the pre-rebase commit: reflog, other branch instances that refer to the right commit, output of git log from before you began, etc. Although none of those are hard, they're all less convenient than the safety of doing the test rebase on a temporary branch.
  • Run git rebase --interactive origin/master. Git will open $EDITOR with a list of all the commits present in your current HEAD that are absent from origin/master. For any commits that are wholly unwanted (perhaps they only added debug logging that you no longer need), delete their lines from the list. For commits that fix mistakes made in earlier listed commits, move the fix commit to be immediately after the to-be-fixed commit. Change the verb on the moved commit from pick to fixup. For commits that contain some useful content and some non-useful content, change the verb to edit. Git will stop at that point in history so that you can amend the commit to remove the non-useful content. After you have amended the commit, run git rebase --continue to resume applying patches. Git will also stop if it encounters a merge conflict. This is usually caused by reordering patches too aggressively (or simply incorrectly). Use git mergetool to visit and fix each conflicted file. Then, run git rebase --continue. If you find yourself facing numerous merge conflicts, you may want to abandon the entire rebase and start over with a different commit reordering. Use git rebase --abort for this. Note that by aborting, you return to your original starting point, discarding all rebase work, including any that was successful.
  • When your rebase is done, run any final tests. Optionally compare its contents to the pre-rebased tree to ensure that you only dropped unwanted changes.
  • When you are satisfied, you now merge this commit, not your original feature branch, into master for publication and long term use. Once merged and pushed, you may delete both the feature branch and the temporary branch, if any.
Back to top
View user's profile Send private message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 4809
Location: The Peanut Gallery

PostPosted: Sun Oct 22, 2017 12:24 pm    Post subject: Reply with quote

One tip I have for general git usage: use git add -p instead of rebasing after the event.

It's much easier to reason about when you think forward in time, rather than trying to "rewrite history" which ends up being a time-sink ime.

OFC sometimes you need to rebase when following upstream; personally I find it better to set that up in the config/ when you branch (a tracking branch.)

I don't have more detail on me atm (there's a couple of tweaks I use in config); but #git are lovely people, and very knowledgeable.
Just follow the usual method of explaining what you're actually trying to do at a high-level, before you get specific.
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 9549
Location: Somewhere over Atlanta, Georgia

PostPosted: Sun Oct 22, 2017 8:15 pm    Post subject: Reply with quote

Hu wrote:
... As I understand the Portage eclasses for working with Git, they sync from their chosen upstream (unless told not to) on every run. So it seems like all you should need to do is arrange for the -9998 ebuild to point to a Git clone that is private to your work, while the -9999 ebuild continues to point to the company IT server. Set EGIT_REPO_URI="file:///home/jrg/work/git" (assuming that ~/work/git is the local clone of the project, and that the path is readable by Portage). If you need more precise advice, would you mind posting appropriately anonymized ebuilds for your projects?
So I thought I had done that before but when I did it again it Just Worked™. Must've been one of those dumb mistakes I was talking about. Thanks so much, Hu, for the sanity check. Here's an ebuild for a project of mine that supports the enhanced aspirational workflow I was talking about above (chickened out, so anonymized):
Code:
# Copyright 1999-2017 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

EAPI=6

inherit eutils autotools git-r3

DESCRIPTION="Package Description"
HOMEPAGE="http://anonymized-url.com/my-package"
EGIT_REPO_URI="git@anonymized-url.com:johngrah/my-package.git"

LICENSE="My-Company-License"
SLOT="0"
KEYWORDS="~x86 ~amd64"
IUSE=""

DEPEND="app-misc/jq dev-libs/json-c"
RDEPEND="${DEPEND}"

# JRG: Special version numbers to access specific branches.
case "${PV}" in
   # Local feature branch.
   9998)
      EGIT_REPO_URI="file:///home/johngrah/Projects/my-package/"
      REFS="refs/heads/feature-branch-name"
      ;;
   # Head of master branch. This is a Gentoo convention.
   9999)
      ;;
   # Normal tagged releases.
   *)
      REFS="refs/tags/${PV}"
      TAG="${PV}"
      ;;
esac

src_unpack() {
   git-r3_fetch ${EGIT_REPO_URI} ${REFS} ${TAG}
   git-r3_checkout ${EGIT_REPO_URI} ${WORKDIR}/${P} ${TAG}
}

src_prepare() {
   eautoreconf
   default
}
Since this package uses an essentially standard *nix autotools build paradigm, the ebuild is nearly empty except for the Git gymnastics, which are all I'm trying to disclose anyway.

The enhanced workflow looks something like this:
  1. Checkout the feature branch.
  2. Develop a little.
  3. Commit. This only commits to the local copy.
  4. emerge --oneshot =my-package-9998
  5. Test a little.
  6. Repeat 2 - 5 as necessary.
  7. Push feature branch to upstream for peer review and/or backup.
  8. Fetch any updates resulting from peer review.
  9. Repeat 4 - 6 as necessary.
  10. Optionally rebase the feature branch per Hu above.
  11. Merge the feature branch with master and tag a new version.
That's the broad strokes. Of course there are a multitude of reasonable variations on that theme.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
Ant P.
Advocate
Advocate


Joined: 18 Apr 2009
Posts: 4445

PostPosted: Mon Oct 23, 2017 3:29 am    Post subject: Reply with quote

I would have gone with 9999-r1 for the custom branch, reasons being that "9999" is understood as a magic number by portage and most utils to mean "this is a live ebuild, handle with extra caution", but also emerge will put the "r1" in ${PR} for you, which may or may not make things easier.

I've seen some main tree ebuilds use a convention of -r${really_high_number} to mean "same package but different results" so this isn't too crazy of an idea. The mysql ebuilds used it once when they were transitioning to slotted, and the most annoying example is gucharmap-3.0.1-r200, which doesn't actually install the program in the source tarball... *groan*
_________________
*.ebuild // /etc/service/*
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 9549
Location: Somewhere over Atlanta, Georgia

PostPosted: Mon Oct 23, 2017 1:32 pm    Post subject: Reply with quote

Hmm. There's no mention of 9999 in the Portage executable source code outside of what appears to be unit testing code. It does get one mention in the main code comments, implying that 9999 ebuilds should have no keywords so as to be ineligible for autounmask, which seems a reasonable practice to me.

Do you know of a utility that explicitly treats 9999 specially?

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
Ant P.
Advocate
Advocate


Joined: 18 Apr 2009
Posts: 4445

PostPosted: Mon Oct 23, 2017 3:23 pm    Post subject: Reply with quote

You may be right... even app-portage/smart-live-rebuild seems to check for vcs eclasses in use rather than the version number. I guess they don't work like that any more. In which case: why is that a convention?
_________________
*.ebuild // /etc/service/*
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 11312

PostPosted: Tue Oct 24, 2017 1:42 am    Post subject: Reply with quote

I think the original motivation was that 9999 was higher than any upstream release was likely to use (until somebody had the bright idea of using YYYYMMDD as their "release number", at which point some live ebuilds switched to using more 9s: 99999999), so a version 9999 package was always newer than any regular release of that package, and would be preferred whenever keywords/masks permitted use of a live ebuild. This would prevent Portage from ever automatically reverting to a regular release, even as new releases with higher numbers were added to the tree.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum