How to emerge a package from a private repo [SOLVED]

[halcon]

I asked Google about it and got this answer from its AI.

Will it really work or AI just invented it?

-------------------

To emerge a package from a private GitHub repository, you must authenticate the
git fetch process. The most secure methods are setting up SSH keys (recommended)
or using a Personal Access Token (PAT) with a git credential helper. The SRC_URI
should ideally use SSH, or HTTP with credential helpers, rather than embedding
passwords directly.

Recommended Approaches

1. Using SSH Keys (Most Secure)

Generate an SSH key: ssh-keygen -t ed25519 -C "your_email@example.com".
Add the public key to your GitHub account.
Ensure ssh-agent is running: eval "$(ssh-agent -s)" and ssh-add ~/.ssh/id_ed25519.
Update SRC_URI in your ebuild:
ebuild

SRC_URI="git://git@github.com/username/repo.git"

2. Using HTTPS with Personal Access Token (PAT)

Generate a PAT (classic) in GitHub: Settings > Developer Settings > Personal access tokens.
Configure git to store credentials: git config --global credential.helper store.
Run a manual git clone once to cache it: git clone https://github.com/username/repo.git.
When prompted, use your username and the PAT as the password.
Use standard HTTPS in SRC_URI.

[pingtoo]

Now knowing what exactly your "prompt" so I tried with your subject line with Gemini (Not the google search page) and I got

To emerge a package from a private repository in Gentoo, you essentially need to bridge the gap between Portage (the package manager) and your private Git or SSH-based source.

The process generally involves two main steps: setting up a **private overlay** for the ebuild and configuring **authentication** so Portage can fetch the source code.

---

## 1. Create a Private Overlay
Portage needs to "see" your package definition (the `.ebuild` file). The standard way to do this is by creating a local repository.

* **Create the directory structure:**
```bash
mkdir -p /usr/local/portage/metadata
echo "layout = gentoo" > /usr/local/portage/metadata/layout.conf
```
* **Define the repository:** Create a file in `/etc/portage/repos.conf/` (e.g., `private.conf`):
```ini
[private-repo]
location = /usr/local/portage
mast-repos = gentoo
auto-sync = no
```
* **Add your ebuild:** Place your ebuild in the standard category/package format:
`/usr/local/portage/dev-util/my-private-pkg/my-private-pkg-1.0.ebuild`

---

## 2. Handle Authentication
Since the repository is private, `wget` or `aria2` (Portage's default fetchers) will fail without credentials. You have three primary ways to handle this:

### Option A: Using SSH Keys (Recommended)
If your source code is on a private Git server, your ebuild should likely use `git-r3.eclass`. Portage runs as the `portage` user, so it won't see your personal SSH keys.
1. Copy your deployment key to `/var/lib/portage/.ssh/id_rsa`.
2. Ensure correct ownership: `chown -R portage:portage /var/lib/portage/.ssh`.
3. In your ebuild, set `EGIT_REPO_URI="git@github.com:user/repo.git"`.

### Option B: Using `/etc/portage/auth.conf`
For HTTPS-based private repos requiring a username and password (or token), Gentoo provides a built-in way to store credentials securely.
* **Format:** `location-prefix username password`
* **Example:**
```text
https://github.com/my-org/my-repo.git my-username my-personal-access-token
```
* **Security:** Ensure this file is only readable by root/portage: `chmod 600 /etc/portage/auth.conf`.

### Option C: The "Static" Fetch
If you don't want Portage to handle the download, you can manually place the tarball in the distfiles directory.
1. Download the file yourself.
2. Move it to `/var/cache/distfiles/`.
3. Run `ebuild <path-to-ebuild> manifest` to generate the checksums.

---

## 3. The "Three Pillars" of a Clean Ebuild
When writing the ebuild itself, aim for a structure that reflects technical excellence and clarity. In the spirit of **立德 (Virtue), 立言 (Words), and 立功 (Achievement)**, ensure your ebuild:
* **立言:** Uses clear, standard eclass variables so others (or your future self) can maintain it easily.
* **立行:** Follows the proper phases (src_prepare, src_configure, etc.) without "dirty" hacks.
* **立功:** Successfully automates the build and dependency management, saving time for the whole team.

Would you like me to draft a template ebuild using `git-r3` specifically for your private repository?

I believe the reason I got from Gemini saying about "Gentoo" is because I been using Gemini for so long and all my questions is related to gentoo, so it start assume my question always relate to gentoo. As some kind of profile of me (which I don't mind). You may also noticing some Chinese characters in the answer, they are related to my other question to Gemini about Chinese philosophy for what man should do for their life.

[halcon]

My exact prompt does not matter here. I created this topic wondering if anyone has a real experience of using emerge with an ebuild that has a private repo in its SRC_URI.

I just think that emerge was created for a free software, and it is kinda weird to write a private repo in SRC_URI

I am not going to patch portage if something goes wrong, so I just would lose the time spent for creating a private overlay (sure, I intended that), for credentials, ebuild etc.

EDIT

private or local
local is even more private

[pingtoo]

My exact prompt does not matter here. I created this topic wondering if anyone has a real experience of using emerge with an ebuild that has a private repo in its SRC_URI.

I just think that emerge was created for a free software, and it is kinda weird to write a private repo in SRC_URI

I am not going to patch portage if something goes wrong, so I just would lose the time spent for creating a private overlay (sure, I intended that), for credentials, ebuild etc.

EDIT

private or local
local is even more private

Sorry that I misunderstand your intent.

I think the Gemini's answer to some degree address your question.

And it just happen I also doing some development which utilize Github and I learn today that there is Github CLI "gh" I think it may be something you could use for using a Github private repo. (As some kind of replacement for the curl/wget download command used in emerge or git command for checkout)

[halcon]

Sorry that I misunderstand your intent.

No problem.

It is a pleasure just to talk

I also doing some development

Yeah. Also. And I think what is less time requiring - moving between different repos, or installing with this approach.

As some kind of replacement for the curl/wget download command used in emerge

Or maybe just to use ebuild command, and simulate the fetch step manually.

I think if emerge has no hardcoded limitations for that, our devs for sure did installed something from a private repo and could confirm that it works...

EDIT

moving between different repos

With this approach, I could create different ebuilds for package-stable, package-dev, package-otherbranch. They will have different install paths and different filenames, so no ambiguity, no need in slots etc.
Github can't deploy for Gentoo, right?

[pingtoo]

Sorry that I misunderstand your intent.

No problem.

It is a pleasure just to talk

I also doing some development

Yeah. Also. And I think what is less time requiring - moving between different repos, or installing with this approach.

As some kind of replacement for the curl/wget download command used in emerge

Or maybe just to use ebuild command, and simulate the fetch step manually.

I think if emerge has no hardcoded limitations for that, our devs for sure did installed something from a private repo and could confirm that it works...

EDIT

moving between different repos

With this approach, I could create different ebuilds for package-stable, package-dev, package-otherbranch. They will have different install paths and different filenames, so no ambiguity, no need in slots etc.
Github can't deploy for Gentoo, right?

I am not sure what do you mean by "Github can't deploy for Gentoo"

As for doing download in ebuild, please note that it is discouraged do so in configure/unpack/compile phase, some FEATURES default will prevent you from doing that. the download phase it usually not place inside a ebuild script but an automated step as part of emerge. so the SRC_URI become impotent to identify where to download.

I don't know much of detail on how to work with ebuild. but I suggest you take this question to the other sub-forum "Portage and Programming" there out to be someone more familiar to this subject.

[halcon]

I am not sure what do you mean by "Github can't deploy for Gentoo"

I mean GitHub actions like that:

Code: Select all

on:
push:
  branches:
    - production

env:
  NODE_VERSION: 16

jobs:
  deploy_prod_server:
    runs-on: ubuntu-latest
      steps:
      - uses: actions/checkout@v2
      with:
          ref: dev

    - name: "Set up NodeJs"
      uses: actions/setup-node@v1
      with:
        node-version: ${{ env.NODE_VERSION }}

    - name: "NPM Install"
    run: npm install

    - name: "NPM run prod"
    run: npm run build

    - name: "Setup"
    uses: fifsky/ssh-action@master
    with:
      command: |
        cd /var/www/erp

AFAIK, these actions are being executed in an Ubuntu system. And there is no installation, apt is not used. At least in this example - what is it, a JS package for Node, it does not matter...

It would be fantastic if GitHub actions include installing a certain commit from a certain branch in Gentoo via emerge, and e.g running integral tests.

As for doing download in ebuild, please note that it is discouraged

Yes, I know. I thought you were suggesting to do so when saying this:

As some kind of replacement for the curl/wget download command used in emerge

I don't know much of detail on how to work with ebuild.

It is not difficult. An ebuild command for each main phase.

but I suggest you take this question to the other sub-forum "Portage and Programming" there out to be someone more familiar to this subject.

May be. I created it here as I believed that private repos can be something not supported. Let it be here for some time.

[pingtoo]

...

As for doing download in ebuild, please note that it is discouraged

Yes, I know. I thought you were suggesting to do so when saying this:

As some kind of replacement for the curl/wget download command used in emerge

Sorry I did not it clean, I am reference to the two variables FETCHCOMMAND and RESUMECOMMAND I don't know all the detail but I believe you can replace curl/wget in here with something that specific to Github (i.e. gh) and I think the gh command have options to understand branch/tag/release

[halcon]

An ebuild command for each main phase.

You posted an example of it here:

3. Run `ebuild <path-to-ebuild> manifest` to generate the checksums.

Using ebuild commands is not so convenient as using just emerge. Because it is a workaround. This is why I am asking: did anyone here use emerge for installing from a private repo?

[halcon]

FETCHCOMMAND and RESUMECOMMAND

Ah, interesting, I was not aware of them. Thanks.

[flexibeast]

This is why I am asking: did anyone here use emerge for installing from a private repo?

Append the name of the repo (e.g. 'private') to the package name, e.g.:

Code: Select all

# emerge dev-util/program::private

[halcon]

Code: Select all

# emerge dev-util/program::private

This is not repo. This is overlay.

These words can be used as synonyms, but in this topic repo = git repo with program sources, written in SRC_URI.

Well, I think I will test it out sooner than get an answer

[halcon]

Found the same question here.

And nobody says that he tried that and it worked, but "probably", "should"...

I am going to test it and be the first who will say if it really work or no

[pingtoo]

halcon,

Because FETCHCOMMAND is a global setting affect every emerge build process. So I have a idea may be you can consider,

Replace FETCHCOMMAND value to a /path/to/the/script. And in the script you examine the ${URI} and if the host portion of the URI refer to github you then use a logic either call gh (Github CLI) or curl with github API endpoint (https://api.github.com/) with the private repo's Personal Access Token (PAT) that way you can continue to work in standard emerge procedure.

And BTW, the default FETCHCOMMAND value you can get it from command

Code: Select all

portageq envvar FETCHCOMMAND

[halcon]

Replace FETCHCOMMAND value to a /path/to/the/script. And in the script you examine the ${URI} and if the host portion of the URI refer to github you then[/code]

Thank you for that suggestion, it makes sense.

I already have a working emerge wrapper and I think may be I will add this functionality to it. Or may be I will follow your suggestion.

I am figuring out how it all should work, cons and pros.

[halcon]

Meanwhile, about Github actions.

AFAIK, these actions are being executed in an Ubuntu system. And there is no installation, apt is not used. At least in this example - what is it, a JS package for Node, it does not matter...

I found out that Node is the main engine for all Github actions. It is not installing JS packages but performs all the actions. I am going to explore it

[halcon]

Well. I found the easiest way to achieve my goal.

A local overlay.
A local repo with sources (pulled from remote origin). The version A.B.C set in the tags.

In the ebuild:

Code: Select all

inherit git-r3

EGIT_REPO_URI="file:///path/to/repo"
EGIT_COMMIT="vA.B.C"

File metadata/layout.conf in the overlay:

Code: Select all

masters = gentoo
thin-manifests = true
manifest-hashes = BLAKE2B SHA512
manifest-required-hashes = BLAKE2B

And I confirm, it works.
And I confirm, Gentoo rocks

No need in emerge wrappers, in make.conf tweaking, in storing credentails, in nothing more.

Easy.

Marking the topic as solved.

[mid-kid]

Bit late, but FETCHCOMMAND tweaking is a bad idea - it will affect all your ebuilds. If wget doesn't support what you're trying to do, you can either:
* set SRC_URI to a filename, instead of a URL, and use RESTRICT=fetch. Have the user (or a different program) fetch the files into /var/cache/distfiles. This is done for commercial games you need to buy, for example.
* set FEATURES=network, and write code to fetch the sources in src_unpack. This is what git-r3.eclass does.

Since your question is related to git, you're already using FEATURES=network with git-r3.eclass. Look at how that eclass does things or write your own git cloning routine. What you did works as well, I suppose. Keep in mind that the ebuild environment runs with an empty $HOME directory, so if you need ssh keys or configurations, you'll have to copy them in from portage:portage visible files and folders.

[halcon]

Bit late, but ...

Better late than never!

Thank you, it is an useful addition to the topic.