ebuild using go get - fails to download packages from go

[KatsuoRyuu]

Hi guys im building a ebuild for minikube

The new minikube is using "go get" and for some reason im getting a bunch of

Code: Select all

go: github.com/spf13/cobra@v0.0.0-20180228053838-6644d46b81fa: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /portage/tmp/portage/sys-cluster/minikube-1.2.0/work/minikube-1.2.0/pkg/mod/cache/vcs/3acf82b7c983ee417907a837a4ec1200962dbab34a15385a11bc6f255dc04d6e: exit status 128:
	fatal: unable to access 'https://github.com/spf13/cobra/': Couldn't connect to server

for all the packages.

Anyone who can put me in the right direction on how to solve this?

Sincerely
Kat

[NeddySeagoon]

KatsuoRyuu,

Does

Code: Select all

emerge --info

show that you have FEATURES=network-sandbox set?
This prevents most ebuild phases from down loading from the internet.

[KatsuoRyuu]

God! now i feel stupid was sure i checked that, and thought it was something special for go... Thanks

[KatsuoRyuu]

@NeddySeagoon do you know if i can force network-sandbox off in the ebuild?

[NeddySeagoon]

KatsuoRyuu,

I don't know. I would hope not.
The idea is to allow the use of the internet only when its safe.

Live ebuilds, -9999 versions, fetch from the upstream repos.
Can you move your internet downloading into another phase function?

[Hu]

That download is an anti-feature in the upstream build system that needs to be burned out. Downloading during the build is fundamentally wrong.

There is an option to inhibit the network sandbox, but the documentation states it is unwelcome in the main tree. If you want to submit your ebuild for inclusion, do not use this option. In my opinion, even if you do not plan to submit your ebuild for inclusion, you should leave the network sandbox enabled and fix the problem by preventing the upstream build from needing a download. Instead, arrange for Portage to install the required prerequisites before building this package.

[KatsuoRyuu]

is there an alternative step in ebuilds that allow for more customized download then? if not we will have problems with go get, or we will need to start supporting all go packages in the portage tree.

might be me, but i think that might be the reason why minikube is old.

[Hu]

By design, ebuilds must download their required files before beginning. Required files are listed in SRC_URI. There is a special exception for live ebuilds, but even those must download during the fetch phase, not the compile phase as minikube is attempting to do. Once the declared downloads are satisfied, a package must be able to build without an Internet connection. Golang is simply broken in the idea that it can just connect to arbitrary Internet servers whenever it wants, and assume those servers will be up and serving a compatible version of the package.

Yes, packaging every required dependency is the correct solution. How else will users be able to trust that emerge --fetchonly has actually downloaded everything necessary?

[KatsuoRyuu]

And Hu this is where we stop talking again, or i might say something wrong.
But again, you help and competence on a whole other level.

[NeddySeagoon]

KatsuoRyuu,

You can't say something wrong.
You may say something that gives away your lack of ebuild writing skills but that's because you want to learn.

Wanting to learn is welcome here.

[KatsuoRyuu]

NeddySeagoon

thanks, I do understand the purpose and idea about not downloading while compiling, but as a whole, the Gentoo community might need to look into the old system, eg you could have a "go get" to get all the data, den after run go to build it. in this instance packing up the files "go get" has gotten from the source. another hook in the build for specialized getting of data might be the idea for this.

THe old system is deprecated when it comes to many languages today, unless we want to handle all packages available for each language, which is starting to become more and more impossible as each language gets its own package manager.

, Again thanks for your input


Sincerely
Kat

[Hu]

Yes, poorly reinventing package management is a recurring problem in several of the younger fad languages. Perhaps when their developers realize what a bad idea it was to have every language have its own incompatible package manager trapped in a silo, some of this will settle down. In the meantime, the correct solution is for each upstream to package their releases in a way that is easily portable: a zip, tar (with optional compression), or similar archive, named to show what version it represents, and with a corresponding source code tag so that interested parties can check out the commit that was packaged to create the archive. Upstream should include good documentation explaining what other packages need to be installed first. Ideally, this would be done by providing at least one of a srpm, deb-src, ebuild, etc. that identifies the supporting packages, but even putting it in easy to find documentation is helpful (and is much less work for upstream). If Go-based projects did this, it would be pretty easy to package them. Put the right archive in SRC_URI. Inherit a go-related eclass, and you're done. Portage already has some eclasses that seem to be for this purpose. Were they not suitable for what you wanted?

If upstream wants to have their build system reiterate the checks for supporting packages, that's fine. If they want to have their build system automatically install the missing software, that's a bad idea, but as long as it's an optional bad idea that can be bypassed by having the system package manager preinstall the required supporting packages, it's tolerable.

[NeddySeagoon]

KatsuoRyuu,

Anything that downloads random stuff from the internet, sometimes as root, is broken by design.
What happens when, not if, such a host site is compromised and users download something nasty.

When broken upstreams don't package their code, the Gentoo solution is to make and host the source tarball.
i.e. do the upstreams packaging job for them.

[KatsuoRyuu]

seems like you are going over the river for water.
no need to download in the compile phase. Also seems like there is a huge misunderstanding about these "younger fad languages", the idea has moved from: hard to develop/easy to deploy, to; easy to develop/hard to deploy.

In go instance the simple thing you have to do is;

Code: Select all

go get -d

to download the source needed for the build

then if you want you can build later, and install without a need of downloading anything.

So again, seems there is a misunderstanding of how they work. I have been happy with gentoo for 14years now, but everything in the gentoo space is starting to become deprecated. My personal ebuild repo is above 200 packages at this point, and now with go ill have to do stuff like remove the network sandbox.

I honestly believed i had missed a "special" custom download function in the build, to exactly do that, custom download.

anyways sorry for that rant, maybe its just about time to move on to another distro

[NeddySeagoon]

KatsuoRyuu,

How do you bootstrap go?

Code: Select all

go get -d 

implies you already have go installed.

I'm not picking on go here. rust is as bad, it won't use the already installed rust to build rust.

My personal ebuild repo is above 200 packages at this point, ...

That's doing it wrong. You need to join proxy-maintainers or the guru project so we can all share your ebuilds, then they wouldn't be in your personal repo. :)

I'm as bad. I run Olde Fashioned Gentooee but I don't share ebuilds I need to modify.
I need to get them into git at least, as the system I'm writing on is 10 years old and needs to be replaced.

[Hu]

no need to download in the compile phase.

That is exactly what we were trying to explain above.

Also seems like there is a huge misunderstanding about these "younger fad languages", the idea has moved from: hard to develop/easy to deploy, to; easy to develop/hard to deploy.

Why not make it easy to develop and easy to deploy, by following the best practices that were accumulated over the course of two decades of distribution management?

In go instance the simple thing you have to do is;

Code: Select all

go get -d

to download the source needed for the build

Great! So you, as the ebuild maintainer, can run go get -d --pretend, note down the download URLs it shows, put those in SRC_URI, and then your ebuild will download the files you need. Your src_unpack can put them where go will find them (probably something in GOPATH). This assumes go get has a --pretend option (not necessarily with that name), as all things that perform network downloads should.

So again, seems there is a misunderstanding of how they work. I have been happy with gentoo for 14years now, but everything in the gentoo space is starting to become deprecated.

Deprecated by whom? From my perspective, some upstarts who don't care about long term maintainability are throwing their weight around, trying to get out of packaging their work properly by making it too painful for anyone else to package properly either, then claiming that it is our fault for expecting them to satisfy some very easily attained quality standards.

maybe its just about time to move on to another distro

Why not instead teach the upstarts how to package their code properly, so that it can be both easy to develop and easy to install?

[KatsuoRyuu]

Why not make it easy to develop and easy to deploy, by following the best practices that were accumulated over the course of two decades of distribution management?

The idea is not to make it harder in general, but make development easy (to speed up the process) and when choosing how which is easier, development should be easiest. Usually just a simple parameter more when moving it to production.

Great! So you, as the ebuild maintainer, can run go get -d --pretend, note down the download URLs it shows, put those in SRC_URI, and then your ebuild will download the files you need. Your src_unpack can put them where go will find them (probably something in GOPATH). This assumes go get has a --pretend option (not necessarily with that name), as all things that perform network downloads should.

So again you want to maintain more instead of less? you are basically saying "ohh even though somebody did this before please do it again". This is also bad practice.
It should be able to build and pack it, itself - that new fancy automation thing - there is no reason why the download part cant do that on its own. A simple download hook that is expected to outputs a tar.gz file would suffice.

Deprecated by whom? From my perspective, some upstarts who don't care about long term maintainability are throwing their weight around, trying to get out of packaging their work properly by making it too painful for anyone else to package properly either, then claiming that it is our fault for expecting them to satisfy some very easily attained quality standards.

Upstarts like google? ups i mean Amazon... ohh wait Netflix? ahh shit mmm... you could see them as upstarts that doesnt care about gentoo in 14 years. But hey, nobody would use Gentoo in 14 years if almost everything is 14 years old right?
Honestly I see this mentality as a hindrance.
in Repo
Tesseract -> deprecated
MiniKube -> deprecated
Helm -> deprecated
... I could continue ... the only way to run a lot of stuff is using GIT which again reduces the build stability.
Anyways seems like over the last 10 years gentoo has gone from a nice frontrunner to a very slow when it comes to updates and software, which is a problem for security etc.

Why not instead teach the upstarts how to package their code properly, so that it can be both easy to develop and easy to install?

I could ask the same, why willingly be stuck in the 18th century when we are in the 21?

[Hu]

Why not make it easy to develop and easy to deploy, by following the best practices that were accumulated over the course of two decades of distribution management?

The idea is not to make it harder in general, but make development easy (to speed up the process) and when choosing how which is easier, development should be easiest. Usually just a simple parameter more when moving it to production.

So we agree that both should be easy. Now, why not make it easy by following good practice, like having ready-to-download archives that, when unpacked, can be built using commonly available tools?

Great! So you, as the ebuild maintainer, can run go get -d --pretend, note down the download URLs it shows, put those in SRC_URI, and then your ebuild will download the files you need. Your src_unpack can put them where go will find them (probably something in GOPATH). This assumes go get has a --pretend option (not necessarily with that name), as all things that perform network downloads should.

So again you want to maintain more instead of less? you are basically saying "ohh even though somebody did this before please do it again". This is also bad practice.

I don't want to maintain it at all, and I don't. I don't use Go, and neither should you, until they learn a bit of discipline. However, since you want to pursue this, I will point out that it's effectively unavoidable that you need to record the download URLs somewhere, so you may as well have the tool do as much work as possible. With that in mind, if there is a go generate ebuild target that would print out the ebuild, that would be nice, although admittedly very specialized. In theory, this should be pretty easy, since the ebuild just needs to download the archive (which go will know the URL of), unpack it (default src_unpack should be sufficient), then run go compile and go install DESTDIR=$D for src_compile and src_install. Therefore, the generated ebuilds should be almost entirely boilerplate inheriting eclass files. This is also great for maintainability and for keeping current with upstream releases, because simple files are easier to proofread and easier to approve.

It should be able to build and pack it, itself - that new fancy automation thing - there is no reason why the download part cant do that on its own. A simple download hook that is expected to outputs a tar.gz file would suffice.

Yes, and where do you put that tar.gz that other people can then download it? Will the author run a webserver on the release management server? The source control server is often a viable place to host that archive file, and if it is hosted over https, Portage can download direct from there, as soon as you give it the download URL in SRC_URI.

Deprecated by whom? From my perspective, some upstarts who don't care about long term maintainability are throwing their weight around, trying to get out of packaging their work properly by making it too painful for anyone else to package properly either, then claiming that it is our fault for expecting them to satisfy some very easily attained quality standards.

Upstarts like google? ups i mean Amazon... ohh wait Netflix?

Yes, Google for Golang and Mozilla for Rust, both of which are horrible names for programming languages. Why does it matter where the authors work? Bad practice can take root anywhere, and has taken root to a disgusting degree in both communities. I would have expected such premier companies to have a better internal culture and for that culture to be reflected in good practice for these projects.

you could see them as upstarts that doesnt care about gentoo in 14 years. But hey, nobody would use Gentoo in 14 years if almost everything is 14 years old right?

I think you meant this to be some sort of pointless attack, but I don't understand your point here. Please rephrase, or switch to a better argument.

Honestly I see this mentality as a hindrance.

Expecting authors to follow good practice, or at least not stand in the way while others follow it for them is not a hindrance.

in Repo
Tesseract -> deprecated
MiniKube -> deprecated
Helm -> deprecated
... I could continue ... the only way to run a lot of stuff is using GIT which again reduces the build stability.

Packages become deprecated if there is no one both willing and able to maintain them. When upstream is openly hostile to good practice, it's not surprising that it's hard to find people who want to maintain the package in a Linux distribution.

Anyways seems like over the last 10 years gentoo has gone from a nice frontrunner to a very slow when it comes to updates and software, which is a problem for security etc.

Are any of these packages affected by known security bugs? Would bumping the version fix those bugs? If yes to both, are the maintainers aware of this?

Why not instead teach the upstarts how to package their code properly, so that it can be both easy to develop and easy to install?

I could ask the same, why willingly be stuck in the 18th century when we are in the 21?

Gentoo did not exist in the 18th century, so your statement doesn't make any sense. If you're asking, "why institute procedures that require packages to be handled in a sane and maintainable way instead of tolerating the latest reckless fad?", then I would say it is because I'd rather my systems be maintainable, predictable, and repeatable, which means following good practice. Build systems that download and compile arbitray blobs in the middle of the install process are not good practice, as several posters have already tried to explain to you.

My question above stands: why not help upstream improve their practices, so that they are easier for everyone, not just Gentoo, to package and to keep current? Gentoo ebuilds are extremely easy to create if a package follows good practice and works according to a standard template. I expect RPMs/DEBs are likewise very easy to create if the packager is not compensating for strange upstream decisions. Thus, if you get upstream to behave, all the Linux distributions can more readily track upstream closely.

[Ant P.]

Upstarts like google? ups i mean Amazon... ohh wait Netflix? ahh shit mmm... you could see them as upstarts that doesnt care about gentoo in 14 years. But hey, nobody would use Gentoo in 14 years if almost everything is 14 years old right?

Nobody would use Gentoo if Gentoo projects became unmaintainable and abandoned at the same rate Google ones do...

[KatsuoRyuu]

Nobody would use Gentoo if Gentoo projects became unmaintainable and abandoned at the same rate Google ones do...

And nobody will use Gentoo when the packages become more and more deprecated and harder and harder to maintain.
I'm specializing in Maintenance, and maintainable code structures, I build our Pipelines my team at work, and thats not a startup. Still with an enterprise company like this we are adjusting to the tech, which Gentoo has seemed to stop doing.

[Ant P.]

Gentoo is a volunteer-driven project and seems to be doing fine for lack of this one language. Veiled threats will not get you free work.

Go packages are poorly integrated because nobody is motivated to fix it (yelling at them will only make it worse), and because Google did not design the language to facilitate third-party self determination, very intentionally, like all its other products. That should be a huge red flag to avoid it for long-term projects.

[KatsuoRyuu]

@Hu - Let skip the part we agree - the part you dont understand was not an attack on anything other, then for me I liked gentoo when it was a bit more bleeding edge, I feel a lot of packages are falling behind.

So lets go to some of the other points.

Lets go back to your question then.

why not help upstream improve their practices, so that they are easier for everyone, not just Gentoo, to package and to keep current?

Im not saying not to do that - but how would you do that? its not like you can go to the Japan National bank and tell them to start using USD or BTC instead of JPY. If it was i would gladly do that.

Opposite you are again talking about bad practice - but you can download the source - it might not come from a blob, but where is the difference from git,svn,ftp,http. You are (in case of go) going to build a GO application so you need to go compiler, there by you have the downloader, so no change in dependencies. You can download the content and repack it putting it in your distfiles folder. If ebuild was made so that it could handle customized download strategies like that its no problem, but if you the only acceptable is tar.gz then why do we have git, or svn for that matter.

wouldnt it just be

Code: Select all

// Download process
go get -d

// Pack it zip, tar etc

// build
go build

// install eg. /opt/go
GOPATH=/opt/go go install

Is this not the process you are actually talking about, if not please explain so i can understand you viewpoint better.

I know know how much more you can simplify a language to build? if you go back to what i said, doing development is even easier.

Code: Select all

// get all dependencies
go get

// build and run your developed program
go your_go_file.go

[KatsuoRyuu]

@Ant P.

Veiled threats will not get you free work.

Nothing is meant as a threat, if you see anything like that, then thats your problem.

Go packages are poorly integrated because nobody is motivated to fix it

Im not yelling or anything. if you notice the original question was if there is a way to do it, but this is again one of those instances where I will just be maintaining my own builds and distribute them to my systems from my binary server and serve my own ebuilds because a feature needs to be disabled to build it. But all good.

and because Google did not design the language to facilitate third-party self determination

If you want it you can make a patch, it is open source, the license is even nice and open. So please fix the issue.

[Hu]

And nobody will use Gentoo when the packages become more and more deprecated and harder and harder to maintain.

Then it's a good thing that the core Gentoo packages are widely used and are not being deprecated.

I'm specializing in Maintenance, and maintainable code structures, I build our Pipelines my team at work, and thats not a startup. Still with an enterprise company like this we are adjusting to the tech, which Gentoo has seemed to stop doing.

When the technology is as poorly considered as this particular Golang feature seems to be, you shouldn't adjust to it. You should work around it, and bug upstream Golang to behave better. In your maintenance workflow, how do you propose to ensure that if you install minikube today, again next month, and a year after that, all nominally from the same commit, that you actually get the same code every time? As far as I can tell, if you let Golang's go get download this supporting package on its own, you can't ensure that. You can't even ensure that the test environment and the production environment install the same code, and you can't install in either of those without granting https access to Github.

I liked gentoo when it was a bit more bleeding edge, I feel a lot of packages are falling behind.

Are they falling behind because upstream is behaving badly or because Gentoo lacks the resources to keep current? If the latter, you could volunteer to proxy-maintain packages that are important to you. If the former, you could volunteer in upstream projects to teach them to behave better. Remember that projects gain and lose contributors over time. The people making a mess of the project now may not be the same visionaries who created it.

Why not help upstream improve their practices, so that they are easier for everyone, not just Gentoo, to package and to keep current?

Im not saying not to do that - but how would you do that? its not like you can go to the Japan National bank and tell them to start using USD or BTC instead of JPY. If it was i would gladly do that.

Join a project that is behaving badly. Contribute patches that make it easier for distributions to package. In the case at hand, Golang needs:

• A way to prevent the automatic download, and instead display a diagnostic telling the user that supporting packages are missing.
• Ebuilds for the supporting packages
• Ideally, a way to generate ebuilds, srpms, deb-srcs, etc. from the control information that Golang already uses to drive go get.

Opposite you are again talking about bad practice - but you can download the source - it might not come from a blob, but where is the difference from git,svn,ftp,http. You are (in case of go) going to build a GO application so you need to go compiler, there by you have the downloader, so no change in dependencies.

First, I consider it a fundamentally wrong design to bundle the compiler and the downloader into a single package. How do you propose to handle the case that I have an Internet-connected system with no Go, and a non-Internet-connected system (that gets all its files via sneakernet) with Go, and I want to install Golang packages on the latter? With Portage packages, this is easy. Get the same Portage tree on both systems. Run emerge --fetchonly --pretend on the build system. Take that list to the download system. Download those files. Bring them to the build system. Run emerge, and it will use the manually provided files. This is a common use case during early setup, because people don't always have a working network, especially if their only network access on the build system comes through a wireless card. Fortunately, for today, most people in that situation can get their environment working without needing anything based on Golang. Will that continue to be the case, or will bootstrap-relevant packages be rewritten in Golang? If Golang is fixed soon to support split installs, we don't need to worry that such rewrites will trouble users later. Second, in the case shown (and as I understand go get, this applies more generally), you can download some source, but what source do you get? Your opening post was trying to fetch the latest commit, not any particular version. For some packages, it is necessary to be able to pin supporting libraries to known good versions. (This is in some cases a workaround for the supporting upstreams making breaking changes, but the reality is that some projects do that, so others need a way to cope.)

I expect that, if you disabled the network-sandbox that affected your opening post, then built minikube, uninstalled it, and built it again, then go get would download that supporting library again. For people with limited bandwidth, that is undesirable. Portage downloads the sources once, then keeps them until told otherwise. This enables offline rebuilds, and ensures that rebuilding a non-live package does not suddenly produce different results through no action of the user.

You can download the content and repack it putting it in your distfiles folder. If ebuild was made so that it could handle customized download strategies like that its no problem, but if you the only acceptable is tar.gz then why do we have git, or svn for that matter.

ebuilds can handle anything that you can reasonably unpack. tar.gz is a common form, but not the only one. We support git, svn, etc. for those projects that refuse to issue releases in a timely manner. This covers both projects that develop so rapidly that a release would promptly be out of date and projects that simply don't issue releases, even for bugfixes, on any useful timetable.

Putting an appropriate archive in SRC_URI, as I described above, would download the content to your distfiles for you, and handle applying appropriate checksum validation.

wouldnt it just be

Code: Select all

// Download process
go get -d

// Pack it zip, tar etc

// build
go build

// install eg. /opt/go
GOPATH=/opt/go go install

You missed the step where we check the signature on the downloaded archive, but generally, yes. However, that path is only appropriate if you are building only for yourself and have no plans to be able to re-package later, or uninstall, or record what version you built. None of what you showed integrates with the system package manager at all, so whether you're on Gentoo, Fedora, or Debian, you won't later be able to use standard tools to check which package installed a file, or check which files were modified post-installation. Registering the installed files with the system (/var/db/pkg for Gentoo) enables common tooling to handle the files effectively. Suppose a supporting package issues a critical security update, and you need to rebuild every Golang program that linked to it. How do you find which ones those are? If we were discussing packages under Portage control, the answer would be to inspect the package database for dependency information.

If you want to make an ebuild, you need one or more URIs where you can download the sources, you need go get to read from distfiles instead of the network, and if upstream doesn't release source archives, then you need some way to make those and upload them to an appropriate mirror.

If you want it you can make a patch, it is open source, the license is even nice and open. So please fix the issue.

The license may be open, but the contribution process is not particularly friendly. According to their documentation:

• Step 1: Contributor License Agreement

[KatsuoRyuu]

@Hu

I understand all your points, and they make sense for most older languages.
But the way we use Go/Rust/Node (this is mainly go) when developing, yes then it's a problem if you use the same for production.
But Go builds statically, so all "essential" libraries are irellevant after production build. A custom Download would force all source code into one dir of your choice, then you can pack it up to distribute it to offline machines.
Actually after this "Download" (go get -d) step we store our stuff in artifact and remove net connection on our CI, and only load from artifact store from that point. This is why I have a hard time seeing the problem.

I would like to know more! So if you have anything I have missed please tell me?