[SOLVED]: stage3 file failed SHA512

[twork]

I'm trying to do my first build of the Build.Dist directory from https://github.com/GenPi64/Build.Dist#

Obvious question, is that the wrong way to go? If not:

Reading the "Usage" section of the README file there, I find that either it's out of date, or I'm misunderstanding stuff, so coach me there...

I've made it as far as making sure that I have "dev-python/lockfile" installed. Then "sudo sh build.sh" reaches this point:

Code: Select all

[...]
+ expected_hash=c5f8cb88806a10a847c1faa062147abcfcf5285117c08ecd857549d56b56944d91d28f566403d29258cc559d5cfcb0859a5608950ee2dfbd31d4314ef88312ad
++ sha512sum /home/mjinks/src/Build.Dist/build/GenPi64OpenRC/build-binary-assets/stage3/stage3-arm64-openrc.tar.xz
++ awk '{print $1}'
+ actual_hash=40511c526d7648f0e40b1b3cd3cbc9426a8f45317d5c21907cdc9e4260ddd07045fbce5fd45a83a80387e691db8c76caca6f3893d5f392caafae9aa5382c0369
+ '[' c5f8cb88806a10a847c1faa062147abcfcf5285117c08ecd857549d56b56944d91d28f566403d29258cc559d5cfcb0859a5608950ee2dfbd31d4314ef88312ad '!=' 40511c526d7648f0e40b1b3cd3cbc9426a8f45317d5c21907cdc9e4260ddd07045fbce5fd45a83a80387e691db8c76caca6f3893d5f392caafae9aa5382c0369 ']'
+ echo 'ERROR: Downloaded stage3 file failed SHA512 verification (attempt 2)'
ERROR: Downloaded stage3 file failed SHA512 verification (attempt 2)
+ rm -f /home/mjinks/src/Build.Dist/build/GenPi64OpenRC/build-binary-assets/stage3/stage3-arm64-openrc.tar.xz
+ '[' 2 -eq 2 ']'
+ exit 1
FATAL: JOB downloadstage3 FAILED with exit code 1
FATAL: JOB gentoo-base FAILED with exit code 1
run complete.

Does that just mean that a key somewhere along the chain is due for an update and I should wait a while and try again? Or have I overlooked a step somewhere? Or...?

Thanks.

[Hu]

As a general comment, seeing sudo sh build.sh is worrisome. You granted the script the ability to write anywhere, so we cannot assume anything about the integrity of system packages anymore.

To your immediate question, no, this is not a cryptographic key issue. The script expected a particular sha512 digest of the downloaded file, and the file it received did not match that digest. This could be because the file was truncated during download, was deliberately replaced upstream (possibly as an improvement; possibly as a mistake; possibly as an attack), or that the upstream server returned the wrong file entirely. I've seen cases where the downloaded file is just an HTML error page that the server sent because the operator wrongly thought all clients are browsers and so wrongly believed serving an error page (with a status code of 200 OK) would be acceptable. We would need to inspect the rejected file to determine why it did not have the right digest.

[pingtoo]

twork,

I think GenPi64 is not a Gentoo project, so it may be better direct your question to project maintainer for better answer.

So I took interest to look at the project code, I can see you posted messages from Build.Dist/parsers/downloadstage3/downloadstage3 script but I am not able to figure out where it got stage3 file. My guess is the build environment have some stale information therefor it need update. May be if you show more build log I can guess where/how to changes. (No guarantee

[twork]

I do recognize that GenPi64 isn't a Gentoo-native undertaking; on the other hand, it does rely on Gentoo tools, its output is a Gentoo machine once the dance is done; my understanding is that it it is used by other Gentoo users; and I haven't been able to find conversations about it, outside of the code itself, so I posted here. Earlier I tried to make Gentoo machines using Gentoo-only information and kept hitting roadblocks (my own fault, no doubt), but that seemed to be caused partially by the fact that Gentoo work has to account for a wide range of machines, while this is an effort to handle the specifics of Pi itself, so...

As a general comment, seeing sudo sh build.sh is worrisome.

Agreed. I wish this process worked in, for instance, "the Portage way". But here's the top of build.sh:

Code: Select all

if [[ ! "$UID" == "0" ]]; then
    echo "Build must run as root!"
    exit 1
fi

Meanwhile I see that activity on this project is quite frequent within the "Builld.Dist" area of GenPi64, unlike that on the "main" section which dates back about five years, left as-is due to obligations of its initiator.

The README page on Build.Dist tells the user to choose and download one's own image, but that build.sh script ignores the path listed there, then looks in a different part of the build structure, fetches the source from ".../config/pyconfig/GenPi64Config.py":

Code: Select all

[...]
    "stage3": os.environ.get("STAGE3", "stage3-arm64-openrc.tar.xz"),
[...]

...and assumes that this is where the source image will be. I see that string referenced in a few places, but only defined there (maybe...?) so my first guess has been not to argue. Second guess: "Hey all, hints?"

Code: Select all

You granted the script the ability to write anywhere, so we cannot assume anything about the integrity of system packages anymore.

Agreed. Thank goodness for doing this on a backed-up VM in a contained network.

Code: Select all

The script expected a particular sha512 digest of the downloaded file, and the file it received did not match that digest.  This could be because the file was truncated during download, was deliberately replaced upstream (possibly as an improvement; possibly as a mistake; possibly as an attack)

Understood. Still, this is a highly regarded GitHub project, or so I read, but parts of the project itself are more active than others. So, Evil? Or, various people putting in what time they can where they can? Problems of both scrambling and Evil are what sha512-etc. are intended to (partly) solve, but the source of the mishaps could go either way. Or -- safe to assume -- it could be my own booboo, so I come here seeking wider insight.

[Hu]

The line you cited delegates to a process environment variable, with a fallback to that fixed name. Therefore, if build.sh set $STAGE3 based on your input, then the cited line could still be respecting your input, albeit in an indirect way.

With the caveat that undiscovered exploits could skew the statistics, I think it more likely that this checksum caught a non-malicious error than that this saved you from an attack. However, even non-malicious errors can still cause havoc, so I wouldn't override the check without understanding why it is failing. From where did the expected value of the checksum come? Is it hardcoded in the script as a literal, computed in the script from a lookup in a table of literals, or read in from outside? Even if this error is your fault (rather than a problem upstream), it could still be the case that you broke it by substituting in a file that won't satisfy what the script wants to do next, which would again leave us with the situation that bypassing the check causes problems later.

[pingtoo]

I use genpi64 image initially on my rpi4. (not building process, just a binary image file)

later I just build gentoo's stage3 using docker. unless you are interesting in alternative building process (alternative as vs Catalyst) you really don't need to build your own stage3.

Just in case you are unaware Catalyst can also build stage4 too.

[twork]

I use genpi64 image initially on my rpi4. (not building process, just a binary image file)

Where do you draw that image from? I thought that was the one I referred to, at: https://github.com/GenPi64 No?

It has worked for me in making a system image, but that image was too old to download and build the latest portage, at least without lots of fixing by me.

Next to it, the "Build.Dist" section looked more live, so I tried that lately, but, as noted, it's wrapped with a shell script which differs from the posted instructions -- maybe because I overlook something in my use of it.

I'm prone to overlooking stuff in GitHub. Should I have downloaded my genpi64 some other way? I hit the green "Code" button, then use whatever "-master" the site sends; that's how I've gotten both the "old" and the "too-root-y" stuff.

Just in case you are unaware Catalyst can also build stage4 too.

Unaware I was indeed, so thanks. (Probably tried there some time ago and fergitt'd about it, let's see how I goof this time.)

Thanks.

[twork]

The line you cited delegates to a process environment variable, with a fallback to that fixed name. Therefore, if build.sh set $STAGE3 based on your input, then the cited line could still be respecting your input, albeit in an indirect way..

Yeah, and, thanks for the nudge, I'd tried to trace its steps before, but didn't look the right way.

After moving the source file from ' ~/src/Build.Dist/build-binary-assets/stage3' to '~/src/Build.Dist/build/GenPi64OpenRC/build-binary-assets/stage3/', the build process is now going a lot further along, as I write. I'll still look at Catalyst as well but this is a step.

Thanks.

[twork]

After moving the source file from ' ~/src/Build.Dist/build-binary-assets/stage3' to '/src/Build.Dist/build/GenPi64OpenRC/build-binary-assets/stage3/', the build process is now going a lot further along, as I write. I'll still look at Catalyst as well but this is a step.

That did run in a chroot, so that's keen, but:

Code: Select all

Traceback (most recent call last):
  File "/home/mjinks/src/Build.Dist/scripts/chroot.py", line 49, in <module>
    pychroot()
    ~~~~~~~~^^
  File "/home/mjinks/src/Build.Dist/scripts/chroot.py", line 17, in pychroot
    os.execvpe('pychroot',
    ~~~~~~~~~~^^^^^^^^^^^^
               ['pychroot',
               ^^^^^^^^^^^^
    ...<3 lines>...
                os.environ['CHROOT_DIR'],
                ^^^^^^^^^^^^^^^^^^^^^^^^^
                *sys.argv[1:]], os.environ)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen os>", line 621, in execvpe
  File "<frozen os>", line 654, in _execvpe
  File "<frozen os>", line 645, in _execvpe
FileNotFoundError: [Errno 2] No such file or directory: b'/usr/bin/pychroot'
FATAL: JOB etc FAILED with exit code 1
FATAL: JOB gentoo-base FAILED with exit code 1
run complete.

From that I'm assuming that the fresh-built environment was missing some package or other. Rudimentary search ('emerge -puv pychroot') didn't turn up anything, so, elsewhere I go, just in case anybody is curious.

[twork]

Just in case you are unaware Catalyst can also build stage4 too.

I am sad to admit how clueless I was about what Catalyst is. Forever I have wondered, How Do They Do This?

[pingtoo]

Just in case you are unaware Catalyst can also build stage4 too.

I am sad to admit how clueless I was about what Catalyst is. Forever I have wondered, How Do They Do This?

English is not my first language, so I am not sure if there is a question here.

If after reading Catalyst and still uncertain the process please be more specific, I'll be happy to try my best to answer the questions.

[twork]

English is not my first language, so I am not sure if there is a question here.

English is my first language but I write as if it were my third or fourth. Long story; I'm grateful for the patience I encounter here.

Much thanks, I'll adjust this as "solved".