Computing portage distfiles directory name hash

[eccerr0r]

Yes this has been done this way for a while but I had a script that was relying on the flat directories to reduce my bandwidth dependence on downloading from distfiles.gentoo.org. However it has stopped working because of the directory split to 256 directories.

While I knew this would be a problem at some point, now I have to figure out how to replicate this on my personal portage distfiles mirror as I download them (again, since I have more than one box that needs updates, I am hoping I download any particular distfile no more than once.)

So in https://archives.gentoo.org/gentoo-dev/ ... ecae2463ba it provides some very vague declaration of how to compute the directory hash...is there a better plaintext way of computing the subdirectory? As far as I can tell, the distfile name is hashed (what hash was it decided on?) and then some esoteric function added. I was kind of surprised, would have thought if the hash was relatively cryptographically secure (which it doesn't need to be since the intent was solely distribution and not security) one would just need to cut off the first byte of the hash and use that as the directory when converted to hex.

So is there a bash script to generate this 2-hexadecimal digit hash?

tldr: http://distfiles.gentoo.org/distfiles/[b]5a[/b]/qtwebengine-5.15.10_p20230815.tar.xz : what's the algorithm to compute the 5a ?

[Genone]

The description in the GLEP seems pretty clear:
- Fetch layout.conf
- Process [structure] section of layout.conf to determine available directory layouts
- If filename-hash is specified, generate hash of filename using algorithm defined in layout.conf
- Extract initial bits of the hash as specified in layout.conf

distfiles.gentoo.org has a layout.conf that specifies:

Code: Select all

[structure]
0=filename-hash BLAKE2B 8

Blake2b hash can be calculated e.g. using python hashlib:

Code: Select all

import hashlib
h = hashlib.blake2b()
h.update(b'qtwebengine-5.15.10_p20230815.tar.xz')
print(h.hexdigest())

which gets us

Code: Select all

5a248ad23276e17f10b44788bddaaf8bdd9cc1982c91163752ea7e129c796eb778b3549863a79c8e42363189ac939597d8f83df8f5b063184d14c8eaebe627f2

Then we take the first 8 bit of that hash, which means the first two bytes of the hex digest (one hex digit = 4 bit), which results in the directory name '5a'

You can also calculate the hash using openssl blake2b512, but you need to pay close attention to whitespace when passing content in bash. E.g.

Code: Select all

echo qtwebengine-5.15.10_p20230815.tar.xz | openssl blake2b512

will generate an incorrect result, you have to use `echo -n` to omit the trailing newline.

But mind that other mirrors may use a different structure, so you really should check layout.conf for what is supported.

[Ionen]

You can also use b2sum from coreutils

Code: Select all

var=$(printf %s qtwebengine-5.15.10_p20230815.tar.xz | b2sum)
echo ${var::2}
5a

Ideally do need to check layout.conf, but odds that this will change often are very low.

[Genone]

Ideally do need to check layout.conf, but odds that this will change often are very low.

True, but I wouldn't be surprised if some mirror operators would opt for a different hash function and/or prefix length. After all, someone must have argued for this functionality in the GLEP.

[sam_]

[..]
So in https://archives.gentoo.org/gentoo-dev/ ... ecae2463ba it provides some very vague declaration of how to compute the directory hash...is there a better plaintext way of computing the subdirectory? As far as I can tell, the distfile name is hashed (what hash was it decided on?) and then some esoteric function added. I was kind of surprised, would have thought if the hash was relatively cryptographically secure (which it doesn't need to be since the intent was solely distribution and not security) one would just need to cut off the first byte of the hash and use that as the directory when converted to hex.

You can read GLEP 75 in full at https://www.gentoo.org/glep/glep-0075.html

Hashes have uses beyond cryptography and there's no need for cryptographic security in a case like this; the hash is used for bucketing. The choice of BLAKE2 is therefore kind of interesting but it's justified within the GLEP.

[eccerr0r]

Thanks, this makes more sense now. Seems there was a lot of discussion on the pages of what the options were and why they were or were not used and thus had a lot of content that didn't make it to the final decision. I could have completely glossed over the discussion about cryptographically secure hashes and got the same result, however a "secure" hash would more likely generate a more random distribution of files - which improves the chances the buckets will end up with similar number of files.

I suppose technically my personal distfiles server I could have my own layout.conf, dump all distfiles into the same directory, and do away with all of this... (a lot of my machines are fairly similar in installed content)? Then again I'll have to make sure layout.conf doesn't get rsynced into my repository and get overwritten...

Then the next question is whether eclean will deal with deleting excess files if I break the directories apart... ahh... more to ponder.