[solved] mirrorselect not outputing servers

[jagdpanther]

I wanted to update the "GENTOO_MIRRORS=" line in my /etc/portage/make.conf because of slow down-load speeds.
Here is today's execution of the same command I ran in June 2023 to get my current GENTOO_MIRRORS servers.

Code: Select all

$ mirrorselect -R "North America" -4 -D -s 5 -b 10 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=North America" hosts. 
* Limiting test to https hosts. 
* Downloading a list of mirrors...
 Got 238 mirrors.
* Downloading 100k files from each mirror... [7 of 7]
No search results found. Check your filter settings and re-run mirrorselect

So I tried some less restrictive options:

Code: Select all

$ mirrorselect -R "North America" -4  -s 5 -b 5 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=North America" hosts. 
* Limiting test to https hosts. 
* Downloading a list of mirrors...
 Got 238 mirrors.
Using netselect to choose the top5 hosts, in blocks of 5. 2 of 2 blocks complete.te.

ERROR: Netselect failed to return any mirrors. Try again using block mode.


$ mirrorselect  -4  -s 5 -b 5 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to https hosts. 
* Downloading a list of mirrors...
 Got 238 mirrors.
Using netselect to choose the top5 hosts, in blocks of 5. 15 of 15 blocks complete.te.

ERROR: Netselect failed to return any mirrors. Try again using block mode.

I was using '-b' which is that block mode right?

Ideas?

[Banana]

There is a stats page about the mirrors: https://mirrorstats.gentoo.org/rsync/
Maybe this can help to narrow it down.

I do not see a -S option here: https://man.linuxreviews.org/man8/mirrorselect.8.html (or maybe I'm blind)

A suggested command from the wiki page would be something like this:

Code: Select all

mirrorselect -s3 -b10 -D -R "North America"

or maybe try -c USA instead of the -R option.

[Genone]

Increase the verbosity level (-d option) to see what is going on, in particular anything netselect might print to stderr.

[freke]

There is a stats page about the mirrors: https://mirrorstats.gentoo.org/rsync/
Maybe this can help to narrow it down.

I do not see a -S option here: https://man.linuxreviews.org/man8/mirrorselect.8.html (or maybe I'm blind)

A suggested command from the wiki page would be something like this:

Code: Select all

mirrorselect -s3 -b10 -D -R "North America"

or maybe try -c USA instead of the -R option.

-S is https-mirrors only

Code: Select all

ns ~ # mirrorselect
Usage: mirrorselect [options]

examples:

         automatic:
                 # mirrorselect -s5
                 # mirrorselect -s3 -b10 -o >> /mnt/gentoo/etc/portage/make.conf
                 # mirrorselect -D -s4

         interactive:
                 # mirrorselect -i -r

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit

  Main modes:
    -a, --all_mirrors   This will present a list of all filtered search
                        results to make it possible to select mirrors you wish
                        to use.  For the -r, --rsync option, it will select
                        the rotation server only. As multiple rsync URL's are
                        not supported.
    -D, --deep          Deep mode. This is used to give a more accurate speed
                        test. It will download a 100k file from each server.
                        Because of this you should only use this option if you
                        have a good connection.
    -i, --interactive   Interactive Mode, this will present a list to make it
                        possible to select mirrors you wish to use.

  Server type selection (choose at most one):
     -c COUNTRY, -country COUNTRY
                        only use mirrors from the specified country NOTE:
                        Names with a space must be quoted eg.:  -c 'South
                        Korea'
    -F, --ftp           ftp only mode. Will not consider hosts of other types.
    -H, --http          http only mode. Will not consider hosts of other types
    -S, --https         https only mode. Will not consider hosts of other
                        types
    -r, --rsync         rsync mode. Allows you to interactively select your
                        rsync mirror. Requires -i or -a to be used.
     -R REGION, -region REGION
                        only use mirrors from the specified region NOTE: Names
                        with a space must be quoted eg.:  -R 'North America'
    -4, --ipv4          only use IPv4
    -6, --ipv6          only use IPv6

  Other options:
     -b BLOCKSIZE, -blocksize BLOCKSIZE
                        This is to be used in automatic mode and will split
                        the hosts into blocks of BLOCKSIZE for use with
                        netselect. This is required for certain routers which
                        block 40+ requests at any given time. Recommended
                        parameters to pass are:  -s 3  -b 10
     -d VERBOSITY, -debug VERBOSITY
                        debug mode, pass in the debug level [1-9]
     -f FILE, -file FILE
                        An alternate file to download for deep testing. Please
                        choose the file carefully as to not abuse the system
                        by selecting an overly large size file.  You must also
                        use the -m, --md5 option.
     -m MD5, --md5=MD5   An alternate file md5sum value used to compare the
                        downloaded file against for deep testing.
    -o, --output        Output Only Mode, this is especially useful when being
                        used during installation, to redirect output to a file
                        other than /etc/portage/make.conf
     -P PROXY, -proxy PROXY
                        Proxy server to use if not the default proxy in the
                        environment
    -q, --quiet         Quiet mode
     -s SERVERS, -servers SERVERS
                        Specify Number of servers for Automatic Mode to
                        select. this is only valid for download mirrors. If
                        this is not specified, a default of 1 is used.
     -t TIMEOUT, -timeout TIMEOUT
                        Timeout for deep mode. Defaults to 10 seconds.
     -e EXCLUDE, -exclude EXCLUDE
                        Exclude host from mirrors list.

For me adding -D seems to result in

Code: Select all

No search results found. Check your filter settings and re-run mirrorselect

Haven't done any real testing nor debugging (-d) to follow up on it though.

Code: Select all

ns ~ # mirrorselect  -R 'Europe' -D -s 5 -b 5 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=Europe" hosts.
* Limiting test to https hosts.
* Downloading a list of mirrors...
 Got 238 mirrors.
* Downloading 100k files from each mirror... [41 of 41]
No search results found. Check your filter settings and re-run mirrorselect
ns ~ # mirrorselect  -R 'Europe' -s 5 -b 5 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=Europe" hosts.
* Limiting test to https hosts.
* Downloading a list of mirrors...
 Got 238 mirrors.
Using netselect to choose the top5 hosts, in blocks of 5. 9 of 9 blocks complete.te.

GENTOO_MIRRORS="https://135.181.111.198/gentoo/gentoo-distfiles/ \
    https://130.236.254.253/gentoo/ \
    https://mirrors.ircam.fr/pub/gentoo-distfiles/ \
    https://129.102.1.37/pub/gentoo-distfiles/ \
    https://130.236.254.251/gentoo/"

Also it ends quite a bit quicker with the -D option, so I wonder if it really works...

Code: Select all

ns ~ # time mirrorselect -D -6 -R 'Europe' -s5 -b5 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=Europe" hosts.
* Limiting test to https hosts.
* Downloading a list of mirrors...
 Got 238 mirrors.
* Downloading 100k files from each mirror... [41 of 41]
No search results found. Check your filter settings and re-run mirrorselect

real    0m19.697s
user    0m1.122s
sys     0m0.175s
ns ~ # time mirrorselect -6 -R 'Europe' -s5 -b5 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=Europe" hosts.
* Limiting test to https hosts.
* Downloading a list of mirrors...
 Got 238 mirrors.
Using netselect to choose the top5 hosts, in blocks of 5. 9 of 9 blocks complete.te.

GENTOO_MIRRORS="https://mirrors.ircam.fr/pub/gentoo-distfiles/ \
    https://mirror.netcologne.de/gentoo/ \
    https://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/ \
    https://[2a0d:f302:136:1d62::1]/gentoo/ \
    https://mirror.kumi.systems/gentoo/"

real    2m40.018s
user    0m2.427s
sys     0m3.377s

Actually adding -d2 shows that the test-file it's supposed to fetch seems to be non-existant on the mirrors?

Code: Select all

ns ~ # time mirrorselect -d2 -D -R 'Europe' -s5 -b5 -S -o
main(); config_path = /etc/portage/make.conf
get_filesystem_mirrors(): config_path = /etc/portage/make.conf
get_filesystem_mirrors(): mirrorlist = ['https://mirrors.ircam.fr/pub/gentoo-distfiles/', '\\', 'https://ftp.fau.de/gentoo', '\\', 'https://mirror.netcologne.de/gentoo/', '\\', 'https://[2a0d:f302:111:83ae::1]/gentoo/', '\\', 'https://[2a0d:f302:136:1d62::1]/gentoo/']
get_filesystem_mirrors(): ignoring non-accessible mirror = \
get_filesystem_mirrors(): ignoring non-accessible mirror = \
get_filesystem_mirrors(): ignoring non-accessible mirror = \
get_filesystem_mirrors(): ignoring non-accessible mirror = \
get_filesystem_mirrors(): fsmirrors = []
using url: https://api.gentoo.org/mirrors/distfiles.xml
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=Europe" hosts.
* Limiting test to https hosts.
getlist(): fetching https://api.gentoo.org/mirrors/distfiles.xml
* Downloading a list of mirrors...
 Got 238 mirrors.
Extractor(): fetched mirrors, 41 hosts after filtering
* Downloading 100k files from each mirror... [1 of 41]
_deeptime(): maxtime is 10
deeptime(): ip's for host gentoo.mirrors.ovh.net: ['213.32.5.7']
deeptime(): testing url: https://213.32.5.7/gentoo-distfiles/distfiles/mirrorselect-test
deeptime(): connection to host gentoo.mirrors.ovh.net
            returned HTTPError: HTTP Error 404: Not Found for ip 213.32.5.7
            Switching back to original url
deeptime(): connection to host gentoo.mirrors.ovh.net
            returned HTTPError: HTTP Error 404: Not Found for ip 213.32.5.7
            Switching back to original url
deeptime(): unable to connect to host gentoo.mirrors.ovh.net
* Downloading 100k files from each mirror... [2 of 41]
_deeptime(): maxtime is 10
deeptime(): ip's for host mirror.leaseweb.com: ['37.58.58.140']
deeptime(): testing url: https://37.58.58.140/gentoo/distfiles/mirrorselect-test
deeptime(): connection to host mirror.leaseweb.com
            returned HTTPError: HTTP Error 404: Not Found for ip 37.58.58.140
            Switching back to original url
deeptime(): connection to host mirror.leaseweb.com
            returned HTTPError: HTTP Error 404: Not Found for ip 37.58.58.140
            Switching back to original url
deeptime(): unable to connect to host mirror.leaseweb.com
* Downloading 100k files from each mirror... [3 of 41]

Which is why it doesn't return any mirrors when using -D I assume.

EDIT:
Found this when I was about to bug.... https://bugs.gentoo.org/921759

[jagdpanther]

Thanks for the replies.

Banana:
I may try to use https://mirrorstats.gentoo.org/rsync/ to pick sites by hand.


Here are my runs from your two suggestions. I added a '-o' because I didn't want to update /etc/portage/make.conf without seeing the update first.

Code: Select all

$ mirrorselect -s3 -b10 -D -R "North America" -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=North America" hosts. 
* Downloading a list of mirrors...
 Got 238 mirrors.
* Downloading 100k files from each mirror... [26 of 26]
No search results found. Check your filter settings and re-run mirrorselect

Code: Select all

$ mirrorselect -s3 -b10 -D -c USA -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "country=USA" hosts. 
* Downloading a list of mirrors...
 Got 238 mirrors.
* Downloading 100k files from each mirror... [17 of 17]
No search results found. Check your filter settings and re-run mirrorselect

Genone:

I tried my original mirrorselect command adding '-d 2'. It appears that something called deeptime() is having issues:

Code: Select all

$ mirrorselect -R "North America" -4 -D -s 5 -b 10 -S -d 2 -o
main(); config_path = /etc/portage/make.conf
get_filesystem_mirrors(): config_path = /etc/portage/make.conf
get_filesystem_mirrors(): mirrorlist = ['https://mirrors.rit.edu/gentoo/', '\\', 'https://mirror.csclub.uwaterloo.ca/gentoo-distfiles/', '\\', 'https://mirror.servaxnet.com/gentoo/', '\\', 'https://mirror.clarkson.edu/gentoo/', '\\', 'https://gentoo.osuosl.org/']
get_filesystem_mirrors(): ignoring non-accessible mirror = \
get_filesystem_mirrors(): ignoring non-accessible mirror = \
get_filesystem_mirrors(): ignoring non-accessible mirror = \
get_filesystem_mirrors(): ignoring non-accessible mirror = \
get_filesystem_mirrors(): fsmirrors = []
using url: https://api.gentoo.org/mirrors/distfiles.xml
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=North America" hosts. 
* Limiting test to https hosts. 
getlist(): fetching https://api.gentoo.org/mirrors/distfiles.xml
* Downloading a list of mirrors...
 Got 238 mirrors.
Extractor(): fetched mirrors, 7 hosts after filtering
* Downloading 100k files from each mirror... [1 of 7]
_deeptime(): maxtime is 10
deeptime(): ip's for host mirror.csclub.uwaterloo.ca: ['129.97.134.71']
deeptime(): testing url: https://129.97.134.71/gentoo-distfiles/distfiles/mirrorselect-test
deeptime(): connection to host mirror.csclub.uwaterloo.ca
            returned HTTPError: HTTP Error 404: Not Found for ip 129.97.134.71
            Switching back to original url
deeptime(): connection to host mirror.csclub.uwaterloo.ca
            returned HTTPError: HTTP Error 404: Not Found for ip 129.97.134.71
            Switching back to original url
deeptime(): unable to connect to host mirror.csclub.uwaterloo.ca
* Downloading 100k files from each mirror... [2 of 7]
_deeptime(): maxtime is 10
deeptime(): ip's for host mirror.reenigne.net: ['172.83.105.10']
deeptime(): testing url: https://172.83.105.10/gentoo/distfiles/mirrorselect-test
deeptime(): connection to host mirror.reenigne.net
            returned HTTPError: HTTP Error 404: Not Found for ip 172.83.105.10
            Switching back to original url
deeptime(): connection to host mirror.reenigne.net
            returned HTTPError: HTTP Error 404: Not Found for ip 172.83.105.10
            Switching back to original url
deeptime(): unable to connect to host mirror.reenigne.net
* Downloading 100k files from each mirror... [3 of 7]
_deeptime(): maxtime is 10
deeptime(): ip's for host gentoo.osuosl.org: ['64.50.233.100', '64.50.236.52', '140.211.166.134']
deeptime(): testing url: https://64.50.233.100/distfiles/mirrorselect-test
deeptime(): connection to host gentoo.osuosl.org
            returned HTTPError: HTTP Error 404: Not Found for ip 64.50.233.100
            Switching back to original url
deeptime(): testing url: https://64.50.236.52/distfiles/mirrorselect-test
deeptime(): connection to host gentoo.osuosl.org
            returned HTTPError: HTTP Error 404: Not Found for ip 64.50.236.52
            Switching back to original url
deeptime(): testing url: https://140.211.166.134/distfiles/mirrorselect-test
deeptime(): connection to host gentoo.osuosl.org
            returned HTTPError: HTTP Error 404: Not Found for ip 140.211.166.134
            Switching back to original url
deeptime(): connection to host gentoo.osuosl.org
            returned HTTPError: HTTP Error 404: Not Found for ip 140.211.166.134
            Switching back to original url
deeptime(): unable to connect to host gentoo.osuosl.org
* Downloading 100k files from each mirror... [4 of 7]
_deeptime(): maxtime is 10
deeptime(): ip's for host mirrors.mit.edu: ['18.7.29.125']
deeptime(): testing url: https://18.7.29.125/gentoo-distfiles/distfiles/mirrorselect-test
deeptime(): connection to host mirrors.mit.edu
            returned HTTPError: HTTP Error 404: Not Found for ip 18.7.29.125
            Switching back to original url
deeptime(): connection to host mirrors.mit.edu
            returned HTTPError: HTTP Error 404: Not Found for ip 18.7.29.125
            Switching back to original url
deeptime(): unable to connect to host mirrors.mit.edu
* Downloading 100k files from each mirror... [5 of 7]
_deeptime(): maxtime is 10
deeptime(): ip's for host mirrors.rit.edu: ['129.21.171.72']
deeptime(): testing url: https://129.21.171.72/gentoo/distfiles/mirrorselect-test
deeptime(): connection to host mirrors.rit.edu
            returned HTTPError: HTTP Error 404: Not Found for ip 129.21.171.72
            Switching back to original url
deeptime(): connection to host mirrors.rit.edu
            returned HTTPError: HTTP Error 404: Not Found for ip 129.21.171.72
            Switching back to original url
deeptime(): unable to connect to host mirrors.rit.edu
* Downloading 100k files from each mirror... [6 of 7]
_deeptime(): maxtime is 10
deeptime(): ip's for host mirror.clarkson.edu: ['128.153.145.19']
deeptime(): testing url: https://128.153.145.19/gentoo/distfiles/mirrorselect-test
deeptime(): connection to host mirror.clarkson.edu
            returned HTTPError: HTTP Error 404: Not Found for ip 128.153.145.19
            Switching back to original url
deeptime(): connection to host mirror.clarkson.edu
            returned HTTPError: HTTP Error 404: Not Found for ip 128.153.145.19
            Switching back to original url
deeptime(): unable to connect to host mirror.clarkson.edu
* Downloading 100k files from each mirror... [7 of 7]
_deeptime(): maxtime is 10
deeptime(): ip's for host mirror.servaxnet.com: ['162.255.226.75']
deeptime(): testing url: https://162.255.226.75/gentoo/distfiles/mirrorselect-test
deeptime(): connection to host mirror.servaxnet.com
            returned HTTPError: HTTP Error 404: Not Found for ip 162.255.226.75
            Switching back to original url
deeptime(): connection to host mirror.servaxnet.com
            returned HTTPError: HTTP Error 404: Not Found for ip 162.255.226.75
            Switching back to original url
deeptime(): unable to connect to host mirror.servaxnet.com
deeptest(): got 7 hosts, and returned []

deeptest(): final rethost []
deeptest(): final md5 failures 0 of 7
No search results found. Check your filter settings and re-run mirrorselect

Using the same -R 'Europe' as you I still get no results after a min. or two.

Code: Select all

$ mirrorselect  -R 'Europe' -D -s 5 -b 5 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=Europe" hosts. 
* Limiting test to https hosts. 
* Downloading a list of mirrors...
 Got 238 mirrors.
* Downloading 100k files from each mirror... [41 of 41]
No search results found. Check your filter settings and re-run mirrorselect

Without the '-D' I receive an instant error

Code: Select all

$ mirrorselect  -R 'Europe'  -s 5 -b 5 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=Europe" hosts. 
* Limiting test to https hosts. 
* Downloading a list of mirrors...
 Got 238 mirrors.
Using netselect to choose the top5 hosts, in blocks of 5. 9 of 9 blocks complete.te.

ERROR: Netselect failed to return any mirrors. Try again using block mode.

[Genone]

deeptime() is just the name of the function within mirrorselect that does most of the testing.

As you've already figured out, the core issue is apparently that mirrorselect hasn't been updated to match the current mirror directory structure. Might be worth to ask on the bug report if there are any status updates.

[user]

Also with --deep option a IP based URL is testing HTTPS connection which often failed because connection and certificate not matched by TLS/SNI method.

Hence, consider mirrorselect option --https with --deep as broken.

Possible workaround command:
- HTTPS only
- TLS v1.3 only
- parallel hosts check
- sorted by speed

Code: Select all

# curl --compressed -qfso - -- https://api.gentoo.org/mirrors/distfiles.xml |\
    grep -oP '\K(https://[^<]*)' |\
    sort -u |\
    xargs -P4 -I{} -- curl -qfso /dev/null --tlsv1.3 --connect-timeout 2 --max-time 30 --write-out '%{speed_download} B/s {}\n' -- {}/snapshots/portage-latest.tar.xz |\
    sort -rn

if needed, reduced by fully validated DNSSEC answers only:

Code: Select all

# curl --compressed -qfso - -- https://api.gentoo.org/mirrors/distfiles.xml |\
    grep -oP '\K(https://[^<]*)' |\
    sort -u |\
    xargs -P4 -I{} -- sh -c "
        delv \$(echo {} | cut -d/ -f3) |&
        grep -q '^; fully validated' &&
        curl -qfso /dev/null --tlsv1.3 --connect-timeout 2 --max-time 30 --write-out '%{speed_download} B/s {}\n' -- {}/snapshots/portage-latest.tar.xz" |\
    sort -rn

[jagdpanther]

Yesterday, mirrorselect was updated so now running =app-portage/mirrorselect-2.5.1, mirrorselect works again. A big Thank You to the coder.

Code: Select all

$ mirrorselect -R "North America" -4 -D -s 5 -b 10 -S -o
* Using url: https://api.gentoo.org/mirrors/distfiles.xml
* Limiting test to "region=North America" hosts. 
* Limiting test to https hosts. 
* Downloading a list of mirrors...
 Got 405 mirrors.
* Downloading f2/mirrorselect-test files from each mirror... [16 of 16]

GENTOO_MIRRORS="https://us.mirrors.cicku.me/gentoo/ \
    https://mirrors.kernel.org/gentoo/ \
    https://ca.mirrors.cicku.me/gentoo/ \
    https://mirrors.mit.edu/gentoo-distfiles/ \
    https://mirrors.rit.edu/gentoo/"