Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Tip: speeding up modprobe bash-completion approximately 500x
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5290

PostPosted: Fri Oct 19, 2018 3:20 am    Post subject: Tip: speeding up modprobe bash-completion approximately 500x Reply with quote

Symptom:
Code:
 ~ $ time _modules

real    1m34.954s
user    0m6.820s
sys     0m3.727s

The _modules command is the bashcomp helper function that finds kernel modules. I was getting a bit annoyed at how slow it was and decided to investigate.

In a discovery that'll probably give SteveL an aneurysm, I found it attempting to parse the output of ls -RL. It's scanning the entire kernel source tree in the /lib/modules/$(uname -r)/source symlink just to find module files under /lib/modules. My kernel source happens to be on an NFS mount, so it hits even harder. Not only that, the results end up being completely useless - it lists every single intermediate object file in the kernel tree as a loadable module.

This is a bit of a rubbish situation imo, so I wrote a sane implementation. Drop this in bashrc.d to fix it:
/etc/bash/bashrc.d/fix_modules_bashcomp.sh:
_modules()
{
    local modpath
    modpath=/lib/modules/$1
    COMPREPLY=( $( compgen -W "$( find -L "$modpath" -type f -xdev -regex '.*\.k?o\(\.[gx]z\)?' 2>/dev/null | \
        command sed -e 's@^.*/\(.*\)\.k\?o\(\.[gx]z\)\?@\1@' )" -- "$cur" ) )
}

Result:
Code:
 ~ $ time _modules

real    0m0.026s
user    0m0.031s
sys     0m0.012s


I'm not the only Gentoo user to have noticed this, but upstream doesn't seem to care: https://github.com/scop/bash-completion/issues/136
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6229
Location: Room 101

PostPosted: Fri Oct 19, 2018 9:29 am    Post subject: Re: Tip: speeding up modprobe bash-completion approximately Reply with quote

Ant P. wrote:
/etc/bash/bashrc.d/fix_modules_bashcomp.sh:
_modules()
{
    local modpath
    modpath=/lib/modules/$1
    COMPREPLY=( $( compgen -W "$( find -L "$modpath" -type f -xdev -regex '.*\.k?o\(\.[gx]z\)?' 2>/dev/null | \
        command sed -e 's@^.*/\(.*\)\.k\?o\(\.[gx]z\)\?@\1@' )" -- "$cur" ) )
}

Ant P. ... hehe, too much to expect a shell to glob for 'files' only, and return only the required part of the name?

Code:
% print -rl /lib/modules/$(uname -r)/(*~(source|build))/**/*.ko*(.:t:r) | head -n 2
acpi-cpufreq
applesmc

The '_modutils' zsh completion has the following:

_modutils:
modules=( $modules_dir/$kver/(*~(source|build))/**/*.(o|ko|ko.gz|ko.xz)(.:t:r:r) )

Anyhow, what I wanted to say was, you can probably improve the speed by excluding 'source' and 'build' from the find.

best ... khay
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5290

PostPosted: Sat Oct 20, 2018 8:31 pm    Post subject: Reply with quote

Oh, good catch, I was using -xdev to exclude /source but that'd only work on my machine, and I completely missed the /build dir.

Here's a revised version, still not as pretty as I'd like—bash doesn't have an inline equivalent to (.:t:r)—but it's correct, and more importantly, faster :)
/etc/bash/bashrc.d/fix_modules_bashcomp.sh:

_modules()
{
    COMPREPLY=( $( compgen -W "$(shopt -s globstar failglob
        printf '%s\n' /lib/modules/"${1:?}"/!(source|build)/**/*.@(o|ko|ko.gz|ko.xz) | \
        sed -e 's@^.*/\(.*\)\.k\?o\(\.[gx]z\)\?@\1@' )" -- "$cur" ) )
}

Code:
 ~ $ time _modules $(uname -r)

real    0m0.008s
user    0m0.006s
sys     0m0.005s

Not a bad speedup.

My numbers were a bit off too. The first one is a 4000× improvement, this is ~12000×.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum