| View previous topic :: View next topic |
| Author |
Message |
haarp
Guru
Joined: 31 Oct 2007
Posts: 535
|
 Posted: Sun Apr 07, 2013 7:45 am Post subject: [solved] Customizing the merge phase |
 |
|
Hey,
I'm trying to customize the merge phase in emerge. For every build, before it is merged into the system, I would like to call jpegoptim and optipng to optimize all images, or delete manpages for languages I don't use but which nevertheless insist on being installed. I can probably come up with more more ideas aswell.
Note that I'm not asking for how to do this in an ebuild, but globally for every merge. It would be trivial for me to write a script that managed these calls. But where and how do I hook it into the emerge session? Is that even possible?
There are obvious uses for this, such as improved load times and reduced disk usage. Yeah, someone inevitably will reply and recommend SSDs or whatever, but that's not the point. There is room for improvement, which I would like to use.
Thanks
Last edited by haarp on Sun Apr 07, 2013 10:13 am; edited 1 time in total |
|
| Back to top |
|
 |
megabaks
Apprentice
Joined: 22 Jan 2012
Posts: 253
Location: Russia && Saint-Petersburg
|
| Posted: Sun Apr 07, 2013 9:55 am Post subject: |
|
|
| Quote: | | I would like to call jpegoptim and optipng to optimize all images |
/etc/portage/bashrc and i.e. | Code: | pre_src_configure ()
{
do something
}
|
you can use prefixes post_/pre_ for any phase. remember - current path for any phase is path/to/sources
| Quote: | | delete manpages for languages I don't use |
emerge localepurge
_________________
sorry my bad english |
|
| Back to top |
|
|
haarp
Guru
Joined: 31 Oct 2007
Posts: 535
|
| Posted: Sun Apr 07, 2013 10:13 am Post subject: |
|
|
| That looks like what I need. Thank you! |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21595
|
| Posted: Sun Apr 07, 2013 3:28 pm Post subject: |
|
|
| For some use cases, INSTALL_MASK and/or PKG_INSTALL_MASK may allow you to eliminate the unwanted man pages. However, it can only delete unwanted files, so the solution from megabaks is required for your more complex goals. You probably want to use post_src_install to modify files under $D. |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Sun Apr 07, 2013 10:25 pm Post subject: |
|
|
haarp ...
You might want to look at mv's app-portage/portage-bashrc-mv which provides some ready-made scripts for removing locales (using localepurge), removing *.la files, and other features. Its available via the mv overlay.
best ... khay |
|
| Back to top |
|
|
haarp
Guru
Joined: 31 Oct 2007
Posts: 535
|
| Posted: Sat Apr 13, 2013 10:14 am Post subject: |
|
|
Thanks everyone, I finished and tested my scripts and they are working pretty well. No known issues with them.
I took some hints from the localepurge package, but rewrote it to better suit my needs. That includes using the LINGUAS variable in make.conf instead of adding annoying extra files in /etc.
The image optimization should reduce space requirements a tiny bit and improve load times.
For anyone interested, here's the portage bashrc:
| Code: | post_src_unpack() {
##
## Optimize images
## needs media-gfx/jpegoptim and media-gfx/optipng
##
[[ -d "${S}" ]] || return #abort if no files (virtuals)
einfo "[H] Optimizing images..."
cpucount=$[ $(awk '/^processor/{print $3}' /proc/cpuinfo | tail -n1) + 1 ]
#we mustn't run on symlinks, therefore -type f
find "${S}" -iname "*.jpg" -type f -print0 | xargs -P$cpucount -n10 -0 jpegoptim -p -q &>/dev/null
find "${S}" -iname "*.jpeg" -type f -print0 | xargs -P$cpucount -n10 -0 jpegoptim -p -q &>/dev/null
find "${S}" -iname "*.png" -type f -print0 | xargs -P$cpucount -n10 -0 optipng -preserve -quiet &>/dev/null
return 0
}
post_pkg_preinst() {
##
## Remove unwanted locales and manpages
## needs valid LINGUAS entry in make.conf
##
[[ -z ${LINGUAS} ]] && return #abort if LINGUAS empty
einfo "[H] Removing unwanted locales and manpages..."
LOCALEDIRS="/opt/sun-jdk-*/jre/lib/locale /opt/sun-jre-bin-*/lib/locale /usr/kde/?.?/share/locale /usr/lib/locale /usr/share/binutils-data/*/*/locale /usr/share/gcc-data/*/*/locale /usr/share/locale"
MANPAGEDIRS="/opt/icedtea*/man /opt/sun-jdk-*/man /opt/sun-jre-bin-*/man /usr/kde/?.?/man /usr/kde/?.?/share/man /usr/local/share/man /usr/man /usr/share/man"
MISCDIRS="/opt/intel/composer*/compiler/lib/*/locale /usr/share/doc/icc* /usr/share/doc/intel-common* /usr/share/doc/xfce4-*/html"
exterminate() {
local whitelist=0
#check if whitelisted by LINGUAS
for lingua in ${LINGUAS}; do
basename "$1" | grep -q ^$lingua && whitelist=1
done
#act if not whitelisted
if [[ $whitelist == 0 ]]; then
echo "rm -r $1" | sed -e "s:/var/tmp/portage/${CATEGORY}/${P}/::" -e "s:///:/:"
rm -r "$1"
fi
}
purgelocale() {
for dir in "$1"/*; do
#check if valid locale dir
if [[ -n $(find "$dir" -name "LC_MESSAGES" 2>/dev/null) ]]; then
exterminate "$dir"
fi
done
}
purgeman() {
local LINGUAS="${LINGUAS} man" #add "man" pseudo lingua
for dir in "$1"/*; do
#check if valid manpage dir
if [[ -n $(find "$dir" -name "man*" 2>/dev/null) ]]; then
exterminate "$dir"
fi
done
}
purgemisc() {
local LINGUAS="${LINGUAS} C" #add "C" lingua
for dir in "$1"/*; do
#check if dir
if [[ -d "$dir" ]]; then
exterminate "$dir"
fi
done
}
for dir in ${LOCALEDIRS}; do
purgelocale "${D}/$dir"
done
for dir in ${MANPAGEDIRS}; do
purgeman "${D}/$dir"
done
for dir in ${MISCDIRS}; do
purgemisc "${D}/$dir"
done
return 0
} |
|
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Sat Apr 13, 2013 11:15 am Post subject: |
|
|
haarp ...
couple of suggestions:
| haarp wrote: | | Code: | | cpucount=$[ $(awk '/^processor/{print $3}' /proc/cpuinfo | tail -n1) + 1 ] |
|
You could replace this with the perhaps more succinct:
| Code: | | cpucount=$(awk '/^processor/{a=$NF};END{print (a + 1)}' /proc/cpuinfo) |
| haarp wrote: | | Code: | find "${S}" -iname "*.jpg" -type f -print0 | xargs -P$cpucount -n10 -0 jpegoptim -p -q &>/dev/null
find "${S}" -iname "*.jpeg" -type f -print0 | xargs -P$cpucount -n10 -0 jpegoptim -p -q &>/dev/null |
|
These can be combined with a regex ...
| Code: | | find "${S}" -regextype posix-egrep -iregex ".*\.(jpg|jpeg)$" -type f -print0 | xargs -P$cpucount -n10 -0 jpegoptim -p -q &>/dev/null |
Also, the (~arch) media-gfx/jpegoptim package can be bumped to 1.3.0 without issues.
best ... khay |
|
| Back to top |
|
|
haarp
Guru
Joined: 31 Oct 2007
Posts: 535
|
| Posted: Sat Apr 13, 2013 2:29 pm Post subject: |
|
|
You're right, thanks. But I can do one better! 
| Code: | for opt in ${MAKEOPTS}; do #get make jobs from make.conf
[[ "$opt" =~ -j ]] && numcpu=${opt#-j}
done
[[ -z $numcpu ]] && numcpu=1 |
|
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Sat Apr 13, 2013 7:23 pm Post subject: |
|
|
| haarp wrote: | | You're right, thanks. But I can do one better! |
haarp ... changing idiom that way would be cheating :) ... but ok:
| Code: | | numcpu=${${MAKEOPTS#-j}:-1} |
sorry ... only works in zsh (one more reason portage should have adopted zsh rather than bash). However, this is bash so:
| Code: | | numcpu=${MAKEOPTS/-j/-P} |
... and omit the '-P' to xargs. If $MAKEOPTS has a value the -j{N} becomes -P{N}, if empty its null, and so xargs defaults to 1 in the absence of -P{N}.
best ... khay |
|
| Back to top |
|
|
haarp
Guru
Joined: 31 Oct 2007
Posts: 535
|
| Posted: Sat Apr 13, 2013 7:34 pm Post subject: |
|
|
Ah. Ok, you win.  Just gotta hope MAKEOPTS does not contain anything else xargs might choke on.
I like bash syntax more in this case, tbh. It resembles sed's.
edit: Wait. Might aswell do this, then 
| Code: | | xargs ${MAKEOPTS/-j/-P} -n10 -0 ... |
|
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21595
|
| Posted: Sat Apr 13, 2013 7:44 pm Post subject: |
|
|
| haarp wrote: |
| Code: | post_src_unpack() {
|
|
What about packages that generate images during build? You might be better off running this as a post_src_install phase to clean $D.
| khayyam wrote: | You could replace this with the perhaps more succinct:
| Code: | | cpucount=$(awk '/^processor/{a=$NF};END{print (a + 1)}' /proc/cpuinfo) |
|
He could also use the even more succinct: | Code: | | grep -c '^processor' /proc/cpuinfo |
| khayyam wrote: | These can be combined with a regex ...
| Code: | | find "${S}" -regextype posix-egrep -iregex ".*\.(jpg|jpeg)$" -type f -print0 | xargs -P$cpucount -n10 -0 jpegoptim -p -q &>/dev/null |
|
True, but he could also use the -o option to find to chain together multiple simple -iname lines.
| haarp wrote: | | Code: |
exterminate() {
local whitelist=0
#check if whitelisted by LINGUAS
for lingua in ${LINGUAS}; do
basename "$1" | grep -q ^$lingua && whitelist=1
done
|
|
You can remove the local variable whitelist and instead return out of the middle of the loop on a match.
| haarp wrote: | | Code: |
echo "rm -r $1" | sed -e "s:/var/tmp/portage/${CATEGORY}/${P}/::" -e "s:///:/:"
|
|
You should use $PORTAGE_TMPDIR instead of hardcoding /var/tmp/portage. You could also use $D and remove the need to use $CATEGORY/$P.
| haarp wrote: | | Code: |
purgelocale() {
for dir in "$1"/*; do
#check if valid locale dir
if [[ -n $(find "$dir" -name "LC_MESSAGES" 2>/dev/null) ]]; then
exterminate "$dir"
fi
done
|
|
You can pass multiple directories to find at once. Combining that with -printf can simplify this construct to: | Code: |
purgelocale() {
find "$1"/* -name LC_MESSAGES -printf '%H\0' 2>/dev/null | while read -d '' dir; do exterminate "$dir"; done
}
|
If multiple LC_MESSAGES can be found beneath a given argument, you will need to add a uniq -z to the pipeline.
| haarp wrote: | | Code: | | xargs ${MAKEOPTS/-j/-P} -n10 -0 ... |
|
Some people put other options in MAKEOPTS than just -j. Limiting load average is relatively popular. |
|
| Back to top |
|
|
haarp
Guru
Joined: 31 Oct 2007
Posts: 535
|
| Posted: Sat Apr 13, 2013 8:22 pm Post subject: |
|
|
Thanks Hu, these are some good improvements!
| Hu wrote: | | What about packages that generate images during build? You might be better off running this as a post_src_install phase to clean $D. |
I debated whether to change this myself. I don't know any packages that generate images, but I do know some which pack images into an archive during their build. gzdoom does it, and I think firefox/thunderbird too. Short of running both before and after a build, there isn't a way of catching both possibilities. I think the current phase is preferable.
| Quote: | | Some people put other options in MAKEOPTS than just -j. Limiting load average is relatively popular. |
Yep, I realized that myself. I went back to my for-loop (did I mention that I love sh-style for-loops?). But it's not perfect either (won't work when there's a whitespace between -j and the number) |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Sat Apr 13, 2013 9:26 pm Post subject: |
|
|
| Hu wrote: | | khayyam wrote: | You could replace this with the perhaps more succinct:
| Code: | | cpucount=$(awk '/^processor/{a=$NF};END{print (a + 1)}' /proc/cpuinfo) |
|
He could also use the even more succinct:
| Code: | | grep -c '^processor' /proc/cpuinfo |
|
Hu ... ah, yes, but I was also incrementing.
| Hu wrote: | | khayyam wrote: | These can be combined with a regex ...
| Code: | | find "${S}" -regextype posix-egrep -iregex ".*\.(jpg|jpeg)$" -type f -print0 | xargs -P$cpucount -n10 -0 jpegoptim -p -q &>/dev/null |
|
True, but he could also use the -o option to find to chain together multiple simple -iname lines. |
Yes, -iname "*.jpeg" -o -iname "*.jpg" would have sufficed, but I was under the impression that a regex was faster becuase the matching less greedy.
| Hu wrote: | | Quote: | | Code: | | MAKEOPTS/-j/-P} -n10 -0 ... |
|
Some people put other options in MAKEOPTS than just -j. Limiting load average is relatively popular. |
Its even popular here, so you'd think I might have paid closer attention :) Anyhow, if I was anything like half awake, and had not tried to follow bash, I would probably have done the following:
| Code: | % MAKEOPTS="-jNN -l xxxx.00" ; print ${${MAKEOPTS}[3,4]}
NN |
... though its probably more reliable to get the value from /proc/cpuinfo
best ... khay |
|
| Back to top |
|
|
|
Portage & Programming
