Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[solved] Customizing the merge phase
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
haarp
Guru
Guru


Joined: 31 Oct 2007
Posts: 354

PostPosted: Sun Apr 07, 2013 7:45 am    Post subject: [solved] Customizing the merge phase Reply with quote

Hey,

I'm trying to customize the merge phase in emerge. For every build, before it is merged into the system, I would like to call jpegoptim and optipng to optimize all images, or delete manpages for languages I don't use but which nevertheless insist on being installed. I can probably come up with more more ideas aswell.

Note that I'm not asking for how to do this in an ebuild, but globally for every merge. It would be trivial for me to write a script that managed these calls. But where and how do I hook it into the emerge session? Is that even possible?

There are obvious uses for this, such as improved load times and reduced disk usage. Yeah, someone inevitably will reply and recommend SSDs or whatever, but that's not the point. There is room for improvement, which I would like to use.

Thanks


Last edited by haarp on Sun Apr 07, 2013 10:13 am; edited 1 time in total
Back to top
View user's profile Send private message
megabaks
Apprentice
Apprentice


Joined: 22 Jan 2012
Posts: 237
Location: Russia && Saint-Petersburg

PostPosted: Sun Apr 07, 2013 9:55 am    Post subject: Reply with quote

Quote:
I would like to call jpegoptim and optipng to optimize all images
/etc/portage/bashrc and i.e.
Code:
pre_src_configure ()
{
    do something
}
you can use prefixes post_/pre_ for any phase. remember - current path for any phase is path/to/sources
Quote:
delete manpages for languages I don't use
emerge localepurge
_________________
sorry my bad english
Back to top
View user's profile Send private message
haarp
Guru
Guru


Joined: 31 Oct 2007
Posts: 354

PostPosted: Sun Apr 07, 2013 10:13 am    Post subject: Reply with quote

That looks like what I need. Thank you!
Back to top
View user's profile Send private message
Hu
Watchman
Watchman


Joined: 06 Mar 2007
Posts: 8592

PostPosted: Sun Apr 07, 2013 3:28 pm    Post subject: Reply with quote

For some use cases, INSTALL_MASK and/or PKG_INSTALL_MASK may allow you to eliminate the unwanted man pages. However, it can only delete unwanted files, so the solution from megabaks is required for your more complex goals. You probably want to use post_src_install to modify files under $D.
Back to top
View user's profile Send private message
khayyam
Veteran
Veteran


Joined: 07 Jun 2012
Posts: 1539

PostPosted: Sun Apr 07, 2013 10:25 pm    Post subject: Reply with quote

haarp ...

You might want to look at mv's app-portage/portage-bashrc-mv which provides some ready-made scripts for removing locales (using localepurge), removing *.la files, and other features. Its available via the mv overlay.

best ... khay
Back to top
View user's profile Send private message
haarp
Guru
Guru


Joined: 31 Oct 2007
Posts: 354

PostPosted: Sat Apr 13, 2013 10:14 am    Post subject: Reply with quote

Thanks everyone, I finished and tested my scripts and they are working pretty well. No known issues with them.

I took some hints from the localepurge package, but rewrote it to better suit my needs. That includes using the LINGUAS variable in make.conf instead of adding annoying extra files in /etc.
The image optimization should reduce space requirements a tiny bit and improve load times.

For anyone interested, here's the portage bashrc:

Code:
post_src_unpack() {
   ##
   ## Optimize images
   ## needs media-gfx/jpegoptim and media-gfx/optipng
   ##
   [[ -d "${S}" ]] || return   #abort if no files (virtuals)

   einfo "[H] Optimizing images..."
   cpucount=$[ $(awk '/^processor/{print $3}' /proc/cpuinfo | tail -n1) + 1 ]
   #we mustn't run on symlinks, therefore -type f
   find "${S}" -iname "*.jpg" -type f -print0 | xargs -P$cpucount -n10 -0 jpegoptim -p -q &>/dev/null
   find "${S}" -iname "*.jpeg" -type f -print0 | xargs -P$cpucount -n10 -0 jpegoptim -p -q &>/dev/null
   find "${S}" -iname "*.png" -type f -print0 | xargs -P$cpucount -n10 -0 optipng -preserve -quiet &>/dev/null

   return 0
}

post_pkg_preinst() {
   ##
   ## Remove unwanted locales and manpages
   ## needs valid LINGUAS entry in make.conf
   ##
   [[ -z ${LINGUAS} ]] && return   #abort if LINGUAS empty

   einfo "[H] Removing unwanted locales and manpages..."

   LOCALEDIRS="/opt/sun-jdk-*/jre/lib/locale /opt/sun-jre-bin-*/lib/locale /usr/kde/?.?/share/locale /usr/lib/locale /usr/share/binutils-data/*/*/locale /usr/share/gcc-data/*/*/locale /usr/share/locale"
   MANPAGEDIRS="/opt/icedtea*/man /opt/sun-jdk-*/man /opt/sun-jre-bin-*/man /usr/kde/?.?/man /usr/kde/?.?/share/man /usr/local/share/man /usr/man /usr/share/man"
   MISCDIRS="/opt/intel/composer*/compiler/lib/*/locale /usr/share/doc/icc* /usr/share/doc/intel-common* /usr/share/doc/xfce4-*/html"

   exterminate() {
      local whitelist=0

      #check if whitelisted by LINGUAS
      for lingua in ${LINGUAS}; do
         basename "$1" | grep -q ^$lingua && whitelist=1
      done

      #act if not whitelisted
      if [[ $whitelist == 0 ]]; then
         echo "rm -r $1" | sed -e "s:/var/tmp/portage/${CATEGORY}/${P}/::" -e "s:///:/:"
         rm -r "$1"
      fi
   }

   purgelocale() {
      for dir in "$1"/*; do
         #check if valid locale dir
         if [[ -n $(find "$dir" -name "LC_MESSAGES" 2>/dev/null) ]]; then
            exterminate "$dir"
         fi
      done
   }

   purgeman() {
      local LINGUAS="${LINGUAS} man"   #add "man" pseudo lingua
      for dir in "$1"/*; do
         #check if valid manpage dir
         if [[ -n $(find "$dir" -name "man*" 2>/dev/null) ]]; then
            exterminate "$dir"
         fi
      done
   }

   purgemisc() {
      local LINGUAS="${LINGUAS} C"  #add "C" lingua
      for dir in "$1"/*; do
         #check if dir
         if [[ -d "$dir" ]]; then
            exterminate "$dir"
         fi
      done
   }

   for dir in ${LOCALEDIRS}; do
      purgelocale "${D}/$dir"
   done
   for dir in ${MANPAGEDIRS}; do
      purgeman "${D}/$dir"
   done
   for dir in ${MISCDIRS}; do
      purgemisc "${D}/$dir"
   done

   return 0
}
Back to top
View user's profile Send private message
khayyam
Veteran
Veteran


Joined: 07 Jun 2012
Posts: 1539

PostPosted: Sat Apr 13, 2013 11:15 am    Post subject: Reply with quote

haarp ...

couple of suggestions:

haarp wrote:
Code:
cpucount=$[ $(awk '/^processor/{print $3}' /proc/cpuinfo | tail -n1) + 1 ]

You could replace this with the perhaps more succinct:

Code:
cpucount=$(awk '/^processor/{a=$NF};END{print (a + 1)}' /proc/cpuinfo)


haarp wrote:
Code:
find "${S}" -iname "*.jpg" -type f -print0 | xargs -P$cpucount -n10 -0 jpegoptim -p -q &>/dev/null
find "${S}" -iname "*.jpeg" -type f -print0 | xargs -P$cpucount -n10 -0 jpegoptim -p -q &>/dev/null

These can be combined with a regex ...

Code:
find "${S}" -regextype posix-egrep -iregex ".*\.(jpg|jpeg)$" -type f -print0 | xargs -P$cpucount -n10 -0 jpegoptim -p -q &>/dev/null


Also, the (~arch) media-gfx/jpegoptim package can be bumped to 1.3.0 without issues.

best ... khay
Back to top
View user's profile Send private message
haarp
Guru
Guru


Joined: 31 Oct 2007
Posts: 354

PostPosted: Sat Apr 13, 2013 2:29 pm    Post subject: Reply with quote

You're right, thanks. But I can do one better! :D

Code:
for opt in ${MAKEOPTS}; do   #get make jobs from make.conf
      [[ "$opt" =~ -j ]] && numcpu=${opt#-j}
done
[[ -z $numcpu ]] && numcpu=1
Back to top
View user's profile Send private message
khayyam
Veteran
Veteran


Joined: 07 Jun 2012
Posts: 1539

PostPosted: Sat Apr 13, 2013 7:23 pm    Post subject: Reply with quote

haarp wrote:
You're right, thanks. But I can do one better!

haarp ... changing idiom that way would be cheating :) ... but ok:

Code:
numcpu=${${MAKEOPTS#-j}:-1}

sorry ... only works in zsh (one more reason portage should have adopted zsh rather than bash). However, this is bash so:

Code:
numcpu=${MAKEOPTS/-j/-P}

... and omit the '-P' to xargs. If $MAKEOPTS has a value the -j{N} becomes -P{N}, if empty its null, and so xargs defaults to 1 in the absence of -P{N}.

best ... khay
Back to top
View user's profile Send private message
haarp
Guru
Guru


Joined: 31 Oct 2007
Posts: 354

PostPosted: Sat Apr 13, 2013 7:34 pm    Post subject: Reply with quote

Ah. Ok, you win. :wink: Just gotta hope MAKEOPTS does not contain anything else xargs might choke on.
I like bash syntax more in this case, tbh. It resembles sed's.


edit: Wait. Might aswell do this, then :P
Code:
xargs ${MAKEOPTS/-j/-P} -n10 -0 ...
Back to top
View user's profile Send private message
Hu
Watchman
Watchman


Joined: 06 Mar 2007
Posts: 8592

PostPosted: Sat Apr 13, 2013 7:44 pm    Post subject: Reply with quote

haarp wrote:

Code:
post_src_unpack() {
What about packages that generate images during build? You might be better off running this as a post_src_install phase to clean $D.
khayyam wrote:
You could replace this with the perhaps more succinct:
Code:
cpucount=$(awk '/^processor/{a=$NF};END{print (a + 1)}' /proc/cpuinfo)
He could also use the even more succinct:
Code:
grep -c '^processor' /proc/cpuinfo

khayyam wrote:
These can be combined with a regex ...
Code:
find "${S}" -regextype posix-egrep -iregex ".*\.(jpg|jpeg)$" -type f -print0 | xargs -P$cpucount -n10 -0 jpegoptim -p -q &>/dev/null

True, but he could also use the -o option to find to chain together multiple simple -iname lines.
haarp wrote:
Code:

   exterminate() {
      local whitelist=0

      #check if whitelisted by LINGUAS
      for lingua in ${LINGUAS}; do
         basename "$1" | grep -q ^$lingua && whitelist=1
      done
You can remove the local variable whitelist and instead return out of the middle of the loop on a match.
haarp wrote:
Code:

         echo "rm -r $1" | sed -e "s:/var/tmp/portage/${CATEGORY}/${P}/::" -e "s:///:/:"
You should use $PORTAGE_TMPDIR instead of hardcoding /var/tmp/portage. You could also use $D and remove the need to use $CATEGORY/$P.
haarp wrote:
Code:

   purgelocale() {
      for dir in "$1"/*; do
         #check if valid locale dir
         if [[ -n $(find "$dir" -name "LC_MESSAGES" 2>/dev/null) ]]; then
            exterminate "$dir"
         fi
      done
You can pass multiple directories to find at once. Combining that with -printf can simplify this construct to:
Code:

   purgelocale() {
      find "$1"/* -name LC_MESSAGES -printf '%H\0' 2>/dev/null | while read -d '' dir; do exterminate "$dir"; done
   }
If multiple LC_MESSAGES can be found beneath a given argument, you will need to add a uniq -z to the pipeline.

haarp wrote:
Code:
xargs ${MAKEOPTS/-j/-P} -n10 -0 ...
Some people put other options in MAKEOPTS than just -j. Limiting load average is relatively popular.
Back to top
View user's profile Send private message
haarp
Guru
Guru


Joined: 31 Oct 2007
Posts: 354

PostPosted: Sat Apr 13, 2013 8:22 pm    Post subject: Reply with quote

Thanks Hu, these are some good improvements!


Hu wrote:
What about packages that generate images during build? You might be better off running this as a post_src_install phase to clean $D.

I debated whether to change this myself. I don't know any packages that generate images, but I do know some which pack images into an archive during their build. gzdoom does it, and I think firefox/thunderbird too. Short of running both before and after a build, there isn't a way of catching both possibilities. I think the current phase is preferable.

Quote:
Some people put other options in MAKEOPTS than just -j. Limiting load average is relatively popular.

Yep, I realized that myself. I went back to my for-loop (did I mention that I love sh-style for-loops?). But it's not perfect either (won't work when there's a whitespace between -j and the number)
Back to top
View user's profile Send private message
khayyam
Veteran
Veteran


Joined: 07 Jun 2012
Posts: 1539

PostPosted: Sat Apr 13, 2013 9:26 pm    Post subject: Reply with quote

Hu wrote:
khayyam wrote:
You could replace this with the perhaps more succinct:
Code:
cpucount=$(awk '/^processor/{a=$NF};END{print (a + 1)}' /proc/cpuinfo)

He could also use the even more succinct:
Code:
grep -c '^processor' /proc/cpuinfo

Hu ... ah, yes, but I was also incrementing.

Hu wrote:
khayyam wrote:
These can be combined with a regex ...
Code:
find "${S}" -regextype posix-egrep -iregex ".*\.(jpg|jpeg)$" -type f -print0 | xargs -P$cpucount -n10 -0 jpegoptim -p -q &>/dev/null

True, but he could also use the -o option to find to chain together multiple simple -iname lines.

Yes, -iname "*.jpeg" -o -iname "*.jpg" would have sufficed, but I was under the impression that a regex was faster becuase the matching less greedy.

Hu wrote:
Quote:
Code:
MAKEOPTS/-j/-P} -n10 -0 ...

Some people put other options in MAKEOPTS than just -j. Limiting load average is relatively popular.

Its even popular here, so you'd think I might have paid closer attention :) Anyhow, if I was anything like half awake, and had not tried to follow bash, I would probably have done the following:

Code:
% MAKEOPTS="-jNN -l xxxx.00" ; print ${${MAKEOPTS}[3,4]}
NN

... though its probably more reliable to get the value from /proc/cpuinfo

best ... khay
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum