Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
stupid bash tricks: multi-emerge-tail
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
gmt
n00b
n00b


Joined: 15 Feb 2013
Posts: 7
Location: meatspace

PostPosted: Sat Jan 04, 2014 11:54 pm    Post subject: stupid bash tricks: multi-emerge-tail Reply with quote

Here is a very ugly script I've whipped up to watch the good ol' meaningless spew scroll by when doing parallel emerges:

Code:

#!/bin/bash

[[ $( id -u ) -eq 0 ]] || {
   [[ -z "$I_WANT_TO_BE_ME" ]] && {
      echo 'becoming root via sudo and reinvoking myself...' >&2
      echo '(or perhaps you [[ -z "$I_WANT_TO_BE_ME" ]] ?)' >&2
      echo >&2
      SCRIPT=`realpath -s $0`
      echo "running: ${SCRIPT} $*" >&2
      echo >&2
      sleep 0.5
      exec sudo "${SCRIPT}" "$@"
      exit 0
   }
}

delay=2.5

vartmpportage="$( portageq envvar PORTAGE_TMPDIR )/portage"

notfirst=
bam=poo
# turns out it surprisingly difficult to figure out what to test here...
# while [[ $( ps -eo args | grep '^/usr/bin/python.*/emerge' ) ]] ; do
while [[ 1 ]] ; do
   oldbam=bam
   bam=
   if [[ $notfirst ]] ; then
      sleep ${delay}
   else
      notfirst=exactly
   fi
   active_emerges=$( qlop -cC | grep '^[[:space:]]\*' | sed 's/^[[:space:]]*\*[[:space:]]*//' | sed 's/[[:space:]][[:space:]]*/\n/g' | sort )
   lines=5
   if [[ ${active_emerges} ]] ; then
      lines=$(( ( $( tput lines ) / $( echo ${active_emerges} | wc -w ) ) - 2 ))
   fi
   (( lines < 2 )) && lines=2
   for merge in ${active_emerges} ; do
      hdr=$( echo -e "\033[1m[[ ${merge} ]]\033[0m" )
      cols=$(tput cols)
      thelines="$( tail -n ${lines} "${vartmpportage}"/${merge}/temp/build.log | fold -w ${cols} | tail -n ${lines} )"
      # if bam already nonempty, add a newline for aesthetic purposes
      if [[ ${bam} ]] ; then
         bam=$( echo "${bam}" ; echo ; echo "${hdr}" )
      else
         bam="${hdr}"
      fi
      bam=$( echo "${bam}" ; echo "${thelines}" )
   done
   if [[ ${oldbam} != ${bam} ]] ; then
      clear
      echo "${bam:-"nuthin'"}"
   fi
done

echo "looks like emerge finished."

_________________
-gmt
Back to top
View user's profile Send private message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 2506
Location: The Peanut Gallery

PostPosted: Tue Jan 07, 2014 2:40 am    Post subject: Reply with quote

You might like to check out update which does a similar thing when using --jobs, and also filters emerge output when not.

It's an awful lot bigger than yours, but started out along the same lines, only there was no --jobs back then, and indeed no --keep-going, which was the original problem to fix.

Your grep | sed | sed can all be done with one sed afaict:
Code:
sed -n '/^[[:blank:]]*\*[[:blank:]]*/{s///;s/[[:blank:]][[:blank:]]*//;p;}'

should give you the basis, though I'm not sure what's going on with \n in there. sed is line-based, so you might as well use [[:blank:]] which is slightly quicker. The only way you could have newlines is if you used the n command. I have a feeling the last substitution may be better as s/[[:blank:]].*/;

Code:
lines=$(( ( $( tput lines ) / $( echo ${active_emerges} | wc -w ) ) - 2 ))
makes little sense to me: you'd be better off using an array for active imo, and then ${#active[@]} is the number of elements in the array, or build it with active[n++]=$pkg to get rid of the echo | wc -w subshell.

Code:
bam=$( echo "${bam}" ; echo ; echo "${hdr}" )
is much easier as
Code:
bam+="
$hdr"
or I use an $EOL readonly so:
Code:
bam+=$EOL$hdr
(shells do not field-split assignment, same as case.)

subshells are the thing which slows stuff down the most, ime, especially with bash which is quite a heavy program in comparison to smaller shells. #bash and #sed on IRC: chat.freenode.net are your friends.

HTH,
steveL
Back to top
View user's profile Send private message
gmt
n00b
n00b


Joined: 15 Feb 2013
Posts: 7
Location: meatspace

PostPosted: Tue Jan 07, 2014 9:22 am    Post subject: Reply with quote

steveL wrote:
You might like to check out update which does a similar thing when using --jobs, and also filters emerge output when not.

It's an awful lot bigger than yours, but started out along the same lines, only there was no --jobs back then, and indeed no --keep-going, which was the original problem to fix.

That looks pretty cool. Many's time time I considered building such a beast myself but, well there's only room in my brain for so many half-finished projects :)

Your grep | sed | sed can all be done with one sed afaict:
Code:
sed -n '/^[[:blank:]]*\*[[:blank:]]*/{s///;s/[[:blank:]][[:blank:]]*//;p;}'


should give you the basis, though I'm not sure what's going on with \n in there. sed is line-based, so you might as well use [[:blank:]] which is slightly quicker. The only way you could have newlines is if you used the n command. I have a feeling the last substitution may be better as s/[[:blank:]].*/;



Since I wrote this code (if you can even call it that hehe) I finally broke down and learned a few basics of how sed works... I'm a way better sed scripter now but I still think I pretty much suck at it :)

To be clear, I'm well aware that this is not anything even vaguely resembling decently written.

I occasionally throw together a crappy little script like this as a one-off but end up addicted to it... this was one of those.

Definitely one from the Jackson Pollack school of software design... Don't think I've used qlop or tput before or since -- today I have no idea what they are (of course I could ask Google but then I'd probably have to forget something to make room for it -- with my luck, it'd be the knowledge of where my genitals are, or how to put on socks, so I'm not gonna risk it :D )

steveL wrote:

subshells are the thing which slows stuff down the most, ime, especially with bash which is quite a heavy program in comparison to smaller shells. #bash and #sed on IRC: chat.freenode.net are your friends.


Many ACKs above. I fucking /love/ #bash. They are shockingly nice there, and as often as not, happy to explore whatever impossibly obscure corner case I've decided I need to obsess over on any given 3am coding bender run amok. And subshells are clearly from the devil, I'm fully on that page with you.

Dunno about other shells. Bash seems pretty damn fast to me ... but I don't have meaningful amounts of experience with anything else and anyhow, my computer has rack ears instead of a megapixel camera on both ends -- I should probably bear in mind that most of theirs will go way slower than mine does when I'm expecting people to consume code I've written...
_________________
-gmt
Back to top
View user's profile Send private message
gmt
n00b
n00b


Joined: 15 Feb 2013
Posts: 7
Location: meatspace

PostPosted: Tue Jan 07, 2014 9:25 am    Post subject: Reply with quote

steveL wrote:
check out update


Looks kinda cool, btw! Or if not cool, useful. I'll check it out.
_________________
-gmt
Back to top
View user's profile Send private message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 2506
Location: The Peanut Gallery

PostPosted: Wed Jan 08, 2014 6:01 am    Post subject: Reply with quote

gmt wrote:
Since I wrote this code (if you can even call it that hehe) I finally broke down and learned a few basics of how sed works... I'm a way better sed scripter now but I still think I pretty much suck at it :)

To be clear, I'm well aware that this is not anything even vaguely resembling decently written.

I occasionally throw together a crappy little script like this as a one-off but end up addicted to it... this was one of those.

Definitely one from the Jackson Pollack school of software design... Don't think I've used qlop or tput before or since -- today I have no idea what they are (of course I could ask Google)

Use the manpages silly ;) With portage-utils (q*) they're pretty minimal, but they don't change much, and are designed for use in scripts. They're all the same binary q, it's multi-call, built for use in rescue situations. Though algorithmically it had a lot of problems last time I looked (over 3 years ago) it's always been very quick. So, you end up just playing with it each time you're writing something that uses it, to check the output. That's a good habit to get into in any case, since a lot of scripting is wrapping other commands; basically building a bit of knowledge in around them, to make routine tasks routine so complex issues are easier to handle.

Hmm tput and terminal handling is a big one: but you're already using #bash so you're in the right place. There's ##tty as well, though it tends to be a bit more on the code side. You'll find some stuff in the libIgli file that's part of update that deals with tput and colours. $CUP is used to move up the terminal for example, and the --jobs filtering uses a lot of that.

As for stuff you've thrown together, don't worry about it: I know how you feel and that's exactly how I started out. update was simply an exercise to learn bash with, and the issue at the time was no --keep-going, but for me it was the pages of spew from every compile output meaning I couldn't see what was happening. So filtering emerges (build-logs essentially) became the major focus, til --jobs came along. It's still used when -j1 or we're installing eg toolchain stuff, where we do one at a time before the main build.

You can try that out using --test for a package you've already installed: it'll just work from the build-log to show you a compacted display as if it had run that compile now (only a lot quicker ofc.:)
Quote:
Dunno about other shells. Bash seems pretty damn fast to me ... but I don't have meaningful amounts of experience with anything else and anyhow, my computer has rack ears instead of a megapixel camera on both ends -- I should probably bear in mind that most of theirs will go way slower than mine does when I'm expecting people to consume code I've written...

I was amazed how far I could push my old 32-bit 1G machine with update; I kept waiting for it to blow up on the script, but it never did. But yeah, most of the "optimisation" in shellscript comes from not shelling out, paradoxically, for minor things that can be done by the runtime. Bash is definitely a step up if you need to write larger glue apps; arrays for instance are essential.
Quote:
Looks kinda cool, btw! Or if not cool, useful. I'll check it out.

Heh thanks :) That's another reason i haven't reworked it and split it into different files: it's had to stay in use, and I don't have the time. Also at some point we'll reimplement it in C or something, so it's easier to keep it as is and save the rewrite for another language, preferably by parsing the bash (which is a much more fun use of time;)

Feel free to patch it and play with it: you can't hurt anyone else's machine ;) and in any case any mistakes you make are unlikely to lead to damage to your machine. For a start it'll likely crash on you immediately, and if not you just hit CTRL-C at any point and it'll bail. Most errors occur in the logic around emerge, not during the actual emerge itself. It's got various safety checks; criticalPkg comes to mind, but there are lots of places where it will abort and give you a stack trace if you pass it dodgy data. And it won't downgrade your toolchain unless you force it to. ;) Having said that, you can also use it for ROOT and PORTAGE_CONFIGROOT builds (an area that has been expanded but not fully-tested to my knowledge) or in a chroot (I did that for 6 months when we were working on ABI upgrades in /etc/warning for expat; from binhosts as well which was kinda spooky: just like a binary distro ;)

Looking at your script, you're outputting from the build-logs: that would be nice to incorporate into the current display which is simply a list of packages and where they are in the process. Something I'd like, so if you fancy it, play with runInstall() which is the multi-job, terminal-aware function. Don't worry too much about keeping it in sync with runInstallPipe() just get something displaying how you'd like it (if you want) and i'll sync anything that needs it (unlikely since the other function doesn't display much by design, and there's no terminal.) If not, no bother, but don't hold back from patching if you feel like it (or asking me to, assuming you have an idea that we can reasonably implement, and you're willing to test.)

People sometimes get put off by how large it is, but each piece is not complex, and that's more an indication of how ad-hoc it is, which should make you feel more relaxed about changing it. Most of the tricky code is in support functions to make handling the tree reasonable, and then higher-layers build on top of that. Oh and some faking of associative arrays, from bash-3 days ;) Tricky stuff is more likely to be found in the lib file, unless it's to do with the portage tree, and most of that is only complex if you're trying to rework version comparison, or something fundamental.

The rest is just dealing with output from various programs and deciding what to do about it. The interesting things from our end are usually making it do something specific for a package, when a hook is not sufficient. The two main mechanisms for that are the warning file, and something being in toolchain (an area I've been working on for last year on and off, and wanted for several more: update -a --toolchain; I'm still working on the last bits for that, around perl, but it tests the version of the compiler used to build stuff now, for example.)

Hmm that went on a bit longer than I intended; blame old age ;)
Regards,
steveL.
Back to top
View user's profile Send private message
Maitreya
Apprentice
Apprentice


Joined: 11 Jan 2006
Posts: 221

PostPosted: Wed Jan 08, 2014 1:27 pm    Post subject: Reply with quote

That is some nice Fu there. Here I am just using a one-liner :
Code:

watch -n 3  -c "ccache -s && genlop -c && tail /var/tmp/portage/*/*/temp/build.log"
Back to top
View user's profile Send private message
gmt
n00b
n00b


Joined: 15 Feb 2013
Posts: 7
Location: meatspace

PostPosted: Thu Jan 30, 2014 10:26 am    Post subject: Re: stupid bash tricks: multi-emerge-tail Reply with quote

gmt wrote:
Here is a very ugly script


This updated version uses lsof to implement a considerably better qlop than qlop is. Oh, since my script stopped working I was forced to figure out what qlop is btw :) It's part of portage-utils. The problem is, it assumes that portage will present its $0 to /proc/${PID} in a certain format that it sometimes doesn't. I should probably fix qlop while I'm at it but I'm pretty lazy...

Code:

#!/bin/bash

[[ $( id -u ) -eq 0 ]] || {
        [[ -z "$I_WANT_TO_BE_ME" ]] && {
                echo 'becoming root via sudo and reinvoking myself...' >&2
                echo '(or perhaps you [[ -z "$I_WANT_TO_BE_ME" ]] ?)' >&2
                echo >&2
                SCRIPT=`realpath -s $0`
                echo "running: ${SCRIPT} $*" >&2
                echo >&2
                sleep 0.5
                exec sudo "${SCRIPT}" "$@"
                exit 0
        }
}

delay=2.5

vartmpportage="$( portageq envvar PORTAGE_TMPDIR )/portage"

notfirst=
bam=poo
# turns out it surprisingly difficult to figure out what to test here...
# while [[ $( ps -eo args | grep '^/usr/bin/python.*/emerge' ) ]] ; do
while [[ 1 ]] ; do
        oldbam=bam
        bam=
        if [[ $notfirst ]] ; then
                sleep ${delay}
        else
                notfirst=exactly
        fi
        # active_emerges=$( qlop -cC | grep '^[[:space:]]\*' | sed 's/^[[:space:]]*\*[[:space:]]*//' | sed 's/[[:space:]][[:space:]]*/\n/g' | sort )
        active_emerges=$( lsof "${vartmpportage}"/*/*/temp/build.log 2>/dev/null | sed -e 's|^.*/\([^/]*/[^/]*\)/temp/build.log|\1|' | tail -n +2 | sort -u )
        lines=5
        if [[ ${active_emerges} ]] ; then
                lines=$(( ( $( tput lines ) / $( echo ${active_emerges} | wc -w ) ) - 2 ))
        fi
        (( lines < 2 )) && lines=2
        for merge in ${active_emerges} ; do
                hdr=$( echo -e "\033[1m[[ ${merge} ]]\033[0m" )
                cols=$(tput cols)
                thelines="$( tail -n ${lines} "${vartmpportage}"/${merge}/temp/build.log | fold -w ${cols} | tail -n ${lines} )"
                # if bam already nonempty, add a newline for aesthetic purposes
                if [[ ${bam} ]] ; then
                        bam=$( echo "${bam}" ; echo ; echo "${hdr}" )
                else
                        bam="${hdr}"
                fi
                bam=$( echo "${bam}" ; echo "${thelines}" )
        done
        if [[ ${oldbam} != ${bam} ]] ; then
                clear
                echo "${bam:-"nuthin'"}"
        fi
done

echo "looks like emerge finished."

_________________
-gmt
Back to top
View user's profile Send private message
gmt
n00b
n00b


Joined: 15 Feb 2013
Posts: 7
Location: meatspace

PostPosted: Thu Jan 30, 2014 11:00 pm    Post subject: Re: stupid bash tricks: multi-emerge-tail Reply with quote

gmt wrote:
Here is a very ugly scrip


Discovered there is vastly superior implementation of this at: https://github.com/mgorny/portage-jobsmon

Unfortunately, it does suffer, somewhat, (as does mine, still, but not nearly so much after I put in the lsof hack) from false negatives when finding the running emerges.

Meanwhile, I figured out a way to split lines without getting confused by color escape codes. Unfortunately it requires some perl modules for which there are no ebuilds:

Code:

#!/bin/bash

[[ $( id -u ) -eq 0 ]] || {
   [[ -z "$I_WANT_TO_BE_ME" ]] && {
      echo 'becoming root via sudo and reinvoking myself...' >&2
      echo '(or perhaps you [[ -z "$I_WANT_TO_BE_ME" ]] ?)' >&2
      echo >&2
      SCRIPT=`realpath -s $0`
      echo "running: ${SCRIPT} $*" >&2
      echo >&2
      sleep 0.5
      exec sudo "${SCRIPT}" "$@"
      exit 0
   }
}

delay=2.5

vartmpportage="$( portageq envvar PORTAGE_TMPDIR )/portage"

notfirst=
bam=poo
# turns out it surprisingly difficult to figure out what to test here...
# while [[ $( ps -eo args | grep '^/usr/bin/python.*/emerge' ) ]] ; do
while [[ 1 ]] ; do
   oldbam=bam
   bam=
   if [[ $notfirst ]] ; then
      sleep ${delay}
   else
      notfirst=exactly
   fi
   active_emerges=()
   while read mrg; do
      active_emerges+=("${mrg}")
   done < <( lsof "${vartmpportage}"/*/*/temp/build.log 2>/dev/null | sed -e 's|^.*/\([^/]*/[^/]*\)/temp/build\.log|\1|' | tail -n +2 | sort -u )
   lines=5
   active_logs=()
   for merge in "${active_emerges[@]}"; do
      active_logs+=("${vartmpportage}"/${merge}/temp/build.log)
   done
   if (( ${#active_logs[@]} )); then
      lines=$(( ( $( tput lines ) / ${#active_emerges[@]} ) - 2 ))
      if (( lines > 8 )) ; then
         poss_active_logs=()
         for amerge in "${active_emerges[@]}" ; do
            poss_active_logs+=("${vartmpportage}"/${amerge}/temp/build.log)
            sublogs=()
            while read nulog; do
               sublogs+=("${nulog}")
            done < <( find "${vartmpportage}"/${amerge}/temp -name 'build-*.log' 2>/dev/null )
            poss_active_logs+=("${sublogs[@]}")
         done
         poss_lines=$(( ( $( tput lines ) / ${#poss_active_logs[@]} ) - 2 ))
         if (( poss_lines >= 3 )) ; then
            lines=${poss_lines}
            active_logs=("${poss_active_logs[@]}")
         fi
      fi
   fi
   (( lines < 2 )) && lines=2
   for log in "${active_logs[@]}" ; do
      merge="${log#${vartmpportage}/}"
      merge="${merge%.log}"
      merge="${merge/\/temp\/build/}"
      
      hdr=$( echo -e "\033[1m[[ ${merge} ]]\033[0m" )
      cols=$(tput cols)
      thelines="$( tail -n ${lines} "${log}" | perl -e "use Text::ANSI::Util; binmode(STDOUT, \":utf8\"); while(<STDIN>) { print Text::ANSI::Util::ta_mbwrap(\$_, ${cols}); }" | tail -n ${lines} )"
      # if bam already nonempty, add a newline for aesthetic purposes
      if [[ ${bam} ]] ; then
         bam=$( echo "${bam}" ; echo ; echo "${hdr}" )
      else
         bam="${hdr}"
      fi
      bam=$( echo "${bam}" ; echo "${thelines}" )
   done
   if [[ ${oldbam} != ${bam} ]] ; then
      clear
      echo "${bam:-"nuthin'"}"
   fi
done

echo "looks like emerge finished."


This requires perl modules:

dev-perl/Text-ANSI-Util/Text-ANSI-Util-0.12.ebuild:
Code:

# $Header: $

EAPI=4

MODULE_AUTHOR="SHARYANTO"
MODULE_VERSION=${PV}
inherit perl-module

DESCRIPTION="Terminal control using ANSI escape sequences"

LICENSE="GPL-1"
SLOT="0"
KEYWORDS="~amd64"
IUSE=""

SRC_TEST="do"

RDEPEND="dev-perl/Data-Dump
      dev-perl/Text-WideChar-Util"


and dev-perl/Text-WideChar-Util/Text-WideChar-Util-0.08.ebuild
Code:

# Copyright 1999-2013 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

EAPI=4

MODULE_AUTHOR="SHARYANTO"
MODULE_VERSION=${PV}
inherit perl-module

DESCRIPTION="Terminal control using ANSI escape sequences"

LICENSE="GPL-1"
SLOT="0"
KEYWORDS="~amd64"
IUSE=""

SRC_TEST="do"

RDEPEND="dev-perl/Unicode-LineBreak"


Anyhow, I doubt I'll work on it any more now that I've discovered mgorny's script which has a far more "practical" (iow: not shit) implementation based on python and a real curses library. Going forward it seems more sensible to contribute to that than to continue to dink around with this.
_________________
-gmt
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum