quietemerge -- provide pretty emerge output

[ppurka]

Updates (reverse chronological order)
Do let me know if you find any bugs. This script is pretty much going into a maintenance stage since I can't think of any more features to add. Adding --jobs was the most difficult for me, but now that is done, and the script is stable. I would like to keep it stable which implies I should leave it as it is.

Note (15 Mar 2015): Google Code is dead. But this script has been on GitHub for quite a few years already. So, no worries

• 20 Dec 2014: Version 20141220.Fix breakage due to changed emerge output.
• 05 Feb 2014: Version 20140205.Small update to change the color of time taken.
• 27 Apr 2013: Version 20130427.Small update for new make.conf location.
• 13 Apr 2012: Version 20120413.Small update to catch new emerge message.
• 06 Apr 2012: Version 20120406.Bugfix: Squash two rusty old bugs.
• 29 Jan 2012: Update zsh-completion instructions.
• 15 Jan 2012: Version 20120115.Bugfix: Reintroduce -q
• 30 Dec 2011: Version 20111230.Bugfix: Fix an infinite recursion.
• 24 Dec 2011: Version 20111224. Execute emerge directly on unsupported args. Fixed some bugs.
• 27 Nov 2011: Version 20111127. Show additional total ETA without binary emerge.
• 13 Nov 2011: Version 20111113. Remove -q from emerge command.
• 27 Aug 2011: Version 20110827. Small bugfixes.
• 26 Jun 2011: Version 20110617. Bugfix for --autounmask-write. Remove references to genlop on this post.
• 11 Jun 2011: Version 20110611. Support for --autounmask-write.
• 21 May 2011: Version 20110521. Mainly one bug fix when SHOW_MERGE_TIME=1
• 20 May 2011: Version 20110520. with --jobs support and some bug fixes. Updated screenshot and usage below
• 30 Apr 2011: Update to version 20110430.Should detect --color=n in command line now. Also bug fix for --resume. Updated screenshot.
• 22 Apr 2011: Version 20110422. Use own genlop function and remove genlop dependency.
• 07 Apr 2010: Version 20100407. Support for a pre-command and a post-command.
• 19 Jan 2010: Version 20100119. Support for multiple quietemerge instances.
• 22 Dec 2009: Update to version 20091222
• 02 Dec 2009: Allow environment variable to override config file setting.
• 21 Nov 2009: Some bug fixes and some security fixes.
• 20 Nov 2009: Respect --ask in EMERGE_DEFAULT_OPTS.
• 12 Nov 2009: Updated to version 20091112. Some bug fixes and some features
• 05 Nov 2009: Workaround the bug in genlop. Again.
• 02 Nov 2009: New config option. SHOW_MERGE_TIME. Set it to 1 to see the merge time for the package instead of "Done!"
  Updated this version to workaround genlop bug.
• 30 Oct 2009: One (not so obvious) bug fix.
• 29 Oct 2009: Version 20091029; support for --keep-going; numerous other changes.
• 26 Oct 2009: Include infon from a different script.
• 23 Oct 2009: New version with compatibility with --ask option of emerge; updated screenshot
• 17 Oct 2009: Added bash completion instruction. Updated zsh completion instruction.

Brief Description:
I am providing this script on a request at the monthly screenshot thread. What this script does is provide a nice timer, while emerging packages. An example screenshot is present in http://oi51.tinypic.com/1ze820g.jpg

How it works:
The script first checks if the emerge process will proceed properly, by doing a emerge --pretend. If the emerge process was successful, then the script launches emerge via

Code: Select all

emerge <rest of the command line options> 2>&1 > output_file &

Since --quiet-build is now the default in emerge, this script now doesn't launch emerge as emerge -q.
The program then repeatedly checks the output of emerge in output_file for updates and accordingly provides an updated terminal output.

Usage:
The script should be run simply as if you were running emerge. To facilitate a nice output as shown in the screenshot, it is recommended to not use emerge options such as -c, -C, --sync. The script itself takes no arguments other than -h or --help and passes all the rest of the arguments to emerge.

If the script detects any unsupported arguments, then it simply executes emerge directly, bypassing all the other setup in the script.

This is a bash script and its compatibility with other shells is unknown. It will definitely produce tons of errors if called as some other shell. So you can either make the script executable (chmod 755 quietemerge) and then just run it as quietemerge, or run it as bash quietemerge.

In order to temporarily override the settings present in the config file, prepend the variable with an underscore _ and provide it as an environment variable.
An example where I want to override the config file setting temporarily, and/or provide a temporary setting:

Code: Select all

 _DELAY=2 _MOUNT_TMPFS=1 quietemerge -a eselect

Download:
You can download the script from: https://github.com/ppurka/quietemerge/r ... 220.tar.gz
You can copy the file quietemerge after extracting it from the tarball.

The files have also been added retroactively to a git repository: http://code.google.com/p/quietemerge/source/list

Installation:
Copy the script to some directory which is in $PATH of root. Personally, I have it installed in /usr/local/sbin.

Dependencies:
Till version 20100407: The script depends on app-portage/genlop.
From version 20110422: No dependency on genlop.

Configuration:
The script creates a configuration file $HOME/.config/quietemerge.config. It is not mandatory to modify that file,- I have tried to keep the defaults sane. However, if you want to mount tmpfs on to /var/tmp/portage or run your favourite config update utility after emerge is over, I would suggest looking at the configuration file.

zsh completion configuration:
If you use zsh, then you can perform the following steps to enable command line autocompletion of quietemerge in zsh.
New instructions for zsh completion:

1. Install zsh-completion:

   Code: Select all

   emerge zsh-completion

   Add the following line to the beginning of your $HOME/.zshrc

   Code: Select all

   [[ -f /etc/zsh/zprofile ]] && . /etc/zsh/zprofile

2. Enable completion by running these commands

   Code: Select all

   echo "compdef quietemerge=emerge" >> $HOME/.zshrc
   exec zsh

Old Instructions for zsh completion:

1. Install zsh-completion:

   Code: Select all

   emerge zsh-completion

   Add the following line to the beginning of your $HOME/.zshrc

   Code: Select all

   [[ -f /etc/zsh/zprofile ]] && . /etc/zsh/zprofile

2. Create a directory /etc/zsh/zsh-functions
3. Enable this new zsh completion path:

   Code: Select all

   echo "fpath=( /etc/zsh/zsh-functions \$fpath )" >> /etc/zsh/zprofile

4. Download _quietemerge and copy it to /etc/zsh/zsh-functions
5. Run the commands

   Code: Select all

   env-update
   exec zsh

[/size]

Bash completion configuration:
Command line completion of quietemerge in bash can be achieved by executing the following steps.

1. Install and configure gentoo-bashcomp

   Code: Select all

   emerge gentoo-bashcomp
   eselect bashcomp enable --global base
   eselect bashcomp enable --global gentoo

2. Add the following two lines to the end of your $HOME/.bashrc

   Code: Select all

   [[ -f /etc/profile.d/bash-completion.sh ]] && source /etc/profile.d/bash-completion.sh
   complete -o filenames -F _emerge quietemerge

3. Run the command

   Code: Select all

   exec bash

Website: http://code.google.com/p/quietemerge/ ( needed some place to host the file )
Git: 1. On Google Code. 2. On Github
README: 1. On Google Code. 2. On Github.
Version: 20141220

[evert_]

Thanks for all the work you've put into posting this . I'm going to test it tonight.

Used it for a few merges now and I'm very happy with it, no bugs yet . Altough there are a few 'features' i would love to see: see how many merges are going to happen. Also, it's in my opinion very scary to do 'the shot in the dark' emerge -uDN world, would it be possible to make it display first what packages with the --ask? .

[ppurka]

Thanks for all the work you've put into posting this . I'm going to test it tonight.

Used it for a few merges now and I'm very happy with it, no bugs yet . Altough there are a few 'features' i would love to see: see how many merges are going to happen. Also, it's in my opinion very scary to do 'the shot in the dark' emerge -uDN world, would it be possible to make it display first what packages with the --ask? .

Interesting. I will give it some thought on how to introduce these features. Give me a couple of days. I will try to script it in the weekend.

My main objective is to have no arguments for the script itself (except -h). All the arguments should be exactly the same as you give to emerge.

EDIT: By the way, you do get the total number of merges, the current merge number and the total time left. Look at your terminal titlebar

[jprobichaud]

Hi,

Since this [great] tool relies on genlop, you might be interested to take a look at this bug. I've submitted a patch so that "genlop -p" is faster at estimating the time emerging a large set of packages should take.

Cheers!

[ppurka]

Hi,

Since this [great] tool relies on genlop, you might be interested to take a look at this bug. I've submitted a patch so that "genlop -p" is faster at estimating the time emerging a large set of packages should take.

Cheers!

Very nice
I know that genlop -p takes significant time. That is why I don't call genlop -p everytime,- it is run only when a new package is being emerged and then it updates the terminal's window titlebar with the total estimated time left. What runs in a loop is genlop -c.

I hope your patch gets accepted. That way everyone will benefit

This is one reason why I run emerge in the background. So, the emerge process runs independently of the time estimating process (genlop). After all, we also care about how quickly the emerge process finishes instead of just some pretty output. I am sure no one would want pretty output if it delayed emerge.

PS: I completed writing the code to take care of --ask last weekend, but I didn't get the opportunity/time to test it completely. I will test it by this weekend and release it after that.

[Bones McCracker]

Nice work, ppurka.

[jprobichaud]

PS: I completed writing the code to take care of --ask last weekend, but I didn't get the opportunity/time to test it completely. I will test it by this weekend and release it after that.

Will we see the actual estimated time before we can say "yes"?

[ppurka]

Nice work, ppurka.

Thank you

PS: I completed writing the code to take care of --ask last weekend, but I didn't get the opportunity/time to test it completely. I will test it by this weekend and release it after that.

Will we see the actual estimated time before we can say "yes"?

I hadn't thought of that! Good idea. I will include this too (It should be quite easy to provide this estimate). However, I think I will provide only the total time and not time for each package since the latter will be a complete beast.

[jprobichaud]

I think I will provide only the total time and not time for each package since the latter will be a complete beast.

Actually, if you take the patched version of genlop, you almost have it for free:

Code: Select all

$ emerge $(eix -I -C kde --format "<category>/<name>\n" | grep -v Found) -pv --nodeps | genlop -pq
/* the emerge pretend output */

                       akonadi [     3m 35s]                                                                                                                     
                     akregator [     1m 55s]                                                                                                                     
                           ark [     1m  3s]                                                                                                                     
                       automoc [        16s]                                                                                                                     
                       dolphin [     1m 53s]                                                                                                                     
                  dragonplayer [        53s]                                                                                                                     
                       drkonqi [        54s]                                                                                                                     
                      gwenview [     1m 57s]                                                                                                                     
                           juk [     1m 45s]                                                                                                                     
                  kaddressbook [     3m 37s]                  
...

[ppurka]

Updated quietemerge to version 20091023. Updated screenshot available at http://omploader.org/vMmx0Zw
Changelog:

• Added compatibility with --ask. --verbose will also work (emerge itself takes care of this when it is piping to somewhere else).
• Fixed bug with --resume ( fortunately, no one noticed )
• Total emerge time is given when --ask is used.

As before, the script itself takes no options. All options are passed to emerge.

[jprobichaud]

oups, there is a little bug:
line 422, you have "infon ..."

I guess that it should have been "info ...."

[ppurka]

oups, there is a little bug:
line 422, you have "infon ..."

I guess that it should have been "info ...."

Shoot! This is really bad! I forgot to include this function from my_bash_functions
Updated version coming quickly.

[ppurka]

@jprobichaud Sorry I should have been more careful.

Update to version 20091026. Changelog:

• Include infon from a different script.
• Now --pretend will also give you time remaining.

[jprobichaud]

The new version is pretty good.

Thanks again ppurka for this great tool, it is really useful and even make emerge process enjoyable!

Cheers

[jprobichaud]

It seems that I'm hitting another potential bug:

Code: Select all

  * Install  media-libs/speex-1.2_rc1.                                            Done!
  * Install  dev-perl/DBI-1.609.                                                  Done!
*** %n in writable segment detected *** Time left:             1 minute and 15 seconds.
  * Install  sys-apps/coreutils-7.5-r1.                                           Done!
  * Install  dev-perl/Locale-gettext-1.05-r1.                                     Done!

Any idea what this "%n in writable..." message means?

[ppurka]

It seems that I'm hitting another potential bug:

Code: Select all

  * Install  media-libs/speex-1.2_rc1.                                            Done!
  * Install  dev-perl/DBI-1.609.                                                  Done!
*** %n in writable segment detected *** Time left:             1 minute and 15 seconds.
  * Install  sys-apps/coreutils-7.5-r1.                                           Done!
  * Install  dev-perl/Locale-gettext-1.05-r1.                                     Done!

Any idea what this "%n in writable..." message means?

That is strange. I have no idea what that was
I am using emerge and taking all stderr and stdout to /tmp/output.log. Can you check /tmp/output.log (if you still have it around) and check what is printing before the coreutils package? My script does not give any such output.

It looks like some output from emerge perhaps overwrote the "Emerging <package>" region, just enough to keep the "Time left:" part alone. This must have happened with the coreutils package. Since emerge will spit out a newline and not erase the rest of the line (the "Time left:" part), my script will start printing from the next line. This makes it pretty certain that it is in the emerge of the coreutils package that that line is being printed. It also gives the time .. you had 1min and 15sec of coreutils emerge left. So, you can even run a normal emerge and see whether emerge spits out that line when around 1min and 15sec is left Some command like this should suffice:

Code: Select all

emerge -q coreutils 2>&1 > tmp.out

EDIT: I just tried it myself and I didn't get that output. So, it is specific to your system.
EDIT 2: Fix the emerge command redirection above. Saw the mistake after many hours

[jprobichaud]

Sadly, I don't have the original /tmp/output/log, but I've follow your suggestion:

Some command like this should suffice:

Code: Select all

emerge -q coreutils 2>&1 > tmp.out

Which gives:

Code: Select all

jrobicha@mt-gen-jprob ~ $ sudo emerge -q coreutils 2>&1 > tmp.out
*** %n in writable segment detected ***
jrobicha@mt-gen-jprob ~ $

If I don't redirect the output to tmp.out, I get:

Code: Select all

>>> Verifying ebuild manifests
>>> Emerging (1 of 1) sys-apps/coreutils-7.5-r1
*** %n in writable segment detected ***
>>> Installing (1 of 1) sys-apps/coreutils-7.5-r1
 * Regenerating GNU info directory index...
 * Processed 181 info files.

I've removed the "-q" option to emerge and saved the Konsole output to see, within the entire emerge output, where that msg could come and I've found that it comes from the 'configure' section:

Code: Select all

 26 ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --lo     calstatedir=/var/lib --libdir=/usr/lib64 --enable-install-program=arch --enable-no-install-program=groups,hostname,kill,su,uptime --enable-largefile --disable-libcap --enable     -nls --enable-acl --enable-xattr --without-gmp                                                                                                                                
  27 checking for a BSD-compatible install... /usr/bin/install -c
  28 checking whether build environment is sane... yes
  29 checking for a thread-safe mkdir -p... /bin/mkdir -p
  30 checking for gawk... gawk
  31 checking whether make sets $(MAKE)... yes
...
 340 checking whether printf supports the 'F' directive... yes
 341 checking whether printf supports the 'n' directive... *** %n in writable segment detected ***
 342 no
 343 checking whether printf supports the 'ls' directive... yes
 ...

I guess this isn't a bug with quietemerge

[ppurka]

I guess this isn't a bug with quietemerge

Maybe it was. I am not sure.

New version 20091029. Changelog:

• Support for failed packages if --keep-going is used.
• Change "Time Left" to "ETA",- it is shorter
• Actually calculate the space required by the part after ETA:
• Rewrite terminal title every $DELAY time, o/w emerge hijacks it
• Use >& instead of 2>&1. Seem to get less overwrites (looks like my bash redirection understanding is flawed?)
• Use separate tmp file for user and for root, and remove the file if emerge was successful.

The --keep-going will look like this if a package failed:

Code: Select all

-------------------------------- Starting emerge -------------------------------

  * Install  rox-base/rox-lib-2.0.5-r1.                                  Error! 
  * Skipped  rox-base/thumbs-0.1.4 due to --keep-going
  * Emerging app-admin/eselect-1.2.3. ETA:                                     

EDIT: Finally realised what I was doing wrong. I used emerge 2>&1 > output_file instead of emerge > output_file 2>&1. The order matters Anyway, I am using the bashism emerge >& output_file in the latest version of the script. So it should be fine.

[ppurka]

Version 20091030. Changelog:
Just one bug fix: Remove the skipped package from tmp file so that the correct count and correct remaining time is given by genlop.

[jw5801]

Great script! I'm not sure if EWARN/ELOG and configuration file update messages are being handled nicely though? It looks to me as though a line has been dropped and the order rearranged a bit in the output below:

Code: Select all

-------------------------------------------- Pretended emerge --------------------------------------------

These are the packages that would be merged, in order:

Calculating dependencies  ... done!
[ebuild     U ] dev-libs/apr-1.3.9 [1.3.8]
[ebuild     U ] app-admin/syslog-ng-3.0.4 [2.1.4] USE="pcre%* ssl%* -caps%" 
[ebuild     U ] net-mail/dovecot-1.2.6 [1.1.7-r1] USE="bzip2%* maildir%* zlib%* -caps% -cydir% -dbox% -sql
ite%" 
[ebuild     U ] dev-python/twisted-8.2.0-r2 [8.1.0]
[ebuild     U ] x11-libs/libgksu-2.0.12 [2.0.9]

                                           Total ETA: 8 minutes.                                          

  * Continue with the emerge process? ([Yes]|No): y
--------------------------------------------- Starting emerge --------------------------------------------
  * Install  dev-libs/apr-1.3.9.                                                                    Done! 
  * Install  app-admin/syslog-ng-3.0.4.                                                             Done! 
  * Install  net-mail/dovecot-1.2.6.                                                                Done! 
  * Install  dev-python/twisted-8.2.0-r2.                                                           Done! 
 * If you are upgrading from Dovecot 1.1, read 
 *  http://wiki.dovecot.org/Upgrading/1.2
  * Install  x11-libs/libgksu-2.0.12.                                                               Done! 
  * There are pending configuration updates:
 * manage the log files.  syslog-ng installs a file in /etc/logrotate.d
 * for logrotate to use.

 * Messages for package net-mail/dovecot-1.2.6:

 * GNU info directory index is up-to-date.

 * IMPORTANT: 4 config files in '/etc' need updating.
 * See the CONFIGURATION FILES section of the emerge
 * man page to learn how to update config files.

    See /tmp/output.log for more information.

The actual EWARN/ELOG messages for this upgrade were:

Code: Select all

>>> Messages generated by process 6559 on 2009-10-31 14:06:19 EST for package app-admin/syslog-ng-3.0.4:

LOG: postinst
It is highly recommended that app-admin/logrotate be emerged to
manage the log files.  syslog-ng installs a file in /etc/logrotate.d
for logrotate to use.


>>> Messages generated by process 6559 on 2009-10-31 14:09:42 EST for package net-mail/dovecot-1.2.6:

WARN: install
If you are upgrading from Dovecot 1.1, read 
 http://wiki.dovecot.org/Upgrading/1.2

EDIT: Ah, I notice that /tmp/output.log contains the proper EWARN and ELOG messages.

EDIT2: Well, I wasn't using the latest version, I was using the 29th Oct version. I upgraded to the most recent version and emerged just syslog-ng to check what happens and it printed correctly, so either you fixed it in the update, or there were conflicts with having several messages to print in my earlier update.

Code: Select all

----------------------------------------- Pretended emerge -----------------------------------------

These are the packages that would be merged, in order:

Calculating dependencies ... done!
[ebuild     U ] app-admin/syslog-ng-3.0.4 [2.1.4] USE="pcre%* ssl%* -caps%" 

                                   Total ETA: less than a minute.                                   


------------------------------------------ Starting emerge -----------------------------------------

  * Install  app-admin/syslog-ng-3.0.4.                                                       Done! 
  * There are pending configuration updates:
 * Messages for package app-admin/syslog-ng-3.0.4:

 * It is highly recommended that app-admin/logrotate be emerged to
 * manage the log files.  syslog-ng installs a file in /etc/logrotate.d
 * for logrotate to use.
 * GNU info directory index is up-to-date.

 * IMPORTANT: 3 config files in '/etc' need updating.
 * See the CONFIGURATION FILES section of the emerge
 * man page to learn how to update config files.

    See /tmp/output.log for more information.

[ppurka]

Great script! I'm not sure if EWARN/ELOG and configuration file update messages are being handled nicely though? It looks to me as though a line has been dropped and the order rearranged a bit in the output below:

The fact is that I don't handle ewarn/elog at all! That is because emerge -q or emerge -qv does not output any ewarn or elog, as far as I could see. If it does output any ewarn/elogs, I think it outputs only a small subset. The output you are seeing (as you wrote under EDIT2) is actually simply the last 10 lines of /tmp/output.log. I output that just to let the user know that configuration updates are pending

ewarn/elogs are better read by a different program such as elogv or elogviewer.

About rearranging the order, it seems you were probably using the version from 26th Oct or earlier. This had the bug that jprobichaud mentions a few posts earlier. The bug was that stderr was not being redirected to the /tmp/output.log. The reason was that I had mistakenly put the 2>&1 before the output redirection (see just two posts earlier). This bug was fixed in the release of 29th Oct. The bug fix in the 30th Oct release is very different and really minor,- the terminal title would not have been updated correctly. One would have hit that bug only if one used --keep-going and if some package failed to merge.

I believe the 30th Oct release is bug free If you can find bugs do let me know

[jw5801]

The fact is that I don't handle ewarn/elog at all! That is because emerge -q or emerge -qv does not output any ewarn or elog, as far as I could see. If it does output any ewarn/elogs, I think it outputs only a small subset. The output you are seeing (as you wrote under EDIT2) is actually simply the last 10 lines of /tmp/output.log. I output that just to let the user know that configuration updates are pending

Ah, cool. No bugs, but a feature request: Would it be possible/simple to print the time taken to emerge each package where you're currently printing 'Done!'?

[ppurka]

The fact is that I don't handle ewarn/elog at all! That is because emerge -q or emerge -qv does not output any ewarn or elog, as far as I could see. If it does output any ewarn/elogs, I think it outputs only a small subset. The output you are seeing (as you wrote under EDIT2) is actually simply the last 10 lines of /tmp/output.log. I output that just to let the user know that configuration updates are pending

Ah, cool. No bugs, but a feature request: Would it be possible/simple to print the time taken to emerge each package where you're currently printing 'Done!'?

It should be possible and simple to do this. I will give it a try.

[ppurka]

Ah, cool. No bugs, but a feature request: Would it be possible/simple to print the time taken to emerge each package where you're currently printing 'Done!'?

This is now available in version 20091102

Changelog:

• Determine the PORTAGE_TMPDIR by portageq, and mount tmpfs on to PORTAGE_TMPDIR/portage (for people who do not compile in /var/tmp/portage)
• Added a new config option: SHOW_MERGE_TIME. If it is 1, then time spent on the merge will be shown instead of "Done!". To set it to 1, do a

  Code: Select all

  echo "SHOW_MERGE_TIME=1" >> ~/.config/quietemerge.config

Adding it was simple but I realised that the script was getting complex and repetitive in many places. I wanted to put some of the repetitive parts in a function. But it seems it will take a bit of time and may introduce unwanted bugs if not done properly, and I want the script to remain bug free. So, cleaning up the script is postponed for later.

[ppurka]

Ha! Bug in genlop!!

Code: Select all

~> genlop -t "=dev-libs/apr-1.3.9" --date 2 minute ago
Use of uninitialized value in subtraction (-) at /usr/bin/genlop line 1039, <_GEN_0> line 23512.
Use of uninitialized value in subtraction (-) at /usr/bin/genlop line 1040, <_GEN_0> line 23512.
Use of uninitialized value in subtraction (-) at /usr/bin/genlop line 1042, <_GEN_0> line 23512.
 * dev-libs/apr

     Tue Nov  3 00:01:33 2009 >>> dev-libs/apr-1.3.9
Use of uninitialized value in subtraction (-) at /usr/bin/genlop line 1078, <_GEN_0> line 23512.
       merge time: 5 days, 5 hours, 1 minute and 33 seconds.
~> genlop -t "=dev-libs/apr-1.3.9"                    
 * dev-libs/apr

     Tue Nov  3 00:01:33 2009 >>> dev-libs/apr-1.3.9
       merge time: 1 minute and 31 seconds.

Updated Version 20091102 to work around genlop bug