app-portage/cfg-update - installation instructions

[xentric]

It will stop interpreting perlcode at that point. saves execution time if you have a big documentation section.

Then you can view the documetnation with ~$ perldoc foo.pl

I want the script to display the usage instructions when it's called without (or with wrong) arguments. And I want to keep things as standard as possible for users, so calling "man cfg-update" will give them the man page. Most users are not familiar with the perldoc functionality.

I have fixed this issue by setting a new variable $indexing_completed to "undefined" in the main loop.
When starting the index_mode subroutine, the variable is set to "false" and at the end of the
subroutine it's set to "true". So when an error occurs and code is skipped, the next step in the
main loop is a line that checks if $indexing_complete is "false". If it is, it exits the script.

I admit this ain't pretty, but it effectively does the trick while keeping the rest of the script-flow intact.

[devsk]

why is it looking at last log file in /var/log/portage/? just curious. Because if you are trying to find who emerged last, then /var/log/emerge.log may be better source. How about if current system time or the timestamps are messed on the file system? (in both cases last is not really the last)

[xentric]

why is it looking at last log file in /var/log/portage/? just curious.

cfg-update needs to update the checksum-index with the MD5-checksums of the configfiles of newly merged packages before a new version of those packages will be emerged because Portage overwrites the old checksum upon installing a new version. If that happends cfg-update can't determine if the current file was modified or not... That's why I needed to create an alias for the emerge command that runs cfg-update first, followed by emerge itself.

cfg-update needs to know which package is emerged last. If it the last emerged package is still the same, cfg-update can skip indexing so there won't be unnecessary 10+ second lag due to indexing on every emerge command...

/var/log/emerge.log may be better source. How about if current system time or the timestamps are messed on the file system? (in both cases last is not really the last)

You have a point. At the time I looked at the emerge.log file and decided it was easier to use

Code: Select all

ls -t /var/log/portage | head -n1

to get the last emerged package.

But I just tried the following command to get the last emerged package:

Code: Select all

cat /var/log/emerge.log | grep ">>> emerge" | tail -n 1 | awk {'print $1$7'}

seems to work just as well, so I'll put that in the new version.

Thanks!

[devsk]

But I just tried the following command to get the last emerged package:

Code: Select all

cat /var/log/emerge.log | grep ">>> emerge" | tail -n 1 | awk {'print $1$7'}

seems to work just as well, so I'll put that in the new version.

Thanks!

Or you could try the the awk only version (no grep or tail):

Code: Select all

awk '/>>> emerge/' /var/log/emerge.log |awk 'END{print $1$7}'

[devsk]

I just timed the operations. grep seems faster than awk in finding regukar expressions in the text:

Code: Select all

$ time grep ">>> emerge" /var/log/emerge.log > /dev/null

real    0m0.005s
user    0m0.005s
sys     0m0.001s

$ time awk '/>>> emerge/' /var/log/emerge.log > /dev/null

real    0m0.024s
user    0m0.018s
sys     0m0.002s

repeated several times to rule out caching effect.

So, the best is:

Code: Select all

grep ">>> emerge" /var/log/emerge.log | awk 'END{print $1$7}'

it executes in 6ms on my system, compared to 14ms for your one-liner and 25ms for awk-only one liner. These will be much slower on bigger emerge.log files and slower systems than mine. Remember, emerge.log only grows, so optimize as much as you can...

BTW, just for comparison sake:

Code: Select all

$ time ls -t /var/log/portage | head -n1
media-sound:sox-12.17.9:20060824-233815.log

real    0m0.041s
user    0m0.016s
sys     0m0.027s

PPS: How do you account for people who have /var/log/portage disabled i.e. logging disabled because /etc/make.conf doesn't have /var/log/portage or directory /var/log/portage is not even present, in your original code (I mean in the current code before you move to one of these emerge.log thingies)?

[xentric]

So, the best is:

Code: Select all

grep ">>> emerge" /var/log/emerge.log | awk 'END{print $1$7}'

it executes in 6ms on my system, compared to 14ms for your one-liner and 25ms for awk-only one liner. These will be much slower on bigger emerge.log files and slower systems than mine. Remember, emerge.log only grows, so optimize as much as you can...

Thanks a lot for your help on this, excellent!

PPS: How do you account for people who have /var/log/portage disabled i.e. logging disabled because /etc/make.conf doesn't have /var/log/portage or directory /var/log/portage is not even present, in your original code (I mean in the current code before you move to one of these emerge.log thingies)?

The script checks if PORT_LOGDIR is set in /etc/make.conf
If it isn't set the script prints some text recommending the user to:
A. enable the PORTLOG variable in /etc/make.conf
or
B. use the bash_helper or php_helper scripts submitted by other users to avoid unnecessary indexing.

I have just removed this check from the new version as it becomes obsolete when we use /var/log/emerge.log to determine if indexing is needed

Hey, you seem pretty creative with your solutions, maybe you (or someone else) can point me in the right direction with two other issues that still bug me:

1. cfg-update checks if a GUI is available before it spawns the GUI tool:
Currently the script checks if the DISPLAY variable is set, by calling "echo $DISPLAY"
If it is not empty, it will assume that X is running and it's possible to spawn a GUI tool like xxdiff.
I figured that X can be running without having a local display, so it's not good enough to simply
check if there's a process running with X as it's name... that's why I check the DISPLAY variable.
Maybe there's a better method?

2. cfg-update creates an alias for the emerge command in /root/.bashrc which calls a wrapperscript
that first runs "cfg-update --index" and then continues with the actual emerge command.
This isn't a nice way to ensure that the index stays up to date...
I have considered cronjobs, looked for a "pre-emerge hook" in portage but have not found
a good alternative for running "cfg-update --index" right before an emerge command.

Maybe you have some ideas?

[Rick]

1. cfg-update checks if a GUI is available before it spawns the GUI tool:
Currently the script checks if the DISPLAY variable is set, by calling "echo $DISPLAY"
If it is not empty, it will assume that X is running and it's possible to spawn a GUI tool like xxdiff.
I figured that X can be running without having a local display, so it's not good enough to simply
check if there's a process running with X as it's name... that's why I check the DISPLAY variable.
Maybe there's a better method?

Checking for $DISPLAY is _the_ method to use. But there are other methods too, for example you could look at the $TERM variable, or the $TTY

[ppurka]

I just timed the operations. grep seems faster than awk in finding regukar expressions in the text:
...

It is strange, but here are the results on my system:

Code: Select all

~> time grep ">>> emerge" /var/log/emerge.log > /dev/null
grep ">>> emerge" /var/log/emerge.log > /dev/null  5.45s user 0.03s system 99% cpu 5.513 total

~> time awk '/>>> emerge/' /var/log/emerge.log > /dev/null
awk '/>>> emerge/' /var/log/emerge.log > /dev/null  0.27s user 0.01s system 99% cpu 0.283 total

~> time grep ">>> emerge" /var/log/emerge.log | awk 'END{print $1$7}'
1156651120:net-nds/ypbind-1.19.1
grep ">>> emerge" /var/log/emerge.log  5.41s user 0.03s system 98% cpu 5.525 total
awk 'END{print $1$7}'  0.00s user 0.00s system 0% cpu 5.476 total

Size of /var/log/emerge.log:

Code: Select all

~> ls -l /var/log/emerge.log 
-rw-rw---- 1 portage portage 2645019 2006-08-26 23:58 /var/log/emerge.log

~> hdparm -tT /dev/sda                                                                                      <root

/dev/sda:
 Timing cached reads:   2120 MB in  2.00 seconds = 1059.76 MB/sec
 Timing buffered disk reads:  130 MB in  3.04 seconds =  42.75 MB/sec

Here is something else which I found interesting:

Code: Select all

~> time tac /var/log/emerge.log| sed -n '/>>> emerge/{p;q;}'                                                <root
1156651120:  >>> emerge (3 of 3) net-nds/ypbind-1.19.1 to /
tac /var/log/emerge.log  0.00s user 0.00s system 34% cpu 0.003 total
sed -n '/>>> emerge/{p;q;}'  0.00s user 0.00s system 77% cpu 0.001 total

[xentric]

Checking for $DISPLAY is _the_ method to use. But there are other methods too, for example you could look at the $TERM variable, or the $TTY

Thanks! Keeping in mind that there are lot's of different terminal-types so I guess sticking with the DISPLAY variable is pretty safe and much easier.

ppurka: Here are my results:

Code: Select all

root@gentoo 852MHz 68C xentric # ls -l /var/log/emerge.log
-rw-rw---- 1 portage portage 2011494 Aug 29 02:39 /var/log/emerge.log

root@gentoo 852MHz 69C portage # hdparm -Tt /dev/hda
/dev/hda:
 Timing cached reads:   444 MB in  2.01 seconds = 220.41 MB/sec
 Timing buffered disk reads:   52 MB in  3.00 seconds =  17.32 MB/sec

root@gentoo 852MHz 65C xentric # time awk '/>>> emerge/' /var/log/emerge.log > /dev/null
real    0m0.111s
user    0m0.058s
sys     0m0.017s

root@gentoo 852MHz 65C xentric # time grep ">>> emerge" /var/log/emerge.log > /dev/null
real    0m0.022s
user    0m0.012s
sys     0m0.003s

Note: this is on an old Pentium3 850Mhz laptop with 256Mb RAM and a slow HD...
(I've tested this multiple times to avoid caching differences)

Try this:

Code: Select all

time LC_ALL="C" grep ">>> emerge" /var/log/emerge.log | awk 'END{print $1$7}'

does that speed up grepping? It seems that locales like UTF-8 slow down grep a lot... (see grep manpage about LC_ALL)
The cfg-update script actually sets the locale to C in every subroutine that uses grep...

[ppurka]

You were right. I have utf-8 set in my LC_*. Here are the new statistics with LC_ALL="C",- grep sure takes way lower time than awk, but the sed beats them both, with far less cpu (btw, it is a 2.2GHz Athlon XP 3200+). I was just thinking you must be having a super fast system there with a 15000 rpm hard disk to get such low numbers.

Code: Select all

~> time LC_ALL="C" grep ">>> emerge" /var/log/emerge.log > /dev/null                                        
LC_ALL="C" grep ">>> emerge" /var/log/emerge.log > /dev/null  0.00s user 0.00s system 95% cpu 0.007 total

~> time LC_ALL="C" awk '/>>> emerge/' /var/log/emerge.log > /dev/null                                        
LC_ALL="C" awk '/>>> emerge/' /var/log/emerge.log > /dev/null  0.02s user 0.00s system 97% cpu 0.029 total

~> time LC_ALL="C" tac /var/log/emerge.log| sed -n '/>>> emerge/{p;q;}'                          
1156651120:  >>> emerge (3 of 3) net-nds/ypbind-1.19.1 to /
LC_ALL="C" tac /var/log/emerge.log  0.00s user 0.00s system 39% cpu 0.003 total
sed -n '/>>> emerge/{p;q;}'  0.00s user 0.00s system 78% cpu 0.001 total

[xentric]

grep sure takes way lower time than awk, but the sed beats them both, with far less cpu

You're right! It takes less time on my system too:

Code: Select all

root@gentoo 852MHz 68C portage # time tac /var/log/emerge.log| sed -n '/>>> emerge/{p;q;}'
1156811850:  >>> emerge (1 of 1) app-misc/beep-1.2.2 to /
real    0m0.009s
user    0m0.003s
sys     0m0.005s

So I've put this in the new version of the script

Next thing is to speed up creating the index... LOL

I currently use:

Code: Select all

root@gentoo 852MHz 75C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | awk '{ print $2"  "$3 }' >> /tmp/checksum-index
real    0m9.742s
user    0m1.989s
sys     0m0.650s

Any ideas?

(BTW, I'm gonna take a nap... it's 05:42am... oops!)

[devsk]

cut is faster for this particular case because it is pretty plain set of args:

Code: Select all

$ time grep "^obj " /var/db/pkg/*/*/CONTENTS | cut -d" " -f2-3 > /tmp/tmper.idx

real    0m0.287s
user    0m0.292s
sys     0m0.148s

$ time grep "^obj " /var/db/pkg/*/*/CONTENTS | awk '{ print $2"  "$3 }' > /tmp/tmper.idx2

real    0m0.576s
user    0m0.548s
sys     0m0.140s

$ l /tmp/tmper.idx*
-rw-r--r-- 1 devsk users 18622047 Aug 28 23:06 /tmp/tmper.idx
-rw-r--r-- 1 devsk users 18834722 Aug 28 23:04 /tmp/tmper.idx2

It also creates a smaller index...

Code: Select all

$ bc
bc 1.06.94
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'. 
18834722-18622047
212675

Code: Select all

$ wc -l /tmp/tmper.idx*
  212675 /tmp/tmper.idx
  212675 /tmp/tmper.idx2
  425350 total

one blank character (which you inserted for an unknown reason) less per line. Otherwise perfect match.

The gains might be larger for slower systems. post your numbers please. I wanna see if its faster than 10 seconds you get currently.

[swimmer]

The problem is more the 'first shot' - when you index after a longer time it
takes quite a time to get through /var/db/pkg ... but don't ask me why - I only
notice that and do not understand

First run:

#> time grep "^obj " /var/db/pkg/*/*/CONTENTS | cut -d" " -f2-3 > /tmp/tmper.idx
real 0m17.422s
user 0m0.402s
sys 0m0.310s
#> time grep "^obj " /var/db/pkg/*/*/CONTENTS | awk '{ print $2" "$3}' > /tmp/tmper.idx2
real 0m0.773s
user 0m0.574s
sys 0m0.162s
#> wc -l /tmp/tmper.idx*
240238 /tmp/tmper.idx
240238 /tmp/tmper.idx2
480476 total
#> ls -l /tmp/tmper.idx*
-rw-r--r-- 1 swimmer users 20704744 2006-08-29 12:44 /tmp/tmper.idx
-rw-r--r-- 1 swimmer users 20944982 2006-08-29 12:45 /tmp/tmper.idx2

Second run:

#> time grep "^obj " /var/db/pkg/*/*/CONTENTS | cut -d" " -f2-3 > /tmp/tmper.idx
real 0m0.606s
user 0m0.378s
sys 0m0.197s
#> time grep "^obj " /var/db/pkg/*/*/CONTENTS | awk '{ print $2" "$3 }' > /tmp/tmper.idx2
real 0m0.782s
user 0m0.573s
sys 0m0.186s
#> wc -l /tmp/tmper.idx*
240238 /tmp/tmper.idx
240238 /tmp/tmper.idx2
480476 total
#> ls -l /tmp/tmper.idx*
-rw-r--r-- 1 swimmer users 20704744 2006-08-29 12:46 /tmp/tmper.idx
-rw-r--r-- 1 swimmer users 20944982 2006-08-29 12:46 /tmp/tmper.idx2

The differences between awk and cut aren't as big ...

Greetz
swimmer

[xentric]

cut is faster for this particular case because it is pretty plain set of args:
...
The gains might be larger for slower systems. post your numbers please. I wanna see if its faster than 10 seconds you get currently.

cut is a little bit faster on my old system too:

Code: Select all

*** YESTERDAY ***

root@gentoo 852MHz 68C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | awk '{ print $2"  "$3 }' >> /tmp/checksum-index
real    0m10.764s
user    0m1.896s
sys     0m0.606s

root@gentoo 852MHz 72C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | awk '{ print $2"  "$3 }' >> /tmp/checksum-index
real    0m10.110s
user    0m1.973s
sys     0m0.625s

root@gentoo 852MHz 75C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | awk '{ print $2"  "$3 }' >> /tmp/checksum-index
real    0m9.742s
user    0m1.989s
sys     0m0.650s

*** TODAY ***

root@gentoo 852MHz 70C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | cut -d" " -f2-3 > /tmp/checksum-index2
real    0m10.017s
user    0m1.431s
sys     0m0.639s

root@gentoo 852MHz 67C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | cut -d" " -f2-3 > /tmp/checksum-index2
real    0m4.482s
user    0m1.458s
sys     0m0.441s

root@gentoo 852MHz 62C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | cut -d" " -f2-3 > /tmp/checksum-index2
real    0m2.382s
user    0m1.486s
sys     0m0.369s

root@gentoo 852MHz 63C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | cut -d" " -f2-3 > /tmp/checksum-index2
real    0m2.428s
user    0m1.457s
sys     0m0.408s

root@gentoo 852MHz 64C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | awk '{ print $2"  "$3 }' >> /tmp/checksum-index
real    0m3.299s
user    0m1.862s
sys     0m0.431s

root@gentoo 852MHz 70C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | awk '{ print $2"  "$3 }' >> /tmp/checksum-index
real    0m3.037s
user    0m1.884s
sys     0m0.423s

On the first run both methods take about 12 seconds to complete. Look at the effect that caching has on the timings... looks like there are a lot of factors that have impact on this kind of benchmarking. Today the awk method is a lot faster after chaching.
The awk method goes down to 5 seconds after a couple of runs and the cut method does it in 4 seconds.

The cut method is a little bit better, so I'll stick to that. Thanks

[xentric]

The problem is more the 'first shot' - when you index after a longer time it
takes quite a time to get through /var/db/pkg ... but don't ask me why - I only
notice that and do not understand

I think that's due to caching. The stuff in /var/db/pkg remains in RAM after the first run, so on sebsequent runs your system can use this data which is already in RAM. That's way faster than your HD.

Just for fun, look on the forums for "shake your files". That method is used to force files into RAM so programs start up a bit faster.

First run:

Code: Select all

#> time grep "^obj " /var/db/pkg/*/*/CONTENTS | cut -d" " -f2-3 > /tmp/tmper.idx 
real    0m17.422s             
user    0m0.402s              
sys     0m0.310s

Looking at your subsequent runs it seems like you have a pretty fast system but it's horribly slow when it uses the HD, maybe you haven't got DMA enabled on your HD? That usually makes it a lot slower. Can you tell me what your system specs are? It makes me curious to see what you get when you execute the hdparm throughput-test command:

Code: Select all

[root@gentoo 852MHz 56C portage # ls -l /var/log/emerge.log
-rw-rw---- 1 portage portage 2012089 Aug 29 06:11 /var/log/emerge.log

root@gentoo 852MHz 56C portage # hdparm -I /dev/hda
/dev/hda:
ATA device, with non-removable media
        Model Number:       TOSHIBA MK8026GAX
        Serial Number:      65TL1869T
        Firmware Revision:  PA001G
Standards:
        Supported: 6 5 4 3
        Likely used: 6
Configuration:
        Logical         max     current
        cylinders       16383   17475
        heads           16      15
        sectors/track   63      63
        --
        CHS current addressable sectors:   16513875
        LBA    user addressable sectors:  156301488
        device size with M = 1024*1024:       76319 MBytes
        device size with M = 1000*1000:       80026 MBytes (80 GB)
Capabilities:
        LBA, IORDY(can be disabled)
        bytes avail on r/w long: 48     Queue depth: 1
        Standby timer values: spec'd by Standard, no device specific minimum
        R/W multiple sector transfer: Max = 16  Current = 16
        Advanced power management level: unknown setting (0x0080)
        DMA: sdma0 sdma1 sdma2 mdma0 mdma1 mdma2 udma0 udma1 *udma2 udma3 udma4 udma5
             Cycle time: min=120ns recommended=120ns
        PIO: pio0 pio1 pio2 pio3 pio4
             Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
        Enabled Supported:
           *    NOP cmd
           *    READ BUFFER cmd
           *    WRITE BUFFER cmd
           *    Host Protected Area feature set
           *    Look-ahead
           *    Write cache
           *    Power Management feature set
                Security Mode feature set
           *    SMART feature set
           *    Mandatory FLUSH CACHE command
           *    Device Configuration Overlay feature set
                SET MAX security extension
           *    Advanced Power Management feature set
           *    SMART self-test
           *    SMART error logging
Security:
        Master password revision code = 65534
                supported
        not     enabled
        not     locked
        not     frozen
        not     expired: security count
        not     supported: enhanced erase
        58min for SECURITY ERASE UNIT.
HW reset results:
        CBLID- above Vih
        Device num = 0 determined by the jumper
Checksum: correct



root@gentoo 852MHz 56C portage # hdparm -Tt /dev/hda
/dev/hda:
 Timing cached reads:   588 MB in  2.01 seconds = 292.51 MB/sec
 Timing buffered disk reads:   62 MB in  3.03 seconds =  20.49 MB/sec

root@gentoo 852MHz 59C portage # hdparm -Tt /dev/hda
/dev/hda:
 Timing cached reads:   588 MB in  2.01 seconds = 292.77 MB/sec
 Timing buffered disk reads:   62 MB in  3.09 seconds =  20.09 MB/sec

root@gentoo 852MHz 57C portage # hdparm -Tt /dev/hda
/dev/hda:
 Timing cached reads:   592 MB in  2.01 seconds = 294.30 MB/sec
 Timing buffered disk reads:   60 MB in  3.00 seconds =  19.98 MB/sec

This is a 2,5" laptop HD: 80Gb with 16Mb cache and 5400 RPM. So if you have a 7200 RPM 3,5" HD you should get better results.

This is my setting for /dev/hda in /etc/conf.d/hdparm:

Code: Select all

disc0_args="-c1 -d1 -m16 -u1 -X66"

Caution: do not use the -X66 unless you are sure that your HD can handle it. Maybe you have a newer driver that supports even better like -X69 (=udma5). Just check "man hdparm" for the -X option. It needs to be (udma_level+64), so mine uses udma2.

[swimmer]

[...]
I think that's due to caching. The stuff in /var/db/pkg remains in RAM after the first run, so on sebsequent runs your system can use this data which is already in RAM. That's way faster than your HD.

... I see

Looking at your subsequent runs it seems like you have a pretty fast system but it's horribly slow when it uses the HD, maybe you haven't got DMA enabled on your HD? That usually makes it a lot slower. Can you tell me what your system specs are? It makes me curious to see what you get when you execute the hdparm throughput-test command [...]

Hmmm - I have a SATA disk and I thought it's not necessary to optimize it with
hdparm since this happens automatically. Correct me if I'm wrong ...

Code: Select all

swimmer ~ # hdparm -I /dev/sda

/dev/sda:

ATA device, with non-removable media
        Model Number:       Maxtor 6Y250M0
        Serial Number:      Y69GHVVE
        Firmware Revision:  YAR511W0
Standards:
        Supported: 7 6 5 4
        Likely used: 7
Configuration:
        Logical         max     current
        cylinders       16383   16383
        heads           16      16
        sectors/track   63      63
        --
        CHS current addressable sectors:   16514064
        LBA    user addressable sectors:  268435455
        LBA48  user addressable sectors:  490234752
        device size with M = 1024*1024:      239372 MBytes
        device size with M = 1000*1000:      251000 MBytes (251 GB)
Capabilities:
        LBA, IORDY(can be disabled)
        Queue depth: 1
        Standby timer values: spec'd by Standard, no device specific minimum
        R/W multiple sector transfer: Max = 16  Current = 1
        Advanced power management level: unknown setting (0x0000)
        Recommended acoustic management value: 192, current value: 254
        DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
             Cycle time: min=120ns recommended=120ns
        PIO: pio0 pio1 pio2 pio3 pio4
             Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
        Enabled Supported:
           *    NOP cmd
           *    READ BUFFER cmd
           *    WRITE BUFFER cmd
           *    Host Protected Area feature set
           *    Look-ahead
           *    Write cache
           *    Power Management feature set
                Security Mode feature set
           *    SMART feature set
           *    FLUSH CACHE EXT command
           *    Mandatory FLUSH CACHE command
           *    Device Configuration Overlay feature set
           *    48-bit Address feature set
           *    Automatic Acoustic Management feature set
                SET MAX security extension
                Advanced Power Management feature set
           *    DOWNLOAD MICROCODE cmd
           *    SMART self-test
           *    SMART error logging
Security:   
        Master password revision code = 65534
                supported
        not     enabled
        not     locked
        not     frozen
        not     expired: security count
        not     supported: enhanced erase
Checksum: correct


swimmer ~ # hdparm -tT /dev/sda
            
/dev/sda:   
 Timing cached reads:   3528 MB in  2.00 seconds = 1763.81 MB/sec
 Timing buffered disk reads:  168 MB in  3.02 seconds =  55.72 MB/sec
swimmer ~ # hdparm -tT /dev/sda
            
/dev/sda:   
 Timing cached reads:   3520 MB in  2.00 seconds = 1759.04 MB/sec
 Timing buffered disk reads:  168 MB in  3.03 seconds =  55.48 MB/sec
swimmer ~ # hdparm -tT /dev/sda
            
/dev/sda:   
 Timing cached reads:   3496 MB in  2.00 seconds = 1747.62 MB/sec
 Timing buffered disk reads:  170 MB in  3.02 seconds =  56.22 MB/sec


swimmer ~ # ll /var/log/emerge.log 
-rw-rw---- 1 portage portage 2226909 Aug 29 07:44 /var/log/emerge.log
swimmer ~ # 

Looks ok to me

Greetz
swimmer

[xentric]

Hmmm - I have a SATA disk and I thought it's not necessary to optimize it with
hdparm since this happens automatically. Correct me if I'm wrong ...
...
Looks ok to me

Yeah, that sure looks good!
Pretty weird that it takes your system about 18 seconds on the first run.

I have /var on a reiser3 filesystem mounted with noatime,nodiratime,notail.
As /var/db/pkg contains lot's of small files, I guess this is where reiserfs with notail makes a difference.

[devsk]

1. cfg-update checks if a GUI is available before it spawns the GUI tool:
Currently the script checks if the DISPLAY variable is set, by calling "echo $DISPLAY"
If it is not empty, it will assume that X is running and it's possible to spawn a GUI tool like xxdiff.
I figured that X can be running without having a local display, so it's not good enough to simply
check if there's a process running with X as it's name... that's why I check the DISPLAY variable.
Maybe there's a better method?

The right way of checking is thru both DISPLAY and xhost, like:

Code: Select all

if ( [[ -z ${DISPLAY} ]] || ! (xhost &>/dev/null) ) ; then echo DISPLAY_NOTTHERE; else echo DISPLAY_PRESENT;fi

So, if DISPLAY is set but user (root can't connect because Joe user didn't do xhost +) can't connect to it, DISPLAY_NOTTHERE will be echoed in this case. xhost fails in few seconds if it can't connect to $DISPLAY. Of course, the check comes out immd. with DISPLAY_NOTTHERE if DISPLAY variable is not set.

[xentric]

So, if DISPLAY is set but user (root can't connect because Joe user didn't do xhost +) can't connect to it, DISPLAY_NOTTHERE will be echoed in this case. xhost fails in few seconds if it can't connect to $DISPLAY. Of course, the check comes out immd. with DISPLAY_NOTTHERE if DISPLAY variable is not set.

Well, I have tried to test this but all I get when I run xhost without any arguments is a message that access control is enabled. It seems like I can't disable access control so I can't test your line of bash-code.
If I run "xhost -localhost" or "xhost -" (with user that started the X server) I still get the same message when I run "xhost" as root. I can't find a file containing an accesslist or anything. The manpage is refering to /etc/X*.hosts but I have no such files.

Next week I'll (finally) get a brand spanking new Core2Duo system on which I will start from scratch by installing the new 2006.1. That's also a good opportunity to test installing cfg-update on a default setup. I'll check if xhost exits with a STDERR message.

[devsk]

So, if DISPLAY is set but user (root can't connect because Joe user didn't do xhost +) can't connect to it, DISPLAY_NOTTHERE will be echoed in this case. xhost fails in few seconds if it can't connect to $DISPLAY. Of course, the check comes out immd. with DISPLAY_NOTTHERE if DISPLAY variable is not set.

Well, I have tried to test this but all I get when I run xhost without any arguments is a message that access control is enabled. It seems like I can't disable access control so I can't test your line of bash-code.
If I run "xhost -localhost" or "xhost -" (with user that started the X server) I still get the same message when I run "xhost" as root. I can't find a file containing an accesslist or anything. The manpage is refering to /etc/X*.hosts but I have no such files.

Next week I'll (finally) get a brand spanking new Core2Duo system on which I will start from scratch by installing the new 2006.1. That's also a good opportunity to test installing cfg-update on a default setup. I'll check if xhost exits with a STDERR message.

the test that I posted checks for exit code of xhost. Try

Code: Select all

DISPLAY=random_host_does_not_exist:0 xhost &>/dev/null
echo $?

exit code indicates if xhost succeeded or failed in getting a status of access control of the X. without any arguments it gives a status if the access control is enabled or disabled. This is easily verified by 'su' in as root in joe's running X xterm. Then type 'xhost'. It will give the status. In another xterm as user joe, do 'xhost -', followed by a 'xhost' in 'su' xterm, followed by 'xhost +' in joe's xterm, followed by 'xhost' in 'su' xterm. You will see status changes in 'su' window.

Trust me that test works if copied verbatim. Just replace the 'echo' commands with whatever you wanna do if X is available and otherwise.

[xentric]

Trust me that test works if copied verbatim. Just replace the 'echo' commands with whatever you wanna do if X is available and otherwise.

Code: Select all

* did "xhost +" in other terminal

        root@gentoo 852MHz 48C / # xhost
        access control disabled, clients can connect from any host

* did "xhost -" in other terminal

        root@gentoo 852MHz 48C / # xhost
        access control enabled, only authorized clients can connect

* did "xhost +localhost" in other terminal

        root@gentoo 852MHz 48C / # xhost
        access control enabled, only authorized clients can connect
        INET:gentoo.homelinux

* test with unknown host

        root@gentoo 852MHz 48C / # DISPLAY=foobar:0 xhost &>/dev/null; echo $?
        1

* test with empty DISPLAY variable

        root@gentoo 852MHz 48C / # DISPLAY= xhost &>/dev/null; echo $?
        1

* test with correct DISPLAY variable

        root@gentoo 852MHz 48C / # DISPLAY=:0 xhost &>/dev/null; echo $?
        0

Verified and works, I will use this in the new version of the script.
Thanks a lot devsk!

[devsk]

Verified and works, I will use this in the new version of the script.

please don't forget to test for the $DISPLAY first and xhost after that, just to optimise the case when DISPLAY isn't even set. Otherwise, cfg-update -u will seem like hung for few seconds because xhost will take few seconds to come out.

Thanks a lot devsk!

You are very welcome.

[xentric]

Otherwise, cfg-update -u will seem like hung for few seconds because xhost will take few seconds to come out.

I don't get any delay when running xhost if I unset the DISPLAY variable, nor when it is set but doesn't contain a value or when it contains an invalid value... it responds immediately with errorcode 0 or 1.
Does xhost "hang" while waiting for a timeout on your system?

I can put the following check in the update_files subroutine:

Code: Select all

if ((using GUI mergetool) and (xhost returns errorcode 1)) {
    print: GUI is unavailable...
    print: if X is installed, run cfg-update in X-terminal otherwise use "-t sdiff" or "-t vimdiff"
    print: if this fails while using an X-terminal, check xhost accesscontrol and DISPLAY variable
    print: if X is not installed, set MERGETOOL in /etc/cfg-update.conf to "sdiff" or "vimdiff"
    print: do you want to [q]uit or continue with [s]diff or [v]imdiff ?
    read keypress
}

This is my preferred way to go... even if xhost "hangs" when trying to connect to the X-server.

PS. I'll treat gvimdiff as a CLI tool because if you run it while X is not available it simply falls back to vimdiff automatically. That's actually pretty cool

[XenoTerraCide]

hmm... cfg-update used to work for me but lately it's not finding the graphical updater has something changed that I would need to update? I'm using kdiff3 (which is installed) and have it set

Code: Select all

MERGETOOL = /usr/bin/kdiff3

anything wrong with this?

[xentric]

hmm... cfg-update used to work for me but lately it's not finding the graphical updater has something changed that I would need to update? I'm using kdiff3 (which is installed) and have it set

Code: Select all

MERGETOOL = /usr/bin/kdiff3

anything wrong with this?

First of all, try "sux" (instead of "su" or "sudo") to become root in your Xterminal, and check if cfg-update can launch the GUI tool. If that works, it probably is simple to fix with some basic troubleshooting:

Try running "cfg-update -vu". The -v option unhides STDERR messages that could give some clues about not being able to start the GUI tool...

Try "xhost +localhost" (as the user who started the X server) and then log in with root and try cfg-update.
Or check if the DISPLAY variable is properly set with "env | grep DISPLAY", it should return something like DISPLAY=:0
Running "source /etc/profile" (when logged in as root as root) before you start cfg-update could also work.

Please post some feedback if you find out what changed, maybe it helps other users with similar problems.