View previous topic :: View next topic |
Author |
Message |
xentric Guru
Joined: 16 Mar 2003 Posts: 410 Location: Netherlands
|
Posted: Mon Aug 28, 2006 10:42 am Post subject: |
|
|
chovy wrote: | It will stop interpreting perlcode at that point. saves execution time if you have a big documentation section.
Then you can view the documetnation with ~$ perldoc foo.pl | I want the script to display the usage instructions when it's called without (or with wrong) arguments. And I want to keep things as standard as possible for users, so calling "man cfg-update" will give them the man page. Most users are not familiar with the perldoc functionality.
I have fixed this issue by setting a new variable $indexing_completed to "undefined" in the main loop.
When starting the index_mode subroutine, the variable is set to "false" and at the end of the
subroutine it's set to "true". So when an error occurs and code is skipped, the next step in the
main loop is a line that checks if $indexing_complete is "false". If it is, it exits the script.
I admit this ain't pretty, but it effectively does the trick while keeping the rest of the script-flow intact. _________________ When all else fails, read the manual...
Registered Linux User #340626 |
|
Back to top |
|
|
devsk Advocate
Joined: 24 Oct 2003 Posts: 2995 Location: Bay Area, CA
|
Posted: Mon Aug 28, 2006 4:30 pm Post subject: |
|
|
why is it looking at last log file in /var/log/portage/? just curious. Because if you are trying to find who emerged last, then /var/log/emerge.log may be better source. How about if current system time or the timestamps are messed on the file system? (in both cases last is not really the last) |
|
Back to top |
|
|
xentric Guru
Joined: 16 Mar 2003 Posts: 410 Location: Netherlands
|
Posted: Mon Aug 28, 2006 9:56 pm Post subject: |
|
|
devsk wrote: | why is it looking at last log file in /var/log/portage/? just curious. | cfg-update needs to update the checksum-index with the MD5-checksums of the configfiles of newly merged packages before a new version of those packages will be emerged because Portage overwrites the old checksum upon installing a new version. If that happends cfg-update can't determine if the current file was modified or not... That's why I needed to create an alias for the emerge command that runs cfg-update first, followed by emerge itself.
cfg-update needs to know which package is emerged last. If it the last emerged package is still the same, cfg-update can skip indexing so there won't be unnecessary 10+ second lag due to indexing on every emerge command...
Quote: | /var/log/emerge.log may be better source. How about if current system time or the timestamps are messed on the file system? (in both cases last is not really the last) | You have a point. At the time I looked at the emerge.log file and decided it was easier to use Code: | ls -t /var/log/portage | head -n1 | to get the last emerged package.
But I just tried the following command to get the last emerged package: Code: | cat /var/log/emerge.log | grep ">>> emerge" | tail -n 1 | awk {'print $1$7'} | seems to work just as well, so I'll put that in the new version.
Thanks! _________________ When all else fails, read the manual...
Registered Linux User #340626 |
|
Back to top |
|
|
devsk Advocate
Joined: 24 Oct 2003 Posts: 2995 Location: Bay Area, CA
|
Posted: Mon Aug 28, 2006 10:39 pm Post subject: |
|
|
xentric wrote: |
But I just tried the following command to get the last emerged package: Code: | cat /var/log/emerge.log | grep ">>> emerge" | tail -n 1 | awk {'print $1$7'} | seems to work just as well, so I'll put that in the new version.
Thanks! |
Or you could try the the awk only version (no grep or tail):
Code: | awk '/>>> emerge/' /var/log/emerge.log |awk 'END{print $1$7}' |
|
|
Back to top |
|
|
devsk Advocate
Joined: 24 Oct 2003 Posts: 2995 Location: Bay Area, CA
|
Posted: Mon Aug 28, 2006 10:55 pm Post subject: |
|
|
I just timed the operations. grep seems faster than awk in finding regukar expressions in the text:
Code: | $ time grep ">>> emerge" /var/log/emerge.log > /dev/null
real 0m0.005s
user 0m0.005s
sys 0m0.001s
$ time awk '/>>> emerge/' /var/log/emerge.log > /dev/null
real 0m0.024s
user 0m0.018s
sys 0m0.002s
|
repeated several times to rule out caching effect.
So, the best is:
Code: | grep ">>> emerge" /var/log/emerge.log | awk 'END{print $1$7}' |
it executes in 6ms on my system, compared to 14ms for your one-liner and 25ms for awk-only one liner. These will be much slower on bigger emerge.log files and slower systems than mine. Remember, emerge.log only grows, so optimize as much as you can...
BTW, just for comparison sake: Code: | $ time ls -t /var/log/portage | head -n1
media-sound:sox-12.17.9:20060824-233815.log
real 0m0.041s
user 0m0.016s
sys 0m0.027s
|
PPS: How do you account for people who have /var/log/portage disabled i.e. logging disabled because /etc/make.conf doesn't have /var/log/portage or directory /var/log/portage is not even present, in your original code (I mean in the current code before you move to one of these emerge.log thingies)? |
|
Back to top |
|
|
xentric Guru
Joined: 16 Mar 2003 Posts: 410 Location: Netherlands
|
Posted: Tue Aug 29, 2006 12:33 am Post subject: |
|
|
devsk wrote: | So, the best is: Code: | grep ">>> emerge" /var/log/emerge.log | awk 'END{print $1$7}' | it executes in 6ms on my system, compared to 14ms for your one-liner and 25ms for awk-only one liner. These will be much slower on bigger emerge.log files and slower systems than mine. Remember, emerge.log only grows, so optimize as much as you can... | Thanks a lot for your help on this, excellent!
Quote: | PPS: How do you account for people who have /var/log/portage disabled i.e. logging disabled because /etc/make.conf doesn't have /var/log/portage or directory /var/log/portage is not even present, in your original code (I mean in the current code before you move to one of these emerge.log thingies)? | The script checks if PORT_LOGDIR is set in /etc/make.conf
If it isn't set the script prints some text recommending the user to:
A. enable the PORTLOG variable in /etc/make.conf
or
B. use the bash_helper or php_helper scripts submitted by other users to avoid unnecessary indexing.
I have just removed this check from the new version as it becomes obsolete when we use /var/log/emerge.log to determine if indexing is needed
Hey, you seem pretty creative with your solutions, maybe you (or someone else) can point me in the right direction with two other issues that still bug me:
1. cfg-update checks if a GUI is available before it spawns the GUI tool:
Currently the script checks if the DISPLAY variable is set, by calling "echo $DISPLAY"
If it is not empty, it will assume that X is running and it's possible to spawn a GUI tool like xxdiff.
I figured that X can be running without having a local display, so it's not good enough to simply
check if there's a process running with X as it's name... that's why I check the DISPLAY variable.
Maybe there's a better method?
2. cfg-update creates an alias for the emerge command in /root/.bashrc which calls a wrapperscript
that first runs "cfg-update --index" and then continues with the actual emerge command.
This isn't a nice way to ensure that the index stays up to date...
I have considered cronjobs, looked for a "pre-emerge hook" in portage but have not found
a good alternative for running "cfg-update --index" right before an emerge command.
Maybe you have some ideas? _________________ When all else fails, read the manual...
Registered Linux User #340626 |
|
Back to top |
|
|
Rick Tux's lil' helper
Joined: 18 Dec 2002 Posts: 141
|
Posted: Tue Aug 29, 2006 12:41 am Post subject: |
|
|
xentric wrote: | 1. cfg-update checks if a GUI is available before it spawns the GUI tool:
Currently the script checks if the DISPLAY variable is set, by calling "echo $DISPLAY"
If it is not empty, it will assume that X is running and it's possible to spawn a GUI tool like xxdiff.
I figured that X can be running without having a local display, so it's not good enough to simply
check if there's a process running with X as it's name... that's why I check the DISPLAY variable.
Maybe there's a better method? | Checking for $DISPLAY is _the_ method to use. But there are other methods too, for example you could look at the $TERM variable, or the $TTY _________________ Have you ever noticed how stable windows is?
Neither have I |
|
Back to top |
|
|
ppurka Advocate
Joined: 26 Dec 2004 Posts: 3256
|
Posted: Tue Aug 29, 2006 1:43 am Post subject: |
|
|
devsk wrote: | I just timed the operations. grep seems faster than awk in finding regukar expressions in the text:
...
| It is strange, but here are the results on my system: Code: | ~> time grep ">>> emerge" /var/log/emerge.log > /dev/null
grep ">>> emerge" /var/log/emerge.log > /dev/null 5.45s user 0.03s system 99% cpu 5.513 total
~> time awk '/>>> emerge/' /var/log/emerge.log > /dev/null
awk '/>>> emerge/' /var/log/emerge.log > /dev/null 0.27s user 0.01s system 99% cpu 0.283 total
~> time grep ">>> emerge" /var/log/emerge.log | awk 'END{print $1$7}'
1156651120:net-nds/ypbind-1.19.1
grep ">>> emerge" /var/log/emerge.log 5.41s user 0.03s system 98% cpu 5.525 total
awk 'END{print $1$7}' 0.00s user 0.00s system 0% cpu 5.476 total
|
Size of /var/log/emerge.log: Code: | ~> ls -l /var/log/emerge.log
-rw-rw---- 1 portage portage 2645019 2006-08-26 23:58 /var/log/emerge.log
~> hdparm -tT /dev/sda <root
/dev/sda:
Timing cached reads: 2120 MB in 2.00 seconds = 1059.76 MB/sec
Timing buffered disk reads: 130 MB in 3.04 seconds = 42.75 MB/sec
|
Here is something else which I found interesting: Code: | ~> time tac /var/log/emerge.log| sed -n '/>>> emerge/{p;q;}' <root
1156651120: >>> emerge (3 of 3) net-nds/ypbind-1.19.1 to /
tac /var/log/emerge.log 0.00s user 0.00s system 34% cpu 0.003 total
sed -n '/>>> emerge/{p;q;}' 0.00s user 0.00s system 77% cpu 0.001 total
|
|
|
Back to top |
|
|
xentric Guru
Joined: 16 Mar 2003 Posts: 410 Location: Netherlands
|
Posted: Tue Aug 29, 2006 2:22 am Post subject: |
|
|
Rick wrote: | Checking for $DISPLAY is _the_ method to use. But there are other methods too, for example you could look at the $TERM variable, or the $TTY | Thanks! Keeping in mind that there are lot's of different terminal-types so I guess sticking with the DISPLAY variable is pretty safe and much easier.
ppurka: Here are my results: Code: | root@gentoo 852MHz 68C xentric # ls -l /var/log/emerge.log
-rw-rw---- 1 portage portage 2011494 Aug 29 02:39 /var/log/emerge.log
root@gentoo 852MHz 69C portage # hdparm -Tt /dev/hda
/dev/hda:
Timing cached reads: 444 MB in 2.01 seconds = 220.41 MB/sec
Timing buffered disk reads: 52 MB in 3.00 seconds = 17.32 MB/sec
root@gentoo 852MHz 65C xentric # time awk '/>>> emerge/' /var/log/emerge.log > /dev/null
real 0m0.111s
user 0m0.058s
sys 0m0.017s
root@gentoo 852MHz 65C xentric # time grep ">>> emerge" /var/log/emerge.log > /dev/null
real 0m0.022s
user 0m0.012s
sys 0m0.003s | Note: this is on an old Pentium3 850Mhz laptop with 256Mb RAM and a slow HD...
(I've tested this multiple times to avoid caching differences)
Try this: Code: | time LC_ALL="C" grep ">>> emerge" /var/log/emerge.log | awk 'END{print $1$7}' | does that speed up grepping? It seems that locales like UTF-8 slow down grep a lot... (see grep manpage about LC_ALL)
The cfg-update script actually sets the locale to C in every subroutine that uses grep... _________________ When all else fails, read the manual...
Registered Linux User #340626 |
|
Back to top |
|
|
ppurka Advocate
Joined: 26 Dec 2004 Posts: 3256
|
Posted: Tue Aug 29, 2006 3:00 am Post subject: |
|
|
You were right. I have utf-8 set in my LC_*. Here are the new statistics with LC_ALL="C",- grep sure takes way lower time than awk, but the sed beats them both, with far less cpu (btw, it is a 2.2GHz Athlon XP 3200+). I was just thinking you must be having a super fast system there with a 15000 rpm hard disk to get such low numbers.
Code: | ~> time LC_ALL="C" grep ">>> emerge" /var/log/emerge.log > /dev/null
LC_ALL="C" grep ">>> emerge" /var/log/emerge.log > /dev/null 0.00s user 0.00s system 95% cpu 0.007 total
~> time LC_ALL="C" awk '/>>> emerge/' /var/log/emerge.log > /dev/null
LC_ALL="C" awk '/>>> emerge/' /var/log/emerge.log > /dev/null 0.02s user 0.00s system 97% cpu 0.029 total
~> time LC_ALL="C" tac /var/log/emerge.log| sed -n '/>>> emerge/{p;q;}'
1156651120: >>> emerge (3 of 3) net-nds/ypbind-1.19.1 to /
LC_ALL="C" tac /var/log/emerge.log 0.00s user 0.00s system 39% cpu 0.003 total
sed -n '/>>> emerge/{p;q;}' 0.00s user 0.00s system 78% cpu 0.001 total
|
|
|
Back to top |
|
|
xentric Guru
Joined: 16 Mar 2003 Posts: 410 Location: Netherlands
|
Posted: Tue Aug 29, 2006 3:38 am Post subject: |
|
|
ppurka wrote: | grep sure takes way lower time than awk, but the sed beats them both, with far less cpu | You're right! It takes less time on my system too:
Code: | root@gentoo 852MHz 68C portage # time tac /var/log/emerge.log| sed -n '/>>> emerge/{p;q;}'
1156811850: >>> emerge (1 of 1) app-misc/beep-1.2.2 to /
real 0m0.009s
user 0m0.003s
sys 0m0.005s | So I've put this in the new version of the script
Next thing is to speed up creating the index... LOL
I currently use: Code: | root@gentoo 852MHz 75C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | awk '{ print $2" "$3 }' >> /tmp/checksum-index
real 0m9.742s
user 0m1.989s
sys 0m0.650s |
Any ideas?
(BTW, I'm gonna take a nap... it's 05:42am... oops!) _________________ When all else fails, read the manual...
Registered Linux User #340626 |
|
Back to top |
|
|
devsk Advocate
Joined: 24 Oct 2003 Posts: 2995 Location: Bay Area, CA
|
Posted: Tue Aug 29, 2006 6:17 am Post subject: |
|
|
cut is faster for this particular case because it is pretty plain set of args:
Code: |
$ time grep "^obj " /var/db/pkg/*/*/CONTENTS | cut -d" " -f2-3 > /tmp/tmper.idx
real 0m0.287s
user 0m0.292s
sys 0m0.148s
$ time grep "^obj " /var/db/pkg/*/*/CONTENTS | awk '{ print $2" "$3 }' > /tmp/tmper.idx2
real 0m0.576s
user 0m0.548s
sys 0m0.140s
$ l /tmp/tmper.idx*
-rw-r--r-- 1 devsk users 18622047 Aug 28 23:06 /tmp/tmper.idx
-rw-r--r-- 1 devsk users 18834722 Aug 28 23:04 /tmp/tmper.idx2
|
It also creates a smaller index...
Code: | $ bc
bc 1.06.94
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
18834722-18622047
212675 |
Code: | $ wc -l /tmp/tmper.idx*
212675 /tmp/tmper.idx
212675 /tmp/tmper.idx2
425350 total
| one blank character (which you inserted for an unknown reason) less per line. Otherwise perfect match.
The gains might be larger for slower systems. post your numbers please. I wanna see if its faster than 10 seconds you get currently. |
|
Back to top |
|
|
swimmer Veteran
Joined: 15 Jul 2002 Posts: 1330 Location: Netherlands
|
Posted: Tue Aug 29, 2006 11:03 am Post subject: |
|
|
The problem is more the 'first shot' - when you index after a longer time it
takes quite a time to get through /var/db/pkg ... but don't ask me why - I only
notice that and do not understand
First run: Quote: |
#> time grep "^obj " /var/db/pkg/*/*/CONTENTS | cut -d" " -f2-3 > /tmp/tmper.idx
real 0m17.422s
user 0m0.402s
sys 0m0.310s
#> time grep "^obj " /var/db/pkg/*/*/CONTENTS | awk '{ print $2" "$3}' > /tmp/tmper.idx2
real 0m0.773s
user 0m0.574s
sys 0m0.162s
#> wc -l /tmp/tmper.idx*
240238 /tmp/tmper.idx
240238 /tmp/tmper.idx2
480476 total
#> ls -l /tmp/tmper.idx*
-rw-r--r-- 1 swimmer users 20704744 2006-08-29 12:44 /tmp/tmper.idx
-rw-r--r-- 1 swimmer users 20944982 2006-08-29 12:45 /tmp/tmper.idx2
|
Second run: Quote: |
#> time grep "^obj " /var/db/pkg/*/*/CONTENTS | cut -d" " -f2-3 > /tmp/tmper.idx
real 0m0.606s
user 0m0.378s
sys 0m0.197s
#> time grep "^obj " /var/db/pkg/*/*/CONTENTS | awk '{ print $2" "$3 }' > /tmp/tmper.idx2
real 0m0.782s
user 0m0.573s
sys 0m0.186s
#> wc -l /tmp/tmper.idx*
240238 /tmp/tmper.idx
240238 /tmp/tmper.idx2
480476 total
#> ls -l /tmp/tmper.idx*
-rw-r--r-- 1 swimmer users 20704744 2006-08-29 12:46 /tmp/tmper.idx
-rw-r--r-- 1 swimmer users 20944982 2006-08-29 12:46 /tmp/tmper.idx2
|
The differences between awk and cut aren't as big ...
Greetz
swimmer |
|
Back to top |
|
|
xentric Guru
Joined: 16 Mar 2003 Posts: 410 Location: Netherlands
|
Posted: Tue Aug 29, 2006 11:24 am Post subject: |
|
|
devsk wrote: | cut is faster for this particular case because it is pretty plain set of args:
...
The gains might be larger for slower systems. post your numbers please. I wanna see if its faster than 10 seconds you get currently. |
cut is a little bit faster on my old system too:
Code: | *** YESTERDAY ***
root@gentoo 852MHz 68C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | awk '{ print $2" "$3 }' >> /tmp/checksum-index
real 0m10.764s
user 0m1.896s
sys 0m0.606s
root@gentoo 852MHz 72C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | awk '{ print $2" "$3 }' >> /tmp/checksum-index
real 0m10.110s
user 0m1.973s
sys 0m0.625s
root@gentoo 852MHz 75C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | awk '{ print $2" "$3 }' >> /tmp/checksum-index
real 0m9.742s
user 0m1.989s
sys 0m0.650s
*** TODAY ***
root@gentoo 852MHz 70C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | cut -d" " -f2-3 > /tmp/checksum-index2
real 0m10.017s
user 0m1.431s
sys 0m0.639s
root@gentoo 852MHz 67C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | cut -d" " -f2-3 > /tmp/checksum-index2
real 0m4.482s
user 0m1.458s
sys 0m0.441s
root@gentoo 852MHz 62C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | cut -d" " -f2-3 > /tmp/checksum-index2
real 0m2.382s
user 0m1.486s
sys 0m0.369s
root@gentoo 852MHz 63C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | cut -d" " -f2-3 > /tmp/checksum-index2
real 0m2.428s
user 0m1.457s
sys 0m0.408s
root@gentoo 852MHz 64C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | awk '{ print $2" "$3 }' >> /tmp/checksum-index
real 0m3.299s
user 0m1.862s
sys 0m0.431s
root@gentoo 852MHz 70C portage # time grep "^obj " /var/db/pkg/*/*/CONTENTS | awk '{ print $2" "$3 }' >> /tmp/checksum-index
real 0m3.037s
user 0m1.884s
sys 0m0.423s | On the first run both methods take about 12 seconds to complete. Look at the effect that caching has on the timings... looks like there are a lot of factors that have impact on this kind of benchmarking. Today the awk method is a lot faster after chaching.
The awk method goes down to 5 seconds after a couple of runs and the cut method does it in 4 seconds.
The cut method is a little bit better, so I'll stick to that. Thanks _________________ When all else fails, read the manual...
Registered Linux User #340626 |
|
Back to top |
|
|
xentric Guru
Joined: 16 Mar 2003 Posts: 410 Location: Netherlands
|
Posted: Tue Aug 29, 2006 12:06 pm Post subject: |
|
|
swimmer wrote: | The problem is more the 'first shot' - when you index after a longer time it
takes quite a time to get through /var/db/pkg ... but don't ask me why - I only
notice that and do not understand |
I think that's due to caching. The stuff in /var/db/pkg remains in RAM after the first run, so on sebsequent runs your system can use this data which is already in RAM. That's way faster than your HD.
Just for fun, look on the forums for "shake your files". That method is used to force files into RAM so programs start up a bit faster.
Quote: | First run: Code: |
#> time grep "^obj " /var/db/pkg/*/*/CONTENTS | cut -d" " -f2-3 > /tmp/tmper.idx
real 0m17.422s
user 0m0.402s
sys 0m0.310s |
| Looking at your subsequent runs it seems like you have a pretty fast system but it's horribly slow when it uses the HD, maybe you haven't got DMA enabled on your HD? That usually makes it a lot slower. Can you tell me what your system specs are? It makes me curious to see what you get when you execute the hdparm throughput-test command:
Code: | [root@gentoo 852MHz 56C portage # ls -l /var/log/emerge.log
-rw-rw---- 1 portage portage 2012089 Aug 29 06:11 /var/log/emerge.log
root@gentoo 852MHz 56C portage # hdparm -I /dev/hda
/dev/hda:
ATA device, with non-removable media
Model Number: TOSHIBA MK8026GAX
Serial Number: 65TL1869T
Firmware Revision: PA001G
Standards:
Supported: 6 5 4 3
Likely used: 6
Configuration:
Logical max current
cylinders 16383 17475
heads 16 15
sectors/track 63 63
--
CHS current addressable sectors: 16513875
LBA user addressable sectors: 156301488
device size with M = 1024*1024: 76319 MBytes
device size with M = 1000*1000: 80026 MBytes (80 GB)
Capabilities:
LBA, IORDY(can be disabled)
bytes avail on r/w long: 48 Queue depth: 1
Standby timer values: spec'd by Standard, no device specific minimum
R/W multiple sector transfer: Max = 16 Current = 16
Advanced power management level: unknown setting (0x0080)
DMA: sdma0 sdma1 sdma2 mdma0 mdma1 mdma2 udma0 udma1 *udma2 udma3 udma4 udma5
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* NOP cmd
* READ BUFFER cmd
* WRITE BUFFER cmd
* Host Protected Area feature set
* Look-ahead
* Write cache
* Power Management feature set
Security Mode feature set
* SMART feature set
* Mandatory FLUSH CACHE command
* Device Configuration Overlay feature set
SET MAX security extension
* Advanced Power Management feature set
* SMART self-test
* SMART error logging
Security:
Master password revision code = 65534
supported
not enabled
not locked
not frozen
not expired: security count
not supported: enhanced erase
58min for SECURITY ERASE UNIT.
HW reset results:
CBLID- above Vih
Device num = 0 determined by the jumper
Checksum: correct
root@gentoo 852MHz 56C portage # hdparm -Tt /dev/hda
/dev/hda:
Timing cached reads: 588 MB in 2.01 seconds = 292.51 MB/sec
Timing buffered disk reads: 62 MB in 3.03 seconds = 20.49 MB/sec
root@gentoo 852MHz 59C portage # hdparm -Tt /dev/hda
/dev/hda:
Timing cached reads: 588 MB in 2.01 seconds = 292.77 MB/sec
Timing buffered disk reads: 62 MB in 3.09 seconds = 20.09 MB/sec
root@gentoo 852MHz 57C portage # hdparm -Tt /dev/hda
/dev/hda:
Timing cached reads: 592 MB in 2.01 seconds = 294.30 MB/sec
Timing buffered disk reads: 60 MB in 3.00 seconds = 19.98 MB/sec | This is a 2,5" laptop HD: 80Gb with 16Mb cache and 5400 RPM. So if you have a 7200 RPM 3,5" HD you should get better results.
This is my setting for /dev/hda in /etc/conf.d/hdparm: Code: | disc0_args="-c1 -d1 -m16 -u1 -X66" | Caution: do not use the -X66 unless you are sure that your HD can handle it. Maybe you have a newer driver that supports even better like -X69 (=udma5). Just check "man hdparm" for the -X option. It needs to be (udma_level+64), so mine uses udma2. _________________ When all else fails, read the manual...
Registered Linux User #340626 |
|
Back to top |
|
|
swimmer Veteran
Joined: 15 Jul 2002 Posts: 1330 Location: Netherlands
|
Posted: Tue Aug 29, 2006 12:50 pm Post subject: |
|
|
xentric wrote: | [...]
I think that's due to caching. The stuff in /var/db/pkg remains in RAM after the first run, so on sebsequent runs your system can use this data which is already in RAM. That's way faster than your HD. |
... I see
xentric wrote: | Looking at your subsequent runs it seems like you have a pretty fast system but it's horribly slow when it uses the HD, maybe you haven't got DMA enabled on your HD? That usually makes it a lot slower. Can you tell me what your system specs are? It makes me curious to see what you get when you execute the hdparm throughput-test command [...] |
Hmmm - I have a SATA disk and I thought it's not necessary to optimize it with
hdparm since this happens automatically. Correct me if I'm wrong ...
Code: | swimmer ~ # hdparm -I /dev/sda
/dev/sda:
ATA device, with non-removable media
Model Number: Maxtor 6Y250M0
Serial Number: Y69GHVVE
Firmware Revision: YAR511W0
Standards:
Supported: 7 6 5 4
Likely used: 7
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
--
CHS current addressable sectors: 16514064
LBA user addressable sectors: 268435455
LBA48 user addressable sectors: 490234752
device size with M = 1024*1024: 239372 MBytes
device size with M = 1000*1000: 251000 MBytes (251 GB)
Capabilities:
LBA, IORDY(can be disabled)
Queue depth: 1
Standby timer values: spec'd by Standard, no device specific minimum
R/W multiple sector transfer: Max = 16 Current = 1
Advanced power management level: unknown setting (0x0000)
Recommended acoustic management value: 192, current value: 254
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* NOP cmd
* READ BUFFER cmd
* WRITE BUFFER cmd
* Host Protected Area feature set
* Look-ahead
* Write cache
* Power Management feature set
Security Mode feature set
* SMART feature set
* FLUSH CACHE EXT command
* Mandatory FLUSH CACHE command
* Device Configuration Overlay feature set
* 48-bit Address feature set
* Automatic Acoustic Management feature set
SET MAX security extension
Advanced Power Management feature set
* DOWNLOAD MICROCODE cmd
* SMART self-test
* SMART error logging
Security:
Master password revision code = 65534
supported
not enabled
not locked
not frozen
not expired: security count
not supported: enhanced erase
Checksum: correct
swimmer ~ # hdparm -tT /dev/sda
/dev/sda:
Timing cached reads: 3528 MB in 2.00 seconds = 1763.81 MB/sec
Timing buffered disk reads: 168 MB in 3.02 seconds = 55.72 MB/sec
swimmer ~ # hdparm -tT /dev/sda
/dev/sda:
Timing cached reads: 3520 MB in 2.00 seconds = 1759.04 MB/sec
Timing buffered disk reads: 168 MB in 3.03 seconds = 55.48 MB/sec
swimmer ~ # hdparm -tT /dev/sda
/dev/sda:
Timing cached reads: 3496 MB in 2.00 seconds = 1747.62 MB/sec
Timing buffered disk reads: 170 MB in 3.02 seconds = 56.22 MB/sec
swimmer ~ # ll /var/log/emerge.log
-rw-rw---- 1 portage portage 2226909 Aug 29 07:44 /var/log/emerge.log
swimmer ~ #
|
Looks ok to me
Greetz
swimmer |
|
Back to top |
|
|
xentric Guru
Joined: 16 Mar 2003 Posts: 410 Location: Netherlands
|
Posted: Tue Aug 29, 2006 1:41 pm Post subject: |
|
|
swimmer wrote: | Hmmm - I have a SATA disk and I thought it's not necessary to optimize it with
hdparm since this happens automatically. Correct me if I'm wrong ...
...
Looks ok to me | Yeah, that sure looks good!
Pretty weird that it takes your system about 18 seconds on the first run.
I have /var on a reiser3 filesystem mounted with noatime,nodiratime,notail.
As /var/db/pkg contains lot's of small files, I guess this is where reiserfs with notail makes a difference. _________________ When all else fails, read the manual...
Registered Linux User #340626 |
|
Back to top |
|
|
devsk Advocate
Joined: 24 Oct 2003 Posts: 2995 Location: Bay Area, CA
|
Posted: Tue Aug 29, 2006 3:56 pm Post subject: |
|
|
xentric wrote: |
1. cfg-update checks if a GUI is available before it spawns the GUI tool:
Currently the script checks if the DISPLAY variable is set, by calling "echo $DISPLAY"
If it is not empty, it will assume that X is running and it's possible to spawn a GUI tool like xxdiff.
I figured that X can be running without having a local display, so it's not good enough to simply
check if there's a process running with X as it's name... that's why I check the DISPLAY variable.
Maybe there's a better method? | The right way of checking is thru both DISPLAY and xhost, like:
Code: | if ( [[ -z ${DISPLAY} ]] || ! (xhost &>/dev/null) ) ; then echo DISPLAY_NOTTHERE; else echo DISPLAY_PRESENT;fi |
So, if DISPLAY is set but user (root can't connect because Joe user didn't do xhost +) can't connect to it, DISPLAY_NOTTHERE will be echoed in this case. xhost fails in few seconds if it can't connect to $DISPLAY. Of course, the check comes out immd. with DISPLAY_NOTTHERE if DISPLAY variable is not set. |
|
Back to top |
|
|
xentric Guru
Joined: 16 Mar 2003 Posts: 410 Location: Netherlands
|
Posted: Thu Aug 31, 2006 11:27 pm Post subject: |
|
|
devsk wrote: | So, if DISPLAY is set but user (root can't connect because Joe user didn't do xhost +) can't connect to it, DISPLAY_NOTTHERE will be echoed in this case. xhost fails in few seconds if it can't connect to $DISPLAY. Of course, the check comes out immd. with DISPLAY_NOTTHERE if DISPLAY variable is not set. |
Well, I have tried to test this but all I get when I run xhost without any arguments is a message that access control is enabled. It seems like I can't disable access control so I can't test your line of bash-code.
If I run "xhost -localhost" or "xhost -" (with user that started the X server) I still get the same message when I run "xhost" as root. I can't find a file containing an accesslist or anything. The manpage is refering to /etc/X*.hosts but I have no such files.
Next week I'll (finally) get a brand spanking new Core2Duo system on which I will start from scratch by installing the new 2006.1. That's also a good opportunity to test installing cfg-update on a default setup. I'll check if xhost exits with a STDERR message. _________________ When all else fails, read the manual...
Registered Linux User #340626 |
|
Back to top |
|
|
devsk Advocate
Joined: 24 Oct 2003 Posts: 2995 Location: Bay Area, CA
|
Posted: Fri Sep 01, 2006 2:01 am Post subject: |
|
|
xentric wrote: | devsk wrote: | So, if DISPLAY is set but user (root can't connect because Joe user didn't do xhost +) can't connect to it, DISPLAY_NOTTHERE will be echoed in this case. xhost fails in few seconds if it can't connect to $DISPLAY. Of course, the check comes out immd. with DISPLAY_NOTTHERE if DISPLAY variable is not set. |
Well, I have tried to test this but all I get when I run xhost without any arguments is a message that access control is enabled. It seems like I can't disable access control so I can't test your line of bash-code.
If I run "xhost -localhost" or "xhost -" (with user that started the X server) I still get the same message when I run "xhost" as root. I can't find a file containing an accesslist or anything. The manpage is refering to /etc/X*.hosts but I have no such files.
Next week I'll (finally) get a brand spanking new Core2Duo system on which I will start from scratch by installing the new 2006.1. That's also a good opportunity to test installing cfg-update on a default setup. I'll check if xhost exits with a STDERR message. | the test that I posted checks for exit code of xhost. Try Code: | DISPLAY=random_host_does_not_exist:0 xhost &>/dev/null
echo $?
| exit code indicates if xhost succeeded or failed in getting a status of access control of the X. without any arguments it gives a status if the access control is enabled or disabled. This is easily verified by 'su' in as root in joe's running X xterm. Then type 'xhost'. It will give the status. In another xterm as user joe, do 'xhost -', followed by a 'xhost' in 'su' xterm, followed by 'xhost +' in joe's xterm, followed by 'xhost' in 'su' xterm. You will see status changes in 'su' window.
Trust me that test works if copied verbatim. Just replace the 'echo' commands with whatever you wanna do if X is available and otherwise. |
|
Back to top |
|
|
xentric Guru
Joined: 16 Mar 2003 Posts: 410 Location: Netherlands
|
Posted: Fri Sep 01, 2006 11:02 am Post subject: |
|
|
devsk wrote: | Trust me that test works if copied verbatim. Just replace the 'echo' commands with whatever you wanna do if X is available and otherwise. |
Code: | * did "xhost +" in other terminal
root@gentoo 852MHz 48C / # xhost
access control disabled, clients can connect from any host
* did "xhost -" in other terminal
root@gentoo 852MHz 48C / # xhost
access control enabled, only authorized clients can connect
* did "xhost +localhost" in other terminal
root@gentoo 852MHz 48C / # xhost
access control enabled, only authorized clients can connect
INET:gentoo.homelinux
* test with unknown host
root@gentoo 852MHz 48C / # DISPLAY=foobar:0 xhost &>/dev/null; echo $?
1
* test with empty DISPLAY variable
root@gentoo 852MHz 48C / # DISPLAY= xhost &>/dev/null; echo $?
1
* test with correct DISPLAY variable
root@gentoo 852MHz 48C / # DISPLAY=:0 xhost &>/dev/null; echo $?
0 | Verified and works, I will use this in the new version of the script.
Thanks a lot devsk! _________________ When all else fails, read the manual...
Registered Linux User #340626 |
|
Back to top |
|
|
devsk Advocate
Joined: 24 Oct 2003 Posts: 2995 Location: Bay Area, CA
|
Posted: Fri Sep 01, 2006 8:41 pm Post subject: |
|
|
xentric wrote: | Verified and works, I will use this in the new version of the script. |
please don't forget to test for the $DISPLAY first and xhost after that, just to optimise the case when DISPLAY isn't even set. Otherwise, cfg-update -u will seem like hung for few seconds because xhost will take few seconds to come out.
xentric wrote: |
Thanks a lot devsk! | You are very welcome. |
|
Back to top |
|
|
xentric Guru
Joined: 16 Mar 2003 Posts: 410 Location: Netherlands
|
Posted: Sun Sep 03, 2006 1:09 am Post subject: |
|
|
devsk wrote: | Otherwise, cfg-update -u will seem like hung for few seconds because xhost will take few seconds to come out. | I don't get any delay when running xhost if I unset the DISPLAY variable, nor when it is set but doesn't contain a value or when it contains an invalid value... it responds immediately with errorcode 0 or 1.
Does xhost "hang" while waiting for a timeout on your system?
I can put the following check in the update_files subroutine: Code: | if ((using GUI mergetool) and (xhost returns errorcode 1)) {
print: GUI is unavailable...
print: if X is installed, run cfg-update in X-terminal otherwise use "-t sdiff" or "-t vimdiff"
print: if this fails while using an X-terminal, check xhost accesscontrol and DISPLAY variable
print: if X is not installed, set MERGETOOL in /etc/cfg-update.conf to "sdiff" or "vimdiff"
print: do you want to [q]uit or continue with [s]diff or [v]imdiff ?
read keypress
} | This is my preferred way to go... even if xhost "hangs" when trying to connect to the X-server.
PS. I'll treat gvimdiff as a CLI tool because if you run it while X is not available it simply falls back to vimdiff automatically. That's actually pretty cool _________________ When all else fails, read the manual...
Registered Linux User #340626 |
|
Back to top |
|
|
XenoTerraCide Veteran
Joined: 18 Jan 2004 Posts: 1418 Location: MI, USA
|
Posted: Mon Sep 11, 2006 1:53 pm Post subject: |
|
|
hmm... cfg-update used to work for me but lately it's not finding the graphical updater has something changed that I would need to update? I'm using kdiff3 (which is installed) and have it set Code: | MERGETOOL = /usr/bin/kdiff3
| anything wrong with this? _________________ I don't hang out here anymore, try asking on http://unix.stackexchange.com/ if you want my help. |
|
Back to top |
|
|
xentric Guru
Joined: 16 Mar 2003 Posts: 410 Location: Netherlands
|
Posted: Mon Sep 11, 2006 5:56 pm Post subject: |
|
|
XenoTerraCide wrote: | hmm... cfg-update used to work for me but lately it's not finding the graphical updater has something changed that I would need to update? I'm using kdiff3 (which is installed) and have it set Code: | MERGETOOL = /usr/bin/kdiff3
| anything wrong with this? |
First of all, try "sux" (instead of "su" or "sudo") to become root in your Xterminal, and check if cfg-update can launch the GUI tool. If that works, it probably is simple to fix with some basic troubleshooting:
Try running "cfg-update -vu". The -v option unhides STDERR messages that could give some clues about not being able to start the GUI tool...
Try "xhost +localhost" (as the user who started the X server) and then log in with root and try cfg-update.
Or check if the DISPLAY variable is properly set with "env | grep DISPLAY", it should return something like DISPLAY=:0
Running "source /etc/profile" (when logged in as root as root) before you start cfg-update could also work.
Please post some feedback if you find out what changed, maybe it helps other users with similar problems. _________________ When all else fails, read the manual...
Registered Linux User #340626 |
|
Back to top |
|
|
|
|
Gentoo Forums Forum Index
Documentation, Tips & Tricks |
All times are GMT Goto page Previous 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 Next
|
Page 7 of 10 |
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|