Bad performance. System cpu% is too high.

Message

Atle · Post by **Atle** » Sat Dec 30, 2006 9:51 pm

My AMD64 is very slow when emerging packages. After 20 hours or so of working, emerge has finished 60 out of 273 packages.

The problem is that for large amounts of time the CPU is busy doing seomthing I don't know what is. This is shown in top by the cpu often spends around 80% time in system/kernel. This does not happen during actual compiling, but on all the tasks between.

I've checked my disks with hdparm, and DMA is turned on for all of them.

So the question is, is there any way I can check what the CPU spends so much time on?

platojones · Post by **platojones** » Sun Dec 31, 2006 3:46 am

How is your swap/memory usage doing? One possibility is that if the machine is low on VM, it will start thrashing and all that paging is very CPU intensive....

Atle · Post by **Atle** » Sun Dec 31, 2006 8:58 am

Thanks for the suggestion, but no, the system is not swapping at all.

Code: Select all

# free
             total       used       free     shared    buffers     cached
Mem:       1542164     899112     643052          0      53184     382316
-/+ buffers/cache:     463612    1078552
Swap:       996020        320     995700
boff ~ #

I've also checked the logs. Couldn't find anything there.

If I'm to guess, I would say it was the IO system that does something unnecessary.

Atle · Post by **Atle** » Sun Dec 31, 2006 9:21 am

Interrupts:

Code: Select all

# cat /proc/interrupts
           CPU0
  0:   42781373    IO-APIC-edge  timer
  8:          0    IO-APIC-edge  rtc
  9:          3   IO-APIC-level  acpi
 14:     684567    IO-APIC-edge  ide0
 15:     953646    IO-APIC-edge  ide1
 16:    6671494   IO-APIC-level  eth0
 18:          2   IO-APIC-level  ohci1394
 19:     329704   IO-APIC-level  ehci_hcd:usb1, uhci_hcd:usb2, uhci_hcd:usb3, uhci_hcd:usb4, uhci_hcd:usb5
 20:   13017531   IO-APIC-level  nvidia
 21:    3746590   IO-APIC-level  VIA8237
NMI:     124906
LOC:   42783412
ERR:          0
MIS:          0

Comparing to another, but very different, system, the only thing that seems high here is the number of interrupts for nvidia.

Atle · Post by **Atle** » Sun Dec 31, 2006 10:20 am

Here is some typical iostat data for ten secs when 'top' shows 'sh' is the task using the most cpu. (the portage directories are on hdd)

Code: Select all

avg-cpu:  %user   %nice    %sys %iowait   %idle
           0.90   25.00   74.10    0.00    0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
hda          0.00   0.40  0.00  1.00    0.00   11.20     0.00     5.60    11.20     0.00    0.00   0.00   0.00
hdd          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

As you can see, %sys is 74.10, and the rest of the system appears to do nothing while the CPU seems to be wasting time in the kernel.

When the compiler, 'cc1plus', runs, the CPU is more fully utilized.

Code: Select all

avg-cpu:  %user   %nice    %sys %iowait   %idle
          82.44    4.29   13.27    0.00    0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
hda          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
hdd          0.00   0.00  0.00  0.20    0.00    1.60     0.00     0.80     8.00     0.00    0.00   0.00   0.00

As far as I can understand, it doesn't seem to be IO-related.

DaggyStyle · Post by **DaggyStyle** » Sun Dec 31, 2006 10:40 am

what were those 60 finished packs?

Atle · Post by **Atle** » Sun Dec 31, 2006 10:57 am

After being without internet at home for a while, I'm updatiing the system and replacing kde with kde-meta. So I assume they were mostly different parts of kde and packages kde depends uppon.

DaggyStyle · Post by **DaggyStyle** » Sun Dec 31, 2006 8:31 pm

Atle wrote:After being without internet at home for a while, I'm updatiing the system and replacing kde with kde-meta. So I assume they were mostly different parts of kde and packages kde depends uppon.

this does takes time.........

david.stevenson · Post by **david.stevenson** » Mon Jan 01, 2007 10:16 pm

For comparison my ints

Code: Select all

Gen64 david # cat /proc/interrupts
           CPU0       
  0:   37499145    IO-APIC-edge  timer
  4:          7    IO-APIC-edge  serial
  7:          0    IO-APIC-edge  parport0
  8:          0    IO-APIC-edge  rtc
  9:          0   IO-APIC-level  acpi
 14:     191280    IO-APIC-edge  ide0
 15:     449962    IO-APIC-edge  ide1
169:      29820   IO-APIC-level  VIA8237, eth0
177:          0   IO-APIC-level  libata
185:          2   IO-APIC-level  ohci1394
193:     431734   IO-APIC-level  ehci_hcd:usb1, uhci_hcd:usb2, uhci_hcd:usb3, uhci_hcd:usb4, uhci_hcd:usb5
201:    2947778   IO-APIC-level  nvidia
NMI:       1387 
LOC:   37518969 
ERR:          0
MIS:          0
Gen64 david # top

top - 22:08:18 up 10:30,  3 users,  load average: 0.16, 0.11, 0.12
Tasks:  87 total,   1 running,  86 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.3% us,  0.0% sy,  0.0% ni, 98.3% id,  0.0% wa,  0.0% hi,  0.3% si
Mem:   1025808k total,  1010656k used,    15152k free,   334308k buffers
Swap:   987956k total,        0k used,   987956k free,    98852k cached

This is a working system, I do not see your problem. But I do get this in dmesg.

Code: Select all

Adding 987956k swap on /dev/hda6.  Priority:-1 extents:1 across:987956k
warning: many lost ticks.
Your time source seems to be instable or some driver is hogging interupts
rip run_timer_softirq+0x2a/0x1c7
ACPI: PCI Interrupt 0000:00:11.5[C] -> GSI 22 (level, low) -> IRQ 169
PCI: VIA IRQ fixup for 0000:00:11.5, from 11 to 9
PCI: Setting latency timer of device 0000:00:11.5 to 64
nvidia: module license 'NVIDIA' taints kernel.

Should I be worried - do you have anything similar?

david.stevenson · Post by **david.stevenson** » Mon Jan 01, 2007 10:27 pm

Whoops - have just seen 7 pages of posts about lost ticks - so ignore me.

joaquin · Post by **joaquin** » Mon Jan 08, 2007 5:45 pm

I have the same problem. When i try to emerge any package, my CPU usage is 90-100%, using runlevel 1, 3 or 5. The problems is only emerging and/or compiling. For example, if i run "emerge -av xxxxx", the "Calculating dependencies" message appear after 30 seconds!!! If i try to compile a package, i go to the kitchen to eat some food and come back in 30 minutes. The memory usage and swap is ok, so no problems with this. With top, iowait is about 30-80%, very very high.

I have AMD Turion64 X2 1.6GHz, 80GB PATA, 512MB, Swap is 1GB. I am using kernel 2.6.19.

Code: Select all

hdparm -tT /dev/hda

/dev/hda:
 Timing cached reads:   1134 MB in  2.00 seconds = 566.66 MB/sec
 Timing buffered disk reads:   50 MB in  3.09 seconds =  16.16 MB/sec

Code: Select all

hdparm /dev/hda

/dev/hda:
 IO_support   =  1 (32-bit)
 unmaskirq    =  1 (on)
 using_dma    =  1 (on)
 keepsettings =  0 (off)
 readonly     =  0 (off)
 geometry     = 16383/255/63, sectors = 156301488, start = 0

Code: Select all

top - 13:43:30 up 28 min,  1 user,  load average: 0.40, 0.35, 0.49
Tasks:  93 total,   1 running,  92 sleeping,   0 stopped,   0 zombie
Cpu(s): 12.6%us, 11.6%sy,  0.8%ni, 38.7%id, 35.9%wa,  0.3%hi,  0.0%si,  0.0%st
Mem:    445028k total,   417716k used,    27312k free,    11840k buffers
Swap:  1044184k total,   104768k used,   939416k free,    94240k cached

Regards