View previous topic :: View next topic |
Author |
Message |
MarcGentoo n00b
Joined: 26 Nov 2020 Posts: 65
|
Posted: Sat May 15, 2021 5:02 am Post subject: Finding the memory hog |
|
|
Hi,
I have a strange case on my server.
The system is using 21 GB ram, but if I add up the top 9 of my most memory hungry programs, I am well below 21 GB!
Code: |
free -lh
total used free shared buff/cache available
Mem: 31Gi 20Gi 6.5Gi 1.5Gi 4.5Gi 9.0Gi
Low: 31Gi 24Gi 6.5Gi
High: 0B 0B 0B
Swap: 31Gi 0B 31Gi
|
Code: |
ps -eo rss --sort=-%mem | head
RSS
1957900
1691220
540296
407216
295736
203748
185288
151528
151332
|
Any ideas what this could be?
Or how to find it out?
After a reboot it is back to normal. |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3334 Location: Rasi, Finland
|
Posted: Sat May 15, 2021 5:47 am Post subject: |
|
|
You were almost there...
Code: | ps -eo rss,cmd --sort=-%mem | head |
_________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
MarcGentoo n00b
Joined: 26 Nov 2020 Posts: 65
|
Posted: Sun May 16, 2021 5:34 am Post subject: |
|
|
Hi,
sorry, this might be a missunderstanding: I know how to get the names of the programs in the list, but the actual issue is that their memory usage just doesn't add up to the memory the system tells me is used.
Sum of Memory Usage reported per process via top / ps << Memory Usage reported by free.
How is this possible? |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3334 Location: Rasi, Finland
|
Posted: Sun May 16, 2021 10:12 am Post subject: |
|
|
So you have the other way around compared to my setup?
Code: | $ ps --no-headers -eo rss | awk '{m+=sprintf("%d",$1)} END {print m/1024}'
6985.8 |
Code: | $ free -m
total used free shared buff/cache available
Mem: 32133 4558 16832 644 10742 26472
Swap: 8191 2 8189 |
In my case combining all values from that ps output yields to greater value than what free shows.
This is probably because ps lists all the threads of a single program.
Since you said it gets back to normal after reboot... Maybe something is filling your tmpfs? _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
tholin Apprentice
Joined: 04 Oct 2008 Posts: 203
|
Posted: Sun May 16, 2021 12:06 pm Post subject: |
|
|
MarcGentoo wrote: | Sum of Memory Usage reported per process via top / ps << Memory Usage reported by free.
How is this possible? |
The memory that is allocated into process address space is just one part of the total system memory. If you assume that used memory always belongs to some process then that is wrong.
Paste the content of /proc/meminfo and /proc/vmstat for a better idea of how memory is used. |
|
Back to top |
|
|
MarcGentoo n00b
Joined: 26 Nov 2020 Posts: 65
|
Posted: Tue May 18, 2021 5:42 am Post subject: |
|
|
Thanks for the response, I'll post the data when it happens next time! |
|
Back to top |
|
|
MarcGentoo n00b
Joined: 26 Nov 2020 Posts: 65
|
Posted: Tue May 18, 2021 5:44 am Post subject: |
|
|
Zucca wrote: | Since you said it gets back to normal after reboot... Maybe something is filling your tmpfs? |
Actually that might be it! I'll check it next time |
|
Back to top |
|
|
MarcGentoo n00b
Joined: 26 Nov 2020 Posts: 65
|
Posted: Thu May 20, 2021 6:35 am Post subject: |
|
|
Hi,
it seems it wasn't the tmpfs after all.
Do you see anything suspicious?
This time my ps yields about 8 GB ram use, but free -lh says I am using 18 GB ...
Where did the 10 GB go?
Thanks a lot for all your Input.
Code: |
total used free shared buff/cache available
Mem: 31Gi 18Gi 617Mi 8.1Gi 11Gi 4.1Gi
Low: 31Gi 30Gi 617Mi
High: 0B 0B 0B
Swap: 31Gi 2.2Gi 29Gi
|
Code: |
ps --no-headers -eo rss | awk '{m+=sprintf("%d",$1)} END {print m/1024}'
7921.99
|
Code: |
cat /proc/meminfo
MemTotal: 32811188 kB
MemFree: 645984 kB
MemAvailable: 4256328 kB
Buffers: 40956 kB
Cached: 12259116 kB
SwapCached: 4896 kB
Active: 6162336 kB
Inactive: 12173796 kB
Active(anon): 3673564 kB
Inactive(anon): 10866004 kB
Active(file): 2488772 kB
Inactive(file): 1307792 kB
Unevictable: 48 kB
Mlocked: 48 kB
SwapTotal: 33554428 kB
SwapFree: 31255292 kB
Dirty: 520 kB
Writeback: 0 kB
AnonPages: 6032272 kB
Mapped: 475772 kB
Shmem: 8503508 kB
KReclaimable: 188808 kB
Slab: 483988 kB
SReclaimable: 188808 kB
SUnreclaim: 295180 kB
KernelStack: 35344 kB
PageTables: 47608 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 49960020 kB
Committed_AS: 24550484 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 243676 kB
VmallocChunk: 0 kB
Percpu: 18688 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 20795024 kB
DirectMap2M: 12703744 kB
DirectMap1G: 1048576 kB
|
Code: |
cat /proc/vmstat
nr_free_pages 150579
nr_zone_inactive_anon 2714189
nr_zone_active_anon 918441
nr_zone_inactive_file 328357
nr_zone_active_file 633467
nr_zone_unevictable 12
nr_zone_write_pending 132
nr_mlock 12
nr_page_table_pages 11989
nr_bounce 0
nr_zspages 198598
nr_free_cma 0
numa_hit 2487517495
numa_miss 0
numa_foreign 0
numa_interleave 31827
numa_local 2487517495
numa_other 0
nr_inactive_anon 2714189
nr_active_anon 918441
nr_inactive_file 328357
nr_active_file 633467
nr_unevictable 12
nr_slab_reclaimable 47350
nr_slab_unreclaimable 73741
nr_isolated_anon 0
nr_isolated_file 0
workingset_nodes 40854
workingset_refault_anon 119692
workingset_refault_file 5322462
workingset_activate_anon 41665
workingset_activate_file 1817215
workingset_restore_anon 1548
workingset_restore_file 1032070
workingset_nodereclaim 22096
nr_anon_pages 1505807
nr_mapped 119523
nr_file_pages 3088924
nr_dirty 142
nr_writeback 0
nr_writeback_temp 0
nr_shmem 2125876
nr_shmem_hugepages 0
nr_shmem_pmdmapped 0
nr_file_hugepages 0
nr_file_pmdmapped 0
nr_anon_transparent_hugepages 0
nr_vmscan_write 837176
nr_vmscan_immediate_reclaim 2094
nr_dirtied 58163294
nr_written 58451972
nr_kernel_misc_reclaimable 0
nr_foll_pin_acquired 0
nr_foll_pin_released 0
nr_kernel_stack 34848
nr_dirty_threshold 211499
nr_dirty_background_threshold 105620
pgpgin 1887921522
pgpgout 232390053
pswpin 119692
pswpout 837176
pgalloc_dma 4
pgalloc_dma32 138327730
pgalloc_normal 2561363647
pgalloc_movable 0
allocstall_dma 0
allocstall_dma32 0
allocstall_normal 1904
allocstall_movable 5095
pgskip_dma 0
pgskip_dma32 0
pgskip_normal 69601430
pgskip_movable 0
pgfree 2717543313
pgactivate 54026282
pgdeactivate 7787812
pglazyfree 1530
pgfault 2837526324
pgmajfault 145590
pglazyfreed 0
pgrefill 9430273
pgreuse 327982667
pgsteal_kswapd 455562361
pgsteal_direct 4596927
pgscan_kswapd 528630270
pgscan_direct 4616570
pgscan_direct_throttle 0
pgscan_anon 5589287
pgscan_file 527657553
pgsteal_anon 838282
pgsteal_file 459321006
zone_reclaim_failed 0
pginodesteal 172068
slabs_scanned 20822429
kswapd_inodesteal 24057074
kswapd_low_wmark_hit_quickly 57268
kswapd_high_wmark_hit_quickly 5617
pageoutrun 77231
pgrotated 2039
drop_pagecache 0
drop_slab 0
oom_kill 0
pgmigrate_success 17617402
pgmigrate_fail 1289909
thp_migration_success 0
thp_migration_fail 0
thp_migration_split 0
compact_migrate_scanned 3179063294
compact_free_scanned 1638263389
compact_isolated 36855415
compact_stall 412
compact_fail 410
compact_success 2
compact_daemon_wake 31328
compact_daemon_migrate_scanned 4505072
compact_daemon_free_scanned 30621770
htlb_buddy_alloc_success 0
htlb_buddy_alloc_fail 0
unevictable_pgs_culled 816972
unevictable_pgs_scanned 0
unevictable_pgs_rescued 806920
unevictable_pgs_mlocked 808366
unevictable_pgs_munlocked 807634
unevictable_pgs_cleared 10
unevictable_pgs_stranded 10
swap_ra 1144
swap_ra_hit 594
nr_unstable 0
|
|
|
Back to top |
|
|
MarcGentoo n00b
Joined: 26 Nov 2020 Posts: 65
|
Posted: Thu May 20, 2021 2:41 pm Post subject: |
|
|
I emptied the caches, but without significant changes
Code: |
sync && echo 3 > /proc/sys/vm/drop_caches
|
|
|
Back to top |
|
|
MarcGentoo n00b
Joined: 26 Nov 2020 Posts: 65
|
Posted: Fri May 21, 2021 11:03 am Post subject: |
|
|
It has changed for the worse...
Any ideas?
Code: |
free -lh
total used free shared buff/cache available
Mem: 31Gi 29Gi 354Mi 191Mi 1.2Gi 1.0Gi
Low: 31Gi 30Gi 354Mi
High: 0B 0B 0B
Swap: 31Gi 28Gi 3.3Gi
|
Code: |
ps --no-headers -eo rss | awk '{m+=sprintf("%d",$1)} END {print m/1024}'
6439.58
|
Code: |
cat /proc/meminfo
MemTotal: 32811188 kB
MemFree: 382812 kB
MemAvailable: 1097688 kB
Buffers: 1640 kB
Cached: 919068 kB
SwapCached: 4848 kB
Active: 3581492 kB
Inactive: 2342832 kB
Active(anon): 3159732 kB
Inactive(anon): 2041304 kB
Active(file): 421760 kB
Inactive(file): 301528 kB
Unevictable: 48 kB
Mlocked: 48 kB
SwapTotal: 33554428 kB
SwapFree: 3457020 kB
Dirty: 936 kB
Writeback: 0 kB
AnonPages: 5000116 kB
Mapped: 474108 kB
Shmem: 197420 kB
KReclaimable: 321560 kB
Slab: 6234692 kB
SReclaimable: 321560 kB
SUnreclaim: 5913132 kB
KernelStack: 34704 kB
PageTables: 50896 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 49960020 kB
Committed_AS: 42172904 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 243180 kB
VmallocChunk: 0 kB
Percpu: 33792 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 22646416 kB
DirectMap2M: 10852352 kB
DirectMap1G: 1048576 kB
|
Code: |
cat /proc/vmstat
nr_free_pages 94671
nr_zone_inactive_anon 510254
nr_zone_active_anon 789966
nr_zone_inactive_file 75581
nr_zone_active_file 106219
nr_zone_unevictable 12
nr_zone_write_pending 272
nr_mlock 12
nr_page_table_pages 12674
nr_bounce 0
nr_zspages 1948643
nr_free_cma 0
numa_hit 3365830325
numa_miss 0
numa_foreign 0
numa_interleave 31827
numa_local 3365830325
numa_other 0
nr_inactive_anon 510254
nr_active_anon 789966
nr_inactive_file 75581
nr_active_file 106219
nr_unevictable 12
nr_slab_reclaimable 80778
nr_slab_unreclaimable 1478329
nr_isolated_anon 0
nr_isolated_file 0
workingset_nodes 49593
workingset_refault_anon 530352
workingset_refault_file 17629331
workingset_activate_anon 126514
workingset_activate_file 7186310
workingset_restore_anon 31477
workingset_restore_file 4776969
workingset_nodereclaim 22214
nr_anon_pages 1250176
nr_mapped 118917
nr_file_pages 232148
nr_dirty 272
nr_writeback 0
nr_writeback_temp 0
nr_shmem 49136
nr_shmem_hugepages 0
nr_shmem_pmdmapped 0
nr_file_hugepages 0
nr_file_pmdmapped 0
nr_anon_transparent_hugepages 0
nr_vmscan_write 8891112
nr_vmscan_immediate_reclaim 26039
nr_dirtied 100965172
nr_written 108999071
nr_kernel_misc_reclaimable 0
nr_foll_pin_acquired 0
nr_foll_pin_released 0
nr_kernel_stack 34656
nr_dirty_threshold 44354
nr_dirty_background_threshold 22150
pgpgin 2557615682
pgpgout 403169881
pswpin 530352
pswpout 8891108
pgalloc_dma 4
pgalloc_dma32 261117936
pgalloc_normal 3384389030
pgalloc_movable 0
allocstall_dma 0
allocstall_dma32 0
allocstall_normal 3598
allocstall_movable 15515
pgskip_dma 0
pgskip_dma32 0
pgskip_normal 97367594
pgskip_movable 0
pgfree 3677238258
pgactivate 79247735
pgdeactivate 23010169
pglazyfree 1530
pgfault 3755873559
pgmajfault 608716
pglazyfreed 0
pgrefill 41470924
pgreuse 410105472
pgsteal_kswapd 619681152
pgsteal_direct 6054297
pgscan_kswapd 710397005
pgscan_direct 6201206
pgscan_direct_throttle 0
pgscan_anon 17416193
pgscan_file 699182018
pgsteal_anon 8894424
pgsteal_file 616841025
zone_reclaim_failed 0
pginodesteal 196280
slabs_scanned 67943223
kswapd_inodesteal 42173740
kswapd_low_wmark_hit_quickly 85294
kswapd_high_wmark_hit_quickly 7229
pageoutrun 115554
pgrotated 32735
drop_pagecache 5
drop_slab 5
oom_kill 0
pgmigrate_success 31524837
pgmigrate_fail 2486998
thp_migration_success 0
thp_migration_fail 0
thp_migration_split 0
compact_migrate_scanned 6612803743
compact_free_scanned 2743272074
compact_isolated 65995607
compact_stall 618
compact_fail 614
compact_success 4
compact_daemon_wake 96236
compact_daemon_migrate_scanned 7089891
compact_daemon_free_scanned 32326709
htlb_buddy_alloc_success 0
htlb_buddy_alloc_fail 0
unevictable_pgs_culled 817254
unevictable_pgs_scanned 0
unevictable_pgs_rescued 807202
unevictable_pgs_mlocked 808650
unevictable_pgs_munlocked 807916
unevictable_pgs_cleared 12
unevictable_pgs_stranded 12
swap_ra 5242
swap_ra_hit 3523
nr_unstable 0
|
|
|
Back to top |
|
|
stonespheres n00b
Joined: 16 Apr 2021 Posts: 16 Location: Australia
|
Posted: Fri May 21, 2021 11:53 am Post subject: |
|
|
MarcGentoo wrote: | It has changed for the worse...
Any ideas?
Code: |
free -lh
total used free shared buff/cache available
Mem: 31Gi 29Gi 354Mi 191Mi 1.2Gi 1.0Gi
Low: 31Gi 30Gi 354Mi
High: 0B 0B 0B
Swap: 31Gi 28Gi 3.3Gi
|
Code: |
ps --no-headers -eo rss | awk '{m+=sprintf("%d",$1)} END {print m/1024}'
6439.58
|
Code: |
cat /proc/meminfo
MemTotal: 32811188 kB
MemFree: 382812 kB
MemAvailable: 1097688 kB
Buffers: 1640 kB
Cached: 919068 kB
SwapCached: 4848 kB
Active: 3581492 kB
Inactive: 2342832 kB
Active(anon): 3159732 kB
Inactive(anon): 2041304 kB
Active(file): 421760 kB
Inactive(file): 301528 kB
Unevictable: 48 kB
Mlocked: 48 kB
SwapTotal: 33554428 kB
SwapFree: 3457020 kB
Dirty: 936 kB
Writeback: 0 kB
AnonPages: 5000116 kB
Mapped: 474108 kB
Shmem: 197420 kB
KReclaimable: 321560 kB
Slab: 6234692 kB
SReclaimable: 321560 kB
SUnreclaim: 5913132 kB
KernelStack: 34704 kB
PageTables: 50896 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 49960020 kB
Committed_AS: 42172904 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 243180 kB
VmallocChunk: 0 kB
Percpu: 33792 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 22646416 kB
DirectMap2M: 10852352 kB
DirectMap1G: 1048576 kB
|
Code: |
cat /proc/vmstat
nr_free_pages 94671
nr_zone_inactive_anon 510254
nr_zone_active_anon 789966
nr_zone_inactive_file 75581
nr_zone_active_file 106219
nr_zone_unevictable 12
nr_zone_write_pending 272
nr_mlock 12
nr_page_table_pages 12674
nr_bounce 0
nr_zspages 1948643
nr_free_cma 0
numa_hit 3365830325
numa_miss 0
numa_foreign 0
numa_interleave 31827
numa_local 3365830325
numa_other 0
nr_inactive_anon 510254
nr_active_anon 789966
nr_inactive_file 75581
nr_active_file 106219
nr_unevictable 12
nr_slab_reclaimable 80778
nr_slab_unreclaimable 1478329
nr_isolated_anon 0
nr_isolated_file 0
workingset_nodes 49593
workingset_refault_anon 530352
workingset_refault_file 17629331
workingset_activate_anon 126514
workingset_activate_file 7186310
workingset_restore_anon 31477
workingset_restore_file 4776969
workingset_nodereclaim 22214
nr_anon_pages 1250176
nr_mapped 118917
nr_file_pages 232148
nr_dirty 272
nr_writeback 0
nr_writeback_temp 0
nr_shmem 49136
nr_shmem_hugepages 0
nr_shmem_pmdmapped 0
nr_file_hugepages 0
nr_file_pmdmapped 0
nr_anon_transparent_hugepages 0
nr_vmscan_write 8891112
nr_vmscan_immediate_reclaim 26039
nr_dirtied 100965172
nr_written 108999071
nr_kernel_misc_reclaimable 0
nr_foll_pin_acquired 0
nr_foll_pin_released 0
nr_kernel_stack 34656
nr_dirty_threshold 44354
nr_dirty_background_threshold 22150
pgpgin 2557615682
pgpgout 403169881
pswpin 530352
pswpout 8891108
pgalloc_dma 4
pgalloc_dma32 261117936
pgalloc_normal 3384389030
pgalloc_movable 0
allocstall_dma 0
allocstall_dma32 0
allocstall_normal 3598
allocstall_movable 15515
pgskip_dma 0
pgskip_dma32 0
pgskip_normal 97367594
pgskip_movable 0
pgfree 3677238258
pgactivate 79247735
pgdeactivate 23010169
pglazyfree 1530
pgfault 3755873559
pgmajfault 608716
pglazyfreed 0
pgrefill 41470924
pgreuse 410105472
pgsteal_kswapd 619681152
pgsteal_direct 6054297
pgscan_kswapd 710397005
pgscan_direct 6201206
pgscan_direct_throttle 0
pgscan_anon 17416193
pgscan_file 699182018
pgsteal_anon 8894424
pgsteal_file 616841025
zone_reclaim_failed 0
pginodesteal 196280
slabs_scanned 67943223
kswapd_inodesteal 42173740
kswapd_low_wmark_hit_quickly 85294
kswapd_high_wmark_hit_quickly 7229
pageoutrun 115554
pgrotated 32735
drop_pagecache 5
drop_slab 5
oom_kill 0
pgmigrate_success 31524837
pgmigrate_fail 2486998
thp_migration_success 0
thp_migration_fail 0
thp_migration_split 0
compact_migrate_scanned 6612803743
compact_free_scanned 2743272074
compact_isolated 65995607
compact_stall 618
compact_fail 614
compact_success 4
compact_daemon_wake 96236
compact_daemon_migrate_scanned 7089891
compact_daemon_free_scanned 32326709
htlb_buddy_alloc_success 0
htlb_buddy_alloc_fail 0
unevictable_pgs_culled 817254
unevictable_pgs_scanned 0
unevictable_pgs_rescued 807202
unevictable_pgs_mlocked 808650
unevictable_pgs_munlocked 807916
unevictable_pgs_cleared 12
unevictable_pgs_stranded 12
swap_ra 5242
swap_ra_hit 3523
nr_unstable 0
|
|
It looks like your kernel is doing a lot of paging there. From superuser.com https://superuser.com/questions/521551/cat-proc-meminfo-what-do-all-those-numbers-mean
Quote: | LowTotal and LowFree — The total and free amount of memory, in kilobytes, that is directly mapped into kernel space. The LowTotal value can vary based on the type of kernel used. |
Now, I'm no expert with Gentoo at this stage so I'd be basically stabbing in the dark. Is it possible your kernel is allocating paging prior to loading? _________________ Every emerge is an opportunity. |
|
Back to top |
|
|
tholin Apprentice
Joined: 04 Oct 2008 Posts: 203
|
Posted: Fri May 21, 2021 12:54 pm Post subject: |
|
|
I'm no expert at reading this but let's see.
Code: |
cat /proc/meminfo
MemTotal: 32811188 kB
MemFree: 382812 kB
MemAvailable: 1097688 kB
Buffers: 1640 kB
Cached: 919068 kB
SwapCached: 4848 kB
Active: 3581492 kB
Inactive: 2342832 kB
Active(anon): 3159732 kB
Inactive(anon): 2041304 kB
Active(file): 421760 kB
Inactive(file): 301528 kB
Unevictable: 48 kB
Mlocked: 48 kB
SwapTotal: 33554428 kB
SwapFree: 3457020 kB
Dirty: 936 kB
Writeback: 0 kB
AnonPages: 5000116 kB
Mapped: 474108 kB
Shmem: 197420 kB
KReclaimable: 321560 kB
Slab: 6234692 kB
SReclaimable: 321560 kB
SUnreclaim: 5913132 kB
KernelStack: 34704 kB
PageTables: 50896 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 49960020 kB
Committed_AS: 42172904 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 243180 kB
VmallocChunk: 0 kB
Percpu: 33792 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 22646416 kB
DirectMap2M: 10852352 kB
DirectMap1G: 1048576 kB
|
AnonPages+Mapped is only 17% of MemTotal. That's the memory that is directly assigned to a process memory so as you've figured out the missing memory is not used by any specific program.
Slab is 19% of MemTotal. The most common use of slab memory is caching for filesystem metadata but that memory is reclaimable. Almost all your slab memory is SUnreclaim. Run "slabtop" from sys-process/procps for more detail about that.
Code: |
cat /proc/vmstat
nr_free_pages 94671
nr_zone_inactive_anon 510254
nr_zone_active_anon 789966
nr_zone_inactive_file 75581
nr_zone_active_file 106219
nr_zone_unevictable 12
nr_zone_write_pending 272
nr_mlock 12
nr_page_table_pages 12674
nr_bounce 0
nr_zspages 1948643
nr_free_cma 0
numa_hit 3365830325
numa_miss 0
numa_foreign 0
numa_interleave 31827
numa_local 3365830325
numa_other 0
nr_inactive_anon 510254
nr_active_anon 789966
nr_inactive_file 75581
nr_active_file 106219
nr_unevictable 12
nr_slab_reclaimable 80778
nr_slab_unreclaimable 1478329
nr_isolated_anon 0
nr_isolated_file 0
workingset_nodes 49593
workingset_refault_anon 530352
workingset_refault_file 17629331
workingset_activate_anon 126514
workingset_activate_file 7186310
workingset_restore_anon 31477
workingset_restore_file 4776969
workingset_nodereclaim 22214
nr_anon_pages 1250176
nr_mapped 118917
nr_file_pages 232148
nr_dirty 272
nr_writeback 0
nr_writeback_temp 0
nr_shmem 49136
nr_shmem_hugepages 0
nr_shmem_pmdmapped 0
nr_file_hugepages 0
nr_file_pmdmapped 0
nr_anon_transparent_hugepages 0
nr_vmscan_write 8891112
nr_vmscan_immediate_reclaim 26039
nr_dirtied 100965172
nr_written 108999071
nr_kernel_misc_reclaimable 0
nr_foll_pin_acquired 0
nr_foll_pin_released 0
nr_kernel_stack 34656
nr_dirty_threshold 44354
nr_dirty_background_threshold 22150
pgpgin 2557615682
pgpgout 403169881
pswpin 530352
pswpout 8891108
pgalloc_dma 4
pgalloc_dma32 261117936
pgalloc_normal 3384389030
pgalloc_movable 0
allocstall_dma 0
allocstall_dma32 0
allocstall_normal 3598
allocstall_movable 15515
pgskip_dma 0
pgskip_dma32 0
pgskip_normal 97367594
pgskip_movable 0
pgfree 3677238258
pgactivate 79247735
pgdeactivate 23010169
pglazyfree 1530
pgfault 3755873559
pgmajfault 608716
pglazyfreed 0
pgrefill 41470924
pgreuse 410105472
pgsteal_kswapd 619681152
pgsteal_direct 6054297
pgscan_kswapd 710397005
pgscan_direct 6201206
pgscan_direct_throttle 0
pgscan_anon 17416193
pgscan_file 699182018
pgsteal_anon 8894424
pgsteal_file 616841025
zone_reclaim_failed 0
pginodesteal 196280
slabs_scanned 67943223
kswapd_inodesteal 42173740
kswapd_low_wmark_hit_quickly 85294
kswapd_high_wmark_hit_quickly 7229
pageoutrun 115554
pgrotated 32735
drop_pagecache 5
drop_slab 5
oom_kill 0
pgmigrate_success 31524837
pgmigrate_fail 2486998
thp_migration_success 0
thp_migration_fail 0
thp_migration_split 0
compact_migrate_scanned 6612803743
compact_free_scanned 2743272074
compact_isolated 65995607
compact_stall 618
compact_fail 614
compact_success 4
compact_daemon_wake 96236
compact_daemon_migrate_scanned 7089891
compact_daemon_free_scanned 32326709
htlb_buddy_alloc_success 0
htlb_buddy_alloc_fail 0
unevictable_pgs_culled 817254
unevictable_pgs_scanned 0
unevictable_pgs_rescued 807202
unevictable_pgs_mlocked 808650
unevictable_pgs_munlocked 807916
unevictable_pgs_cleared 12
unevictable_pgs_stranded 12
swap_ra 5242
swap_ra_hit 3523
nr_unstable 0
|
You have 1948643 nr_zspages so I assume you are using zram or swap? Do you put swap on zram? What is your swap situation like? Run "zramctl --output-all" and "swapon -s" for info about that.
Summing up anon+file+slab+zspages (assuming zspages are 4K, not sure about that) we only get about 60% of MemTotal. Something else is using the rest and it doesn't show up in these stats. Are you using zfs or any other out of kernel modules? Have you compiled the kernel with your own .config and if so did you enable any exotic debug capabilities? The memory could be lost to a kernel memory leak so if you can't figure out where the memory goes try running an older prebuilt sys-kernel/gentoo-kernel-bin and see if the problem goes away. |
|
Back to top |
|
|
MarcGentoo n00b
Joined: 26 Nov 2020 Posts: 65
|
Posted: Fri May 21, 2021 1:17 pm Post subject: |
|
|
Hi,
I do indeed use the zram. At first swap is empty, when ram fills up, swap is filled until both are full.
I have a custom kernel but not fancy debug fs enabled.
As I had a lot of trouble setting up gentoo because of device drivers, I am hesitant to try the default kernel.
All my modules are compiled in the kernel and are not a lot, no debug capabilities are enabled.
Code: |
cat .config | grep DEBUG | grep =y | cut -d "=" -f 1
CONFIG_SLUB_DEBUG
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC
CONFIG_X86_DEBUGCTLMSR
CONFIG_PNP_DEBUG_MESSAGES
CONFIG_CIFS_DEBUG
CONFIG_DEBUG_BUGVERBOSE
CONFIG_DEBUG_FS
CONFIG_DEBUG_FS_ALLOW_ALL
CONFIG_ARCH_HAS_DEBUG_WX
CONFIG_HAVE_DEBUG_KMEMLEAK
CONFIG_ARCH_HAS_DEBUG_VM_PGTABLE
CONFIG_ARCH_HAS_DEBUG_VIRTUAL
CONFIG_DEBUG_MEMORY_INIT
CONFIG_LOCK_DEBUGGING_SUPPORT
|
|
|
Back to top |
|
|
|