zram not swapped? [SOLVED]

[Zucca]

I left my laptop run overnight to compile stuff. Most notably firefox and rust.
I have my portage temp on zram device. I also have around 20GB of swap in addition to 8GB of RAM.

During compilation of rust oom-killer stepped into play and the emerge process was killed off.
Indeed my memory usage was almost at 100% but almost none of the swap was used. After rm -r'ing everything that was left from compilation in portage temp dir most of the RAM was freed up.

Is this behavior normal or did I encounter a bug? Does kernel refrain swapping out memory pages that belong to zram?

EDIT: I'm now running emerge again, but tmpfs as portage temp dir and swap usage increases as one would expect.

[pingtoo]

Did you have zswap or swap on /dev/zramX?

zram need to setup backing_dev (a disk partition) for write back.

zswap is part of kernel overall swap, so either you have zswap + swap (partition or swapfile), in this case zswap is hidden. (it is known as frontswap)

you can examine zram stats by looking at /sys/block/zramX/mm_state

you can examine zswap in two place

• /sys/kernel/debug/zswap
• /sys/module/zswap/parameters

For more information please see

• zram -- https://docs.kernel.org/admin-guide/blockdev/zram.html
• zswap -- https://www.kernel.org/doc/html/v5.16/a ... ight=zswap
• frontswap -- https://www.kernel.org/doc/html/v5.16/v ... ight=zswap

[Hu]

I briefly read through the Kconfig and kernel documentation for zram, and I don't see anywhere that it says that pages written to zram will be implicitly swapped. Considering the existence of zswap as a separate feature, it seems likely that zram is intended for cases where you want the speed of a tmpfs, but greater information density than a pure tmpfs would offer. I think zswap+tmpfs is intended to cover the usage case described here, where you want tmpfs as long as possible, and compressed swapping when the kernel is finally forced to swap.

[Zucca]

@pingtoo: I use(d) zram as my portage temporary directory. Not as swap.
I don't use zswap. Let's not mix zswap into the issue here. ;)

I don't have any backing devices for my zram devices as they only host temporary files.

I think I'll convert all (or most) places where I used zram to tmpfs. And maybe enable zswap to have some compression.

[pingtoo]

@pingtoo: I use(d) zram as my portage temporary directory. Not as swap.
I don't use zswap. Let's not mix zswap into the issue here.

I don't have any backing devices for my zram devices as they only host temporary files.

I think I'll convert all (or most) places where I used zram to tmpfs. And maybe enable zswap to have some compression.

zram by default does not have backing storage, it will not use normal swap as back storage. You need to manual configure a back store.

With CONFIG_ZRAM_WRITEBACK, zram can write idle/incompressible page to backing storage rather than keeping it in memory. To use the feature, admin should set up backing device via:

Code: Select all

echo /dev/sda5 > /sys/block/zramX/backing_dev

If you have LVM installed you can setup like this (This is how I use zram from another machine)

Code: Select all

nbd-client -N zram littleboy /dev/nbd0
pvcreate -y /dev/nbd0
vgextend vg0 /dev/nbd0
lvcreate --extents 100%PV -n ramdisk vg0 /dev/nbd0
lvextend -L +10G vg0/ramdisk
mkfs.ext4 -O ^has_journal /dev/vg0/ramdisk
mount -o noatime /dev/vg0/ramdisk /chroot/var/tmp

If you don't use NBD, then just replace /dev/nbd0 with your /dev/zramX.

My littleboy serve zram as follow

Code: Select all

nbd-server

littleboy for nbd-server have configuration file

Code: Select all

# This is /etc/nbd-server/config
# This is nbd-server configuration file stored in /etc/nbd-server/config
# This is used in docker container
# listenaddr = 0.0.0.0 bind to docker IPv4 address
# max_threads = 4 because RPI 4 only have 4 CPU cores.
# splice = true for <1MiB none TLS I/O

[generic]
   listenaddr  = 0.0.0.0
   root        = root
   group       = root
   max_threads = 4
   splice      = true

[zram]
   exportname  = /dev/zram0
   virtstyle   = none
   copyonwrite = false

[Zucca]

zram by default does not have backing storage, it will not use normal swap as back storage. You need to manual configure a back store.

With CONFIG_ZRAM_WRITEBACK, zram can write idle/incompressible page to backing storage rather than keeping it in memory. To use the feature, admin should set up backing device via:

Code: Select all

echo /dev/sda5 > /sys/block/zramX/backing_dev

A-ha! I had completely misunderstood the meaning of this backing device. This explains it! Thanks! [SOLVED]

[tnt]

I have set up zram in order to test it for swap purposes (200MB zram device, 10GB backing_dev SSD partition), but although I have added backing_dev, pages never get written to it. I run out of memory sooner.
So, my question is: should zram writeback to backing_dev automatically, or only by explicit command like:

Code: Select all

echo idle > /sys/block/zramX/writeback

[pingtoo]

I have set up zram in order to test it for swap purposes (200MB zram device, 10GB backing_dev SSD partition), but although I have added backing_dev, pages never get written to it. I run out of memory sooner.
So, my question is: should zram writeback to backing_dev automatically, or only by explicit command like:

Code: Select all

echo idle > /sys/block/zramX/writeback

tnt,

You could do the echo idle ... command at the beginning of setup. for example

Code: Select all

echo 1 > /sys/block/zramX/reset
echo /dev/<dev_partition> > /sys/block/zramX/backing_dev
echo NNNg > /sys/block/zramX/disksize
echo 200m > /sys/block/zramX/mem_limit
# follow is optinal
echo huge > /sys/block/zramX/writeback
# end option
echo all > /sys/block/zramX/idle
echo idle > /sys/block/zramX/writeback

[tnt]

I have tried suggested approach and it seems it writes pages out to backing device once, but next memory pressure produces OOM event instead of writing more pages to backing device in the real-time

[pingtoo]

I have tried suggested approach and it seems it writes pages out to backing device once, but next memory pressure produces OOM event instead of writing more pages to backing device in the real-time

While you create pressure to memory, can you also do

Code: Select all

cat /sys/block/zramX/bd_stat

and

Code: Select all

cat /sys/block/zramX/mm_stat

and

Code: Select all

cat /sys/kernel/debug/zram/zram0/block_state

How do you tested to know your pressure size is not greater than mem_limit?

Remember the backing_dev is not a swap concept, it will always hold zram pages unless those pages are free. it will only move zram page into backing_dev when the page is marked idle (i.e no read access) if those pages are needed, it will need zram space to pull them back.

EDIT: add check mm_stat.

[tnt]

I was making pressure by starting two concurrent mprime torture tests (selecting test type 3).
One was successfully handled in RAM, occupying most of it.
The second one would crash/be killed as memory would run out.

During this, I was following:

Code: Select all

cat /sys/block/zram0/bd_stat

and I saw no new writes to it - numbers were staying the same.

Remember the backing_dev is not a swap concept, it will always hold zram pages unless those pages are free. it will only move zram page into backing_dev when the page is marked idle (i.e no read access) if those pages are needed, it will need zram space to pull them back.

This explains a lot.
It seems by design I can run with filled up zram and still having empty backing device.

[pingtoo]

tnt,

This go back to the original question, zram is a block device. if it is full you will not be able to write to it any more.

In your case you make zram as swap device so now you have fast swap. (fast as in compare to dist storage device)

when physical memory are short. kernel wrote unused page(s) in to swap. And when swap full, you start getting OOM.

But zram backing_dev is not itself a swap for zram. backing_dev is just a place when instruction given to zram to flush out known unused pages to backing_dev. remember zram is block device. It does not have memory management system.

The instruction you can give to zram are "idle | huge", you issue the instruction into zram instrument (writeback) that is when zram will perform freeze the block device system and examine pages for marker of "idle" or "huge (as incompressible)" write pages match the instruction then unfree the block device.

If you make you zram larger and wait until it also full, try the idle, and you examine your process(es) you will find them in the "D" stat which mean they are waiting.

May be you want to use ZSWAP?

[tnt]

Yes, now I see that zswap seems to fit better my needs.

Nevertheless, current state of my zram-only machine is quite odd:

Code: Select all

# cat /proc/meminfo |grep -i swap
SwapCached:      1903236 kB
SwapTotal:       4194300 kB
SwapFree:        1643268 kB
# zramctl
NAME       ALGORITHM DISKSIZE   DATA  COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lz4             4G 633.9M 191.8M  200M       4 [SWAP]
# cat /sys/block/zram0/bd_stat
       0        0        0
# swapon -s
Filename                                Type            Size            Used            Priority
/dev/zram0                              partition       4194300         2551032         16000
#

zram seems to hold 634MB of data,
backing device has no data written to it,
and yet my swap seen by system contains 2.5GB of swapped data

[pingtoo]

tnt,

It is Z-RAM, a RAM disk with compression builtin. Kind of like BTRFS with compression.

Try this code

Code: Select all

#print various stats info about zram swap device
zram_stats()
{
	local zdev="/sys/block/$( basename "$1" )"

	printf "\nGathering stats info for zram device \"$( basename "$1" )\"\n\n"

	printf "Z-RAM\n-----\n"
	printf "%-25s - %s\n" "Block device" $zdev
	awk '{ printf "%-25s - %d MiB\n", "Device size", $1/1024/1024 }' <$zdev/disksize
	printf "%-25s - %s\n" "Compression algo" "$(cat $zdev/comp_algorithm)"

	awk 'BEGIN { fmt = "%-25s - %.2f %s\n"
		fmt2 = "%-25s - %d\n"
		print "\nDATA\n----" }
		{ printf fmt, "Original data size", $1/1024/1024, "MiB"
		printf fmt, "Compressed data size", $2/1024/1024, "MiB"
		printf fmt, "Compress ratio", $1/$2, ""
		print "\nMEMORY\n------"
		printf fmt, "Memory used, total", $3/1024/1024, "MiB"
		printf fmt, "Allocator overhead", ($3-$2)/1024/1024, "MiB"
		printf fmt, "Allocator efficiency", $2/$3*100, "%"
		printf fmt, "Maximum memory ever used", $5/1024/1024, "MiB"
		printf fmt, "Memory limit", $4/1024/1024, "MiB"
		print "\nPAGES\n-----"
		printf fmt2, "Same pages count", $6
		printf fmt2, "Pages compacted", $7 }' <$zdev/mm_stat

	awk '{ printf "%-25s - %d\n", "Free pages discarded", $4 }' <$zdev/io_stat
}

zram_stats zram0

I guess you have good compress ratio for your swap pages.

Disclamer: I did not wrote this code. I got it from OpenWRT.

[tnt]

this is my output:

Code: Select all

Gathering stats info for zram device "zram0"

Z-RAM
-----
Block device              - /sys/block/zram0
Device size               - 4096 MiB
Compression algo          - lzo lzo-rle [lz4] lz4hc zstd

DATA
----
Original data size        - 624.99 MiB
Compressed data size      - 190.28 MiB
Compress ratio            - 3.28

MEMORY
------
Memory used, total        - 199.23 MiB
Allocator overhead        - 8.96 MiB
Allocator efficiency      - 95.50 %
Maximum memory ever used  - 200.06 MiB
Memory limit              - 200.00 MiB

PAGES
-----
Same pages count          - 36465
Pages compacted           - 5
Free pages discarded      - 5154109