How To Set And Use ZRAM swap

Continuing from the topic and making guide for those interested in, as well as a reminder for myself after clean install someday.

Goals / Possible Benefits

  • To minimize dedicated amounts of RAM
  • To maximize utilizing total amount of RAM
  • To extend lifespan of a SSD
  • Hopefully to reduce Qubes footprint
  • To increase overall performance
  • To prevent performance hits to a level at least as without zram.

Why?

By default swappiness is set to 60. This means that when 40% of RAM is used, swapping is started. According to experience, even decreasing swappiness to 0 didnā€™t prevent swapping to start much earlier than RAM is exhausted. So, obviously, different approach is needed (beside disabling swap at all)

Prerequsite

It is advisable to disable zswap prior to activate zram module, in order to prevent zswap intercepting memory pages being swapped out before they reach zram. The difference compared to zram is that zswap works in conjunction with a swap device while zram is a swap device in RAM that does not require a backing swap device, but in my guide I will as well set a swap as a backup device to avoid system crash once the (z)RAM is exhausted.

How To

I have added kernel parameter to disable zswap just in case

GRUB_CMDLINE_LINUX=ā€œā€¦ quiet zswap.enabled=0ā€

For other qubes, it can be done by adding kerenlopts

qvm-prefs -s VMname kernelopts ā€œzswap.enabled=0ā€

Or, we can do it the other way

$ sudo bash -c ā€œecho 0 > /sys/module/zswap/parameters/enabledā€

Now, we will disable all active swaps

$ sudo swapoff --all

If you want to completely disable swap devices in fstab, edit it

$ sudo nano /etc/fstab

and comment out swap entries.

I) Starting zram for the first time

The zram module is controlled by systemd, so thereā€™s no need for a fstab entry, so letā€™s load the module

$ sudo modprobe zram num_devices=1

Please note that it is advisable to set one module per CPU. If you have more than 1 CPU, set num_devices above accordingly.

Check supported compression algorithms

$ cat /sys/block/zram0/comp_algorithm

the one in [] is currently set. Letā€™s set lz4hc

$ sudo bash -c ā€œecho lz4hc > /sys/block/zram0/comp_algorithmā€

Now we have to set disk size. I have decreased dom0ā€™s maxmem size to 1536MB so for sure all of that will be used, and the rest itā€™ll be taken - from RAM! So I blindly set 8GB and watching its use.

$ sudo bash -c ā€œecho 8G > /sys/block/zram0/disksizeā€

Now to activate zram0

$ sudo mkswap --label zram0 /dev/zram0

Since I will use swap as a backup device, I will set it with the highest possible priority to ensure it will be used first

$ sudo swapon --priority 32767 /dev/zram0

Now to check what weā€™ve done so far

zramctl

NAME       ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lz4hc            8G   4K   64B    4K       4 [SWAP]

Everything seems to be OK.

II) Letā€™s stop zram swap

$ sudo swapoff /dev/zram0

Letā€™s free already allocated memory to device, reset disksize to 0, and unload the module

$ sudo bash -c ā€œecho 1 > /sys/block/zram0/resetā€
$ sudo modprobe -r zram

III) Starting zram at boot
In order zram to start on boot we will need to create 2 scripts (zram_start and zram_stop - copy these files to /usr/local/bin) and a service - zram_swap.service.

zram_start script

#!/usr/bin/env bash
# Create a swap device in RAM with the 'zram' kernel module. Copy this file to /usr/local/bin.

# Show supported compression algorithms...
#  cat /sys/block/zram0/comp_algorithm
compress="lz4hc"

disksize="8G" #Set this accordingly to available RAM
priority="32767"  # give zram device highest priority

# Disable zswap  in order to prevent zswap intercepting memory pages being swapped out before they reach zram
echo 0 > /sys/module/zswap/parameters/enabled
# Disable any active swaps (I don't want to disable swap to prevent crashes that - uncomment if want to completely disable swap)
# swapoff --all
# Load module
modprobe zram num_devices=1
# Set compression algorithm
echo $compress > /sys/block/zram0/comp_algorithm
# Set disk size
echo $disksize > /sys/block/zram0/disksize
# Activate
mkswap --label zram0 /dev/zram0
swapon --priority $priority /dev/zram0

# View info with zramctl

zram_stop script

#!/usr/bin/env bash
# Deactivate zram0 swap device in RAM. Copy this file to /usr/local/bin.

swapoff /dev/zram0

# Free already allocated memory to device, reset disksize to 0, and unload the module
echo 1 > /sys/block/zram0/reset

sleep 1
modprobe -r zram

Create service file as /etc/systemd/system/zram_swap.service

[Unit]
Description=Configure zram swap device
After=local-fs.target

[Service]
Type=oneshot
ExecStart=/bin/bash /usr/local/bin/zram_start
ExecStop=/bin/bash /usr/local/bin/zram_stop
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Enable service

$ sudo systemctl enable zram-swap.service

IV) (Additional) Kernel parameters

We can fine tune the kernel to get the most of the zram.
I have chosen these parameters:

vm.vfs_cache_pressure - increase frequency of clearing caches to free RAM
vm.swappiness - increase percentage to start using zram earlier
vm.dirty_background_ratio and vm.dirty_ratio - amount of memory pages permitted to be ā€œdirtyā€ before writing to zram
vm.oom_kill_allocating_task - if set to non-zero, OOM killer kills the task that triggered the out-of-memory condition. This avoids the expensive tasklist scan which is unlikely to ever be finished in such a situation. Set as a precaution.

Check default values

$ cat /proc/sys/vm/vfs_cache_pressure
100
$ cat /proc/sys/vm/swappiness
60
$ cat /proc/sys/vm/dirty_background_ratio
10
$ sudo cat /proc/sys/vm/dirty_ratio
20

Set new parameters /etc/sysctl.d/99-sysctl.conf or if you prefer ( it is not advisable) directly to /etc/sysctl.conf

vm.vfs_cache_pressure=500
vm.swappiness=100
vm.dirty_background_ratio=1
vm.dirty_ratio=50
vm.oom_kill_allocating_task=1

You may now reload the deamon

sudo systemctl daemon-reload

and check the parameters:

$ sudo sysctl -p
vm.vfs_cache_pressure = 500
vm.swappiness = 100
vm.dirty_background_ratio = 1
vm.dirty_ratio = 50
vm.oom_kill_allocating_task = 1

V) Reboot and enjoy
Checking

$ cat /proc/swaps (or `swapon --show)

Filename				Type		Size		Used	Priority
/dev/dm-x                               partition	4136956		0	0
/dev/zram0                              partition	6291452		463616	32767

Obviously, SSD swap partition is not used.

Findings

  • It looks that swapping starts almost at the same time after ballooning starts too. Given the fact (?), that is one reason more to try zram to avoid swappines and performances hits.
  • Since RAM can hold much more information, for obvious reasons CPU will be used more, so this is something you might want to consider when deciding. I have too old CPU and using zram anyway (see below).
  • Still, it is much quicker than swapping to a SSD. Since I set dom0's maxmem to 1536MB (thus leaving more for qubes), system often falls back to swap, and this improves responsiveness while achieving goals from the beginning.
  • Iā€™m using zram0 for more than 2 weeks and I havenā€™t faced any performance hits, or at least not other than without zram0, again goals achieved.
  • ZRAM has to be set for each template if youā€™d like to use it everywhere.

#To Do
Creating salt formula to implement zram as highstate to avoid manual qube-by-qube setting. Contribution with this highly appreciated.

Please let me know if I made some mistakes here, or if the guide can be improved, Iā€™d be happy to edit it.

Helpful links

2 Likes

Thank you very much for your Post! I think i will give this a shot.

1 Like

Since Fedora 33, zram is officially recognized and introduced

https://fedoraproject.org/wiki/Changes/SwapOnZRAM

This is great, since all you now have to do for templates is to install

$ sudo dnf install zram-generator zram-generator-defaults

to reboot and to check both swap and zram are there, with zram having higher priorityā€¦

Since at the moment dom0 is based on fedora 32, I hope we will soon have this possibility by default for dom0 as wellā€¦

Thanks for posting this.
Some follow up questions:

  1. Are there any advantages in doing all that after a clean install?

  2. Which are the additional steps (if any) during/after a clean install which you consider useful in regards to doing all that? IOW how would you approach a clean install?

I am asking because I may reinstall soon.

  1. You mention the possible benefits. What are the possible disadvantages? (besides the obvious extra CPU load related to de/compression)
  1. What about using zswap for that? Couldnā€™t the compression of swapping (less bytes written) reduce the write cycles and wear of SSD?

  2. Can you suggest a way to benchmark and compare a system in each combination:
    5.1 zram, no zswap
    5.2 zswap, no zram
    5.3 zram and zswap
    5.4 no zram, no zswap

  1. How about Windows templates and their derivative qubes? Will Xen zram/zswap them implicitly? Or is anything additional necessary? (Suppose swapping/paging is explicitly disabled in a Windows VM)
  1. No. itā€™s just that for me - reminder what and how to do it after the Qubes OS clean install I plan.
  2. No additional steps. My newest finding is that itā€™s not necessary to provide dom0 with more than 2GB zram.

Itā€™s offtopic, but if you asked

  1. Beside CPU, none that Iā€™m aware of, especially if swap as a backup device is set to avoid system crash once the (z)RAM is exhausted. The fact that zram is included both in Fedora and Debian now should say enough.

As I said above, I donā€™t want zswap to compete with and intercept zram and write to swap. I want to avoid zswap acting as a swap cache in front of it. If they are both enabled also leads to wrong zramctl statistics because obviously zram is then mostly unused; this is because zswap intercepts and compresses memory pages being swapped out before they can reach zram. Swap is there allowed in very rare cases, and I can accept wearing my SSD in those rare cases. Not to say that I can format zram and use it temporary as any other block device, but much faster, right? Also, zswap decompressing to swap makes me extremely uncomfortable since under specific circumstances it can be very dangerous.

Itā€™s all there in those links I posted

I havenā€™t thought about this, but Iā€™d probably use sysbench and vmstat. And systemd-analyze?
This could give you an idea, too.

Iā€™m not sure I understand this. Beside disabling paging in Win qubes in order to spare SSD, I see creating RamDisk meaningful only to put browser cache there for example.

Thank you for the quick reply.

My newest finding is that itā€™s not necessary to provide dom0 with more than 2GB zram.

Could you please explain how you found that? I am asking in order to find the correct number for a system with N GiB of RAM (and based on other factors which you may have considered too).

The fact that zram is included both in Fedora and Debian now should say enough.

I donā€™t quite understand. Enough for what? I suppose you are not saying that if a package is included in a distro automatically makes it free of disadvantages, so I hope you can clarify.

Also, in regards to that inclusion, does it somehow change the steps you suggest?

Not to say that I can format zram and use it temporary as any other block device,

Can you suggest how to do this in the thread where I asked about RAM disk? That might be really helpful if it provides additional benefits to a standard RAM disk.

but much faster, right?

I donā€™t know. It sounds so but perhaps it needs benchmarking because performance depends on the bottleneck in the particular system, be that the CPU (de/compression), RAM speed, HDD/SSD speed, zram size, compression algorithm, perhaps even the way the system is used (workload specifics). Looking at one of the links you shared I see zram can make things slower or faster.

Also, zswap decompressing to swap makes me extremely uncomfortable since under specific circumstances it can be very dangerous.

Could you please elaborate? (sorry if it is in the links you posted, I just canā€™t see it)

Iā€™m not sure I understand this.

OK, I am clarifying: The question is in relation to your ā€œFor other qubes, it can be done by adding kerenlopts [ā€¦]ā€ which seems not to apply to a Windows VM. So, what I am asking is: how will the memory used by a Windows VM be swapped (if and when necessary) and are there any specific steps required in order to ā€œzswap itā€ like the Linux qubes?

Beside disabling paging in Win qubes in order to spare SSD, I see creating RamDisk meaningful only to put browser cache there for example.

It is also meaningful for improving performance while working on big files, as I mentioned in the RAM disk thread.

I regularly check cat /proc/swaps for its usage and it never went above 1.3GB. Also, not that much things to do in dom0 anyway. Also, Changes/SwapOnZRAM - Fedora Project Wiki?

I have posted many links above, which I read before I decided to use zram. So basically, you should do the same if thinking about it.

I wasnā€™t clear. When developed, SwapOnZRAM was considered to be installed by default in Fedora 33. I donā€™t know if it happened, but seriously discussed. Combining the fact with other advantages I already emphasized, made me start to use it. Please do not point zswap is already there. Anyone has to decide for itself. Iā€™m not forcing anyone, nor pledging for zram use.

When an application allocates many pages and writes to them data that compresses very well (say a sequence of zeros), zswap happily stores them in kernel slab memory. However, when something triggers actual disk swapping, then the stored data suddenly ā€˜burstsā€™ ā€“ that many zeros on the pages which took ā€œonlyā€ gigabytes in memory now decompress to hundreds of gigabytes on disk.
ā€¦
An attacker might try to store low entropy data on a server. When something triggers swapping, the server would be dead

Yes, and I decided to use it. What will you do?

Itā€™s not something that interests me, but simple sudo mkfs.ext4 --label zram0 /dev/zram0 should be enough? Will you try it?

Most of these is not Qubes specific and (better) answers could be and should be found out of this community, except the thing that Qubes is based on fedora-32 and thereā€™s no SwapOnzram there, so this guide is necessary if wanting to use it.
I think now there are enough things explained and clarified anyone to decide for him/herself whether or not to use it. At least to try to use it in addition and then to finally decide.

Please do not point zswap is already there. Anyone has to decide for itself. Iā€™m not forcing anyone, nor pledging for zram use.

I just asked about possible disadvantages. Perhaps we got carried away from it. No problem :slight_smile:

[ā€¦] An attacker might try to store low entropy data on a server. When something triggers swapping, the server would be dead

Thanks for explaining. Thatā€™s an interesting (kind of DoS) approach to attack a system. I wonder what protections against such swap ā€œexplosionsā€ exist for public servers (besides having plenty of RAM in the first place).

What will you do?

I will try it and look for ways to benchmark the difference.

Will you try it?

Yes.

Thank you.

1 Like

Splendid! It would be great to get a feedback from you here.