I am in the process of migrating my sys-qubes (-net, -firewall, and -usb) into debian-11-minimal templates. Thus, I have been examining the existing templates for these qubes. On this topic, I have a few questions:
Why are the default startup memory values set to 400 MB and 300 MB for sys-net and sys-usb, respectively? Are these values an educated guess? How are these values estimated, that is, how can one decide that 300 MB startup memory (with no memory balancing, so that value is the ceiling value for the sys-usb qube) is enough for a USB hub (that is, sys-usb)?
I noticed that by default the sys-net is exempt from memory balancing, while sys-firewall makes use of it (up to 4000 MB RAM). Why is this difference between those qubes? Afaik, sys-firewall is connected to sys-net qube like two pipes being concatenated one after the other. So, if sys-net is limited to 400 MB RAM while sys-firewall can memory balance up to 4000 MB RAM, isn’t the whole chain of these two effectively limited to 400 MB RAM?
If you are trying to save memory, fedora minimal may be a better option, in my limited experience. I originally built sys-qubes with debian minimal (debian is my preference) but when I built equivalent qubes using fedora minimal, these used slightly less memory. The starting values are educated guesses but you need to consider the fact that reducing the startup memory may lead to paging/swapping and that may not be a good thing for your SSD in the long term. In order to see what works for you, I would start with some value, run some load and see the memory footprint, both in memory and in swap. You can also reduce swappiness to 1 (or even 0, if you don’t mind the occasional OOM killer triggering at some point) to reduce the wear in your SSD (assuming you use an SSD).
Sys-net needs to have access to hardware, so it runs HVM, which does not support ballooning/balancing. Same applies to sys-usb. Sys-firewall, on the other hand, is a standard pvh, so balancing works well with it.
No, your analysis about the effect of the memory limits in those two vm’s when they are connected in series is not correct. The use of memory is very different. Most of the activity that sys-net does happens in kernel space and usually memory utilization doesn’t increase much over heavy traffic (exception being when you use sys-net for Dom0 updates too). Sys-firewall, in the other hand, could be running proxies or other user space services that take additional memory, but 4,000MB is still a very generous amount and that memory is usually never used and reclaimed by Qubes if there is memory pressure somewhere else.
But if you want to really save memory, you can replace sys-firewall with the mirage firewall instead, and only use 64MB in that sys-firewall qube, as described here: https://github.com/mirage/qubes-mirage-firewall
I don’t want to dabble with swappiness settings. I would prefer least amount of changes enacted on the QubesOS by myself, as I am fallible and my information is limited on Linux OS stuff.
So, what would be the “appropriate” value for the max memory limit for the sys-firewall (considering the fact that it is based on debian-11-minimal). Would you say 1,000 MB is fine?
Edit: I am aware of the Mirage firewall, which sounds fantastic (I remember we have chatted on that before, too). However, currently I am taking things slow and deciding against moving MirageOS’s ISO to dom0 and installing it from there. I would much rather wait for the official support for its installation by the QubesOS team.
Yes, 1,000MB for max mem is fine. Again, if there is free memory in this qube and you are trying to start a new qube, Xen may reclaim some or all of this (all the way down to the starting mem value).
Well, I increased swappiness to 100 to save my SSD
It’s much more than 64MB, but others might want to know that my sys-firewall works flawlessly with 300MB, based on fedora 35 (planning to migrate it to fedora-36-minimal)
These work for me on my hardware. They will not fit every use case.
If you want to get in to this, you can start higher and then trim away
until you hit a pain point.
Here, QubesOS is doing a full system update. And I am seeing that sys-firewall is using ~1500 MB of RAM, and sys-whonix is also using the same amount of RAM.
So, what would happen if I limit sys-firewall to 500 MB as you guys did? And if I disable memory balancing for it, is sys-firewall going to use swap space? Or is it going to crash due to out-of-memory situtaions?
You’d need powerful enough CPU. But I’m not sparing mine. They are build to work, so working 100% while temperature is low enough is something I practice…
So you are effectively saying that lowering the RAM usage for sys-firewall will result in that qube using more CPU percentage? I think I, too, am OK with that.
But I am failing to see the connection between lowering RAM causing more CPU usage. Can you explain?
I guess it’s when you don’t have swap and have low RAM cache is filling more often, or faster. I’m not definitive with this, but i clearly see it happens.
I don’t understand why sys-firewall is often allocated max memory in the first place. All it does is passing through traffic.
Same applies to the supposed trade-off CPU-or-RAM-usage. sys-firewall isn’t compressing traffic, is it? This is not a video-codec using CPU-power to spare memory.
