A single misbehaving qube slowed my whole system. Wondering about DOS potential / security implications

General Discussion
1

I ran a few dd commands in parallel in a disposable vm. (For those unfamiliar, dd is a copy command, able to copy files or data from drives, etc.)

I notcied my entire system slowed down badly as dd worked, other appVMs became stuttery, even text input to a terminal was significantly delayed. Running dd with a bigger block size resolved the issue.

Apparently, dd has bad defaults, and if the block size is not specified, only 512 bytes are copied at a time. There is a system call for every 512 bytes. I think this is the reason it slows the whole system unless the block size is bigger (reducing system calls). Worth noting my disposable had a small fraction of the total vcpus so this is not CPU is exhaustion.

It made me wonder if any appVM can DOS the entire system by spamming system calls and if this is in a sense a minor security issue.

Is my theory correct or is there a better explanation for the slowdown? Any thoughts on this?

2

I use this script to measure the CPU load in a generic monitor on panel (period 3 sec):

Maybe if you retry your test it will give you an idea how much CPU load your VM actually consumes.

3

There is no solution right now, but what can potentially be more secure against DoS, is very taxing on the CPU, if you can get the magic commands right, you can make a domain vCPUs have parity to physical CPUs/pCPU (pinning), plus allow only that domain from using those pCPUS. This doesn’t work on scale, cause you’d be limited by the number of pCPUS. I can’t write all the commands that you should use, cause I haven’t done all of this myself (just vCPU pinning, which isn’t enough for this issue).

1 Like
4

@ben-grande

Would using xl sched-credit2 with a low weight be helpful in such situations?

5

Changing the scheduling credit is for another purpose, it can allow the domain to be preferred or less preferred for spending the CPU time, when there is a lot of requests. You can try reducing the potentially malicious qube scheduling credits compared to the other qubes, but I am not hopeful on this approach to solve the problem from the first post, as Qubes already sets the dom0 credit to a high value, that is not enough, as even dom0 being preferred over all others, it can still behave badly on DoS.

6

I also would like to emphasize “potentially”, cause I don’t know if it will work, cause if there is shared state in the CPU that locks up on high load, then there is nothing that consumers can do besides using a separate machine. This is one reason some hyperscalers have “dedicated servers” which are separate hardware, instead of having to restrict and isolate state of CPU and disk and memory to X amount.

7

Then I guess emergency pause is the only way out of this… if one even has the chance to activate it.

9

What about running the dd command on a tmpfs drive?
Has anyone compared that?

10

Actually, I think that what I said about limiting CPU won’t make sense it doesn’t have to max out on CPU to max out hogging the disk.