My personal experience of attempt to harden qubes VM

General Discussion
1

Just saw this discussion More practical security for Qubes (and more realistic threat model) - #42 by 51lieal @arkenoi so I would like to share some of my experience on harden qubes VM. Basically my opinion is qubes is fantastic about compartmentalization, but abandon in VM security completely is a bad idea. Even for a amateur user like me I can feel the VM security is quite terrible. It seems the developer’s idea is to use seperate VM for every activity, so it is irrelevant to secure each single VM. My belief is XEN is not holy grail, it is not invincible like SEL4, you cannot count everything on XEN. A good secured system consists of many layers of security mechanisms, like an iPhone you need to use a chain of exploits to perform a successful and persistent hack. After spend a lot of time I just found it is almost impossible for a normal user like me to properly harden a VM, and it require systematic work plus a lot of money to bring serious OS security model like Mac OSX to qubes VM. But I will still share some of my experience, hopefully this will help people who are also thinking about this problem.

Disclaimer: I am not a professional at all, there might be (big) mistakes in my list

All things I mentioned below only applied to debian family VMs because I do not use fedora.

1.Root
Qubes leaves a passwordless root which I think is the most controversial part. A door with any crappy lock is better than an unlocked door. With an unrestricted root every other security mechanism seems pretty much pointless

This tutorial gives a dom0 root prompt solution, but I am not a security researcher and cannot verify this security model. So I took another simpler way: cut the root access.

This will purge passwordless root access and all the root operation need to be invoked from dom0 with qvm-run VMName xterm -u root

This will remove SUID and hopefully make root escalation a little bit more difficult. Because we do all the root escalation in dom0 completely, there is no point giving normal user access to sudo/su.
Comment out everything in /etc/permission-hardening.d/25_default_whitelist_sudo.conf so normal user will just get a permission denied when try to sudo.

We need to give permission back to mount with /usr/lib/security-misc/permission-hardening-undo /usr/bin/mount, otherwise HVM usb qubes do not boot.

2.Kernel
Model Linux Kernel is monstrously complicated for a normal end user to seriously customize/audit/secure it. Also most harden options are sadly commercial solutions. Most open projects attempt to harden kernel are dead because no body is interested/no fund to hire enough people.

Whonix has a kernel harden project but it is not seriously tested/audited, and it is still in alpha stage pretty much like all other dead kernel harden projects. It has two version: VM and host, when I test it host version simply connot compile, and VM version seems working.

This kernel harden approach is pretty aggressive, it disabled a lot of things and accordingly it breaks most applications. It is far from ready and not recommended at all.

I took another approach which is lkrg. This project exists for a really long time and is developed by reputable developers, hopefully this can increase the ability for our buggy kernel to fend off exploits.

ht tps://www.qubes-os.org/doc/managing-vm-kernels/

First install in vm kernel and all other dependencies, and then you should install grub2-xen-pvh in dom0. This part is not mentioned in the previous document and I have spent a hell amount of time to figure out how to have pvgrub2-pvh in my dom0 setting. Use uname -a in vm terminal, if you do not see qubes in kernel name everything worked.

Then follow this tutorial to install lkrg

Use dkms status to make sure lkrg is loaded

Also if you do not use the stock kernel shipped by dom0, VMs cannot boot when you try to launch them with over 4GB ram with your customized kernel. Kernel equipped with lkrg also breaks firejail+chromium combo.

3.GUI
GUI is the most tricky and annoying part. Qubes VM are still using X11 which is notoriously insecure. It is not designed with security in mind and a huge escalation vector. Obviously there is no way for me to replace it by wayland, this requires a huge overhaul of all there qubes service software. The only thing I can do is to attempt to sandbox applications, so it cannot simply escalate through X11 and make all of my previous hardening attempt pointless. Current the solution I can find is to use the graphic server isolaion function of firejail. Firejail is not a very good sandbox, it has a huge codebase and has many CVE discovered in the past, but anyway it is better than a big exposed X11 server. There are three possible solutions that possibly work(yes possibly) under qubes.

First is firejail --X11=xorg firefox/chromium/etc… This is the solution that work out of the box, it puts programs in a untrusted domain of xorg. This disable the pasteboard function, and it breaks some programs like libreoffice because most programs just assume they are in a trusted xorg domain.

Second is firejail -X11=xpra The windows crashes several seconds no matter what you do, cannot figure this out

Third is firejail --X11=xephyr This will not crash, but it display things in a extremely weird resolution and I cannot figure out how to fix it

Also there are some document says you need to set up a separate network device with --net option for programs otherwise they can escape through network socket. But I do not have enough knowledge to verify and tinker this thing

4.MAC
I seriously cannot understand why qubes do not enable MAC by default, MAC is the standard equipment for all modern desktop and mobile OS.

Also whonix has a fancy new toy called apparmor-profile-everything which intends to bring the way how mobile OS confines app to whonix, but it is still highly experimental and it breaks qubes VM.

5.Sandbox
I can feel there are still a huge amount of potential vulnerability in the OS, because security is a systematic work and it requires everything to be perfect otherwise everything is pointless. So I decide to sandbox all the applications to further increases the escalation difficulty of a potential hack.
Whonix also has a fancy toy called bubblewrap, but again it is not production ready. The only option we have right now is firejail. Again firejail is buggy, but debian only use the lts version of it, and plus we will apply some hardening options hopefully this increase the overall security instead of make things worse.

This document talked about how to use and harden firejail

Also some posts say firejail might increase fingerprint of torbrowser, but I do not have proper knowledge to verify that

6.Root of trust
Qubes OS has something called AEM, which is to be honest not quite user friendly and most important it is not compatible with TPM 2.0 . Most modern devices is only equipped with TPM 2.0, and obviously AEM do not work on my device. Also there is a solution called heads which is a serious secure boot solution, but again it only works on those ancient machines, and at the time I tested it it only worked out of box for x230. If qubes cannot boot from a ensure trusted state, all the glorious work like XEN, compartmentalization, and qrexec will be pretty much pointless. Currently I did some little tricks to determine if someone has tried to open the back cover of my computer, but qubes really need some strong secure boot mechanism that works out of box like MAC OSX.

This is everything I have done currently, and I think I have reached the limit of an end user. I will add more when I have time. A good secured OS requires some serious security engineering and design from the ground up. I have used Qubes OS for many years and saw how it grew up from an idea, and I have seen no reason it should not further increases its security.

3 Likes
2

Welcome to the forum @zphaskqjrm, it appears you have been a member of the community for a long time.

1.Root> 2.Kernel> 3.GUI
I challenge that all three of this topics have been addressed over the years over and over. The basic question to you is: what do root privileges or a kernel/X11 exploit buy the attacker? How does any of it make a XEN exploit more likely? The attacker is already in domU and all the data there is exposed. What more can the attacker achieve now?

4.MAC
It’s coming

5.Sandbox

Not a bad idea, I suppose if you run a lot of stuff in a single qube … but why wouldn’t you just distribute it over multiple qubes instead?

6.Root of trust

Any ideas beyond AEM / heads?

I have used Qubes OS for many years and saw how it grew up from an idea, and I have seen no reason it should not further increases its security.

There is no reason, just money and time constraints. Anyone can donate and contribute!

1 Like
3

first word, as fsflover sad

however, don’t think i don’t want this to be true, i love it if this can become default in someway (like create a version for it, make a switch, …)

# WTF?! Have you lost your mind?!
#
# In Qubes VMs there is no point in isolating the root account from
# the user account. This is because all the user data is already
# accessible from the user account, so there is no direct benefit for
# the attacker if she could escalate to root (there is even no benefit
# in trying to install some persistent rootkits, as the VM's root
# filesystem modifications are lost upon each start of a VM).
#
# One might argue that some hypothetical attacks against the
# hypervisor or the few daemons/backends in Dom0 (so VM escape
# attacks) most likely would require root access in the VM to trigger
# the attack.
#
# That's true, but mere existence of such a bug in the hypervisor or
# Dom0 that could be exploited by a malicious VM, no matter whether
# requiring user, root, or even kernel access in the VM, would be
# FATAL. In such situation (if there was such a bug in Xen) there
# really is no comforting that: "oh, but the mitigating factor was
# that the attacker needed root in VM!" We're not M$, and we're not
# gonna BS our users that there are mitigating factors in that case,
# and for sure, root/user isolation is not a mitigating factor.
#
# Because, really, if somebody could find and exploit a bug in the Xen
# hypervisor -- as of 2016, there have been only three publicly disclosed
# exploitable bugs in the Xen hypervisor from a VM -- then it would be
# highly unlikely if that person couldn't also found a user-to-root
# escalation in VM (which as we know from history of UNIX/Linux
# happens all the time).
#
# At the same time allowing for easy user-to-root escalation in a VM
# is simply convenient for users, especially for update installation.
#
# Currently this still doesn't work as expected, because some idotic
# piece of software called PolKit uses own set of policies. We're
# planning to address this in Beta 2. (Why PolKit is an idiocy? Do a
# simple experiment: start 'xinput test' in one xterm, running as
# user, then open some app that uses PolKit and asks for root
# password, e.g.  gpk-update-viewer -- observe how all the keystrokes
# with root password you enter into the "secure" PolKit dialog box can
# be seen by the xinput program...)
#
# joanna.

only do this when you make sure you don’t need root anymore

also, Linux is not designed to be secure

x11 is more common than wayland

not always ancient machines, you can build heads for your machine even if it not supported but keep in mind that it only maintained by you so you might miss some security patches

in order to do that, you need a computer/laptop specifically designed to only run qubes

after all, i feeling that qubes os only use default configuration of not security-oriented os

3 Likes
4

The point is when you see a door open, and you do not see a good reason to keep it open, lock it. If you do not use that door frequently, lock it. Qubes is like a mini corporate netowork, we have all the internal air gapped things like vault and archive vm for confidential data, and all the vms that face the public internet, and disposable vm for potential dangerous content analysis like a sandbox. What will a company do for all the machines under its network? Will they say okay we already have ABC security measures implemented, we already have XEN for infrastructure isolation/IDS for intrusion detection/Regular wipe and reset of potentially dangerous machines, so there is no need to further enhance single machine security? No, they patch / lockdown everything like there is no tomorrow. A good company that value the corporate network security have a paranoid check list, like secure boot lockdown, bios lockdown,kernel runtime integrity and control flow check, give only necessary minimal permission, remove unused authorization, cut all the unnecessary software, regular vulnerability self scan, etc… Because somehow someday something might screw you up in an unexpected way.

A penetration of a company network very likely will not succeed in one shoot, you will likely end up in possible situations like 1.Your successfully screwed an old windows computer with eternal blue but rootkit/trojan is deleted by windows defender or other antivirus 2.You payload landed in the IDS analysis sandbox and nothing real happened 3.You have spent a lot up time sending phishing emails, and finally successfully invoked a chrome browser bug in one your their employee computer, but your payload is trapped inside their customized sandbox and cannot figure out how to escape 4.You have screwed up their nginx program and prepared an esix escape exploit(they cost you 0.5M usd) but found out they implemented some fancy ASLR/grsecurity/other harden method so your exploit just somehow did not work! Even an adversary controls some sort of overkill exploits like XEN exploit, the intrusion might not succeed because of other measures in place(like their web payload stopped by adblock) or they are simply uncertain about your attack surface and their payload failed to work(like a nicely prepared payload vmware payload landed in a proprietary vm software) . And leave everything unlocked will exactly increase the attacker’s ability to gather info and finally figure out a way to screw up the victim. Doing layers of harden will decrease the possibility of an adversary performing a successful hack even they really know how to hack every single part of your system. Hacking is like blind shooting, even best hackers might fail sometimes because of unexpected things.

Also I am not asking too much, I just say we should at least follow what the mainstream linux desktop have already done to improve security. Like debian official version already use wayland by default, apparmor works out of the box, secure boot works out of the box, and has a somehow properly implemented user/root isolation. These things already have very mature solution, and not too difficult to implement them in qubes VM. In the future we can do something like graphene OS to further increases the VM security, but right now we should just followed what everybody is doing.

1 Like
5

You can always qvm-run xterm - u root, and you can set up an shortcut on desktop specially for that

This is not a good reason to stick with it, maybe it is hard to switch because of some qubes render/graphic mechanism is only implemented in X11, but change it should be placed in schedule.

I just want a machine when you open it you can make sure it is not tampered. Developing a boot firmware for a new machine without insider document is already a highly professional work. Most likely you will need to reverse engineer the machine, figure out how things work and reimplement it, also you need to fight with the vendor who is exactly trying to stop you to do this. Best approach is just stick with the commercial solution which is making qubes compatible with uefi secure boot like debian already did that.

1 Like
6

i just remember xwayland

can you ask M$ to accept the bootloader signature of qubes ?!?

7

I think most secure boot mechanism(at least for lenovo and HP as I know) allows you to enroll your own key. Also a simpler solution is use a minized debian, verified boot into that debian and kexec into qubes like what heads did.

8

dell too, but it quite complex to do that

great idea! (it better to use some security/minimal focused distro)

9

Not just on ancient machines: Purism Librem 14 v1.

10

It depends on the cost. I think if your attacker has enough resources to get a Xen exploit, it’s highly likely they will also have a Linux exploit. You probably should find other security measures like a good firewall, instead of using Linux for protection.

11

Thanks for sharing your experience @zphaskqjrm

Would you be willing to elaborate?

I’m aware of Glitter Nail Polish but am always interested in new tricks.

12

candle wax on the screw
fake warranty/genuine device stamp

(this different)

13

That is a quite simple and great solution. I have deployed sensors with a separate chip on the back cover, it will be triggered when back cover is detached. I have designed a simple challenge response. If back cover is not removed in a designed way then the chip wipes the keys which cannot be undone by an adversary. When the challenge response fail I will be 100% sure my laptop is tampered.

14

how you do that (i curious the power source for sensors vulnerabilities)

15

Purism maybe has a good ideology, but their products still cannot compare to the reliability of those big vendors. I have seen some posts complain about their laptops with many kinds of problems. Also mainstream products are cheap and powerful because of mass production, a good OS should be compatible with these new mainstream products.

16

It is a quite simple project, like those beginner level Arduino project you turn on led when a switch is pressed. I sacrificed a usb port for charging the battery, I soldered the power cable directly on the usb port. Those low power Soc should be able to stay for months even laptops is power off without any problem.

17

interesting, thought i like to drill a hole on laptop case (or use magnetic usb-c cable + plug) for charging that battery rather sacrifice a usb port
what sensors (and soc if not some arduino or pi pico) you use?

18

Well actually it is just a simple pressure switch like how your refrigerator open light when you open the gate.

19

so what if they (someone who want to tamper you laptop) find where that sensors then loosen the screw and put something thin to not trigger the sensors?

20

I guess nobody will know that if you do not take a picture of your laptop internal structure and show it in your twitter profile