Proxmox vs. Qubes

General Discussion
1

I have considered switching to ProxMox just because of more nested virtualization with KVM and considerations that it may be more flexible.

I like Qubes, but sometimes Xen is frustrating. I want to be able to run multiple levels of virtualization at times and can’t do that easily in Xen. Xen also requiring a dedicated GPU if attaching a GPU for AI functionality, something not required by KVM, is also annoying.

Is there really that much of a security benefit of Xen over KVM and specifically Qubes over Proxmox?

Any people who have used Qubes and Proxmox?

2 Likes
2

Something a bit similar was discussed here What do you think SmartOS as lightweight version of Qubes os?

1 Like
3

I’m running Proxmox on several PCs with a bunch of VMs and Linux Containers… Then I have a slim Notebook with Windows 10 and a big Notebook (with 64Gig RAM) for Qubes… Qubes serves a whole different purpose imho with a lot of unique features like disposable qubes, net qubes, the whole desktop with windows of different qubes… In my dreams qubes would use KVM as Hypervisor like it is in Proxmox with great passthrough and nested virtualization… I think both are absolutely great Projects with diferent objective and none can replace the other…

4 Likes
4

I wish Qubes ran KVM at this point too.

It would be so much easier. I don’t know if Xen is much better in security, but KVM is so much easier than Xen.

1 Like
5

well, the only thing I ask for those who prefer nested virtualization - how would you detect one more nesting introduced by some apt rootkit with ProxMox?

3 Likes
6

A bit dated, but the main part still holds true:

5 Likes
7

it would be easy to detect

just wait for evidence of an account breach implying there’s been data exfiltration

1 Like
8

I love reading anything from Joanna! She’s so smart! omg. I feel so privileged to read that.

3 Likes
9

why we should detect it?

2 Likes
10

Well. Imagine you have a job with data marked as state-secret, or even simpler - only commercially secret. Say you have got the job and you also paid for data leak from that job. One of the possibilities to get out of restrictions made by admin stuff with operating system restrictions is execute the code, that will pop out of the current OS, w/o crashing it. The blue pill (AFAIR Joanna marked their attack tool with that name) did right that with the host OS. Once you execute outside of OS & its virtualizator restrictions you are the god number two there. The god number one is cracked OS and its virtualization software. The user is victim here. You may write a module that will steal almost anything and pass out via any of available channels. So if you’re writing “secure” solution you should have something against being owned by virtualizing your OS with code execued within one of your VMs.

Joanna already told something like this, but with better English. :slight_smile:

3 Likes
11

I need to install Qubes again and file the github issue about the problem I’m having. Thanks for reminding me.

1 Like
12

I think this was being said ironically i.e. we’re talking hypothetically, but if it’s a good rootkit, why would we ever even know? :wink:

1 Like
13

ну я думал считать вм с малварью бай дефолт скомпрометированной, вне зависимости от того как работает вирус, разве нет?

Просто вы почему то выделяете это в отдельный вектор, хоть для его использования злоумышленником nested виртуализации, вм должна быть заражена.

1 Like
14

Switched to proxmox after qubes. I don’t regret anything, I was able to implement most of qubes functionality myself, including seamless windows, netvm, safe clipboard, file sharing. There are also several advantages, as proxmox can use KVM and LXC, you can use for example lightweight Alpine LXC images as virtual routers, for AppVM - KVM. There is also nested virutalization, which allows you to run things like an android emulator inside AppVM. I’m also very happy that now I don’t have to deal with the “guest footprint” (qubes delivers its Python software to each AppVM, which leads to package conflicts when upgrading). Anyway, I have a lot of questions about the strange decisions made by the Qubes developers, from choosing Xen as hypervisor to creating templates for VMs.

image
image1440×810 224 KB

image
image1440×810 196 KB

2 Likes
15

Some key differences (that to my knowledge still hold true since the early 2000s):

  • LOC: linux kernel is about 10x LOC (currently 2.000.000 LOC vs Xen 200.000 LOC), which means a larger attack surface.
  • service isolation: The microkernel design of Xen inherently reduces the attack surface by isolating essential services and running them in separate, unprivileged domains. This isolation – inspired by the arsenic project[1] – enhances fault tolerance and security because a breach in one service domain is less likely to compromise the entire system. By keeping the hypervisor minimal and delegating most system services to separate drivers or extensions, Xen reduces the risk of vulnerabilities within the hypervisor impacting the broader system. This design inherently limits the attack surface, enhancing overall security.
  • security features from the ground up: Xen incorporated robust security features[2] from the start, which further isolate guest VMs from one another, preventing potential cross-VM attacks. They were an explicit design goal.
  • robust processes: Xen’s security process is highly robust, featuring a pre-disclosure list, a clear method for reporting security issues, and a dedicated security response team, making it one of the best security models in the industry. This meticulous security framework is far superior to that of Linux, which runs KVM.
  • inter-VM communication: In Xen, inter-VM communication is mediated by the hypervisor, which acts as an arbiter between VMs that do not trust each other.

I don’t know that much about current kernel development, but AFAIK Proxmox is built on a regular Linux kernel, a monolithic kernel that integrates all system services and drivers directly into the core operating system. That means a way bigger TCB. Furthermore it runs KSM by default[3], though it could be disabled (and probably should for Qubes-like duties).

Concerning templates[4] in QubesOS: They are elegant and efficient at the same time. What’s not to like?

  1. https://www.cl.cam.ac.uk/research/srg/netos/projects/archive/arsenic/gige.ps (PS file) ↩︎

  2. https://www.cl.cam.ac.uk/research/srg/netos/papers/2003-xensosp.pdf (PDF file) ↩︎

  3. Kernel Samepage Merging (KSM) - Proxmox VE ↩︎

  4. Templates | Qubes OS ↩︎

6 Likes
16

Yes I realize that the Xen hypervisor can be more compact, but to me an operating system should perform the range of tasks assigned to it, here we face a balance between security and convenience. If you take my statement to the point of absurdity, you can get the following: “A brick is perfectly safe and secure, but you can’t do anything with it”. I believe that an operating system should first and foremost fulfill the user’s tasks, which Qubes does not do well. Now let’s move on to a set of theses why I don’t like Qubes, and why I still can’t consider the solution that the developers use reliable.

  1. Well it obviously doesn’t cover my tasks, I work as a pentester, and Qubes doesn’t do well with things like Android emulation and networking.
  2. Qubes only relies on the hypervisor. Above you compared Xen and KVM, but in my solution there is physical isolation of two different devices as Proxmox is physically located on a different device.
  3. Qubes templates are terribly done, you can’t just put “guest software” in any VM and make it work. Which makes it multiples more difficult to work with it. Users are still waiting for a lot of templates to be created, such as for example NixOS templates. “Guest software” is highly dispersed on the system, which as mentioned above makes it difficult to create templates. And also complicates the work of the users themselves. For example Qubes configures iptables/nftables in each qube.
  4. Implicit behavior. And users have to guess why the network between qubes is not working. Also do you know how NetVM works in Qubes? Why can’t I as a user choose which interface to pass traffic through myself? Why are all qube’s on the same subnet 10.137.0.0/24? Why is the Qubes Firewall GUI missing some settings, including the UDP setting? Why is the size of /tmp tmpfs 1GB? Why does checking for updates with “update via tor” enabled check for updates without using tor? Why do I have to perform updates through the built-in update tool in Qubes?
  5. Qubes thinks that security is when you have an encrypted disk, I don’t think it’s safe, I don’t know how it is in Europe, but in CIS they will stick a soldering iron up your ass until you tell them the password.
  6. Xen hypervisor is not a panacea (it is written in C which is unsafe memory language), many Qubes users switch to Libreboot/Heads and their analogs to protect themselves from hardware bookmarks in the processor. Which to me does not guarantee them any protection. You could take a better approach here.
  7. No nested virtualization and GPU emulation. Some software just doesn’t work in Qubes.

Anyway, these are the main things I don’t like about Qubes. Unfortunately I consider the approach of relying on the Xen hypervisor to be dead by default. So I think I can design/build a solution that provides more reliability and security. I really tried to use Qubes in my work, but it’s painful, also i developed a tool which IMO Qubes developers have to develop. GitHub - r3t4k3r/qubes-forward-gui: Qt GUI for simple port forwarding. In my practice, I’ve spent too much time fighting Qubes behavior instead of doing my job (I don’t want to spend that much time on system maintenance)

1 Like
17

I understand. It depends on what a certain user expects it to do. I’m working in security, too, and it perfectly fits my needs. (Though I wouldn’t setup some vulnhub scenarios on it, either.)

That’s exactly what Qubes solved for me: carrying two to three laptops with me. Do you transfer data between the instances?

I still don’t understand. From a sys admin perspective it’s very nice: there is a single source, that gets updates. System parts of appVMs are ro or disposable. Data parts are mounted in. For inter-VM-communication there are special Qubes tools:

Concerning Nix:

I agree on the networking part in general. It’s a bit puzzling (to say the least). But for a start (even with some links on historical decisions):

Nothing could save you from that. It couldn’t be different with any other OS.

While that’s not about Xen or even Hypervisors in general … I’m using heads myself. Apart from AEM hardening it’s the nicer boot environment. Did you try it? What do you suggest?

I’m curious to see (and possibly try) it.

3 Likes
18

You don’t have to carry it with you, in my case the server part of Proxmox is hosted by my friend from work, I connect to it from my lightweight laptop

The update mechanisms in different linux distributions are different, I wouldn’t use a tool to merge them. For example, updates in nix/arch/debian obviously work in different ways. Also, I don’t like the implicit update checking behavior. Even in case you turn off updates in Qubes, it will still notify you when you start AppVM

qubes-core-agent-linux

As you can see the documentation for this tool is very scarce. As a user, I just want to put one or two programs in AppVM and have it work.

I actually don’t really like the idea of AppVMs as such. The core idea of Qubes is that your data is functionally separated—conditionally, one VM is for storage, another for messengers. In this case, the software in “Storage” and the software in “Messengers” is different. With this approach, using AppVMs only creates difficulties. To install new software in an AppVM and keep it after a reboot, you have two options:

  1. Install the software in the template VM and reboot the AppVM (which disrupts your entire workflow).
  2. Install the software in both the template VM and the AppVM, which results in downloading and installing the same package twice.

It’s clear that AppVMs should be used in cases where you need to maintain multiple copies of a single VM and update them together. But in my case, once again, I tried to use AppVMs as little as possible because of the reasons described above.

That’s not true. You can use tools like VeraCrypt (VeraCrypt - Free Open source disk encryption with strong security for the Paranoid), which allows:

  1. Using a fake OS in case someone forces you to reveal the password.
  2. Making it impossible to prove that there’s another OS on the PC besides the one for which you provided the password above.

I suggest using a physical killswitch router. Essentially, you create a NetVM that wraps all your traffic in a proxy, for example, socks5://user:pass@10.13.37.1:31337. You use this NetVM for all qubes that need internet access. On the router, you use a killswitch that blocks all traffic except for tcp://10.13.37.1:31337. This way, even if your processor/OS/hardware has a remote access backdoor, it cannot be used because any data transmission attempts will be blocked by the router. In fact, hardware backdoors become useless, as you know they may have access to network interfaces, so limiting yourself to a virtual router won’t work—only physical isolation will.

19

Thanks for the tip. Plausible deniability is of no use to me, as I have a very different threat model, it seems. Still, I would prefer steganographic approaches (and only for code/data, not for an entire OS). I am no longer involved in this, but I remember about 9 to 10 years ago it was possible to do forensic analysis to detect hidden volumes by entropy patterns (as could sigbit et al. detect LSB pattern in steganocraphic files):

OK. That’s for the IME/HAP part. But that’s what CoreBoot does over here (and it speeds things up considerably). heads is just to prove that nothing has been tampered with. (I agree that even though things are verified, that doesn’t mean they’re secure. It just ensures that my boot environment still contains the bugs it contained before, while the only perfectly secure computer is on it’s way out of the galaxy. Not because it’s technically perfect, but because it’s too much trouble to get Voyager 1 back. Again, we are dealing with useless bricks).

I do use some OpenWRT travel routers (cheap GLi boxes), too, but not as part of a trusted chain. I still distrust the infrastructure (including sys-net, hotel APs, ISP DNSs etc.). Securely initiating connections with an entity you’ve never communicated with before over a network is the killer problem, the elephant in the room, and the mixed metaphor of Internet security protocols (and their problems). I don’t see how there’s a viable way to do this. It seems to me that splitting hardware functions is just moving the problems to other platforms/devices, while increasing the complexity (by adding physical devices with more code to run in a secure way). As far as the conclusions from “programming Satan’s computer”[1] are concerned (rules of thumb for trusted computing on untrusted platforms): It seems to me that the sheer size and scope of the problem today makes the original approach all but impossible. All that’s left is to maintain the barriers for valuable data while assuming you’ll get hacked anyway. And that’s where Qubes does a very good job (for my purposes).

Last but not least: In my business, the network is (by far) not the only entry vector. My main opponent (among others) is my own human condition/stupidity when dealing with complexity. That’s why I love Qubes and it’s UX, because it enforces me to think about data domains and helps me make (more) conscious and deliberate decisions. If I were – as it seems is your case – more bound to certain applications and workflows, things would be certainly different.

  1. https://www.cl.cam.ac.uk/archive/rja14/Papers/satan.pdf (PDF file) ↩︎

1 Like
20

You have a few good points. In a more standard environment it is certainly easier to run VMs that are not supported by qubes suite of linux guest utils or qubes windows tools.

A physical different machine is definitely more secure than a system which shares the same graphic and audio drivers and firmware (running in dom0). dom0 having no network access is a myth, btw, an exploit would just have to have a few more lines of code to establish a tunnel to sys-net, implanting rootkits in both.

Unless the RAT knows the address of your proxy. I assume your browser knows the address of your proxy, doesn’t it? I would consider the browser to be one of the main possible points for foothold.

1 Like