After a decade of use, I'm considering moving on from Qubes

I have been using Qubes as a daily driver for a little over a decade. I love the concept. I wanted a way to keep my work and personal life separate, and being able to create VMs/domains for each worked very well. I also like the ability to spin up a fresh disposable browser. I’ve wanted a client hypervisor ever since XenClient hit the market (and was subsequently abandoned). Qubes seemed like the perfect alternative.

However, I’ve recently been thinking about moving on and trying something else. Qubes is great from a security standpoint, but it’s not the best for performance or developer use. For example, the lack of GPU acceleration really slows things down and limits the ability to use programs that can leverage GPUs. Chromium-based browsers are able to use GPUs to improve performance, and recently, I had trouble launching Joplin because it apparently now uses GPU acceleration. I could just add more RAM and pick a different note taking app, but it’s becoming a frustration for me.

I’ve also never been able to do certain development work because Qubes makes it difficult. Since GPU acceleration isn’t available, I’ve never gotten into AI development. I’d really like to, but I can’t do it with Qubes.

Also, Android development is a massive challenge. Perhaps there’s a way to make it work, but I’ve never been able to use the built-in virtual Android environment in Android Studio. I’ve gotten around this by using USB debugging, but this requires frequently reattaching my device to the target VM. It’s not ideal.

My hope when I started using Qubes was that I could use it to quickly spin up VMs to use for development, but the default network isolation makes this impractical. Yes, there are ways around this, but they are incredibly manual. The Qubes Network Server project seems to make this easier, but I wish this was a default option for the OS.

Other little things are just a pain. I’ve never been able to get Bluetooth headphones to work despite following community guides. Also, my Logitech wireless mouse sends two clicks for every single click, so that makes it unusable. Heck, I can’t even get printing to work right and have to reconfigure printers after every restart. There may be a fix for this, but it shouldn’t require following an online guide just to make printing work.

I really wish Qubes offered some less secure options that could be enabled for users who want the functionality of a client type 1 hypervisor but aren’t quite as concerned with security. I know this isn’t the core focus of Qubes, but it would make the OS a viable option for power users. A lot of developers would probably consider Qubes if it was a little more flexible on this front.

Until then, I’m thinking of going to a vanilla Debian installation with KVM enabled and virt-manager installed. This will allow me to create and manage VMs for development with network access. To emulate separate domains, I’m thinking of rolling my own setup using separate user accounts. By modifying launcher entries under a global desktop user, I’ll launch programs using sudo so that they launch as the correct user but run natively in a single desktop session.

This type of setup won’t be as secure as Qubes, but it provides similar functionality with additional features and flexibility. I don’t know of any other OS that provides something like this, so if you are aware of projects that do, I would really be interested in feedback from other users.

Thanks again for this great project. While it may not fit my personal needs as well as I hoped, it is a valuable platform for those in the security industry. I just wish it was a little more flexible in order to appeal to a broader audience of users.

1 Like

It will absolutely frustrating when you attempt to update all your kvm virtual machines and all your snapshots.Unless your can work out a way to create template vm and app vm,otherwise things will start to get very messy.

1 Like

Here a some one-men Qubes clones (I have no experience with them):

1 Like

I have no problem with BT headphones, for AI, gpu pass thru works and every app qubes can have network access. Accessing qubes is some hurdle only.
With android people make it work, dunno about development. I couldn’t connect my GOS Pixel thou.

I’m not sure if I’m misunderstanding you or you’re misunderstanding me (either is possible, so forgive my poor explanation or lack of understanding). I’m planning to use my base Debian install as my main “domain” and desktop with a user called simply user. Then, I will create separate standard user accounts, say personal and work. Then, I’ll modify the launchers in the application menu of the user account to launch programs from my personal and work accounts via sudo. All of them will open within the desktop of the user account.

In addition, I’ll use KVM and virt-manager to run and manage VMs for various development needs. They won’t be “templates” for anything as far as the system is concerned, so I can just manage them as regular VMs.

Yes, this setup completely obliterates the security of Qubes’ network stack and domain isolation. While I appreciate these features, I care much more about functionality at the end of the day. This configuration will allow me to keep my work and personal files separate while providing me with a standard type 1 VM platform. It will also allow me to use my GPU for whatever I want, Bluetooth headphones, a wireless mouse, and printers without jumping through hoops.

If Qubes could accomplish the things in that last sentence in a turnkey manner, I’d be on board. However, due to the firm security posture of developers (which I fully understand and respect), I doubt any of these things will ever happen.

Boy, if you’ve gotten that stuff to work, good on ya. However, I think it’s frustrating that these things require lots of extra configuration and can’t be easily enabled. Also, I really wish that Qubes was compiled with VirtIO-GPU because per my understanding, that would allow the GPU to be shared between VMs. Yes, I understand the security implications of this, but I wish the option was available and easy enough to enable.

As is, anything outside the norm with Qubes that doesn’t follow strict security standards either requires a lot of extra configuration or is simply not possible. As I said in my other response above, I completely understand and respect this posture. However, I really wish these features were available as turnkey options for those who want them and are willing to forego the security issues.

On a final note, if someone could help me compile Qubes/Xen with VirtIO-GPU enabled so that it can be used by VMs, I might stick around. I tried doing it and posted about it here. I’ve just never compiled a system like this from scratch, so I don’t know where to begin. Furthermore, I don’t know if it would even work. However, it would be nice to be able to take advantage of the powerful card in my fancy laptop.

1 Like

It should work in the qube with USB controller passed through to it, you can try to use it in sys-usb directly:

1 Like

Are you about local AI? You can’t work with it in debian also if you are talking about your notebook GPU, it’s almost useless for AI.

I run (or access) frontend at (from) Qubes and use server with GPU(s) to run models. This is the best way

I tried this, and even went with Void Linux as the base OS, to get rid of systemd. It works, the “KVM + virt-manager” setup is supported and well-documented even on Void. You can then run “immutable” Whonix using snapshots, which is as good a setup as using disposable whonix-workstation qubes.
But in the end, QubesOS offers the convenience and a good balance for my use case. :man_shrugging:

Nah. Connected it to my gaming pc.

If one want to play with local AI it should either go M series mac or pc with at least 2 gfx cards in system: something to display (iGPU is sufficient) and 4090/5090 for LLM

1 Like

Perhaps you should create a dedicated topic for that, because there are many other questions being answered here.

1 Like

My notebook has discreet graphics - it’s not just an iGPU. Yes, I can use it if I set up passthrough for AI, but I wish VirtIO was possible because that makes things a lot easier.

For example, I’d like to let the browser in each of my different domains use GPU acceleration. Even Joplin is now refusing to launch without specifying a --no-sandbox flag because it throws the error GPU process isn't useable. Goodbye. When even a basic note taking app is refusing to run properly, that’s massively frustrating.

Lastly, multiple monitors causes my system to crawl because all of the rendering is being done by the CPU from what I understand. What good is my Dell Precision laptop and docking station if I can’t use it to its full potential by connecting multiple monitors?

I did create a separate topic for VirtIO-GPU that I linked in my post:
https://forum.qubes-os.org/t/building-vmm-xen-with-virtio-gpu/

I also see that Marek made some comments about the CONFIG_XEN_VIRTIO kernel option in a GitHub patch:
https://github.com/QubesOS/qubes-linux-kernel/commit/f6940588e19cd3c207f313108bb62da67609b91c.patch

Perhaps I should start a conversation over on Google Groups?


In conclusion, I suppose what I want to do with Qubes is:

  1. Keep different projects and parts of my life separate and isolated (Qubes does this well).
  2. Use local VMs for development work (Qubes stinks at this because of network isolation, so I don’t even bother).
  3. Be able to spin up disposable VMs when fresh environments are needed (Qubes does this well).
  4. Have everyday functionality available if I want to accept the security risks (Qubes has made progress on this front over the years, but it’s still lagging).

My frustrations with Qubes are:

  1. I can’t utilize my hardware (primarily GPU) to its fullest potential, so simple things like multiple monitors and programs that can use acceleration just crawl.
  2. Everyday functionality that requires accepting decreased security like Bluetooth, a wireless mouse, and printers don’t have obvious turnkey solutions.
  3. Networking between VMs when doing development work is such a pain that it’s almost not worth it.
2 Likes

The Android emulator doesn’t work under Qubes because of the lack of nested vritualization support. Networking should work with custom nftables rules to set up port forwarding, but I agree that that is a pain.

@Demi Is there a GUI that could be added to make networking easier? Something so I can easily allow networking between two qubes without much hassle. Yes, I know this can done manually and there are guides for it, but when you need to do things on the fly, it would be helpful.

2 Likes

It’ll be nice when we have a more official option for GPU acceleration. If your hardware supports SRIOV then you can use that to split up your GPU and pass the hardware to the VM.
Some AMD and NVidia pro/server cards support SR-IOV. 6.18 XE’s driver will have SR-IOV support for Tiger Lake (11th), Alder Lake (12th), Panther Lake (Ultra 3), and Battlemage Arc Pro Cards. The Alder Lake support may also work with Raptor Lake (13th), but I haven’t seen any indication either way.

the intel-linux-lts branch has i915 SRIOV support for 11th-13th gen and Meteor Lake (Ultra 1). I created a fork of the qubes stable kernel repo that uses this as its source: qubes-linux-intel-kernel.

@burnskp My system supports SR-IOV. Are you familiar with VirtIO? Maybe you could help work with me on building the kernel and configuring the guests to make it work.

You don’t need VirtIO with SRIOV. You use SRIOV to split your card into multiple devices and then pass those devices to the VM.

My understanding is that VirtIO allows you to share your GPU to multiple guests as a virtualized device instead of passthrough to a single guest.

I’m not familiar with using VirtIO-GPU with Xen. SR-IOV with Intel allows splitting up the GPU into 7 devices, which is enough for my use case.