Qubes monitoring and forensics

General Discussion
1

Apologies if this is overly broad or devolves immediately - Is there a general high level guide on how to monitor and defend Qubes for Enterprise IT offices?

This is the point where I’m not 100% comfortable yet in trying to promote Qubes to others. There is an (I believe false) view I typically encounter that Qubes is simply security through obscurity. This I can counter explaining the architecture and they will be reassured learning that Qubes is essentially a stripped down Xen and Linux virtual machines. At this point I’m generally asked about how you defend it. The advice here is sound and awesome and i share this of course: Compromise recovery in Qubes OS | Qubes OS

However - this is not enough to an organization if you have a fleet of people running Qubes counting on you. You need to know how your adversary got in, what they did, and can it happen again. An incident responder needs to be able to review the logs, do forensics on the memory if the device hasn’t been restarted, and image the drive.

I basically want to be able to do this myself and describe to others:

Logging - Remote logging I am going to try with Elastic in a AppVM, the necessary pieces are implemented: WORKING: Guide for creating a logging qube (I.E. sys-log) - WIP
IDS - That’s been discussed a lot in forums, Tripwire seems a decent solution.
Drive analysis - A forensic image of the physical drive is usually simple enough. Doing it on a running system is a little trickier and I have to think about.
Memory capture - Here’s the hard part. Analyzing dom0 and the individual VM’s using volatility shuld be straightforward enough with the map files. However, the dump itself is stumping me conceptually. It seems like the options are:

  1. Virtual machine introspection using Xen tools to dump VM’s and dump dom0 separately
    -Not probable, while Xen has virtual machine introspection, Qubes is a minimized version of Xen and may not have the required modules, so you would need to add an unknown number of components to make this functional.
    -Additional wrinkle - Qubes uses it’s own memory manger qmemman so this would probably be a futile direction to take.

  2. Dump everything
    -While you conceivably could, the issue here is that the custom Qubes tool qmemman is doing memory distribution amongst the vm’s and does not appear to have a native dump feature, while native features in dom0 (e.g. free) only show dom0. So while you could analyze dom0 using traditional means, looking into your dump of all the VM’s is an issue to reconstruct.

  3. Dump dom0 using LiME and then dump individual running qubes.
    -This seems achievable, mount a USB drive large enough for the memory and containin LiME. Dump dom0, then dump all the appVM’s with a series of commands from Dom0. This could probably be scripted.

3 Likes
2

“Security through obscurity” is usually defined as something like “the reliance in security engineering on design or implementation secrecy as the main method of providing security to a system or component.”

Considering that all the code created by the Qubes OS Project is open-source on GitHub for everyone in the world to see and download, it’s almost impossible to imagine how that view could be true.

ITL (Invisible Things Lab, the company behind Qubes) specializes in helping enterprise clients do exactly this sort of thing. The devs (most of whom are on both the ITL and Qubes teams) can customize Qubes OS, help deploy it at scale to a client’s specifications, and set up whatever kind of auxiliary systems and processes the client needs. (Such inquiries should be directed to business@qubes-os.org.) If you’re in charge of IT security at an enterprise organization that uses (or wants to use) Qubes, you should absolutely contact them.

[Please note that this is one of the primary ways that ITL earns an income, which in turn is used to fund the Qubes OS Project (without which the project would quickly die). ITL already partially works for free by providing Qubes for free as open-source software. It can’t also provide enterprise support for free, since devs need to eat and usually don’t work for free. Someone has to pay the bills and keep the lights on, or else we lose open-source Qubes too.]

3 Likes
3

I tested and wrote [1] for that purpose a while back.
Also, the volatility3 author very kindly implemented basic support for Qubes VM images upon my request.

On a side-note the logging part [2] was also implemented by me. However it didn’t make it upstream (not sure why, but I usually close my PRs after a year of inactivity by the maintainer), i.e. you’ll have to maintain and use it at your own risk. I haven’t used it in a while and thus won’t maintain it at the moment.

In total, Qubes OS enterprise class usability is still somewhat lacking imho.
Possibly ITL wants to sell it separately? It might be worth to contact them for consulting.

[1] Contents/docs/security/forensics.md at master · Qubes-Community/Contents · GitHub
[2] qubes.Syslog by 3hhh · Pull Request #321 · QubesOS/qubes-core-agent-linux · GitHub

4

This is just what I was hoping to hear about, thank you for your kind replies!