Standard overlay with upperdir in tmpfs
Updated log and metadata cleanup script. Now all metadata in these directories /etc/lvm/archive/, /etc/lvm/backup/ is removed if the filenames contain these words: ram, zram, test, ephemeral, tmpfs. Also added commands to delete all backups except the last two from /var/lib/qubes/backup
1 Like
I’ve refined the script for deleting logs and metadata and studied how metadata works in dom0. Script now mounts these directories: /var/log , /etc/lvm/archive , /etc/lvm/backup , and /var/lib/qubes/backup into tmpfs (why I previously had issues with this is unknown - perhaps I did it during an active session or had applied some additional mounts beforehand). Mounting directories from /home/user/.local/share is pointless - it would only break appmenu, since we’d merely be removing empty directories from /home/user/.local/share/qubes-appmenus/ (Qubes doesn’t create metadata in this directory for disp-vms). Achieving 100% anti-forensics is impossible, as these two directories write metadata to disk: /etc/libvirt/libxl, /var/lib/qubes and mounting them into tmpfs would break Qubes. Therefore, for 100% anti-forensics, you need to follow this guide and run the entire dom0 on an overlay / zram-disk.
FYI, using e.g. symlink to /dev/null, bind mount, or an unmodifiable file (chattr +i) won’t work either.
Congrats 
Achieving 100% anti-forensics is impossible, as these two directories write metadata to disk:
/etc/libvirt/libxl,/var/lib/qubesand mounting them into tmpfs would break Qubes.
It is libvirtd that writes there and my earlier investigation (when I made the RAM qubes) revealed there was no way to disable its logging completely. If you want to continue, you should perhaps check its services boot and probably have a tmpfs mount before them. If that doesn’t work - see libvirtd source.
1 Like
Forensic-Timestamp Metadata Risk in Qubes DVMS (ZRAM) and Why It Matters
Question
In /home/user/.local/share/applications there are many .desktop launcher files for dom0-visible apps.
When a user starts (for example) a DVM such as anon-whonix-zram created from a zram_pool (amnesic), a related .desktop file may exist such as:
/home/user/.local/share/applications/org.qubes-os.qubes-vm-settings._anon_dwhonix_dzram.desktop
Those launchers contain an Exec= line (and related entries like X-Qubes-*).
For example, a desktop entry like:
org.qubes-os.vm._anon_dwhonix2_dzram.pcmanfm-qt.desktop
may include:
Icon=/home/user/.local/share/qubes-appmenus/anon-whonix2-zram/apps.icons/pcmanfm-qt.pngExec=qvm-run -q -a --service -- anon-whonix2-zram qubes.StartApp+pcmanfm-qt
I don’t fully understand all the Qubes mechanics here, but my understanding is: when you click an app/menu icon, Qubes reads the .desktop file and executes the command in Exec (or uses X-Qubes-DispvmExec depending on disposable mode).
Then the key point: it may not run automatically when the VM boots, but it does get accessed when the user launches the app.
That access can update file metadata timestamps. An attacker conducting forensic analysis could examine and correlate time relationships, including:
- when
.desktoplaunchers were accessed - the same for this .png and others like that → /home/user/.local/share/qubes-appmenus/anon-whonix2-zram/apps.icons/pcmanfm-qt.png
- when the DVM/app was started or stopped (repeated over many sessions)
- correlation with other online activity happening at matching times
If an app like anon-whonix-zram was started and terminated many times, and on the same days and at the same times some online activity occurred within the same time windows—using Tor, and remembering that anon-whonix-zram uses Tor—then it can become useful timing evidence, potentially enabling de-anonymization.
Because of this, I believe timestamps associated with .desktop launchers (and similar files), .png etc… should be prevented from being written in durable storage.
What “file timestamps” mean
File timestamps are date/time values stored by the filesystem that represent different kinds of events that occurred for that file. Mainly:
- mtime (modification time): last time the file content (or relevant file data) changed.
- atime (access time): last time the file was read/accessed.
- ctime (status change time): last time file metadata/status changed.
- birth time (creation time): when the file was created (not always available/maintained depending on the filesystem).
Why clicking the menu can leave forensic traces
If, every time a user clicks an icon to launch an app inside a ZRAM-backed DVM/VM, the system accesses one or more .desktop files, then their timestamps (atime/mtime/ctime) can be updated.
Even if Qubes uses tmpfs-based amnesic logs for command history (or keeps many operational logs ephemeral), the .desktop file metadata may still persist on disk.
An attacker could then collect those timestamps and infer usage patterns.
That is why the mitigation strategy should not be limited only to logs. Any filesystem objects related to VM launching that produce timestamps (including .desktop, icons, and related Qubes menu/appmenu files) should be handled in a way that keeps them ephemeral (e.g., tmpfs/zram/overlayfs), or otherwise prevents timestamp persistence.
If those objects are not made ephemeral, the setup is not guaranteed to be “100% anti-forensics,” and remaining timestamp artifacts can still be dangerous. From an usability standpoint, a fully ZRAM/overlayfs-based approach may be preferable to brittle low-usability methods.
Tails as a reference model (amnestic + usable hybrid)
Tails enables amnesia while still allowing useful persistence via a hybrid approach.
Example: the user may store a Bitcoin wallet in an encrypted persistent folder. Data and logs related to operations in that persistent folder remain there, but other activity outside that encrypted folder remains ephemeral/amnesic.
Qubes would need an approach similar to hybrid amnesia + persistence to match Tails usability.
One referenced path is the “live mode / dom0 in RAM / non-persistent boot / RAM wipe protection … Tails mode hardening … ephemeral encryption” approach from the Qubes forum thread:
However, approaches like that may be difficult to use in practice (usability issues).
Ideal goal (concept)
The ideal approach would be:
- Keep the DVMS in a
zram_pool(amnestic behavior). - Also ensure that directories/files that produce persistent timestamps are placed in ephemeral storage (RAM) or otherwise made non-forensic.
- Avoid any workflow where launcher-related metadata persists on disk and can be correlated.
In that design, Qubes would resemble a Tails-style hybrid!
/home/user/.local/share/ is one example, but other directories and other files can leave traces. Something in the DVM within a zram_pool created by user commands is also necessary to investigate, and all of it needs to be examined!
Yes, these directories /home/user/.local/share/ don’t contain any forensically significant files… except for the creation timestamps of templates and the disp launch. But that’s not the main issue. I tried mounting these directories in tmpfs and wrote a script that updates the app menu in every session. And it works.
Our real problem is these two directories: /etc/libvirt/libxl , /var/lib/qubes. If we mount them in tmpfs (or an overlay), we risk breaking Qubes - and not just DVMs, but all Qubes. It’s very hard to design a mechanism that removes all forensics for DVMs without affecting other Qubes services: new AppVMs, new apps, and other settings.
In theory, we could create an overlay for these directories and discard it after closing disp, but that would be very inconvenient (it’d be easier to run the entire dom0 in an overlay). I will think about it some more - maybe we can create an isolated overlay space just for running disp…
I poked around in dispvm.py and you can add this code for extra anti-forensics:
- This will redirect metadata about DispVM creation to
/run/qubesinstead of/var/lib/qubes/appvms:
- **Used**: The qube is marked as used and may be unpaused/resumed (if
applicable). Only in this phase, GUI applications treat the qube as any
other unnamed disposable and the qube object is returned to the caller if
requested.
"""
# dispvm metadata lives in tmpfs (RAM)
dir_path_prefix = "/run/qubes/dispvm" # add this line
template = qubes.VMProperty(
╭─user ⚔ dom0 │ overlay │ ─
│ ~
╰─[\]❯ ls /run/qubes/dispvm
disp3969 disp6357 disp7678
- This will also make the root ephemerally encrypted for all disps:
if (
"ephemeral" not in self.volume_config[name]
and "ephemeral" in config
):
self.volume_config[name]["ephemeral"] = config[
"ephemeral"
]
# root and volatile with ephemeral encryption #add this lines
if "root" in self.volume_config:
self.volume_config["root"]["rw"] = False
if "volatile" in self.volume_config:
self.volume_config["volatile"]["ephemeral"] = True
[user@disp2701 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
xvda 202:0 1 20G 1 disk
├─xvda1 202:1 1 200M 1 part
├─xvda2 202:2 1 2M 1 part
└─xvda3 202:3 1 19.8G 1 part
└─dmroot 252:0 0 19.8G 0 dm /
xvdb 202:16 1 5G 0 disk /home
/usr/local
/rw
xvdc 202:32 1 12G 0 disk
├─xvdc1 202:33 1 1G 0 part [SWAP]
└─xvdc2 202:34 1 11G 0 part
└─dmroot 252:0 0 19.8G 0 dm /
xvdd 202:48 1 760.7M 1 disk
╭─user ⚔ dom0 │ overlay │ ─
│ ~
╰─[\]❯ qvm-volume info disp2701:volatile
pool varlibqubes
vid appvms/disp2701/volatile
rw True
source
save_on_stop False
snap_on_start False
size 12884901888
usage 12709888
revisions_to_keep 0
ephemeral True
is_outdated False
List of available revisions (for revert): none
After making these changes, you will need to restart qubesd. However, these changes will not persist after qubes-core-admin update.