I spent my second month with Qubes working on a better way to support veracrypt volumes. (The first, was with minimal qubes.) I didn’t let the fact that I was Qubes n00b stop me from poking around a lot.
I managed to accomplish this, but since I am a n00b it’s entirely possible that out of ignorance, I did some very clumsy things that could be done more simply another way.
Wen I started using Qubes, I had a VM with sole access to the veracrypt volumes, and my initial plan was to use it to mount the volumes, then work on them there–which of course would have entailed installing all sorts of stuff on that qube. That’s exactly how I worked it on my previous oracle virtualbox machine.
Then, realizing that it’s actually pretty easy to move files from one qube to another, I decided I would use copy-to-vm to move files to the qube where I would work on them. That lets me use one of Qubes’ strengths: compartmentalization.
But it could be problematic if the files are large. So that is a push in the other direction–towards having a bunch of qubes, each specialized with one application, access the place where the volumes are kept.
But I already liked compartmentalization…so I didn’t want a bunch of AppVM accessing those veracrypt volumes…especially when they are on a network and I don’t want those AppVMs accessing any network.
My physical system is connected to WiFi (to get to the internet) and a (hopefully) disconnected-from-the-internet ethernet. Qubes access one or the other, not both. And this is reflected in my color scheme (which doesn’t correlate perfectly with “trust”). Red, orange and yellow qubes connect to Wifi, and purple qubes connect to ethernet. Blue and green are offline linux and windows, respectively. And the veracrypt volumes are out on the Ethernet.
So what I really wanted was “split veracrypt” where the system that uses the volume is connected to it without seeing where the encrypted volume resides or otherwise accessing the location it’s stored.
Well eventually I did get such a thing. And I then worked on making it more sophisticated–and thereby easier to use.
How I’m going to organize this: First I’ll present a verbal outline of what you have to do to get this to work.
Then, I’m going to go through the key command lines. With these, you should be able to open the different VMs that are involved in the process, issue the commands, and eventually have a decrypted volume mounted on the “target” VM.
The next level of sophistication is to use qrexec so you can run all of the commands from one VM, or even create a combined script to run them for you. I’ll talk a little bit about this.
Finally I created a couple of GUIs, very similar to each other. The most important one is one where I can simply select my volume off a list of volumes, press a button for either read-only or read-write access, then wait for a password prompt, and then voila, the volume is decrypted and mounted on the VM on which the GUI was running. (Alternatively, have it run on some other machine. Which brings up the question as to whether it’s more secure to run this from dom0 than it is from the target machine. I welcome security-centered comments on this!)
There is one object: The veracrypt volume. Which for this description, will be a file named volume.vrc. It can be either “locked,” with the contents inaccessible because it’s not decrypted, or “unlocked.” Notice that some of the actors I’m about to describe below may see it as locked, while others see it as unlocked.
[Incidentally, when one starts doing scripts and especialy GUI front ends, it’s handy to have a naming convention of some kind to be able to identify veracrypt volumes, which would otherwise just look like files with noise in them or more accurately, files you don’t know how to read. My convention is the .vrc at the end of the name. If your threat model is such that you don’t want people to be able to see which files are veracrypt volumes, then obviously, don’t do this just because I suggested it; think about it first!]
There are multiple actors here. There is the user, i.e., you or some other carbon unit. But more importantly there are the following qubes, which I will give notional names to. (I don’t actually have qubes with these names; my names are more in keeping with the all-lower-case, use-dashes-for-spaces convention Qubes developers tend to use. You and I will have to replace those names with real names, in command lines I show below.)
Store has direct access to the actual files which are veracrypt volumes.
Decryptor actually decrypts the volumes.
Receiver is the qube that will actually access what’s on the now-decrypted volume; he’s the one that wants to work on something in the volume.
Keys (optional) a qube which manages the veracrypt decryption keys, letting Decryptor have access when the user approves.
Also optionally, you can have a Manager, which issues the commands to the other qubes. That comes into play when you start writing remote scripts. If you’re just doing things at the command prompt, then you’re likely opening a terminal on every qube that’s involved and banging on your keyboard. YOU are the manager in that case. One you start writing scripts, you will be deciding, even if implicitly, which qube or qubes will take on the management job for you. It may still be split among individual qubes, or there could be another dedicated qube to do it…or it could be the Receiver, basically “pulling” veracrypt volumes towards itself. In my case I do both. Linux Receivers do their own management; Windows qubes (or rather, my one windows qube) have need a different VM, a Linux VM, to act as a Manager since Windows won’t run bash scripts.
A question for the security gurus: Is having the Receiver do the management insecure for some reason? If so, would dom0 make a good manager, or would a dedicated Manager qube be sufficient?
OK, we’ve identified the actors. What’s the sequence of events?
(Implicit, unless on a gui front end, in which case it will be explicit) selecting volume.vrc, making sure it’s where Store can see it.
Store needs to mount volume.vrc to a loop; so it becomes /dev/loop1. This is called “Checkout.” I’ll talk about loops a bit in the Local Command line section. But the main point is that once the loop is mounted, not only can Store see it, dom0 can see it. (But, importanly, dom0 hasn’t mounted it.)
The loop needs to be made available to Decryptor–as near as I can tell this must be done by dom0. The Decryptor will see a /dev/xvdi device. This is called “Mounting”
(part a) Decryptor does the heavy lifting. It actually runs veracrypt. It needs to prompt for password or keyfiles and decrypt the loop it has been given. This is called “Unlocking.” Now for the first time files on volume.vrc are accessible, but right at this moment, it’s actually mounted as /media/veracrypt1 on the decryptor (note that the name doesn’t resemble the name of the volume file as seen by Store).
(part b) Decryptor needs to mount the decrypted volume to another loop, and this entails dismounting it from /media/veracrypt1. Doing this again makes it the new loop visible to dom0.
The loop must be made available to Receiver (Attaching). Again it looks like this must be done by dom0.
Receiver mounts the loop. Receiver can then access what’s on the decrypted volume, as if it was part of Receiver’s file system, but it is not limited to the file system space that Reciver has set for it in settings, rather it’s limited by the size of the volume.vrc file. (Another benefit of this is you are not doing read-writes to the SSD you’ve installed QubesOS on.)
The process of relinquishing access works in the opposite direction (and we will count the steps downward): 5-relinquish (done by Reciever), 4-detaching (done by dom0), 3-relock (done by Decrytor), 2-dismount (done by dom0), and finally 1-checkin, done by Store). You will see that there’s one substep of step 3 done in getting the volume onto Receiver, that has no corrsponding undo in step 3 of the reverse process; otherwise it’s pretty logical, the command names will obviously correspond.
With that, it’s time to get down to brass tacks.
LOCAL COMMAND LINE
1 CHECKOUT, done by Store
I’m going to assume that the Store qube has access to the actual file that is the veracrypt volume, and that it’s in /home/user/Volumes. So the full path is /home/user/Volumes/volume.vrc. Now it could be on a thumb drive (which you’ll have to mount) or a NAS (which [wait for it…] you’ll have to mount) or a separate SSD (which is completely different; in this case you’ll have to mount it), or it could just be stored in the Store VM. Which ever one it is, I’m assuming you’ve already taken care of that, and volume.vrc resides in /home/user/Volumes.
[Obviously if your names are different, you make substitutions.]
The Store qube handles all veracrypt volumes. There could be five or ten distinct VMs running that might want to be receivers, with perhaps some of them having two or more decrypted volumes mounted. It makes no difference, there’s one Store and he doesn’t even have to know the name of the Receiver. Or Receivers. [In my case Store does know the name, but only so he can display status in my GUI front end, as to who’s connected to which veracrypt volume.] The only restriction, which seems to be enforced by veracrypt itself, is that ONE user can access a particular decrypted volume at a time.
Essentially, this step is a matter of taking volume.vrc and making a block device out of it. But instead of being set up as a disk or partition (e.g., /dev/xvdi), it will be mounted as a loop. In fact, loops are typically how you turn a file (as opposed to an ssd, thumbdrive or hard drive) into a block device.
Unlike thumbdrives, that automatically get set up as a disk device and just have to be mounted, a file must be explicitly “deviceified.” (Once you do that, IF the file has the right layout, it can then be mounted as a file system. However, a veracrypt volume that is not unlocked is just random-looking bits, so we can’t do that here.) To make the volume.vrc file into a loop device, you use the losetup command.
So to check volume.vrc out, you can type the following:
sudo losetup loop1 /home/user/Volumes/volume.vrc
If you want Receiver to have read only access to the volume, do the following instead:
sudo losetup loop1 /home/user/Volumes/volume.vrc -r
Unless you’ve got things set up a lot differently than I do, you will need the sudo on the command; without it my Store qube claims to have no idea what losetup is.
(If you have multiple veracrypt volumes checked out, then you’ll have loop2, loop3 and so forth; please substitute appropriately. It appears that losetup wants to use a number, though so keep that in mind.)
Once you do this, you will see loop1 on an lsblk listing (with no indication of what it is, but the full listing will give you the size). The lsblk command shows you all block devices (including loops, but also including disks and partitions). Your loop will also appear on losetup’s listing mode, “sudo losetup -l”, where only loops are shown. There, you will see it’s /dev/loop1 and you will see the file that got mounted (/home/user/Volumes/volume.vrc). In both cases (lsblk and losetup -l) you will see an indication that the loop is read only if that’s how you did it.
Also, dom0 can see this loop. The point of this step, in fact, is to ensure that dom0 can see it. More on that in the next step.
If you are undoing this whole process, going in the reverse direction, the last step of that process is:
losetup -d loop1
That dismounts the volume from the loop device, and now it’s just a file, volume.vrc, sitting on Store.
2 MOUNT, done by dom0
Now dom0 can see the loop1 device, that is actually volume.vrc.
And you as the user can see it in two different ways, on dom0.
The first is to issue the qvm-block command in a dom0 terminal, without any parameters.
The command will show a bunch of devices (such as USB devices) but you should see Store:loop1 listed. And it will actually give the path on Store, in this case /home/user/Volumes/volumes.vrc. There is also a “used by” command but this should be blank, for now.
The second way is to look at the qui-devices menu on your desktop. This is the one that has a little picture of an SD card and a usb thumb drive (at least I think that’s what they’re supposed to be), when you bring that menu up it typically lists your thumb drives and so forth. Well, when you do it now, it will list your new loop device in bold, near the top. Store:loop1 - /home/user/Volumes/volume.vrc
IMPORTANT: At this point Decryptor must be up and running. So I have to discuss that, as it could be trivial or not, depending on how you choose to set up Decryptor.
Decryptor could be an ordinary AppVM, or it could be a disposable VM.
You could just have a regular AppVM as decryptor, in which case it can get multiple volumes attached and you will see xvdj, xvdk, and so on. In fact you’ll probably want to do this at first as you learn the process (then later if you go with a DVM, delete that AppVM). If that’s what you’re doing, it might have been running before you even checked out volume.vrc; otherwise you just have to start it at now. Either way, you’re ready to do the actual mount.
But I chose to use a disposable VM (and I think you will want to, as well, once you know what you are doing), so that each of these could handle its own volume, which simplifies the script (it can assume /dev/xvdi). Another advantage is the disposable VM must, however briefly, know your decryption key, and a disposable VM will forget it when shut down–an AppVM won’t. If you’re doing that, then part of dom0’s job at this step is to fire up the disposable VM. I’m going to assume a named disposable VM, i.e., one where you specify the name, rather than one of the random ones (e.g., disp1248) If you do follow the Disposable VM route, you’ll need a disposable VM template. I’m going to assume you know how to set that up (especially since we have GUIs that can do it for you), and assume you’ve named the template “Decryptor-Template”. The command to fire it up is:
qvm-create --class DispVM --template=Decryptor-Template --label purple Decryptor
(Pick whatever color you like, obviously. I went with purple because Store is purple on my system; Store is purple because Store is on the Ethernet.)
That command will start the Decryptor qube, and you’re ready to do the mount.
[Note: if using DVMs you will need different names for different Decryptors, perhaps you could name this DVM as “Decryptor-volume” since it’s decrypting volume.vrc.)
If you’re totally manual you can just use the qui-devices menu to attach the volume to Decryptor (Decryptor has to be running): Just hover over that line on the menu, move right, and a flyout menu listing all running qubes shows up. Select Decryptor and you’ve done Step 2. (In fact, I’d do it this way the very first time since that very first time, you’re trying to prove the concept.)
But, there’s a command line to do this, and if you ever want to automate with scripts or a gui front end, you need to know it. So here it is: In a dom0 terminal:
qvm-block attach Decryptor Store:loop1
Which simply means attach the block device named Store:loop1 to Decryptor.
It’s not necessary, in this step, to specify read-only if you want read only.
Now decryptor will see the thing in /dev/xvdi (or if it’s got multiple things connected to it, it could be /dev/xvdj, etc.). We’re now ready for the next step.
To undo this step, on Dom0:
qvm-block detach Decryptor Store:loop1
And, if Decryptor is a disposable VM, shut it down.
3 UNLOCK, done by Decryptor
Decryptor must have veracrypt command line version installed if you want to automate the process. If one were willing to go with a more manual mode requiring you to open a terminal on Decryptor, then you might be able to run the GUI version of veracrypt on Decryptor. I never tried it. However one key command must still be issued at the command line (because it is not a command to Veracrypt). I don’t know how the GUI version would respond to the results of this command.
I’m going to assume here that there’s one veracrypt volume per Decryptor VM. If you’re not doing it that way, then please allow for the possibility of multiple disk devices being attached to Decryptor at one time.
This step starts with a device mounted to /dev/xvdi. This is the still-encrypted veracrypt volume.vrc file, but the Decryptor has no idea where the thing actually resides; it’s just an object that needs to be mounted.
In this step, veracrypt will decrypt this, then mount it, thereby unlocking it. Then we partially dismount it and make a loop available to dom0–except this loop is the decrypted volume.vrc
The Decryptor must remain up while the decrypted volume is in use; its role is to decrypt/encrypt things on the disk behind the scenes.
Veracrypt’s command line version will decrypt the volume with the following command:
veracrypt ... -k "keyfile" --pim=nnn --non-interactive --stdin --slot=1 /dev/xvdi /media/veracrypt1
veracrypt ... -k "keyfile" --pim=nnn --password="yourpassword" --slot=1 /dev/xvdi /media/veracrypt1
Where I show quotes, you actually need to type the quotes.
The … should be replaced by “
--mount” for read-write, and “
-m ro” for readonly. Note that if you originally checked out the volume as read only, it’s really going to remain readonly, even if you were to specify read-write here. Receiver might even think it can write to the decrypted volume, but no changes made will stick; if you relock and unlock the volume, they’ll be gone. It’s best to match them, though, so the user doesn’t accidentally make changes that will vanish. If Receiver sees a readonly system, it won’t let the user do anything at all to the volume.
If there is no keyfile, then type
-k "" (i.e., explcitly type the empty pair of quotes.)
If you are not using a pim, then use zero, i.e., “
The first form of the command will read your password out of standard input, which means it should be part of a command something like this:
cat passwordfile.txt | [that whole command] or
echo $PASSWORD | [that whole command]
The second form of the command seems to require you to type the password on the command line, which is a very bad idea in a script (it’s almost as bad to put it on the command line if typing manually). Be aware that your command line might end up in a journalctl file somewhere. But it is possible to use that form of the command without actually typing the password; say it’s stored in a file: (I’m assuming, for this example, read-write, no PIM, and no key file, to keep things simple.)
veracrypt --mount -k "" --pim=0 --password="cat password.txt
" --slot=1 /dev/xvdi /media/veracrypt1
cat password.txt part, surrounded by backward ticks (upper leftmost key of a US keyboard) causes the system to execute that command and put its output there, so your password ends up between the quotes.
Now someone might object that storing your passwords in a file is a bad idea, and normally they’d be right–the file would usually be stored in the DVM template somewhere (or worse the TemplateVM that the DVM template is based on). But if the file is being brought in from somewhere else, it’s not so bad.
You have the flexibility to determine how to keep that password (and keyfile, and PIM number, as appropriate) secure. I actually created a rather complex scheme where yet another VM actor, called Keys, prompts the user for the password; Keys then encrypts it and puts it on a RAM disk on Decryptor, which then decrypts it and feeds it to the veracrypt command. The password is thus never saved to disk. That’s a subject worthy of its own post, possibly.
The result of a successful decrypt is:
There is a new device, /dev/mapper/veracrypt1. This will show in lsblk. The ‘1’ matches the slot number in the veracrypt command. If you are doing multiple decrypts on one VM, you’ll need to number other ones with slot 2, and so forth.
Veracrypt created this and will maintain it. But it’s not usable as a drive, so veracrypt automatically mounts it. And if you look, you will see that…
There is also a drive mounted to /media/veracrypt1. This is your veracrypt volume, decrypted. You COULD access it here if you wanted to. Note that this name was given on the command line; you could conceivably mount it anywhwere. However, we do NOT plan to leave things this way for long–so I used the Veracrypt GUI’s default here because I’m used to it. Nonetheless, it’s probably not a bad idea (if doing multiple decrypts on one VM) to make the number match the slot number just to be able to keep things straight.
We do NOT want this decrypted container mounted on this Decryptor VM, we want to mount it on Receiver!
So dismount it.
This is the hack that is key to the whole process. Veracrypt will normally mount decrypted drives wherever it is run, but it’s possible with this hack to mount them on a different VM! (This is not, so far as I know, documented by Veracrypt; I was simply goofing around and said “gee I wonder what happens if I do this?”)
sudo umount /media/veracrypt1
and then make it into a loop:
losetup loop1 /dev/mapper/veracrypt1
Or, for read only, you should do
losetup -r loop1 /dev/mapper/veracrypt1
Again I recommend replacing the ones with some other number if you are dealing with multiple containers in one VM (I don’t because I use the DVM method). This is for consistency so you can tell which things are associated with which container.
[There is one potential exception to the statement that the losetup should be done with a -r for a volume that started out from Store as read only, and that is if it is to be attached to a Windows VM. Windows VM won’t mount it as a drive if it thinks it’s a read-only drive. If you want a read only drive on windows, do NOT specify it here. Windows will mount it, and it will behave like a read-write drive, but no change you make will actually “stick”–once the drive is detached and relocked, the changes will be gone. It’s potentially confusing if you’re doing things on a Windows box and forget that D: is really only read only, so you might just prefer to never mount a read only drive to Windows.]
Now you have a loop that dom0 can see. This should seem eerily familiar! But the key difference is, this loop is decrypted!
To RELOCK, when you are done, first remove the loop:
losetup -d /dev/loop1
The decrypted drive is already dismounted and we like it that way. Veracrypt command line will not fail if it’s not mounted. So:
veracrypt --dismount --slot=1
gets rid of /dev/mapper/veracrypt1.
The volume is now locked again. If you have done this in a disposable VM, you can shut it down now.
4 ATTACH, done by dom0
This is the step where the decrypted volume is given to Receiver. Receiver will simply see /dev/xvdi (or some later letter; the Receiver can have multiple veracrypt volumes attached to it). He’ll have no idea it’s a decrypted volume and that the volume “lives” on Store.
Just like Step 2 you can use qui-devices to do the attach, or the command line in dom0:
qvm-block attach Receiver Decryptor:loop1
There is again, no need to distingush read-only from read write. Because the loop was created at the end of step 3 as read only or read write, the /dev/xvdi block device will be marked the same way, and that will carry through when Receiver actually does the mount.
Dismounting works the opposite way, like before:
qvm-block detach Reciver Decryptor:loop1
5 MOUNT the decrypted drive, done by Receiver
If Receiver is a Windows box, nothing need be done. The loop, once assigned to the machine by dom0 with the qvm-block command, automatically gets mounted as D: drive (or E: drive, etc).
Linux qubes, on the other hand, will see a new entry in lsblk, /dev/xvdi, /dev/xvdj, and so on.
Linux Receivers have to figure out which one of these is the new drive, then mount them.
Of course it’s your choice where to mount them; this is the part that in my scripts, is the most customizable.
There is also one more wrinkle you should be aware of. The mount command will differ according to what file system is on the decrypted drive.
You can determine the file system with:
lsblk -f /dev/xvdi -o FSTYPE -n
(where, obviously you use xvdj or k or whatever, as appropriate)
If the result of this commnd is either vfat, exfat, or ntfs, you must mount like this:
sudo mount -o uid=1000,gid=1000 /dev/xvdi /home/user/your-chosen-mountpoint
Otherwise the disk will mount read-only no matter what (it will look like root is the owner and you, as “user” have no write privilege, and you won’t be able to change that ownership, even with sudo. And even if you can, it’s a pain in the ass to do so).
If it’s any other file system then just do:
sudo mount /dev/xvdi /home/user/your-chosen-mountpoint
If you try to use the -o uid/gid stuff, it will give you a rather nasty error message and refuse to comply.
To undo, i.e., to relinquish the volume, just do
sudo umount /home/user/your-chosen-mountpoint
Though you can actually just rip the drive away by having dom0 do a qvm-block dismount, it’s more graceful to do this first, then do the qvm-block dismount.
COMMAND LINE Conclusions
If you’re still with me you should be able to walk through the process, rather tediously, and mount a veracrypt volume visible on one qube, on a very different qube that can’t see the volume file at all. Once you’ve done that, you will positively want to write scripts, even ones that run on the actors, and perhaps even automate some stuff that has to be done manually–for example, the decryptor script could determine whether /dev/xvdi is readonly from the lsblk command and decide on its own whether to do read-only.
REMOTE SERVICE CALLS
You may want to run these steps from another machine other than the one they take place on. In fact, if you want to use a manager (or to have Receiver take on that role), you must be able to do this.
One way (the only way I know of) is to use qrexec-client-vm. But this requires a fair bit of setup. In fact, I’ve already myself forgotten a lot about how to do it, and I’ll be refreshing my memory as I write this.
Client Side of a Service Call
The general command line structure of qrexec-client-vm is like this. I’m using Store as an example “doer” of the command, and will be discussing the check out.
qrexec-client-vm Store vera.store-checkout client.sh arg1 arg2 arg3...
Store, is of course the VM you want to perform the command…in this case it will be to check out, or check in, a container. The servicename (here vera.store-checkout) specifies this. client.sh is a script on the calling machine (i.e., one where you issue the command). And then there are the command arguments.
Let me discuss the client.sh first, it’s simplest and provides some context for the rest.
During a qrexec-client-vm call, the client.sh script is invoked; it writes things to standard output. The doer (Store) reads things from standard input. Store then executes the command and can write things to its standard output, which client.sh will see on its standard input. Store then returns a status, and client.sh exits. So note: in qrexec-client-vm, all communication between the client and service (other than the return code) are done on standard input and standard output.
I’ve fiddled with this a bit and have never been able to get it to stay open for more interaction between the two systems; no doubt it’s possible some way I haven’t tried. (I am a bit of a n00b after all.) In fact, since for calling checkout, I’m just telling someone else to check out the file, I really don’t need anything back from Store other than its return code; a way to indicate it failed somehow (e.g., maybe I fatfingered and gave it the name of a non-existent veracrypt volume).
I ended up writing so many different client.shs with names specific to the services that did nothing but pass command line parameters to the service, that I eventually just wrote five generic versions of it and was done; each version took a different number of command line parameters.
So my client that takes 3 parameters, I named vera-3-param-client. Where to put it? Well, do you want this stuff in your template, or in your AppVM? You’ll probably figure out where best it goes as you move towards more and more automation. (I settled on /usr/lib/qubes in a templateVM qube, for things installed on templates; for things installed into AppVMs I went with /home/usr/.local/vera-client. Most of my Receivers have things installed in their templates.
This is one case where I will give source code, because this file is so simple:
exec cat >&$SAVED_FD_1 # print result here, not there.
When run by qrexec-client-vm it simply copies three command line arguments to standard output (so that the service on the other VM sees them) and awaits anything on std output (it does pass standard output back, so you could have your service return error messages and the like if you want). Any return code from the service will be returned by this script.
Note that this script doesn’t need to know the name of the service it’s communicating with; that’s handled, apparently, by qrexec-client-vm.
It should be obvious what to do for vera-0-param-client, vera-1-param-client, and so on.
[32000 character limit…this continues in the first comment.]