AppVM, or at least some of its data, on a USB external drive

User Support
1

I’m looking out for the best solution to resolve this problem : having some of my qube VMs, or at least their data, on an external USB hard drive.

I already managed to follow the Secondary storage documentation page on another machine with an internal HDD. But, as far as I understand, it’s not possible with an external HDD without security risks, because this drive should be attached to dom0.

My current solution is to create a normal AppVM in dom0, attach the corresponding logical volume to the AppVM and link some directories to it. My external HDD is encrypted and contains a LVM thin pool with logical volumes. I’m able to decrypt it in an AppVM dedicated to this task called disks-manager. So, I can attach the drive to my test-automount AppVM on qvm-start:

qvm-start test-automount --hddisk=disks-manager:dm-1

This is my /rw/config/rc.local inside test-automount:

if [ ! -b /dev/xvdi ]; then
    shutdown now
fi

mkdir -p /mnt/vm-external-data
mount /dev/xvdi /mnt/vm-external-data

if [ ! -h /home/user/Music ]; then
        rmdir /home/user/Music
        ln -s /mnt/vm-external-data/home/user/Music /home/user
fi

It’s okay but think I can look for some better solution because I can have a lot of different directories to link … I’m not sure if doing this is safe:

rm -r /home/user
ln -s /mnt/vm-external-data/home/user /home/user

Of course, I tried to play with bind-dirs and /etc/fstab. This is the content of my /etc/fstab:

/dev/mapper/dmroot          /                       ext4 defaults,discard,noatime        1 1
/dev/xvdb                   /rw                     auto    noauto,defaults,discard,nosuid,nodev    1 2
/dev/xvdi                   /mnt/vm-external-data   auto    defaults,discard,nosuid,nodev   0 2
/mnt/vm-external-data/home  /home                   none    noauto,bind,defaults,nosuid,nodev 0 0
/rw/usrlocal                /usr/local              none    noauto,bind,defaults 0 0
# ... [default AppVM lines here] ...
/dev/xvdj                   /mnt/removable          auto noauto,user,rw 0 0

I only edited the 3rd, 4th and last line, but without success. findmnt doesn’t output errors or warnings and mount -a ignore everything I added in /etc/fstab …

There is a lot of topics about this on this forum but I can’t find a good solution. Is there something I’m missing ? Any ideas to improve my rc.local, get fstab to work or anything else ?

2

You can attach block device to qube persistently:
qvm-block attach --persistent test-automount disks-manager:dm-1
And then start test-automount using GUI.

You can use mount --bind instead of ln -s.

Remove noauto then it’ll mount with mount -a.

2 Likes
3

qvm-block attach --persistent test-automount disks-manager:dm-1 is a really good solution to part of my problem ! Thank you very much !

Now, to mount the drive as /dev/xvdb, I’ve discovered this file: /etc/libvirt/libxl/test-automount.xml I’m trying to edit it, via the jinja2 template in /usr/share/qubes/templates/libvirt/xen/by-name/test-automount.xml. I’m try to play with this but I’ve got some security concerns about this …

Any idea of a way to force running a shell script in dom0 or the disks-manager vm startup when starting the test-automount vm ?

4

You can use libxl hook:

5

Thank you very much for your help. I’m still testing an libxl hook script of my own, but my problem is very different from the solution given in your link: I want to start a VM when starting an other, waiting for it to do something, a bit like the Net qube requirement. Even with the prepare libvirt operation, it fails because the persistent block device is not attached.

I need more time to test this … maybe I can remove the persistent drive and do the attachment in the libxl hook … Thank you again because you give me a lot of hints.

1 Like
6

I could only think up a dirty solution like this (not tested):

#!/bin/bash
guest_name="$1"
libvirt_operation="$2"
timeout=60
guest_to_start="test-automount"
guest_dm="disks-manager"
dm_block_dev="dm-1"

if [ "$guest_name" = "$guest_to_start" ] && [ "$libvirt_operation" = "prepare" ]; then
    (
        exec 0</dev/null
        exec 1>/dev/null
        exec 2>/dev/null
        for i in $(seq 1 $timeout);
        do
            if qvm-ls --running $guest_dm | grep -q Running; then
                if qvm-block list -q $guest_dm | grep -q "$guest_dm:$dm_block_dev"; then
                    qvm-run -q $guest_to_start xfce4-terminal
                    break
                fi
            fi
            sleep 1
        done
    ) & disown
fi

It’ll show you an error about failing because of missing block device and you’ll have to hardcode which app to run on qube start.

7

Sorry, I didn’t post mine. I’ve tried different things and the logic is a bit different inside the forloop… But it’s painful to test, I created another VM to quickly test drive management with a loop device (loop1), I can get it to work if I don’t use persistent attachment.

I even tried to remove the (...) &disown part to “block” the starting process, waiting for the manager VM to start but it seems to break everything.

This is my current script:

#!/bin/bash

# Assure that a drive is unlocked on a disk manager vm before
# starting an automount vm
guest_name="$1"
libvirt_operation="$2"

manager_name="test-dm-automount"
drive_name="loop1"
timeout_unit=15 # seconds
timeout_cycles=8 # ±2min

unlock_user=$( id -un 1000 )
unlock="/home/$unlock_user/scripts/test_automount_unlock_disks.sh"

if [ "$guest_name" = "test-to-automount" ] && [ "$libvirt_operation" = "prepare" ]; then
    (
        exec 0</dev/null
        exec 1>/dev/null
        exec 2>/dev/null

        manager_started=0
        for i in $(seq 1 "$timeout_cycles");
        do
            if ! $( qvm-block list -q "$manager_name" | grep -q "$drive_name" ); then
                if [ $manager_started = 0 ]; then
                    manager_started=1;
                    source $unlock # attach the drive, decrypt with key and scan disk
                fi
                sleep "$timeout_unit"
            else
                break
            fi
        done

        if ! qvm-block list -q "$manager_name" | grep -q "$drive_name"; then
            echo "Timeout when waiting for $drive_name in $manager_name" >&2
            exit 1
        fi
    ) & disown
fi

I need to spend more time on this, then I may have questions :slight_smile:

8

Another way would be to start test-automount without persistent attachment and in libxl hook start disks-manager on test-automount start. In disks-manager configure to either automatically decrypt the drive on start or do it manually before libxl script timeout. Then in libxl hook wait for disks-manager:dm-1 block device to appear and attach it to test-automount. In test-automount configure udev rules to automatically mount the attached block device when it appears.

9

Yep but with this solution I need to forget about using /etc/fstab to replace the /home mountpoint …

Maybe I can’t get exactly what I want, but thanks again for your help !

10

You can try to automount using systemd:

I think it should replace /home mountpoint without a problem.

11

I’ve got the impression that a first version of /etc/fstab exist before launching the bind-dirs version … So systemd won’t help much I think. I’ve spend so much time on this, I think I will stick to my first solution with this /rw/config/rc.local file:

#!/bin/sh

# maybe quite useless with persistent attachment ...
if [ ! -b /dev/xvdi ]; then
    shutdown now
fi

external_data="/mnt/data"

mkdir -p "$external_data"
mount /dev/xvdi "$external_data"

links_list="${external_data}/.qube_external_data_links.lst"

while IFS= read -r dir_name; do
	dir_path="/home/user/$dir_name"
	if [ ! -h "$dir_path" ]; then
		if [ -d "$dir_path" ]; then
			rmdir "$dir_path"
		fi
		
		echo "ln -s "${external_data}$dir_path" $( dirname "$dir_path" )"
		ln -s "${external_data}$dir_path" $( dirname "$dir_path" )
	fi
done < "$links_list"

I store a simple list of files to link in /mnt/data/.qube_external_data_links.lst, like:

Downloads
Music
...

It’s not perfect but after all it’s simple enough. There is only two files to edit. I’m not sure about the pros or cons of using mount --bind instead of ln -s…

Edit: now i’m thinking about my backup process and maybe, mount --bind is a better solution…

12

Yes, the system starts with your templates version of /etc/fstab, but when /rw/config/rc.local is run the bind-dirs version is used so you need to run systemctl daemon-reload in /rw/config/rc.local to load new fstab:

13

A few alternatives to the previously suggested solutions:

14

Sorry, I should have paid more attention to jour previous link. Now I get it, I will try this.

Yes, the qcrypt tool is something I came across while searching and it seems quite interesting BUT I don’t want to install something that I don’t fully understand in dom0.

15

So, I had this post as a draft for a while now and this is where I am : I’m focusing on trying to mount (or “bind” ?) /home on my external drive mounted in /mnt/data. I have a fresh test-fstab vm, I added /etc/fstab to bind dirs, rebooted the vm and changed the file content to:

# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/dmroot /                       ext4 defaults,discard,noatime        1 1
/dev/xvdb               /rw                     auto    noauto,defaults,discard,nosuid,nodev    1 2
/dev/xvdi               /mnt/data               auto    defaults,discard,nosuid,nodev   1 2
/mnt/data/home        /home                     none    bind,defaults,nosuid,nodev 0 0
/rw/usrlocal        /usr/local                  none    noauto,bind,defaults 0 0
/dev/xvdc1      swap                            swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults,size=1G        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
#/dev/xvdj      /mnt/removable  auto noauto,user,rw 0 0

And then, I added this to /rw/config/rc.local:

mkdir -p /mnt/data
systemctl daemon-reload
mount -a

This is the result of lsblk after a reboot:

NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
xvda    202:0    1    10G  0 disk 
├─xvda1 202:1    1   200M  0 part 
├─xvda2 202:2    1     2M  0 part 
└─xvda3 202:3    1   9.8G  0 part /
xvdb    202:16   1     2G  0 disk /etc/fstab
                                  /usr/local
                                  /home
                                  /rw
xvdc    202:32   1    10G  0 disk 
├─xvdc1 202:33   1     1G  0 part [SWAP]
└─xvdc3 202:35   1     9G  0 part 
xvdd    202:48   1 526.5M  1 disk 
xvdi    202:128  1   100M  0 disk /rw/home
                                  /home
                                  /mnt/data

And this is the result of findmnt:

TARGET                                                  SOURCE                          FSTYPE          OPTIONS
/                                                       /dev/mapper/dmroot              ext4            rw,noatime,discard
├─/proc                                                 proc                            proc            rw,relatime
│ └─/proc/sys/fs/binfmt_misc                            systemd-1                       autofs          rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13616
├─/sys                                                  sysfs                           sysfs           rw,relatime
│ ├─/sys/kernel/security                                securityfs                      securityfs      rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup                                      cgroup2                         cgroup2         rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot
│ ├─/sys/fs/pstore                                      pstore                          pstore          rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/bpf                                         bpf                             bpf             rw,nosuid,nodev,noexec,relatime,mode=700
│ ├─/sys/kernel/debug                                   debugfs                         debugfs         rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/tracing                                 tracefs                         tracefs         rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/config                                  configfs                        configfs        rw,nosuid,nodev,noexec,relatime
│ └─/sys/fs/fuse/connections                            fusectl                         fusectl         rw,nosuid,nodev,noexec,relatime
├─/dev                                                  devtmpfs                        devtmpfs        rw,nosuid,size=4096k,nr_inodes=492908,mode=755,inode64
│ ├─/dev/shm                                            tmpfs                           tmpfs           rw,size=1048576k,nr_inodes=35348,inode64
│ ├─/dev/pts                                            devpts                          devpts          rw,relatime,gid=5,mode=620,ptmxmode=000
│ ├─/dev/hugepages                                      hugetlbfs                       hugetlbfs       rw,relatime,pagesize=2M
│ └─/dev/mqueue                                         mqueue                          mqueue          rw,nosuid,nodev,noexec,relatime
├─/usr/lib/modules                                      none                            overlay         rw,relatime,lowerdir=/tmp/modules,upperdir=/sysroot/lib/modules,workdir=/sysroot/lib/.modules_work
├─/run                                                  tmpfs                           tmpfs           rw,nosuid,nodev,size=56560k,nr_inodes=819200,mode=755,inode64
│ ├─/run/credentials/systemd-tmpfiles-setup-dev.service ramfs                           ramfs           ro,nosuid,nodev,noexec,relatime,mode=700
│ ├─/run/credentials/systemd-sysctl.service             ramfs                           ramfs           ro,nosuid,nodev,noexec,relatime,mode=700
│ ├─/run/credentials/systemd-tmpfiles-setup.service     ramfs                           ramfs           ro,nosuid,nodev,noexec,relatime,mode=700
│ ├─/run/user/1000                                      tmpfs                           tmpfs           rw,nosuid,nodev,relatime,size=28276k,nr_inodes=7069,mode=700,uid=1000,gid=1000,inode64
│ │ └─/run/user/1000/gvfs                               gvfsd-fuse                      fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000
│ └─/run/credentials/systemd-resolved.service           ramfs                           ramfs           ro,nosuid,nodev,noexec,relatime,mode=700
├─/tmp                                                  tmpfs                           tmpfs           rw,size=1048576k,nr_inodes=35348,inode64
├─/rw                                                   /dev/xvdb                       ext4            rw,nosuid,nodev,relatime,discard
│ └─/rw/home                                            /dev/xvdi[/home]                ext4            rw,nosuid,nodev,relatime,discard
├─/home                                                 /dev/xvdb[/home]                ext4            rw,nosuid,nodev,relatime,discard
│ └─/home                                               /dev/xvdi[/home]                ext4            rw,nosuid,nodev,relatime,discard
├─/usr/local                                            /dev/xvdb[/usrlocal]            ext4            rw,nosuid,nodev,relatime,discard
├─/etc/fstab                                            /dev/xvdb[/bind-dirs/etc/fstab] ext4            rw,nosuid,nodev,relatime,discard
└─/mnt/data                                             /dev/xvdi                       ext4            rw,nosuid,nodev,relatime,discard

Seems quite good BUT … I don’t understand why /rw/home is mounted on /mnt/data/home ?

16

On startup the system will mount all entries from default fstab from your template.
That means it’ll mount:

/dev/xvdb		/rw			auto	noauto,defaults,discard,nosuid,nodev	1 2
/rw/home        /home       none    noauto,bind,defaults,nosuid,nodev 0 0

Then your /rw/config/rc.local script will change fstab and mount all entries from there but it won’t unmount all entries that were previously mounted from old fstab but are missing from it now, it’ll only mount the new ones.
So with this new entry in fstab:

/mnt/data/home        /home                     none    bind,defaults,nosuid,nodev 0 0

It’ll mount /mnt/data/home over /home which is currently a mount from /rw/home.
You can unmount the /home before mount -a if you want but it doesn’t really matter.

17

Once again, thank you for your help ! I consider this problem solved now.

18

This solution is not working with Qubes 4.3 anymore (because of the new custom-persist feature).