Guide: Rsync from multiple qubes to one over qrexec

Community Guides
1

Hi everyone,

It took me a while to figure this procedure out properly, so I thought it might be useful also for others.

Big thanks to @unman who pointed me in this direction and who wrote very helpful instructions which are the basis of this procedure.

As I’m still new to QubesOS, I’m also curious about feedback/comments from the qubes community on this guide.

Context
Let’s assume the following qube structure:

  • data1 and data2: Offline storage qubes with large amount of data that changes often and needs to be backed up regularly (there could be many more like those)
  • backupsync: An offline storage qube which is only in charge of collecting the backups and writing them to an external harddrive

The below mechanism lets you efficiently sync the data changes of the data qubes to the backup qube using rsync and qrexec.

Of course, this mechanism is not as secure as the official QubesOS backup mechanism but probably secure enough in specific cases (depending on the actual threat model, of course).

The guide is using the Fedora templates. On Debian, several commands and package names might be different.

Step 1: Build the data templates

  • Make a copy of the Fedora 33 Minimal Template and name it “fedora-33-mini-data”
  • In Dom0: qvm-run -u root fedora-33-mini-data xterm
    • dnf upgrade
    • dnf install rsync-daemon
    • Write the file /etc/rsyncd.conf
port = 873

[syncdata]
path = /home/user/
uid=1000
gid=1000
comment = Read-only data to be synced
read only = yes

Note: uid/gid are needed to ensure that rsyncd has the same permissions as the default user “user” (who has the id 1000)

  • Write the file /etc/qubes-rpc/qubes.Rsync
#!/bin/sh
exec socat STDIO TCP:localhost:873
  • systemctl enable rsyncd
  • Shutdown this template VM

Step 2: Build the appvm “data1”

  • Create a new VM:
    • Name: data1
    • Type: Qube based on template (AppVM)
    • Template: fedora-33-mini-data
    • Networking: None
  • In Dom0 run: qvm-run data1 xterm
    • touch /home/user/testdata1.txt
    • Ensure that the rsync service is running: systemctl status rsyncd

Step 3: Build the appvm “data2”

  • Create a new VM:
    • Name: data2
    • Type: Qube based on template (AppVM)
    • Template: fedora-33-mini-data
    • Networking: None
  • In Dom0 run: qvm-run data2 xterm
    • touch /home/user/testdata2.txt
    • Ensure that the rsync service is running: systemctl status rsyncd

Step 4: Build the “backupsync” template

  • Make a copy of the Fedora 33 Minimal Template and name it “fedora-33-mini-backupsync”
  • In Dom0: qvm-run -u root fedora-33-mini-data xterm
    • dnf upgrade
    • dnf install rsync
      • Note: qubes-core-agent-nautilus, cryptsetup and lvm2 are useful for mounting an external harddrive with encryption based on luks.
    • Now we create the service and socket files and enable the service for data1:
      • Write the file /lib/systemd/system/qubes-rsync-forwarder1@.service
[Unit]
Description=Forward connection to rsync daemon on data1 over qrexec to port 1837

[Service]
ExecStart=/usr/bin/qrexec-client-vm data1 qubes.Rsync
StandardInput=socket
StandardOutput=inherit
  • Write the file /lib/systemd/system/qubes-rsync-forwarder1.socket
[Unit]
Description=Forward connection to rsync daemon on test-server1 over qrexec to port 1837

[Socket]
ListenStream=127.0.0.1:1837
BindToDevice=lo
Accept=true

[Install]
WantedBy=multi-user.target

* `systemctl enable qubes-rsync-forwarder1.socket`
  • Now we create the service and socket files and enable the service for data2:
    • Write the file /lib/systemd/system/qubes-rsync-forwarder2@.service
[Unit]
Description=Forward connection to rsync daemon on data2 over qrexec to port 2837

[Service]
ExecStart=/usr/bin/qrexec-client-vm data2 qubes.Rsync
StandardInput=socket
StandardOutput=inherit
  • Write the file /lib/systemd/system/qubes-rsync-forwarder2.socket
[Unit]
Description=Forward connection to rsync daemon on test-server2 over qrexec to port 2837

[Socket]
ListenStream=127.0.0.1:2837
BindToDevice=lo
Accept=true

[Install]
WantedBy=multi-user.target

* `systemctl enable qubes-rsync-forwarder2.socket`
  • Shutdown this template VM

Step 5: Set the RPC policy in Dom0

  • Allow the RPC-based communication between those VMS by creating this file: /etc/qubes-rpc/policy/qubes.Rsync
backupsync data1 allow
backupsync data2 allow

Step 6: Build the appvm “backupsync”

  • Create a new VM:
    • Name: backupsync
    • Type: Qube based on template (AppVM)
    • Template: fedora-33-mini-backupsync
    • Networking: None

Step 7: Trigger the backup

  • In Dom0 run: qvm-run backupsync xterm
    • Ensure that the data1 rsync service is running: systemctl status qubes-rsync-forwarder1.socket
    • Ensure that the data2 rsync service is running: systemctl status qubes-rsync-forwarder2.socket
    • Mount a luks encrypted external harddrive in this vm
    • Navigate to the directory where this harddrive is mounted and execute rsync to synchronize the data (of course, you can synchronize the data into two different directories):
      • rsync -a --delete --stats --port=1837 localhost:syncdata
      • rsync -a --delete --stats --port=2837 localhost:syncdata

Step 8
Enjoy the feeling of having a fast and reasonably secure backup mechanism for your large data :slightly_smiling_face:

4 Likes
2

Thanks for this helpful guide!

@qpost135’s guide demonstrates some core Qubes principles when it comes to networking and Qubes RPC, if I may, I’ll add a little more info in case it helps in the understanding:

AppVMs with data are running the rsync network daemon on port 873.

The “backupsync” AppVM, which would have an external drive mounted, connects to the rsync daemon of the data AppVMs to pull the data.

But since Qubes purposely makes intra-qube networking difficult, @qpost135 uses Qubes-provided facilities to simulate intra-qube networking with the Qubes RPC framework. This means:

  • The data AppVMs runs the rsync daemon on localhost:873
  • A qubes.Rsync RPC service is created in the data AppVMs that connects the standard input/output [that will come from other qubes], to the rsync daemon on localhost:873.
  • This way another qube will be able to talk to the rsync daemon in the data AppVM as long as dom0 allows it to, based on Qubes RPC policies that were created at /etc/qubes-rpc/policy/qubes.Rsync.
  • But, the rsync client in the backupsync qube needs to speak “IP” and not Qubes RPC.
  • Therefore @qpost135 created systemd listener sockets within the backupsync qube, that, when connected to, would forward input/output to the qubes.Rsync RPC service in the respective data AppVM.
  • Then, the rsync client in backupsync only needs to connect to the right port number in the systemd forwarder to reach the right rsync daemon in the data AppVMs.

Example flow:

  1. backupsync: rsync client: pull data from 127.0.0.1:1837
  2. backupsync: systemd qubes-rsync-forwarder1.socket: any connections to port 1837, I will redirect it to the qubes.Rsync RPC service of the data1 AppVM
  3. dom0: is backupsync qube allowed to call the qubes.Rsync service of the data1 AppVM? Check /etc/qubes-rpc/policy/qubes.Rsync
  4. dom0: backupsync, you’re allowed, here’s a vchan to directly talk to qubes.Rsync in the data1 AppVM
  5. backupsync <–> data1 vchan representing qubes.Rsync: Hey data1 rsync daemon, gimme some data
  6. data1: qubes.Rsync service: New client, let me connect the client to the real rsync daemon on port 873
  7. data1: rsync daemon: Ok, here’s your data backupsync rsync client
2 Likes
3

Using some sync method between VMs for backups is rather overkill from my point of view.

Just attach the disk of the VM that you want to backup to your backup VM (in read-only at best) and run whatever backup tool you like from there. If you like rsync, go with that.

If you don’t want to trust your backup VM, use the Qubes OS official backup method.

qvm-sync should be useful for small subsets of files that need to be shared between VMs which have lots of other files.
Since you usually want to backup all your files though, attaching all of them to another VM is the faster way to go.
Only disadvantage might be that you can’t use your VMs during the backup then.

There’s also [1] for adventurous users, but I never tried it.

[1] GitHub - tasket/wyng-backup: Fast Time Machine-like backups for logical volumes

4

Thanks, @icequbes1! Your explanations are a very good addition to the procedure!

5

Thanks, @tripleh! Those are interesting thoughts!

I assume your comment about attaching the VM disk to the backup VM was referring to this mechanism: How to mount LVM images | Qubes OS

That’s a valid point and for some reason, I didn’t consider it yet. I’ll do some testing in this direction. But as you say, they slight disadvantage is that the VMs can’t be used then at the same time.

With qvm-sync you are referring to this, right? I did have a quick look at it but wasn’t sure on how reliable it is. It felt safer to rely on the well proven rsync tool.

I did have a quick look at Wyng in the past, which is indeed interesting. In the end however I decided against it because I feel somewhat more in control when I can see the actual files on my backup hard drive instead of some abstract data which needs extra steps/tools to be readable. Maybe I’m just too much in love with rsync :wink:

6

It seems to me that being able to use a qube during backup is important

  • I don’t want to have to take qubes down just to make backups.

Personally, I would run things backward from this write up, which has
the advantage of not needing to write separate configs for each qube.
I’d push from the qubes to the backup qube, using 127.0.0.1:837 - this
has the added advantage that 837 is designated for use by the rsync
daemon, so you are doing the right thing™.

7

I was referring to [1], which I had incorrectly memorized as qvm-sync.

Anyway security is also a point for the attach approach I mentioned. If one of your VMs is compromised, it may attempt to compromise your backup VM via some bug in rsync and then jump from there to all other VMs. That’s less likely for VMs that aren’t running.

And yes, I was referring to [2]. I also wrote some code for it at [3].

rsync makes “backup verification” a lot easier indeed.

[1] GitHub - unman/qubes-sync: Simple syncin between qubes over qrexec
[2] How to mount LVM images | Qubes OS
[3] blib/dom0 at 72d2efd6b4180d739fa2b205f2a59bfa8275ab1d · 3hhh/blib · GitHub

8

Unfortunately, I somehow cannot edit my first post.
In case someone finds and wants to use this guide in the future, here are some corrections to it:

This should be /etc/qubes-rpc/qubes.Rsync

Also one important addition: The policy file needs to have the exeuction permission (somehow it did work without it in the past but now it seems to be needed - maybe due to a recent update?).

Command to be executed in “fedora-33-mini-data” as root:
chmod +x /etc/qubes-rpc/qubes.Rsync