Docker in /rw vs. /rw/.autorelabel

User Support General
1

I’ve tried to move Docker data to AppVMs both by bind-mounts and by {"data-root": "/rw/docker-data"} in /etc/docker/daemon.json. In both cases, it stops working after reboot:

% docker run -it debian bash 
bash: error while loading shared libraries: /lib/x86_64-linux-gnu/libc.so.6: cannot apply additional memory protection after relocation: Permission denied

According to my investigations, it looks like being caused by SELinux. I’ve fixed it multiple times, but I don’t know the exact steps (might add it later). It was related to changing SELinux contexts.

Further investigations (and one accident with timeout due to qubes-relabel-rw.service) have pointed to /rw/.autorelabel. It looks like due to presence of .autorelabel, the whole /rw is getting changed SELinux labels on every boot. Can I safely remove it? What’s the purpose of /rw/.autorelabel?

2

Two ideas:

  1. Alternatively, you can try using a Debian-based template
  2. the /rw/.autorelabel is needed to fix the SELinux lables after you modify files in /rw, which are then used in other places (for example, under /rw/config/NM-system-connections which is linked under /etc/NetworkManager) - which otherwise will fail.
3

For now, I’ve removed /rw/.autorelabel. Also, I’ve adjusted the labels:

sudo chcon -Rt container_var_lib_t -u system_u /rw/docker-data
sudo chcon -Rt container_ro_file_t /rw/docker-data/overlay2

This seems to have restored most of the permissions. Some volumes still need to be connected with z flag, though.

4

It looks like /rw/.autorelabel is automatically created. So, after every boot, I need to run the two commands above. Not nice, but at least some workaround.

5

Removing /rw/.autorelabel file will actually trigger relabeling on next boot (yeah, it behaves differently than /.autorelabel).

But also, the actual issue is because Fedora’s SELinux policy applies container labels to /var/lib/docker, not the alternative directory. But, if you keep the original directory in the configuration, but move it to /rw using bind-dirs, it will work. See How to make any file persistent (bind-dirs) — Qubes OS Documentation

1 Like
6

Hmm, with bind-dirs, it works, thank you! (Tested with clean template.) I remember having issues even with bind-dirs (I used bind-dirs before data-root), but maybe this is a past issue that has been fixed.

So, maybe it’s time to use /etc/qubes-bind-dirs.d, as this can configure it in TemplateVM.

7

Why does it behave differently?

8

Because it got implemented this way in Relabel / and /rw if needed by DemiMarie · Pull Request #541 · QubesOS/qubes-core-agent-linux · GitHub, ask the author :stuck_out_tongue:

9

@Demi :sob:

10

Well, now, the /rw/bind-dirs/var/lib/docker seems to have been relabeled and I started getting errors:

one-container  | Error relocating /lib/ld-musl-x86_64.so.1: RELRO protection failed: No error information
one-container  | Error relocating /entrypoint.sh: RELRO protection failed: No error information

one-container exited with code 127
other-container      | /bin/sh: error while loading shared libraries: /lib/x86_64-linux-gnu/libc.so.6: cannot apply additional memory protection after relocation: Permission denied
other-container exited with code 127

I fixed it by this command:

sudo chcon -Rt container_var_lib_t -u system_u /rw/bind-dirs/var/lib/docker &&
  sudo chcon -Rt container_ro_file_t /rw/bind-dirs/var/lib/docker

IIUC, the /rw is also getting relabeled when context file is changed. So, I will still get this issue after some updates.

However, I am unable to reproduce this in my testing AppVM (TemplateVM: fedora-42 testing + updates + docker, otherwise vanilla), even removing /rw/.autorelabel doesn’t reproduce it. In my production AppVM (TemplateVM: fedora-42 with many packages and some adjustments), it still occurs. Not sure what’s the important difference.