Verifying installation

General Discussion
1

Not sure if the category is good, feel free to reassign where it belongs.

Maintaining OEM disk image, which is standard ISO installation + additional packages + updates + tweaks to provide up to date templates, is not an easy task.
Doing so comes with a lot of valid questions from end users on integrity, and providing proof/auditability is also not so easy, but interesting in itself and a filed of needed improvements.

I’m posting here to try to get an understanding of what touches what after deployment. From Qubes, I understand that sometimes, its done post-install by salt recipes, but I cannot really get my head over some of the changes, so here i’m posting the whole list of files changed after package installation to launch discussion.

dom0 (Fedora) currently relies on RPM databases. RPM provides some interesting tools.
Installed package list: rpm -qa
Integrity validation checks of package files: rpm -V package_name.
To see if a file was deployed by a package and which package: rpm -q --whatprovides /path/to/file

rpm -V will report a ‘5’ (..5...... ) when integrity digest of a certain file is different from what is expected in rpm database.

With the above commands, we can do some interesting checks.

The following takes the list of packages installed, verifies integrity against digest and keep the filenames that were reported to differ from digests.
It then take each of the individual files reported to be different from the digest (ownership, timestamp etc combined) and filter the output only to keep checksum differences (“5”, not optimal, I know, and then needs to be curated).
It then keeps in bad_check the package names that contained files that were modified (checksum different).

[user@dom0 ~]$ bad_check=$(rpm -qa | while read package; do sudo rpm -V $package; done | grep "5"| grep -o '/[^"]*'| while read file; do sudo rpm -q --whatprovides $file; done| uniq|sort)

The following package list contains packages that also contained files having a “5” in their filenames.

[user@dom0 ~]$ echo $bad_check 
anaconda-core-32.24.5-5.fc32.x86_64 anaconda-gui-32.24.5-5.fc32.x86_64 dnf-data-4.6.0-1.fc32.noarch kernel-5.10.104-3.fc32.qubes.x86_64 kernel-5.10.109-1.fc32.qubes.x86_64 kernel-5.10.112-1.fc32.qubes.x86_64 kernel-qubes-vm-5.10.104-3.fc32.qubes.x86_64 kernel-qubes-vm-5.10.109-1.fc32.qubes.x86_64 kernel-qubes-vm-5.10.112-1.fc32.qubes.x86_64 libxfce4ui-4.14.1-4.fc32.x86_64 plymouth-0.9.4-14.20200325gite31c81f.fc32.x86_64 python3-blivet-3.2.1-2.fc32.noarch python3-kickstart-3.24-2.fc32.noarch python3-libvirt-6.6.0-5.fc32.x86_64 python3-qubesadmin-4.1.22-1.fc32.noarch qubes-audio-dom0-4.1.21-1.fc32.x86_64 qubes-core-dom0-4.1.26-1.fc32.noarch qubes-core-qrexec-4.1.18-1.fc32.x86_64 qubes-input-proxy-1.0.26-1.fc32.x86_64 qubes-mgmt-salt-admin-tools-4.1.13-1.fc32.noarch qubes-usb-proxy-dom0-1.1.1-1.fc32.noarch selinux-policy-minimum-3.14.5-46.fc32.noarch xen-runtime-4.14.4-4.fc32.x86_64 xfce4-panel-4.14.4-1.fc32.x86_64 xfce4-power-manager-1.6.6-1.fc32.x86_64 xfce4-session-4.14.2-1.fc32.x86_64 xfce4-settings-4.14.3-1.fc32.x86_64 xscreensaver-base-5.45-5.fc32.x86_64

The curated package list (without kernels, which contained filenames containing “5”):
sudo rpm -V anaconda-core-32.24.5-5.fc32.x86_64 anaconda-gui-32.24.5-5.fc32.x86_64 dnf-data-4.6.0-1.fc32.noarch libxfce4ui-4.14.1-4.fc32.x86_64 plymouth-0.9.4-14.20200325gite31c81f.fc32.x86_64 python3-blivet-3.2.1-2.fc32.noarch python3-kickstart-3.24-2.fc32.noarch python3-libvirt-6.6.0-5.fc32.x86_64 python3-qubesadmin-4.1.22-1.fc32.noarch qubes-audio-dom0-4.1.21-1.fc32.x86_64 qubes-core-dom0-4.1.26-1.fc32.noarch qubes-core-qrexec-4.1.18-1.fc32.x86_64 qubes-input-proxy-1.0.26-1.fc32.x86_64 qubes-mgmt-salt-admin-tools-4.1.13-1.fc32.noarch qubes-usb-proxy-dom0-1.1.1-1.fc32.noarch selinux-policy-minimum-3.14.5-46.fc32.noarch xen-runtime-4.14.4-4.fc32.x86_64 xfce4-panel-4.14.4-1.fc32.x86_64 xfce4-power-manager-1.6.6-1.fc32.x86_64 xfce4-session-4.14.2-1.fc32.x86_64 xfce4-settings-4.14.3-1.fc32.x86_64 xscreensaver-base-5.45-5.fc32.x86_64

provides the following output:

..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/__pycache__/network.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/core/__pycache__/constants.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/core/__pycache__/users.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/core/__pycache__/util.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/core/configuration/__pycache__/storage.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/modules/common/structures/__pycache__/partitioning.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/modules/network/__pycache__/network.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/payload/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/payload/__pycache__/utils.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/storage/__pycache__/partspec.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/ui/lib/__pycache__/space.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/ui/gui/spokes/__pycache__/datetime_spoke.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/ui/gui/spokes/__pycache__/root_password.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/ui/gui/spokes/__pycache__/user.cpython-38.pyc
S.5....T.  c /etc/dnf/dnf.conf
S.5....T.  c /etc/xdg/xfce4/xfconf/xfce-perchannel-xml/xfce4-keyboard-shortcuts.xml
S.5....T.  c /etc/plymouth/plymouthd.conf
.M.......  g /var/lib/plymouth/boot-duration
..5....T.    /usr/lib/python3.8/site-packages/blivet/devices/__pycache__/lvm.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/pykickstart/commands/__pycache__/partition.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/pykickstart/commands/__pycache__/repo.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/__pycache__/libvirt.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/__pycache__/libvirtaio.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/app.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/base.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/config.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/devices.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/exc.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/features.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/firewall.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/label.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/log.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/spinner.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/storage.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/tags.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/utils.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/events/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/events/__pycache__/utils.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qubes_prefs.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_check.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_features.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_kill.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_ls.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_prefs.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_remove.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_run.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_shutdown.cpython-38.pyc
..5...GT.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_start.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_template.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/vm/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesguidaemon/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesguidaemon/__pycache__/mic.cpython-38.pyc
S.5....T.  c /etc/qubes/qmemman.conf
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/app.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/backup.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/config.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/devices.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/events.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/exc.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/features.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/firewall.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/log.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/utils.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/api/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/api/__pycache__/admin.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/api/__pycache__/internal.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/api/__pycache__/misc.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/admin.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/audio.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/backup_restore.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/block.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/core_features.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/gui.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/pci.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/r3compatibility.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/services.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/supported_features.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/windows.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/qmemman/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/qmemman/__pycache__/algo.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/qmemman/__pycache__/client.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/storage/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/storage/__pycache__/file.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/storage/__pycache__/kernels.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/storage/__pycache__/lvm.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/storage/__pycache__/reflink.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/tests/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/tests/__pycache__/devices.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/tests/__pycache__/never_awaited.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/tools/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/tools/__pycache__/qmemmand.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/tools/__pycache__/qubesd.cpython-38.pyc
..5...GT.    /usr/lib/python3.8/site-packages/qubes/tools/__pycache__/qubesd_query.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/__pycache__/adminvm.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/__pycache__/appvm.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/__pycache__/dispvm.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/__pycache__/qubesvm.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/__pycache__/standalonevm.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/__pycache__/templatevm.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/mix/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/mix/__pycache__/dvmtemplate.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/mix/__pycache__/net.cpython-38.pyc
.M.......    /var/run/qubes
..5....T.    /usr/lib/python3.8/site-packages/qrexec/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/__pycache__/client.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/__pycache__/exc.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/__pycache__/server.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/__pycache__/utils.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/policy/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/policy/__pycache__/parser.cpython-38.pyc
.M5...GT.    /usr/lib/python3.8/site-packages/qrexec/policy/__pycache__/parser_compat.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/policy/__pycache__/utils.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/tools/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/tools/__pycache__/qrexec_policy_daemon.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/tools/__pycache__/qrexec_policy_exec.cpython-38.pyc
S.5....T.  c /etc/qubes-rpc/policy/qubes.InputMouse
..5....T.    /usr/lib/python3.8/site-packages/qubessalt/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesusbproxy/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesusbproxy/__pycache__/core3ext.cpython-38.pyc
..5....T.    /var/lib/selinux/minimum/active/commit_num
S.5....T.    /var/lib/selinux/minimum/active/file_contexts
S.5....T.    /var/lib/selinux/minimum/active/homedir_template
S.5....T.    /var/lib/selinux/minimum/active/policy.kern
.M.......  g /var/lib/selinux/minimum/active/policy.linked
S.5....T.    /var/lib/selinux/minimum/active/seusers
.M.......  g /var/lib/selinux/minimum/active/seusers.linked
.......T.    /var/lib/selinux/minimum/active/users_extra
.M.......  g /var/lib/selinux/minimum/active/users_extra.linked
.M....G..    /etc/xen
S.5....T.  c /etc/xen/xl.conf
.M....G..    /var/lib/xen
.M....G..    /var/log/xen
.M....G..    /var/log/xen/console
S.5....T.  c /etc/xdg/xfce4/panel/default.xml
S.5....T.    /etc/xdg/xfce4/xfconf/xfce-perchannel-xml/xfce4-power-manager.xml
S.5....T.    /etc/xdg/xfce4/xfconf/xfce-perchannel-xml/xfce4-session.xml
S.5....T.  c /etc/xdg/xfce4/xfconf/xfce-perchannel-xml/xsettings.xml
S.5....T.  c /etc/xscreensaver/XScreenSaver.ad.tail

Looking at files having ..5...... above gives a list of all files that were modified since package installation (which files are tracked by the package database). See man rpm and check “Verify” section for more information:

The following curated list without kernel packages:

sudo rpm -V anaconda-core-32.24.5-5.fc32.x86_64 anaconda-gui-32.24.5-5.fc32.x86_64 dnf-data-4.6.0-1.fc32.noarch libxfce4ui-4.14.1-4.fc32.x86_64 plymouth-0.9.4-14.20200325gite31c81f.fc32.x86_64 python3-blivet-3.2.1-2.fc32.noarch python3-kickstart-3.24-2.fc32.noarch python3-libvirt-6.6.0-5.fc32.x86_64 python3-qubesadmin-4.1.22-1.fc32.noarch qubes-audio-dom0-4.1.21-1.fc32.x86_64 qubes-core-dom0-4.1.26-1.fc32.noarch qubes-core-qrexec-4.1.18-1.fc32.x86_64 qubes-input-proxy-1.0.26-1.fc32.x86_64 qubes-mgmt-salt-admin-tools-4.1.13-1.fc32.noarch qubes-usb-proxy-dom0-1.1.1-1.fc32.noarch selinux-policy-minimum-3.14.5-46.fc32.noarch xen-runtime-4.14.4-4.fc32.x86_64 xfce4-panel-4.14.4-1.fc32.x86_64 xfce4-power-manager-1.6.6-1.fc32.x86_64 xfce4-session-4.14.2-1.fc32.x86_64 xfce4-settings-4.14.3-1.fc32.x86_64 xscreensaver-base-5.45-5.fc32.x86_64 | grep "5"

Outputs:

..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/__pycache__/network.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/core/__pycache__/constants.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/core/__pycache__/users.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/core/__pycache__/util.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/core/configuration/__pycache__/storage.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/modules/common/structures/__pycache__/partitioning.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/modules/network/__pycache__/network.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/payload/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/payload/__pycache__/utils.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/storage/__pycache__/partspec.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/ui/lib/__pycache__/space.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/ui/gui/spokes/__pycache__/datetime_spoke.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/ui/gui/spokes/__pycache__/root_password.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/pyanaconda/ui/gui/spokes/__pycache__/user.cpython-38.pyc
S.5....T.  c /etc/dnf/dnf.conf
S.5....T.  c /etc/xdg/xfce4/xfconf/xfce-perchannel-xml/xfce4-keyboard-shortcuts.xml
S.5....T.  c /etc/plymouth/plymouthd.conf
..5....T.    /usr/lib/python3.8/site-packages/blivet/devices/__pycache__/lvm.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/pykickstart/commands/__pycache__/partition.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/pykickstart/commands/__pycache__/repo.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/__pycache__/libvirt.cpython-38.pyc
..5....T.    /usr/lib64/python3.8/site-packages/__pycache__/libvirtaio.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/app.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/base.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/config.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/devices.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/exc.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/features.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/firewall.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/label.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/log.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/spinner.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/storage.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/tags.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/__pycache__/utils.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/events/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/events/__pycache__/utils.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qubes_prefs.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_check.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_features.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_kill.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_ls.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_prefs.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_remove.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_run.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_shutdown.cpython-38.pyc
..5...GT.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_start.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/tools/__pycache__/qvm_template.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesadmin/vm/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesguidaemon/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesguidaemon/__pycache__/mic.cpython-38.pyc
S.5....T.  c /etc/qubes/qmemman.conf
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/app.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/backup.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/config.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/devices.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/events.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/exc.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/features.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/firewall.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/log.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/__pycache__/utils.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/api/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/api/__pycache__/admin.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/api/__pycache__/internal.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/api/__pycache__/misc.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/admin.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/audio.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/backup_restore.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/block.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/core_features.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/gui.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/pci.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/r3compatibility.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/services.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/supported_features.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/ext/__pycache__/windows.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/qmemman/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/qmemman/__pycache__/algo.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/qmemman/__pycache__/client.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/storage/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/storage/__pycache__/file.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/storage/__pycache__/kernels.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/storage/__pycache__/lvm.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/storage/__pycache__/reflink.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/tests/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/tests/__pycache__/devices.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/tests/__pycache__/never_awaited.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/tools/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/tools/__pycache__/qmemmand.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/tools/__pycache__/qubesd.cpython-38.pyc
..5...GT.    /usr/lib/python3.8/site-packages/qubes/tools/__pycache__/qubesd_query.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/__pycache__/adminvm.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/__pycache__/appvm.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/__pycache__/dispvm.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/__pycache__/qubesvm.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/__pycache__/standalonevm.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/__pycache__/templatevm.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/mix/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/mix/__pycache__/dvmtemplate.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubes/vm/mix/__pycache__/net.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/__pycache__/client.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/__pycache__/exc.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/__pycache__/server.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/__pycache__/utils.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/policy/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/policy/__pycache__/parser.cpython-38.pyc
.M5...GT.    /usr/lib/python3.8/site-packages/qrexec/policy/__pycache__/parser_compat.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/policy/__pycache__/utils.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/tools/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/tools/__pycache__/qrexec_policy_daemon.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qrexec/tools/__pycache__/qrexec_policy_exec.cpython-38.pyc
S.5....T.  c /etc/qubes-rpc/policy/qubes.InputMouse
..5....T.    /usr/lib/python3.8/site-packages/qubessalt/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesusbproxy/__pycache__/__init__.cpython-38.pyc
..5....T.    /usr/lib/python3.8/site-packages/qubesusbproxy/__pycache__/core3ext.cpython-38.pyc
..5....T.    /var/lib/selinux/minimum/active/commit_num
S.5....T.    /var/lib/selinux/minimum/active/file_contexts
S.5....T.    /var/lib/selinux/minimum/active/homedir_template
S.5....T.    /var/lib/selinux/minimum/active/policy.kern
S.5....T.    /var/lib/selinux/minimum/active/seusers
S.5....T.  c /etc/xen/xl.conf
S.5....T.  c /etc/xdg/xfce4/panel/default.xml
S.5....T.    /etc/xdg/xfce4/xfconf/xfce-perchannel-xml/xfce4-power-manager.xml
S.5....T.    /etc/xdg/xfce4/xfconf/xfce-perchannel-xml/xfce4-session.xml
S.5....T.  c /etc/xdg/xfce4/xfconf/xfce-perchannel-xml/xsettings.xml
S.5....T.  c /etc/xscreensaver/XScreenSaver.ad.tail

@marmarek @fepitre @Demi : do we have a way to validate from where those files were modified from and why? (same would apply for the fedora template).

1 Like
2

Not a solution, but some ideas…

One way is an analysis of the {pre,post}install scripts from the rpm packages.

See the scripts with the rpm -q --scripts <package name> command, for example the qubes-core-dom0 package modifies the /etc/xen/xl.conf file from your list.

Another way for pyc files is a verification if the source python file (*.py) compilation matches the byte-compile file (*.pyc) from your list (see the python compilation documentation).

No idea for Salt, I don’t know it enough.

3

For the sake of clarity, the above list of modified files reported per rpm -V above relates to files reported as tampered on a default installation.

4

With Heads’s related public key injected in dom0 (gpg --import of such trusted public key) prior of upgrading dom0:

sha256sum $(find /boot/kexec*.txt) | gpg --verify /boot/kexec.sig - && cd /boot && sudo sha256sum -c /boot/kexec_hashes.txt

Example output (with my own key)

[user@dom0 boot]$ sha256sum $(find /boot/kexec*.txt) | gpg --verify /boot/kexec.sig - && cd /boot && sudo sha256sum -c /boot/kexec_hashes.txt 
gpg: Signature made Thu 02 Jun 2022 12:44:01 PM EDT
gpg:                using RSA key ACF4B7893D4D05C8F18069BAE7B4A71658E36A93
gpg: Good signature from "Insurgo Technologies Libres / Open Technologies <insurgo@riseup.net>" [unknown]
gpg:                 aka "[jpeg image of size 9521]" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: ACF4 B789 3D4D 05C8 F180  69BA E7B4 A716 58E3 6A93
./initramfs-5.10.112-1.fc32.qubes.x86_64.img: OK
./initramfs-5.10.109-1.fc32.qubes.x86_64.img: OK
./xen-4.14.4.config: OK
./.auditing-0: OK
./config-5.10.112-1.fc32.qubes.x86_64: OK
./System.map-5.10.109-1.fc32.qubes.x86_64: OK
./efi/EFI/qubes/initramfs-5.10.112-1.fc32.qubes.x86_64.img: OK
./efi/EFI/qubes/xen-4.14.4.efi: OK
./efi/EFI/qubes/xen.efi: OK
./efi/EFI/qubes/initramfs-5.10.90-1.fc32.qubes.x86_64.img: OK
./loader/entries/ab22ef20d85a490f8e042b251de691a9-5.10.104-3.fc32.qubes.x86_64.conf: OK
./loader/entries/ab22ef20d85a490f8e042b251de691a9-5.10.112-1.fc32.qubes.x86_64.conf: OK
./loader/entries/ab22ef20d85a490f8e042b251de691a9-5.10.109-1.fc32.qubes.x86_64.conf: OK
./config-5.10.109-1.fc32.qubes.x86_64: OK
./symvers-5.10.104-3.fc32.qubes.x86_64.gz: OK
./vmlinuz-5.10.112-1.fc32.qubes.x86_64: OK
./config-5.10.104-3.fc32.qubes.x86_64: OK
./symvers-5.10.109-1.fc32.qubes.x86_64.gz: OK
./initramfs-5.10.104-3.fc32.qubes.x86_64.img: OK
./vmlinuz-5.10.104-3.fc32.qubes.x86_64: OK
./vmlinuz-5.10.109-1.fc32.qubes.x86_64: OK
./System.map-5.10.104-3.fc32.qubes.x86_64: OK
./symvers-5.10.112-1.fc32.qubes.x86_64.gz: OK
./xen-4.14.4.gz: OK
./System.map-5.10.112-1.fc32.qubes.x86_64: OK
./grub2/grubenv: OK
./grub2/device.map: OK
./grub2/i386-pc/gcry_idea.mod: OK
./grub2/i386-pc/loopback.mod: OK
./grub2/i386-pc/search_fs_file.mod: OK
./grub2/i386-pc/tga.mod: OK
./grub2/i386-pc/gcry_serpent.mod: OK
./grub2/i386-pc/font.mod: OK
./grub2/i386-pc/zfs.mod: OK
./grub2/i386-pc/gcry_rsa.mod: OK
./grub2/i386-pc/f2fs.mod: OK
./grub2/i386-pc/bitmap.mod: OK
./grub2/i386-pc/read.mod: OK
./grub2/i386-pc/affs.mod: OK
./grub2/i386-pc/gcry_rmd160.mod: OK
./grub2/i386-pc/udf.mod: OK
./grub2/i386-pc/acpi.mod: OK
./grub2/i386-pc/lspci.mod: OK
./grub2/i386-pc/at_keyboard.mod: OK
./grub2/i386-pc/lsapm.mod: OK
./grub2/i386-pc/uhci.mod: OK
./grub2/i386-pc/aout.mod: OK
./grub2/i386-pc/usb_keyboard.mod: OK
./grub2/i386-pc/vbe.mod: OK
./grub2/i386-pc/keystatus.mod: OK
./grub2/i386-pc/elf.mod: OK
./grub2/i386-pc/shift_test.mod: OK
./grub2/i386-pc/cbls.mod: OK
./grub2/i386-pc/gcry_tiger.mod: OK
./grub2/i386-pc/part_sun.mod: OK
./grub2/i386-pc/sfs.mod: OK
./grub2/i386-pc/fat.mod: OK
./grub2/i386-pc/div_test.mod: OK
./grub2/i386-pc/help.mod: OK
./grub2/i386-pc/extcmd.mod: OK
./grub2/i386-pc/iso9660.mod: OK
./grub2/i386-pc/part_dvh.mod: OK
./grub2/i386-pc/video_bochs.mod: OK
./grub2/i386-pc/pxechain.mod: OK
./grub2/i386-pc/eval.mod: OK
./grub2/i386-pc/linux.mod: OK
./grub2/i386-pc/minix2_be.mod: OK
./grub2/i386-pc/lsacpi.mod: OK
./grub2/i386-pc/relocator.mod: OK
./grub2/i386-pc/procfs.mod: OK
./grub2/i386-pc/truecrypt.mod: OK
./grub2/i386-pc/pbkdf2_test.mod: OK
./grub2/i386-pc/div.mod: OK
./grub2/i386-pc/archelp.mod: OK
./grub2/i386-pc/hello.mod: OK
./grub2/i386-pc/multiboot2.mod: OK
./grub2/i386-pc/modinfo.sh: OK
./grub2/i386-pc/morse.mod: OK
./grub2/i386-pc/crc64.mod: OK
./grub2/i386-pc/ext2.mod: OK
./grub2/i386-pc/crypto.mod: OK
./grub2/i386-pc/mdraid1x.mod: OK
./grub2/i386-pc/videotest_checksum.mod: OK
./grub2/i386-pc/cbfs.mod: OK
./grub2/i386-pc/cpio_be.mod: OK
./grub2/i386-pc/video_fb.mod: OK
./grub2/i386-pc/cmosdump.mod: OK
./grub2/i386-pc/hfsplus.mod: OK
./grub2/i386-pc/probe.mod: OK
./grub2/i386-pc/newc.mod: OK
./grub2/i386-pc/core.img: OK
./grub2/i386-pc/terminfo.mod: OK
./grub2/i386-pc/ntfs.mod: OK
./grub2/i386-pc/part_amiga.mod: OK
./grub2/i386-pc/btrfs.mod: OK
./grub2/i386-pc/gcry_md5.mod: OK
./grub2/i386-pc/vga.mod: OK
./grub2/i386-pc/memdisk.mod: OK
./grub2/i386-pc/gfxterm.mod: OK
./grub2/i386-pc/gfxmenu.mod: OK
./grub2/i386-pc/gcry_sha1.mod: OK
./grub2/i386-pc/cpio.mod: OK
./grub2/i386-pc/fs.lst: OK
./grub2/i386-pc/gcry_dsa.mod: OK
./grub2/i386-pc/hashsum.mod: OK
./grub2/i386-pc/dm_nv.mod: OK
./grub2/i386-pc/pata.mod: OK
./grub2/i386-pc/boot.mod: OK
./grub2/i386-pc/random.mod: OK
./grub2/i386-pc/tar.mod: OK
./grub2/i386-pc/net.mod: OK
./grub2/i386-pc/gcry_des.mod: OK
./grub2/i386-pc/mdraid09_be.mod: OK
./grub2/i386-pc/mul_test.mod: OK
./grub2/i386-pc/terminal.lst: OK
./grub2/i386-pc/ctz_test.mod: OK
./grub2/i386-pc/xnu.mod: OK
./grub2/i386-pc/hfs.mod: OK
./grub2/i386-pc/zstd.mod: OK
./grub2/i386-pc/file.mod: OK
./grub2/i386-pc/gcry_md4.mod: OK
./grub2/i386-pc/part_plan.mod: OK
./grub2/i386-pc/gcry_cast5.mod: OK
./grub2/i386-pc/lzopio.mod: OK
./grub2/i386-pc/raid6rec.mod: OK
./grub2/i386-pc/gfxterm_menu.mod: OK
./grub2/i386-pc/functional_test.mod: OK
./grub2/i386-pc/play.mod: OK
./grub2/i386-pc/odc.mod: OK
./grub2/i386-pc/terminal.mod: OK
./grub2/i386-pc/exfctest.mod: OK
./grub2/i386-pc/hfspluscomp.mod: OK
./grub2/i386-pc/all_video.mod: OK
./grub2/i386-pc/bsd.mod: OK
./grub2/i386-pc/ntfscomp.mod: OK
./grub2/i386-pc/legacy_password_test.mod: OK
./grub2/i386-pc/memrw.mod: OK
./grub2/i386-pc/usbserial_common.mod: OK
./grub2/i386-pc/ls.mod: OK
./grub2/i386-pc/command.lst: OK
./grub2/i386-pc/ufs2.mod: OK
./grub2/i386-pc/diskfilter.mod: OK
./grub2/i386-pc/gcry_seed.mod: OK
./grub2/i386-pc/adler32.mod: OK
./grub2/i386-pc/msdospart.mod: OK
./grub2/i386-pc/gcry_arcfour.mod: OK
./grub2/i386-pc/minix3.mod: OK
./grub2/i386-pc/hdparm.mod: OK
./grub2/i386-pc/pgp.mod: OK
./grub2/i386-pc/iorw.mod: OK
./grub2/i386-pc/macho.mod: OK
./grub2/i386-pc/offsetio.mod: OK
./grub2/i386-pc/nativedisk.mod: OK
./grub2/i386-pc/bfs.mod: OK
./grub2/i386-pc/legacycfg.mod: OK
./grub2/i386-pc/fshelp.mod: OK
./grub2/i386-pc/raid5rec.mod: OK
./grub2/i386-pc/loadenv.mod: OK
./grub2/i386-pc/pci.mod: OK
./grub2/i386-pc/cs5536.mod: OK
./grub2/i386-pc/squash4.mod: OK
./grub2/i386-pc/usbserial_ftdi.mod: OK
./grub2/i386-pc/jfs.mod: OK
./grub2/i386-pc/usbms.mod: OK
./grub2/i386-pc/bufio.mod: OK
./grub2/i386-pc/minix.mod: OK
./grub2/i386-pc/serial.mod: OK
./grub2/i386-pc/syslinuxcfg.mod: OK
./grub2/i386-pc/minix2.mod: OK
./grub2/i386-pc/usb.mod: OK
./grub2/i386-pc/gcry_rijndael.mod: OK
./grub2/i386-pc/gcry_blowfish.mod: OK
./grub2/i386-pc/cmdline_cat_test.mod: OK
./grub2/i386-pc/xnu_uuid.mod: OK
./grub2/i386-pc/videoinfo.mod: OK
./grub2/i386-pc/tftp.mod: OK
./grub2/i386-pc/part_msdos.mod: OK
./grub2/i386-pc/cbtable.mod: OK
./grub2/i386-pc/part_bsd.mod: OK
./grub2/i386-pc/exfat.mod: OK
./grub2/i386-pc/xzio.mod: OK
./grub2/i386-pc/ohci.mod: OK
./grub2/i386-pc/signature_test.mod: OK
./grub2/i386-pc/video_colors.mod: OK
./grub2/i386-pc/cmostest.mod: OK
./grub2/i386-pc/cbtime.mod: OK
./grub2/i386-pc/freedos.mod: OK
./grub2/i386-pc/biosdisk.mod: OK
./grub2/i386-pc/cmp.mod: OK
./grub2/i386-pc/part_dfly.mod: OK
./grub2/i386-pc/gfxterm_background.mod: OK
./grub2/i386-pc/gcry_crc.mod: OK
./grub2/i386-pc/setjmp.mod: OK
./grub2/i386-pc/mda_text.mod: OK
./grub2/i386-pc/mmap.mod: OK
./grub2/i386-pc/priority_queue.mod: OK
./grub2/i386-pc/blocklist.mod: OK
./grub2/i386-pc/search_fs_uuid.mod: OK
./grub2/i386-pc/bswap_test.mod: OK
./grub2/i386-pc/video.mod: OK
./grub2/i386-pc/strtoull_test.mod: OK
./grub2/i386-pc/linux16.mod: OK
./grub2/i386-pc/trig.mod: OK
./grub2/i386-pc/cpuid.mod: OK
./grub2/i386-pc/disk.mod: OK
./grub2/i386-pc/pxe.mod: OK
./grub2/i386-pc/sleep.mod: OK
./grub2/i386-pc/video.lst: OK
./grub2/i386-pc/gcry_twofish.mod: OK
./grub2/i386-pc/lvm.mod: OK
./grub2/i386-pc/datehook.mod: OK
./grub2/i386-pc/boot.img: OK
./grub2/i386-pc/romfs.mod: OK
./grub2/i386-pc/macbless.mod: OK
./grub2/i386-pc/halt.mod: OK
./grub2/i386-pc/tr.mod: OK
./grub2/i386-pc/minicmd.mod: OK
./grub2/i386-pc/testload.mod: OK
./grub2/i386-pc/efiemu.mod: OK
./grub2/i386-pc/sleep_test.mod: OK
./grub2/i386-pc/ufs1_be.mod: OK
./grub2/i386-pc/setpci.mod: OK
./grub2/i386-pc/wrmsr.mod: OK
./grub2/i386-pc/plan9.mod: OK
./grub2/i386-pc/regexp.mod: OK
./grub2/i386-pc/testspeed.mod: OK
./grub2/i386-pc/luks.mod: OK
./grub2/i386-pc/vga_text.mod: OK
./grub2/i386-pc/gcry_sha512.mod: OK
./grub2/i386-pc/moddep.lst: OK
./grub2/i386-pc/part_acorn.mod: OK
./grub2/i386-pc/video_cirrus.mod: OK
./grub2/i386-pc/part_sunpc.mod: OK
./grub2/i386-pc/gcry_rfc2268.mod: OK
./grub2/i386-pc/configfile.mod: OK
./grub2/i386-pc/sendkey.mod: OK
./grub2/i386-pc/search.mod: OK
./grub2/i386-pc/cat.mod: OK
./grub2/i386-pc/scsi.mod: OK
./grub2/i386-pc/nilfs2.mod: OK
./grub2/i386-pc/cbmemc.mod: OK
./grub2/i386-pc/ufs1.mod: OK
./grub2/i386-pc/bitmap_scale.mod: OK
./grub2/i386-pc/progress.mod: OK
./grub2/i386-pc/zfscrypt.mod: OK
./grub2/i386-pc/gcry_whirlpool.mod: OK
./grub2/i386-pc/ntldr.mod: OK
./grub2/i386-pc/ata.mod: OK
./grub2/i386-pc/increment.mod: OK
./grub2/i386-pc/date.mod: OK
./grub2/i386-pc/usbserial_pl2303.mod: OK
./grub2/i386-pc/png.mod: OK
./grub2/i386-pc/afs.mod: OK
./grub2/i386-pc/reiserfs.mod: OK
./grub2/i386-pc/pbkdf2.mod: OK
./grub2/i386-pc/search_label.mod: OK
./grub2/i386-pc/gcry_camellia.mod: OK
./grub2/i386-pc/chain.mod: OK
./grub2/i386-pc/reboot.mod: OK
./grub2/i386-pc/datetime.mod: OK
./grub2/i386-pc/partmap.lst: OK
./grub2/i386-pc/gptsync.mod: OK
./grub2/i386-pc/gdb.mod: OK
./grub2/i386-pc/mpi.mod: OK
./grub2/i386-pc/gzio.mod: OK
./grub2/i386-pc/true.mod: OK
./grub2/i386-pc/backtrace.mod: OK
./grub2/i386-pc/hexdump.mod: OK
./grub2/i386-pc/zfsinfo.mod: OK
./grub2/i386-pc/geli.mod: OK
./grub2/i386-pc/part_gpt.mod: OK
./grub2/i386-pc/ldm.mod: OK
./grub2/i386-pc/crypto.lst: OK
./grub2/i386-pc/jpeg.mod: OK
./grub2/i386-pc/gettext.mod: OK
./grub2/i386-pc/parttool.mod: OK
./grub2/i386-pc/usbserial_usbdebug.mod: OK
./grub2/i386-pc/keylayouts.mod: OK
./grub2/i386-pc/parttool.lst: OK
./grub2/i386-pc/setjmp_test.mod: OK
./grub2/i386-pc/spkmodem.mod: OK
./grub2/i386-pc/part_apple.mod: OK
./grub2/i386-pc/xnu_uuid_test.mod: OK
./grub2/i386-pc/rdmsr.mod: OK
./grub2/i386-pc/ehci.mod: OK
./grub2/i386-pc/test.mod: OK
./grub2/i386-pc/lsmmap.mod: OK
./grub2/i386-pc/normal.mod: OK
./grub2/i386-pc/http.mod: OK
./grub2/i386-pc/videotest.mod: OK
./grub2/i386-pc/password.mod: OK
./grub2/i386-pc/time.mod: OK
./grub2/i386-pc/cmp_test.mod: OK
./grub2/i386-pc/xfs.mod: OK
./grub2/i386-pc/verifiers.mod: OK
./grub2/i386-pc/mdraid09.mod: OK
./grub2/i386-pc/ahci.mod: OK
./grub2/i386-pc/gcry_sha256.mod: OK
./grub2/i386-pc/password_pbkdf2.mod: OK
./grub2/i386-pc/usbtest.mod: OK
./grub2/i386-pc/cryptodisk.mod: OK
./grub2/i386-pc/pcidump.mod: OK
./grub2/i386-pc/test_blockarg.mod: OK
./grub2/i386-pc/echo.mod: OK
./grub2/i386-pc/minix_be.mod: OK
./grub2/i386-pc/drivemap.mod: OK
./grub2/i386-pc/multiboot.mod: OK
./grub2/i386-pc/minix3_be.mod: OK
./grub2/fonts/unicode.pf2: OK
./grub2/grub.cfg: OK
./grub2/themes/qubes/progress_bar_c.png: OK
./grub2/themes/qubes/qubes.png: OK
./grub2/themes/qubes/progress_bar2_c.png: OK
./grub2/themes/qubes/unifont-regular-16.pf2: OK
./grub2/themes/qubes/progress_bar_hl_c.png: OK
./grub2/themes/qubes/unifont-bold-16.pf2: OK
./grub2/themes/qubes/theme.txt: OK
./grub2/themes/qubes/unifont-regular-32.pf2: OK
./grub2/themes/qubes/unifont-regular-14.pf2: OK

A single bit change on /boot/kexec* file content will invalidate the detached signature validation made by Heads, where calling manually sha256sum -c on thedigest will show what file changed.

This is manual, and valid until the user regenerates digest on reboot if changes are detected and resigns the digests.

On Dom0, that could be done manually to verify changes prior of an upgrade and after, to know better what would be signed into Heads, and manually check with rpm -V (and rpm -q --what-provides call to locate rpm package) to make sure nothing but system deployed upgrades changed /boot content.

5

Also, prior of rebooting, a user could run the following on dom0 to validate /boot from RPM database:

[user@dom0 ~]$ sudo find /boot|while read entree; do sudo rpm -q --whatprovides "$entree"; done| grep -v "not owned by any package"|sort|uniq | while read package; do sudo rpm -V $package; done
.M.......  c /boot/grub2/grub.cfg
.M.......  g /boot/initramfs-5.10.104-3.fc32.qubes.x86_64.img
.......T.    /lib/modules/5.10.104-3.fc32.qubes.x86_64/modules.builtin.alias.bin
.......T.    /lib/modules/5.10.104-3.fc32.qubes.x86_64/modules.builtin.bin
.......T.    /lib/modules/5.10.104-3.fc32.qubes.x86_64/modules.devname
.......T.    /lib/modules/5.10.104-3.fc32.qubes.x86_64/modules.softdep
.M.......  g /boot/initramfs-5.10.109-1.fc32.qubes.x86_64.img
.......T.    /lib/modules/5.10.109-1.fc32.qubes.x86_64/modules.builtin.alias.bin
.......T.    /lib/modules/5.10.109-1.fc32.qubes.x86_64/modules.builtin.bin
.......T.    /lib/modules/5.10.109-1.fc32.qubes.x86_64/modules.devname
.......T.    /lib/modules/5.10.109-1.fc32.qubes.x86_64/modules.softdep
.M.......  g /boot/initramfs-5.10.112-1.fc32.qubes.x86_64.img
.......T.    /lib/modules/5.10.112-1.fc32.qubes.x86_64/modules.builtin.alias.bin
.......T.    /lib/modules/5.10.112-1.fc32.qubes.x86_64/modules.builtin.bin
.......T.    /lib/modules/5.10.112-1.fc32.qubes.x86_64/modules.devname
.......T.    /lib/modules/5.10.112-1.fc32.qubes.x86_64/modules.softdep

Where the following removes the above benign time .......T. and mode .M....... changes (see man rpm) and returns nothing, stating that no file integrity per see was impacted:

[user@dom0 ~]$ sudo find /boot|while read entree; do sudo rpm -q --whatprovides "$entree"; done| grep -v "not owned by any package"|sort|uniq | while read package; do sudo rpm -V $package; done| grep -v ".M......."| grep -v ".......T."
[user@dom0 ~]$

And where we learned earlier that

[user@dom0 ~]$ sudo find /boot|while read entree; do sudo rpm -q --whatprovides "$entree"; done| grep -v "not owned by any package"|sort|uniq
filesystem-3.14-2.fc32.x86_64
grub2-common-2.04-2.fc32.noarch
grub2-pc-2.04-2.fc32.x86_64
grub2-qubes-theme-5.14.4-2.fc32.x86_64
kernel-5.10.104-3.fc32.qubes.x86_64
kernel-5.10.109-1.fc32.qubes.x86_64
kernel-5.10.112-1.fc32.qubes.x86_64
xen-hypervisor-4.14.5-4.fc32.x86_64

So basically, to quickly verify /boot related packages, one could do:

[user@dom0 ~]$ sudo rpm -V filesystem grub2-common grub2-pc grub2-qubes-theme kernel xen-hypervisor
.M.......  c /boot/grub2/grub.cfg
.M.......  g /boot/initramfs-5.10.104-3.fc32.qubes.x86_64.img
.......T.    /lib/modules/5.10.104-3.fc32.qubes.x86_64/modules.builtin.alias.bin
.......T.    /lib/modules/5.10.104-3.fc32.qubes.x86_64/modules.builtin.bin
.......T.    /lib/modules/5.10.104-3.fc32.qubes.x86_64/modules.devname
.......T.    /lib/modules/5.10.104-3.fc32.qubes.x86_64/modules.softdep
.M.......  g /boot/initramfs-5.10.109-1.fc32.qubes.x86_64.img
.......T.    /lib/modules/5.10.109-1.fc32.qubes.x86_64/modules.builtin.alias.bin
.......T.    /lib/modules/5.10.109-1.fc32.qubes.x86_64/modules.builtin.bin
.......T.    /lib/modules/5.10.109-1.fc32.qubes.x86_64/modules.devname
.......T.    /lib/modules/5.10.109-1.fc32.qubes.x86_64/modules.softdep
.M.......  g /boot/initramfs-5.10.112-1.fc32.qubes.x86_64.img
.......T.    /lib/modules/5.10.112-1.fc32.qubes.x86_64/modules.builtin.alias.bin
.......T.    /lib/modules/5.10.112-1.fc32.qubes.x86_64/modules.builtin.bin
.......T.    /lib/modules/5.10.112-1.fc32.qubes.x86_64/modules.devname
.......T.    /lib/modules/5.10.112-1.fc32.qubes.x86_64/modules.softdep

Or again, remove the time and mode changes not related to integrity change, which should return empty output:

[user@dom0 ~]$ sudo rpm -V filesystem grub2-common grub2-pc grub2-qubes-theme kernel xen-hypervisor| grep -v ".M......."| grep -v ".......T."
[user@dom0 ~]$

So a quick PoC could look like:

[user@dom0 ~]$ if [ -z $(sudo rpm -V filesystem grub2-common grub2-pc grub2-qubes-theme kernel xen-hypervisor| grep -v ".M......." | grep -v ".......T.") ] ; then echo "Good"; else echo "Bad"; fi
Good
[user@dom0 ~]$
1 Like
6

The same commands above will succeed without sudo from Heads recovery shell.

I have deployed the following under /boot on my side to ease such verification:

[user@dom0 ~]$ cat /boot/verify_detached_signatures.sh
sha256sum $(find /boot/kexec*.txt) | gpg --verify /boot/kexec.sig -
[user@dom0 ~]$ cat /boot/verify_hashes.sh
cd /boot
sha256sum -c /boot/kexec_hashes.txt
1 Like
7

I love this thread. Keep it up.

8

A reminder that man is a forgotten best friend sometimes.

man rpm tells us:

The format of the output is a string of 9 characters, a possible attribute marker:

       c %config configuration file.
       d %doc documentation file.
       g %ghost file (i.e. the file contents are not included in the package payload).
       l %license license file.
       r %readme readme file.

       from  the  package  header,  followed  by  the  file  name.   Each of the 9 characters denotes the result of a comparison of
       attribute(s) of the file to the value of those attribute(s) recorded in the database.  A single "." (period) means the  test
       passed,  while  a  single "?" (question mark) indicates the test could not be performed (e.g. file permissions prevent read‐
       ing). Otherwise, the (mnemonically emBoldened) character denotes failure of the corresponding --verify test:

       S file Size differs
       M Mode differs (includes permissions and file type)
       5 digest (formerly MD5 sum) differs
       D Device major/minor number mismatch
       L readLink(2) path mismatch
       U User ownership differs
       G Group ownership differs
       T mTime differs
       P caPabilities differ

So that today:

[user@dom0 ~]$ sudo rpm -V xen-hypervisor kernel
.M.......  g /boot/initramfs-5.15.64-1.fc32.qubes.x86_64.img
.......T.    /lib/modules/5.15.64-1.fc32.qubes.x86_64/modules.builtin.alias.bin
.......T.    /lib/modules/5.15.64-1.fc32.qubes.x86_64/modules.builtin.bin
.......T.    /lib/modules/5.15.64-1.fc32.qubes.x86_64/modules.devname
.......T.    /lib/modules/5.15.64-1.fc32.qubes.x86_64/modules.softdep
.M.......  g /boot/initramfs-5.15.74-1.fc32.qubes.x86_64.img
.......T.    /lib/modules/5.15.74-1.fc32.qubes.x86_64/modules.builtin.alias.bin
.......T.    /lib/modules/5.15.74-1.fc32.qubes.x86_64/modules.builtin.bin
.......T.    /lib/modules/5.15.74-1.fc32.qubes.x86_64/modules.devname
.......T.    /lib/modules/5.15.74-1.fc32.qubes.x86_64/modules.softdep
.M.......  g /boot/initramfs-5.15.74-2.fc32.qubes.x86_64.img
.......T.    /lib/modules/5.15.74-2.fc32.qubes.x86_64/modules.builtin.alias.bin
.......T.    /lib/modules/5.15.74-2.fc32.qubes.x86_64/modules.builtin.bin
.......T.    /lib/modules/5.15.74-2.fc32.qubes.x86_64/modules.devname
.......T.    /lib/modules/5.15.74-2.fc32.qubes.x86_64/modules.softdep

Time and Mode of files differs only, otherwise we would have empty output.
This is why we have nothing said on xen related files under boot: their Mode and Time being fine.

Why those are different would be a question for @fepitre @Demi @marmarek, as other unanswered questions of this thread on why files are modified on a clean installation, which should otherwise be marked as configuration files and not reported as being tampered by our friend rpm -V, under OP’s question:

9

Anaconda-related files shouldn’t differ, unless you modified related .py files (even if you changed them later back, .pyc may differ, as they may have modification time embedded). I checked on two systems, and the don’t get reported by rpm -Va for me.

As for initramfs, it’s generated at install time (and includes drivers required for specific systems), so it is expected to differ (the fact that 5 is not there, is because its digest isn’t even recorded by rpm). Having pre-built initramfs with all relevant modules is technically possible, but will require bigger /boot.

10

@marmarek if initramfs files increases to 50mb or even 100mb files because everything boot drivers are included into them, only 3 are kept under /boot which is by default 1GB in size. Even if I keep ireelevant states under /boot (measured by Heads so practical in my testing use cases), /boot would not be more then 50% occupied here. Qubes could increase /boot size in next release, but I do not see current /boot size as being a limitation to have possible integrity contract verification under Qubes deployed binaries, waiting for Qubes to detach sign their binaries or have kexec verify them prior of launching them.

I continue to think that files under /boot should be signed/detached signed per distro key as previously discussed. That would be a great improvement, alongside of Aiming to a readonly dom0 · Issue #7839 · QubesOS/qubes-issues · GitHub

11

Synchronicity made people ask similar questions in PM and on Heads matrix channel today.

Under heads, /boot/kexec_hashes.txt contains /boot digests, which are detached signed, as all other
kexec_*.txt files under kexec.sig. Heads verifies detached signature on runtime generated digest file to make sure that /boot files are in the state expected. That is, that kexec_default*.txt default boot option is as expected (grub options, individual referred grub referred binaries hashes and /boot global hashes) are in expected state.

Unfortunately, that doesn’t include new files that were deployed (those new files not being part of existing digest!).

Head should probably report on the differences (including new files that are to be included in digest) prior to asking for signing for the changes.

Today new dom0 updated my /boot content (was running testing-repo xen so may not be relevant to other users, but interesting to see that same files are touched here).

If we reuse what Heads does at signing (kexec-sign-boot script), but instead of overwriting /boot/kexec_hashes.txt, we save those new hashes under /tmp/kexec_hashes.txt and compare them:
EDIT: the forum mangled '*kexec*' into 'kexec'.
The following is edited:

[user@dom0 ~]$ cd /boot
[user@dom0 boot]$ sudo find ./ -type f ! -name '*kexec*' -print0 | sudo xargs -0 sha256sum > /tmp/kexec_hashes.txt
[user@dom0 boot]$ diff /boot/kexec_hashes.txt /tmp/kexec_hashes.txt 
8,9c8,9
< 3e5f6ccc321ac18ac47d581db40f840b76507884d233fee0e0ad85bdc42d0345  ./efi/EFI/qubes/xen.efi
< 3e5f6ccc321ac18ac47d581db40f840b76507884d233fee0e0ad85bdc42d0345  ./efi/EFI/qubes/xen-4.14.5.efi
---
> 7df8ea17b75e0a38d0840769732e3192006611c10e5a516beb732e47560e35ae  ./efi/EFI/qubes/xen.efi
> 7df8ea17b75e0a38d0840769732e3192006611c10e5a516beb732e47560e35ae  ./efi/EFI/qubes/xen-4.14.5.efi
22c22
< 49de5d9fd4286e5ff6d61d21bd7ae5982f765efe71d19d01e2e81e7a5cfc4794  ./xen-4.14.5.gz
---
> 43467546e0a7787eeb93d49523b01c5d4e15deddbbbd1315df95048350139def  ./xen-4.14.5.gz

We see that the same files were modified/overwritten.

Further investigating, asking rpm what packages provided the changed files:

[user@dom0 boot]$ sudo rpm -q --whatprovides ./efi/EFI/qubes/xen.efi ./efi/EFI/qubes/xen-4.14.5.efi ./xen-4.14.5.gz
file /boot/efi/EFI/qubes/xen.efi is not owned by any package
xen-hypervisor-4.14.5-12.fc32.x86_64
xen-hypervisor-4.14.5-12.fc32.x86_64

We see that xen.efi is generated. We don’t care about this one under Heads, since Heads kexec’s grub.cfg referred multiboot binaries (initrd, xen, kernel files with grub boot options)

So are the two other files in expected states?

[user@dom0 boot]$ sudo rpm -V xen-hypervisor-4.14.5-12.fc32.x86_64
[user@dom0 boot]$

Yes.

Heads should show not only the files modified/deleted, but also new files deployed. This is to be considered under Extend /boot digest verification to not only report on changed files · Issue #1108 · osresearch/heads · GitHub

1 Like
12

Question was asked again this morning: “Should I follow Verifying Installation post over QubesOS Forum everytime I update dom0?”

I answered under Heads official Matrix channel at You're invited to talk on Matrix

The discussion on how to tackle the issue should happen upstream under Provide OS integration to sign /boot and/or root during updates · Issue #1615 · linuxboot/heads · GitHub