Boot hash mismatch

User Support
#1

OK so my HOTP/TOTP is good, but I got the ‘boot hash mismatch’ trying to default boot for xen.efi, xen-4.14.5.efi and xen-4.14.5.gz verifications asking me to update checksums. Is it possible this was an evil maid attack or did I just forget that last time I booted, I updated dom0 and forgot to reboot to update the checksums? Is there a way to see if there was an update applied one of my last sessions?

#2

Check dnf history. Newer heads versions also show you the files that changed.

#3

Newer heads shows files that were added. Prior Heads only showed versions that changed.

1 Like
#4

@marmarek I think this is a design bug under QubesOS, which overwrites the same xen binaries since a version of Qubes (4.1) is sticking to the same Xen version (4.14) and backport and patches Xen, while Xen binaries do not contain the same revision name scheme as rpm does.

[user@dom0 ~]$ sudo rpm -q --whatprovides /boot/xen-4.14.5.gz /boot/efi/EFI/qubes/xen-4.14.5.efi
xen-hypervisor-4.14.5-20.fc32.x86_64
xen-hypervisor-4.14.5-20.fc32.x86_64

Then doing integrity verification through rpm -V returns nothing (no altering):
[user@dom0 ~]$ sudo rpm -V xen-hypervisor-4.14.5-20.fc32.x86_64

While asking verbose gives no alteration “details”:

[user@dom0 ~]$ sudo rpm -V xen-hypervisor-4.14.5-20.fc32.x86_64 --verbose
.........    /boot/efi/EFI/qubes/xen-4.14.5.efi
.........    /boot/xen-4.14.5.config
.........    /boot/xen-4.14.5.gz
.........  a /usr/lib/debug/.build-id
.........  a /usr/lib/debug/.build-id/c5
.........  a /usr/lib/debug/.build-id/c5/c694c51299448bfd1ca4f4907b7a1d624d12ab
.........  a /usr/lib/debug/.build-id/c5/c694c51299448bfd1ca4f4907b7a1d624d12ab.debug
.........    /usr/lib/debug/xen-4.14.5.efi.map
.........    /usr/lib/debug/xen-syms-4.14.5
.........    /usr/lib/debug/xen-syms-4.14.5.map

Heads reports new files presence (since track files in /boot in kexec_tree.txt by 3hhh · Pull Request #1262 · osresearch/heads · GitHub) while files being present in kexec_hashes.txt are verified on each boot, after hashes are verified against detached signature under kexec.sig. File changes always reported alteration notice, which is where there is confusion for many. When dom0 upgrade only updates Xen without updating a kernel, there is no grub.cfg changes, and the user only sees that the same xen files could have been tampered, selecting default boot warning about a possible compromise under Heads, since grub.cfg has not changed. Only Xen binary did.

Heads cannot know if the changes themselves are good or bad. But normally, changes imply old files removal and new file addition, and for example in the case of a new kernel, since old ones are removed, and new ones are added, it also triggers grub.cfg update.

@marmarek the current process of having rpm pack same binary filenames (/boot/xen-4.14.5.gz /boot/efi/EFI/qubes/xen-4.14.5.efi), but under a new rpm version containing revision numver 20 (xen-hypervisor-4.14.5-20.fc32.x86_64) is confusing for a lot of people. Agreed, this is an opsec problem, while also an opsec problem (users should be warned constantly through notification that dom0 was updated and needs a reboot) but the following is still confusing and should be modified for clarity:

  • xen-hypervisor-4.14.5-20.fc32.x86_64 contains revision 20, but still deploys files not containing the same revision number /boot/xen-4.14.5.gz, not /boot/xen-4.14.5-20.gz

If this is a bug, feel free to open issue and point here for clarification, or have documentation on versioning be clear also on the core components, which Xen will overwrite its files and that not being tampering from an outside validation perspective. Otherwise, for a user, /boot/xen-4.14.5.gz actually seems tampered and the only way to know is to boot inside of thought tampered system to verify traces of dom0 past intallation, rpm database integrity reports etc.

@user229910 : You most probably forgot to reboot after deploying dom0 upgrade. You can boot Heads through unsafe boot option without signing boot configuration through boot options, show boot options and choosing to boot without verifying integrity. Check integrity reports, dnf history and logs, reboot and then sign.

1 Like
#5

This is a really good shout. Theres a change, but not a versioning change as we would expect following normal nomenclature.

1 Like
#6

I’d say a package update updating files belonging to the package is normal thing. The kernel packages are abnormal here (allowing multiple versions of the same package to be installed at the same time), and that’s because sometimes regressions happen and user needs a way to boot older version. In ideal world, we wouldn’t need that either.