Packages from mirror differ from packages saved in Qubes OS

Hello,

today I updated dom0. The following packages were updated:

qubes-manager-4.3.23-1.fc41.noarch.rpm
qubes-core-dom0-4.3.49-1.fc41.noarch.rpm
qubes-mgmt-salt-dom0-virtual-machines-4.3.17-1.fc41.noarch.rpm

Everything went fine. No warnings or errors. However, since I am quite paranoid, I tried to check the rpms downloaded vs. rpms on the mirror manually.

What I found is, that the files differ. Each file in the /var/lib/qubes/updates/rpm/-folder shows a different sha256-hash, compared to the file at

I would expect, that the files are identical.

Interestingly, all local files are 3600 bytes smaller, than the files for download. I also checked the signature of the local files via:

rpm --checksig -v

everything looks fine. The key id is 7f3fada4. And this key is identified as “Qubes OS Release 4.3 Signing Key” via https://keyserver.ubuntu.com.

However, it bugs me, that the rpms differ. Can someone explain why?

Best regards and thank you for your help,

bimbelbi

What I tested so far:

The code from below is given for qubes-manager, but can be used analogously for the other packages.

I unpacked the packages from the repository (in some fedora vm as well as in a debian like os running on independent real hardware) as well as the local packages (dom0) via:

mkdir qubes-core
cd qubes-core
rpm2cpio ../qubes-manager-4.3.23-1.fc41.noarch.rpm | cpio -idmv

The number of files in the corresponding qubes-manager directories are the same:

find . -name "*" | wc -l
183

Then I tried to compare the result via:


find . -name "*" -type f -exec sha256sum '{}' \; | sort | sha256sum

The hashes for the repository and local package are the same:

931a172c38e9b9c3efab1b29de86d49af7b3d51acc39d65ec3823ba888db1beb  -

I did the same without “-type f” and compared the error output. The same.

But I found no explanation for differing packages, local compared to the repository, on the internet. Everything says, that the packages should be the same. And what drives me crazy is, that I could not find the local packages on any remote repository. This is a fact. The local files are exactly 3600 bytes smaller than the remote ones.

Next try:

rpm -qp --dump qubes-manager-4.3.23-1.fc41.noarch.rpm | sha256sum
bd1de05e5d96b74e445eeef890c585977b6d46e1b1fc262e8bdb1da8da69160d  -

This is the same for the local and remote packages.

I also went through all mirrors given in

https://yum.qubes-os.org/r4.3/current/host/fc41/repodata/repomd.xml.metalink

and downloaded qubes-manager-4.3.23-1.fc41.noarch.rpm to check for size and sha256-hash. The files are all the same and not the 3600 byte smaller version.

I’m running out of ideas. Is qubes altering the packages when updating, e.g., unpacking the packages, verifying and then rebuilding them again? In my opinion, this would make no sense.

Hello,

Qubes OS recently updated my dom0. And just for fun, I compared the downloaded packages with the packages in the repository. I found some issues, the sha256-digest differ and the file size is exactly 3600 bytes smaller for each package locally compared to the remote files. However, everything else seems fine.

I found no explanation for that fact yet and are running out of ideas. This is not bugging me anymore, it is not making me crazy anymore, it really drives me nuts! I already opened a thread, where you can find the stuff I already did. However, the place is badly chosen (or the topic is uninteresting or I am too impatient) so that almost nobody looked at it.

I hope it has simple explanation, and I hope information about this behavior is easy to find, when you know the correct keyword. I hope, I am only a stupid, paranoid fool.

Thank you in advance,

bimbelbi

For consistency:

mkdir qubes-core
cd qubes-core

from the first post should be

mkdir qubes-manager
cd qubes-manager

(copy and paste …)

Thread is in “slow mode” so that I cannot edit …

Yes. Packages arrive in dom0 via /etc/qubes-rpc/qubes.ReceiveUpdates, which passes each package through rpmcanon from qubes-rpm-oxide to canonicalize it:

I think the idea is to have some modern Rust code defang the package file before it is passed to the relatively complex rpm legacy codebase for inline signature validation. See the end of the Summary section of QSB-067.

Hi rustybird,

thank you for your answer. That calms me down for today. I will go into detail in the next days. You gave enough hints.

Is there a way to canonicalize an rpm package manually? For testing. Like a command line call that uses qubes-rpm-oxide?

Moreover, I think this information is missing in:

(maybe I missed it?)

It would be good, to add or make clear, that the rpm-packages, that end up in /var/lib/qubes/updates/rpm/-folder might look different (size) than the originals from the repository. Actually, I would prefer the canonicalization being done in the background so that the original rpm is saved in the update folder. But I guess Qubes OS has it’s reasons to do it this way.

bimbelbi

rpmcanon --allow-old-pkgs -- input.rpm output.rpm

That’s the syntax used by the linked snippet from qubes.ReceiveUpdates.

Hi rustybird,

thank you again!

Is there a way to get this working outside of dom0 so that I do not mess up the most important part of qubes os, i.e., in a vm?

bimbelbi

Yes, you can install the qubes-rpm-oxide package containing rpmcanon in a VM('s template) too. Although I don’t know if its canonicalized output there is necessarily identical to that in dom0.

Should be fine to just run it in dom0 though. After all, that’s what it’s for.

Don’t pass the output package to dnf though, at least not before verifying the package signature.

Hi rustybird,

it worked like a charm.

How everyone can check it:

  1. create a new fedora vm (in my case fedora 43) with internet access
  2. install qubes-rpm-oxide in vm: sudo dnf install qubes-rpm-oxide
  3. download original package in vm, e.g.: curl -O https://yum.qubes-os.org/r4.3/current/host/fc41/rpm/qubes-manager-4.3.23-1.fc41.noarch.rpm
  4. canonicalize package in vm, e.g.: rpmcanon --allow-old-pkgs -- qubes-manager-4.3.23-1.fc41.noarch.rpm qubes-manager-4.3.23-1.fc41.noarch.canonicalized.rpm
  5. compute sha256-digests in dom0 (the automatically downloaded and already canonicalized package): sha256sum qubes-manager-4.3.23-1.fc41.noarch.rpm
  6. compute sha256-digest in vm: sha256sum qubes-manager-4.3.23-1.fc41.noarch.canonicalized.rpm
  7. compare the sha256sum-digests

In my case, all tested packages were identical (whew).

Thank you again rustybird :clap:!

Best regards,
bimbelbi

Problem solved thanks to rustybird. You find the relevant information in the link above.

Best regards,
bimbelbi