How to verify Heads integrity

hi @Insurgo

hopefully my question is on topic,
is it possible to generate checksum for heads’ ROM,
that has been flashed into chip,
maybe via heads recovery shell ?

thanks & regards,

At compilation and under CircleCI, a file named hashes.txt contains the hashes of all individual files that were packed.

Under CircleCI, each board compiled for a commit also prints all the hashes of the files that were contained in the firmware in an individual step, which should stay there forever.

So if you downloaded your ROM from CircleCI, you could go under Heads Options->“System Information” to get your ROM’s commit ID and find relative hashes.txt file for initrd contained files packed in the firmware under CircleCI’s commit ID’s board build’s “Output hashes” step.
Exemple:


If you are talking about taking a backup of the firmware, and are willing/searching for ways to unpack its content for inspection against hashes, that will be another subject and a really long one…

hi @Insurgo

do you mean for example this github hashes.txt ?

do you mean for example this circle CI output hashes

hmm, which one is system information ?

is it possible to use heads recovery shell, to generate checksum,
for heads ROM, that have been flashed into chip ?

thanks and regards,

@deeplow this post (Compiling and flashing heads - #11) down should be splitted and moved to another thread “How to verify Heads integrity”.

@newbie This seems to be getting far of Qubes. I opened an issue under Notes on how to audit a maximized flashed firmware image · Issue #107 · linuxboot/heads-wiki · GitHub which should answer some of your questions. I did not intend to answer and dig into all that today

This exposes the hashes of the neutered ME binary blob and GBE+IFD configuration blobs that are consistent across all xx30 boards, which can be extracted from ifdtool and verified externally from an internally/externally taken backup, as discussed in opened issue above with coreboot utilities installed on top of debian-12 (bookworm).

More tools could be packed into the firmware (which would require precious space), including ifdtool, to be able to extract those as well from within Heads.

But that wouldn’t really make any sense if the goal is to investigate untrustworthyness of an already flashed firmware image, if I follow you and all the other opened thread questions you opened recently.

The only way to validate this without any doubt would be to reduce trust to its possible minimal and take a backup of the firmware (externally is best while combining firmware images for investigation: that would be zero trust approach, or internally, trusting Heads compiled flashrom binary tool, packed under initramfs to take a backup internally to an external USB thumb drive and investigate externally (to me this seems enough, since that backup will include the initrd, and that initrd can be extracted to verify the hashes of flashrom and everything else packed into the initrd from hashes.txt produced for that commit’s build).

Yes, exactly. But those hashes are valid only for that commit https://github.com/osresearch/heads/commit/3184bf7a8c6404551871e1353a391710cc52f426

So the starting point is to get your FW_VER (with the commit used to build that image) and start from there, not the other way around. The commit you picked up was of master today, as of today, which is most probably not the version you have flashed. You can get that information from “System Information” or from FW_VER to get the filename of what you flashed, which includes the commit id at the end of the filename after -gCOMMIT.rom.

Having that hashes.txt file on a usb thumb drive alongside of the the same ROM version, would permit you (if you trust trust integrity of busybox which exposes sha256sum binary) to do from Heads recovery shell:

mount-usb
cd /
sha256sum -c /media/hashes.txt

This is why taking a backup from the firmware, and doing the job outside of Heads would be more trustworthy and consequent to verifying integrity of the firmware, which is why I opened an issue for heads-wiki documentation I pointed out here. This definitely would need more automatization/documentation.

Where again, as said in other posts, the possibility of having Heads report for good firmware measurements reported by coreboot which measures itself and the payload (heads) (cbmem -L from Heds recovery shell, which extends measurements inside of the TPM prior of booting into Heads payload) is already verifying and reporting integrity which were sealed by totp-seal under TPM nvram and exposed through TOTP/HOTP (hotp-unseal/totp-unseal operations) for remote attestation.

But if the need here is to verify this even more throroughly in total distrust, extremely paranoid people would most probably want to do a backup trough an external programmer, combine the two 4mb and 8mb SPI images and do what is exposed (now partial) into the heads-wiki issue (participation there could lead to a good heads-wiki page that doesn’t exist right now).

Yes, that would be, as said previously, to verify decompressed initrd (Heads), decompressed from initrd.cpio.xz, currently running environment exposed binaries, libraries and scripts.

But this would not be consequential of someone not trusting already reported measurements as possibly forged to trust sha256sum to do the right thing here (even though chances being that busybox would be tampered with, kernel being tampered with and coreboot measurements hardcoded into bootblock, ramstage, romstage and possibly having scripts modified to also hardcode kernel modules measurements to be reported as good.) are extremely low, while possible.

All of this would need high targeting of a specific user to have consistent measurements reported to the TPM even if initrd would be modified, to have totp-unseal calls being successful, which is also discussed extensively under heads issues. Just like modifying one bit or of file generates a totally different hash (test it you will see), the hashes taken first from coreboot are extended. This means that to have the same final extended measurements (XOR between measurements from TPM) would require precise collision calculation to be able to provide a good, final measurement equivalent to the same final value sealed inside of the TPM nvram, needed to unseal TOTP/HOTP/TPM Disk unlock key. Otherwise, it would mean that all measurements reported to the TPM are now forged to match reported values to the TPM extend operations. That would mean that coreboot’s bootblock, ramstage, romstage would need to be hardcoded into those parts which talk to the TPM through extend operations, and Heads be modified as well to forge measurements that are TPM extended, including measurements of kernel modules, LUKS header hash etc (as detailed under Keys and passwords in Heads | Heads - Wiki) to have the distinct TPM nvram region unsealed at default boot if you have a TPM Disk encryption key setuped. Of course, if you resealed those measurements after compromise, there is no guarantee… And therefore, comparing with firmware that was flashed is the only possibility to verify that flashed firmware is in good state, comparing its parts with the firmware that was flashed.

Also as said, what is nice about maximized firmware images is that they flash the whole combined 12mb SPI flash space. So I would recommend taking a backup today and inspect later, but meanwhile, simply flash a new maximized firmware from master and reseal TOTP/HOTP, Generate /boot digest and sign, and show boot OS options, define new boot default and setup a TPM disk encryption key when asked. You will be able to investigate later, meanwhile trusting the state of your current firmware. Be sure you already use a maximized board (pretty obvious, main screen should say x230-hotp-maximized) prior of internally flashing. Otherwise simply and carefully follow instructions under Upgrading Heads | Heads - Wiki

I will wait for this thread to be splitted up to answer more specific questions. Heads relative questions should be under heads-wiki if documentation related, or heads if code related. Qubes is being booted from Heads on only a subset of machines, and hence is not completely related. I would also want to use this opportunity to remind users that not all Heads users who would have the same questions would come to Qubes forum: I try to answer here in the goal of having people go over Heads related issues to centralize the documentation/code questions and also to not duplicate my own efforts in multiple platforms. It might seem a good idea for a user to use this platform to have answers, but not all users would come here, even less in this current threads for “Compiling and flashing heads” to get integrity validation answers.

I’m not convinced into investing time to extend Heads internal supplemental validation if Heads is untrusted. That validation should be done externally, where as said previously, the quick fix is to flash the same firmware which should result into the same measurements/remote attestation as said under Upgrading Heads | Heads - Wiki. One should take an internal backup of the firmware to further investigate tampering if needed, but the chances of having flashrom tampered here are also really small, where hashes.txt could be used to verify flashrom internal binary hashes, but yet again, if nothing is trusted, the only thing that could be done is to flash externally. I understand the “do not trust anything” but then again, if this is the case, I would not advice pressing a computer’s power button.

Little extract from current Notes on how to audit a maximized flashed firmware image · Issue #107 · linuxboot/heads-wiki · GitHub

mount-usb rw
flashrom -p internal -r /media/$CONFIG_BOARD-$(echo $FW_VER|awk -F ' ' '{print $2}'_backup.rom
umount /media

Where the result in that example is

/media/x230-hotp-maximized-Heads-v0.2.0-1296-g139ecb8_backup.rom

And where 139ecb8 is the Heads commit leading to be able to visualize the hashes that were produced for a Heads specific commit, for a downloaded CircleCI build.

All those informations are injected into a ROM under what you can see under Heads recovery shell doing:
cat /etc/config
Which file is sourced from Heads early init code as environement variables, which can also be seen from recovery shell doing:
env

So basically, the snippet above is just naming a flashrom internally taken backup to correspond to the CONFIG_BOARD (here x230-hotp-maximized) with FW_VER stripped to just include Heads versioning information, in a similar way “System Information” menu is preparing the information to be showed on screen.

“System Information” is a menu option under Heads Options menu for a long time now.


Once again, let’s try to stay on topic, and Heads homes are:

2 Likes

hi @deeplow,
thanks for splitting the thread.

hi @Insurgo

sorry, before i was worried about creating duplicate tickets,
because usually, verification is part of compiling & flashing,
but then end up off-topic,

i saw your post in another thread, but forget which one,
saying that, it is okay to discuss about heads in Qubes forum,
because most of Heads’ users are Qubes’ user.

maybe all options are good.

either there is another forum, that specializing in firmware discussion,
such as: heads, coreboot, libreboot, skull, etc,

or maybe, also good, if there is new category inside Qubes forum,
to discuss about firmware (heads, coreboot, libreboot, skull, etc),
maybe for 3 reasons:

  • it can help to promote each other, i.e. between Qubes & Heads
  • most of Heads’ users are Qubes’ user
  • open firmware can add more security for Qubes
  • etc

also, it is good, to discuss about Heads, in the Heads Github wiki,
do you prefer to discuss about Heads in its wiki ?
so that can add more documentation,
if so then i will ask in separate more specific questions in the wiki

thanks and regards,

Yep, I questioned that, so I am @Confused too now.

It’s the other way around. Certified Qubes laptops today are based on Heads firmware. This may change in the future. And not all Heads users are Qubes users.

Therefore I do my best to answer questions through Qubes forum, but some if not all questions asked here are benefiting all Heads users, therefore not all Heads users will search Qubes forums to find answers and this duplicates the work, since the ultimate goal would be that no documentation is needed (clearly impossible) and therefore, documentation should be clearer for all Heads users, not only Qubes users.

I mostly agree, while not all Heads users are compiling firmware themselves. Having build logs and artifacts actually help testing things faster and ease collaboration. Ultimately, local builds and cloud builds should produce the same exact ROMs, but we are not there yet where funded work will happen soon to finally have reproducible builds again.

The hashes.txt is part of artifacts produced in a build, and tarballs, and dependencies hashes are things validated as part if the build process as well.

I prefer to direct people into inspecting public artifacts then locally produced ones until all reproducibility issues are resolved. This permits to have hashes known for a specific commit, which would vary today if you built the rom yourself, since paths and other OS variables are bleeding into the builds. This is documented under Heads issues, which I won’t detail here.

As any open source projects, discussing outside of their roof is good. But having users contribute back where they exist and evolve is better, reducing duplication efforts. Discussions should lead to documentation improvements and code improvements and understanding of mechanics, so users reading current documentation can propose improvements that will benefit everybody else later on. I’m not a documentation expert, unfortunately. And I would love to see more involvement directly under Heads.

I will try to open issues and entice discussions over Heads more directly in the future, so that discussions here can point there more directly.

:smile: I not much to contribute here. Often confused but try to follow :arrow_down_small: advice.

and

I consider all Insurgo contributions tremendous gifts, personally and, to Qubes community.
And respect

2 Likes

Absolutely.

4 Likes

hi @Insurgo

okay, i will prepare more specific questions,
so then each will have more specific answer,
then post in Heads Github wiki,
so hopefully can improve the documentation.

thanks a lot and regards,

2 Likes