Heads testing

General Discussion
,
1

Hello there! You use heads? You flashed it yourself? You still have an external programmer to unbrick your device in case something goes wrong? Then your help is needed!

Heads is a community driven project. It needs board owners to test and report once in a while when massive changes are happening. This is a time like that right now.

  • Heads recently enabled PR0, aka smi triggered chipset locking, where coreboot prepares the platform to have SPI accessible for read only access when locked, and where the payload effectively trigger locking just before booting the OS, which OS can only read the SPI now, even if iomem=relaxed is passed as kernel command line option. User can deactivate it through configuration settings menu at runtime, as can be seen in PR. heads is consequently the only internal flasher of firmware through flashrom. This was merged https://github.com/osresearch/heads/pull/1373.
    • It was not tested on all boards, but since enough similar boards were tested, it got merged nonetheless.
  • Now Heads is bumping internal kernel from 4.14 to 5.10.5, bumping coreboot 4.13 to 4.19, enabling libgfxinit to show bootsplash early at boot while i915frmfb kernel driver drives the framebuffer again, upgraded gnupg toolstack to 2.4 and flashrom from 1.0 to 1.3 to support new boards and new security dongles and preparing next steps for new upcoming functionalities. This is massive, and testing is needed under Staging branch for merging 5.10 kernel changes, gnupg2.4 and flashrom 1.3 (testing needed) by tlaurion · Pull Request #1398 · osresearch/heads · GitHub today.

As of right now, we need more willing testers owning boards and an external programmer (ch341a 1.6+ with voltage selector is cheapest recommended one available).

The following boards need more known board owners/testers:

  • t430-maximized, t430-legacy
  • x220-maximized, x220 (legacy locally built with backup extracted ME,IFD nad GBE. This will probably become deprecated soon since its really not that well tested…)
  • t420-maximized
  • t520-maximized
  • t530-maximized (with/without dgpu)
  • w530-maximized (with/without dgpu)
  • x230-fhd/edp mod (high res panel)
  • t440p-maximized
  • kgpe-d16
  • ASUS P8Z77 M PRO
  • Talos 2 (irrelevant for QubesOS as of today though)

@moderators feel free to move this post where it should be. Thought all-around-qubes but would be limited in numbers, support but not sure it applies.

Edit: also opened Heads needs testers with external programmers · Issue #8302 · QubesOS/qubes-issues · GitHub pointing back here. was closed.

3 Likes
2

To be clear - is external programmer needed only to unbrick if something goes wrong, or is it expected normal way to flash/update a development build (until proper update method is added?).

And also - what’s expected life time of flash in x230, how many flashing it’s supposed to survive?

3

I searched for MX25L6408E, but all Macronix chips are saying the same standard thing: 100,000 erase cycles
2023-06-29-112755

External programmer is needed for initial external flashing and in case something goes wrong (to unbrick). A development build can be flashed internally through normal procedures:

The difference there is that instead of downloading CircleCI roms coming from master latest commit, one has to download from CI test results of a PR, select the board flavor in question (maximized or not, HOTP or not, dGPU or not) to be tested.

Example for t520-maximized (without HOTP support) to be downloaded from Staging branch for merging 5.10 kernel changes, gnupg2.4 and flashrom 1.3 (testing needed) by tlaurion · Pull Request #1398 · osresearch/heads · GitHub on main computer:

Boot Heads computer that is going to be used to test downloaded firmware image.
For extreme precaution (flash/filesystem can become corrupted), we will validate the hashes once more.

  • Go to recovery shell. (Options-> Exit to recovery shell)
  • Connect prepared USB thumb drive to USB port.
  • Type mount-usb
  • Type cd /media
  • Type sha256sum *.rom
  • Type grep rom hashes.txt
  • Both hashes should match:
    signal-2023-06-29-104515
    signal-2023-06-29-1045152048×390 139 KB
  • Type reboot
  • Select Options->Flash/Update the BIOS->Flash the firmware with a new ROM, retain settings-> Yes
  • Select the rom filename to flash, Yes.

Re-owning states on flashed computer

Default boot

signal-2023-06-29-113551
signal-2023-06-29-1135512048×1152 175 KB

signal-2023-06-29-113642
signal-2023-06-29-1136421819×1023 363 KB

In case of bricking
External flashing needs to be done with latest version known working for that board. This means downloading top/bottom images for xx30 boards from CI, or single rom image for other platforms. That should be discussed under issues over github, ideally under pull request tested prior of it being merged, or under a new issue specifying the rom complete image name that was tested (which contains the commit ID, platform and exact flavor in question).

Example filename here:

  • heads-t520-maximized-v0.2.0-1613-g92e29c4.rom
    • heads : branding
    • t520-maximized: platform + flavor
    • v0.2.0-1613: release 0.2.0 + 1613 commits since that release
    • g92e29c4: commit 92e29c4
2 Likes
4

@marmarek : As of now, the “normal way” to update Heads is still what is described above. Once Heads finally moves to a dedicated repository under an organization, and proper rights are given to maintainers to create releases, CircleCI and rolling releases will continue to be the way Heads produces ROMs for supported boards: artifacts, available for download for 30 days. Releases will then be the next step, and releases should be added to LVFS, and then land to the user through fwupd updates.

Note that hashes.txt are produced not only as artifacts, but also a CircleCI step output and kept as step output forever there. So at any point in time, a Heads user could go into “System information”, get the commit ID from which his ROM is coming from and have access to hashes.txt output to verify ROM integrity against CircleCI produced ROM images.

7

As of today, and this is hopefully gonna change, the following boards have been renamed to UNTESTED platforms to be clear with the community:

  • UNTESTED_kgpe-d16_server
  • UNTESTED_kgpe-d16_server-whiptail
  • UNTESTED_kgpe-d16_workstation
  • UNTESTED_kgpe-d16_workstation-usb_keyboard
  • UNTESTED_leopard
  • UNTESTED_p8z77-m_pro-tpm1-hotp-maximized
  • UNTESTED_p8z77-m_pro-tpm1-maximized
  • UNTESTED_qemu-linuxboot
  • UNTESTED_r630
  • UNTESTED_s2600wf
  • UNTESTED_t420
  • UNTESTED_t520-hotp-maximized
  • UNTESTED_t520-maximized
  • UNTESTED_t530-dgpu-hotp-maximized
  • UNTESTED_t530-dgpu-maximized
  • UNTESTED_t530-hotp-maximized
  • UNTESTED_t530-maximized
  • UNTESTED_tioga
  • UNTESTED_w530-dgpu-K1000m-hotp-maximized
  • UNTESTED_w530-dgpu-K1000m-maximized
  • UNTESTED_w530-dgpu-K2000m-hotp-maximized
  • UNTESTED_w530-dgpu-K2000m-maximized
  • UNTESTED_w530-hotp-maximized
  • UNTESTED_w530-maximized
  • UNTESTED_winterfell
  • UNTESTED_x220
  • UNTESTED_x230-hotp-maximized-fhd_edp
  • UNTESTED_x230-maximized-fhd_edp

Hopefully, we, as a community, will be able to rename the UNTESTED boards back to their original names… after having been tested and reported as functional by board owners themselves stepping up.

Don’t be scared though, those are a mix of linuxboot boards (untested for a while) and variations of boards for which maximized versions counterpart are tested. You can pick any Heads commit (green mark) to see the CircleCI status for boards that are currently tested and maintained. I just took a stance to lower importance of boards for which owners are unknown for the moment and for which importance seems to need to be reconsidered.

Now would be a good time to step up if you own such board under Platform blobs, collaborators/maintainers/testers for faster problems resolution · Issue #692 · osresearch/heads · GitHub

2 Likes
8

@Insurgo I can test on W530. Does it matter if I simply flash it via USB versus flashing the SPI chip if I already have HEADS running and it was hardware flashed initially?

If I’m not mistaken once HEADS has been flashed on to the SPI chips you can internally flash it to the same effect.

Let me know and I will update and notify if there are any issues. Is there anything specific I should be looking out for? I have a fairly basic range of use, so if there are tasks to perform that might help elucidate issues let me know.

1 Like
9

It’s actually best to flash internally keeping settings (key material, config override etc) then flashing externally (which would require to reinject your public key and reconfigure things manually as documented on https://osresearch.net)

Exactly as replied above. Just make sure the hash of the firmware on your USB thumb drive matches the build rom hash.

That greatly depends on how old your actual firmware version is compared to what the current version is on master.

The best way I have found to compare this is through github.
Note once again Heads filename/versioning scheme:

So here, if you go through “System Information” option within Heads, you will get that information, which would help me know which flavor of w530 you are actually testing (gGPU, HOTP and whatnot).

The commit is the important part to get the list of changes and commits that happened since the version you are currently running, replaced in the following URL: replace the commit part of the url (92e29c4 here) with your current commit id:

Scroll down on that page, click on Load more commits and you will then be able to search the whole page (Ctrl-F) with text Merge pull request which will let be aware of “feature/bugfixes” Pull Requests (PR) where discussions happened prior of the actual merges since your currently used firmware version.

@KarlinQubes May I ask you to write a line under Legacy and untested boards now considered untested until tested · Issue #1421 · osresearch/heads · GitHub with the filename of the W530 rom you will be testing from master? (w530 is now named UNTESTED_w530* until confirmed tested, and will be renamed once tested and reported working by board owner.)

@KarlinQubes thanks for replying here! Means a lot!

2 Likes
10

To clarify, I did not mean in order to compare my current (older) version of HEADS with the new version UNTESTED_*, I was just asking if there is any functionality of HEADS that I should also test from the menu that might be helpful to see if everything works.

I am not a technical user and so my use of HEADS is just basics.

I believe my hardware is w530 k1000m dgpu. Would it be helpful if I tested each board that is applicable? So: w530-maximized / hotp-maximized / k1000m-maximized / k1000m-hotp-maximized?

Currently I am just running the w530-hotp-maximized version. Perhaps I will try to get the dgpu version working with external screen with the untested board.

Thanks for your hard work, I’m excited to be able to contribute in some minuscule way. I’m building HEADS now and relearning the details so I can produce all these boards.

Also, what is the address for the HEADS matrix server?

2 Likes
11

That is what i meant by checking what is new for you to discover and test since your last firmware update though. When it comes to menu option, those are already tested across qemu and other boards, so ita basically coreboot config + linux config for that board flavor.

So in your case, what would be interesting to report would be for w530-hotp-maximized, and if you have a spare board with k1000m gdpu, k1000m-hotp-maximized as well. The risk of bricking lies there, since Heads recently enabled libgfxinit for coreboot native graphic initialization across all Intel iGPU, where k1000m-hotp-maximized depends on VGA bios to do the graphic initialization on dGPU.

As to test non-hotp boards, if you only have one HOTP enabled security dongle(nitrokey pro/librem key/ nitrokey storage), i would suggest testing HOTP on main computer, and non-hotp on spare, since a HOTP dongle can only be bound to a single computer and non-HOTP variant will rely only on TOTP+Smartphone for firmware integrity attestation on boot after Qr code is generated and scanned on smartphone.

Room address is #OSFW-Heads:matrix.org

1 Like
12

I would love to take a second to say that I really appreciate the collaboration that came out from this post and other sources. There is an increase in participation resulting in more feedback and speed of testing from the community.

I will not keep the list of boards in this thread updated, but the OP here is updated accordingly Legacy and untested boards now considered untested until tested · Issue #1421 · osresearch/heads · GitHub

2 Likes
13

2 posts were split to a new topic: Coreboot/Heads flashing help/services?

14

And… This is another time for board owners to willingly test and report success/failure, for those of you having external programmers.

The current PR needing testing:

  • Improves boot speed by 3 seconds (bringing x230 to under 6 seconds with a beautiful bootsplash coming from coreboot native ram initialization)
  • Is the result of important collaboration with coreboot developers!!! (Thanks Nico Huber/@icon for the fix to fbwhiptail, efifb trampoline/exposition to kernel and imagemagick programmatically doing needed voodoo to have bootsplash and efifb today! See added patches under that PR for more details).
  • Replaces i915drmfb which was previously driving graphics to a much simpler efifb kernel driver, which coreboot enables correctly (gain of 500kb firmware space)
  • Makes fbwhiptail snappier! Refreshes barely noticible.
  • Qemu/KVM targets now using same graphic init path, also permitting to boot into VESAFB enabled OSes, incluging tinycore a lot of people wanted for testing.
  • No more flickering/screen corruption at kexec!
  • No more hack needed under kexec to boot into next kernel.

The reason I need this PR to be merged is because of the 500kb freed space, and because this is a massive change that cannot be tested without boards owners testing new boot path. That is, most of the boards are affected by this change and it changes the way we use both the linux kernel and coreboot to boot our platforms from the graphic initialization routine up to how we kexec into the final OS kernel.

To whom I talked personally before, this permits to kexec into PureTinyCore, which was not working prior. Why? Because PureTinyCore, as the future of linux distributions trying to deprecate Legacy boot, PureTinyCore was only providing VESA framebuffer drivers. That is, it was impossible to boot into those OSes that were only providing VESA drivers into their initrd.

Fedora 36+ tried to deprecate Legacy boot before. And they are still planning to. This change in Heads makes it more resilient to the current planning, since it piggybacks into coreboot native graphic initialization to prepare a EFI compatible framebuffer (which can be initialized by GOP, Optional video rom or libgfxinit currently) and permits to use EFIFB provided by linux kernel by default. It is also compatible with simplefb and simpledrm on kexec’ed OS, so we do not care about the deprecation plan of VESA/higher DRM drivers into initrd and unified kernels anymore from the Heads side.

The PR to be tested this time, and the boards not currently have test reports is in the main post at libgfxinit/nativegfx init: efifb enforced fb (+coreboot ramstage enabled bootsplash) by tlaurion · Pull Request #1403 · osresearch/heads · GitHub

Instructions to test PR is at the current bottom post at libgfxinit/nativegfx init: efifb enforced fb (+coreboot ramstage enabled bootsplash) by tlaurion · Pull Request #1403 · osresearch/heads · GitHub

If you would be willing to be a recurrent tester of Heads, please notify me @tlaurion under Platform blobs, collaborators/maintainers/testers for faster problems resolution · Issue #692 · osresearch/heads · GitHub

Stay tuned for massive features coming soon! (Which will need less drastic testing then now, but still!!! Each time there is a coreboot/linux version bump or a change that is board specific, we need testers. So the more official willing testers we have, the faster testing occurs and the richer Heads will become!

Note on recent changes under Heads:

Thank you all! Next step is Add secure thumb drive creation premisses by tlaurion · Pull Request #1446 · osresearch/heads · GitHub going toward risk-free Authentication under Heads (preventing unauthenticated: recovery shell access, flashing, unsafe boot or USB boot options).

1 Like
15

EDIT: Merged on 05/09/2023!

Today lands another Pull Request that was long awaited: TPM Disk Unlock Key setup/renewal, even more upon internal firmware upgrades, has been eased quite a lot.

@Sven I tagged you there, but this is the standard usage anticipated for all Qubes users of Heads: Ease TPM Disk Unlock Key sealing/resealing after TOTP mismatch (firmware upgrade) + warn and die changes by tlaurion · Pull Request #1482 · osresearch/heads · GitHub

Why this change is important:

  • QubesOS /boot/initrd was not parsed previously: Heads was generating a generic crypttab override (which was working, but if QubesOS changed the way it decided to pass LUKS options, those were not reflected from Heads which was static: for example it was forcing discard on LUKS). As of this PR, only /secret.key is overriden from dom0’s crypttab, which can be modified by the OS into boot’s initrd, and now reparsed when setting a new boot option (which codepath is triggered when grub.cfg is updated by OS, and can update its crypttab and consequently modify its initrd).
  • This PR also fixes long term pending Debian manual fixes that were needed into host’s /etc/crypttab file to hardcode /secret.key. Now Heads parses initrd crypttab (cryptroot/crypttab) and overrides it correctly.
  • Heads now detects LUKS containers and only suggests the LUKS volumes/LVM+LUKS instead of asking the user to know those details. Basically all LUKS drives suggested should normally be selected, code validates input against existing devices, unless advanced custom setup is known by the user. Note that default installation normally has a single LUKS container on disk and in this case Heads doesn’t prompt the user and just uses it by default. Easy.
  • Heads now proposes to reseal TPM Disk Unlock Key upon firmware upgrade when TOTP could not be unsealed (firmware has changed… Past sealed measurements cannot unseal) when Heads TOTP/HOTP is resealed by user choice. It now automatically prompts the user into resealing TPM DUK if previously configured, ask to reuse past configured devices and regenerates /boot hashes (since LUKS header dump is measured and Heads refuses to boot if those changed) and ask the user to detach sign /boot hashes and boot options.
  • At any time (if a user decided to add a new drive and configured it under QubesOS’s crypttab to be also unlocked+mapped and usable eg. for backups), the user can decide to have TPM DUK also added into the new drive LUKS header and unsealed by TPM DUK on boot. To do so, he needs to set a new boot default and when asked to reuse previously configured LUKS drive, to choose to not use previoulsy configured ones and retype all the LUKS devices suggested. Heads then regenerates a new TPM DUK, adds it under each LUKS drives slot 1, and overrides Qubes OS initrd provided crypptab to tell final OS to use initramfs’s crypttab to take the TPM DUK passphrase to also unlock that additioanal drive. You can see this in action at Ease TPM Disk Unlock Key sealing/resealing after TOTP mismatch (firmware upgrade) + warn and die changes by tlaurion · Pull Request #1482 · osresearch/heads · GitHub
  • 3 issues are fixed with this PR

There is no regression under this PR and this one can safely be flashed internally without external programmer.

@moderators Should I create another thread for testing needed without the need of users to have external programmers? (This one was for request of users to do some risky testing, requiring to have a way to unbrick).

I take the opportunity of writing this to say to all that I will take some time away of keyboard. Expect a little more delay in my answers then normal, my soul needs to get away of screens a little bit and enjoy the sun before autumn strikes.

Be safe!
tlaurion

4 Likes
16

I think we just keep this as the ‘Heads testing’ super thread in ‘General Discussion’?

Enjoy!

1 Like
17

Good for renaming the thread! (Tired but didn’t found how!)

Thanks!

1 Like
18

Note for future posts reading : I will put in bold if testing requests should be done only by users having an external programmer in case changes are risky to test and might require unbricking.

2 Likes
19

User authentication prior of landing into recovery shell/USB boot landed upstream

It is now possible to generate keys in memory and create a USB Thumb drive encrypted backup over LUKS of the master keys and subkeys, while the subkeys can optionally (recommended) be copied to smartcard. If such USB Thumb drive backup exists, Heads authentication is enabled by default!

This is available under OEM Factory-reset/Re-Ownership wizard where the flow is explained under https://github.com/linuxboot/heads/pull/1515#issuecomment-1808964736

What does that mean
Well. You do not need a USB Security thumb drive (smartcard) anymore to use Heads. That dedicated USB thumb drive alone can be used to authenticate you with a passphrase you chose (which will unlock the LUKS encrypted partition) and load and use the subkeys, matching the public key fused in the firmware, which is measured/attested through sealed secrets inside the TPM.

Wait. What does that really mean?
Well, if you decide to generate master key and subkeys in ram, and copy those subkeys to smartcard, you’re all set. You can use either of them to authenticate yourself when you do anything else then booting your default boot option. That also means that if an evil maid can boot your computer when you are away, that person cannot boot external media or go to recovery shell to wipe your QubesOS installation anymore. And as a bonus, shoulder surfing of that passphrase you could have typed before is not enough. That evil maid would need to have both something you have (USB Security dongle/USB Thumb drive backup of key material) and something you know (USB Security Dongle’s User PIN/USB Thumb drive encrypted key passphrase) to do those potentially harmful actions on your computer.

Again, the TPM sealed Disk Unlock Key is the best security precaution you can enable on your Heads laptop, and doing so is documented in the main documentation under Step 4 - Installing Qubes and other OSes | Heads - Wiki

Enjoy!

DISCLAIMER: IF YOU DECIDE TO DO THE OEM FACTORY RESET/RE-OWNERSHIP WITH A USB SECURITY DONGLE YOU ALREADY OWN, THE KEY MATERIAL ON THE SMARTCARD WILL BE LOST.

Thanks:

  • Jonathon Hall from Purism for his amazing support on this long awaited feature and collaboration. That reviewing process was the longest I have been through… ever. Bonus: code, terminology and unification/refactoring of code has been streamlined in the process.
  • Thanks to NLnet to believe in me and support my R&D efforts for so many years already. That project was realized through NLnet; Heads-OpenPGP funded work. NLnet: you support the next internet and the kind of world I want to live in. Thank you for that.
1 Like
20

Hello everyone!

Desperately looking for owners of the following boards

  • t420
  • t520
  • t530
  • Asus p8Z77 M PRO
  • HP Z220 CMT
  • w530

with hexternal programmers to test WiP: FB_EFI (EFIFB kernel module's framebuffer on top of libgfxinit or GOP) next steps related cleanups by tlaurion · Pull Request #1522 · linuxboot/heads · GitHub prior of merging!!!

This is final step getting rid of Intel i915+drm kernel driver that is in firmware to use efifb only and standardize the booting process across all boards. As a bonus that gives a magnificient bootsplash on boot and speed up boot process and getting console on final OS booted. Please reach out!

Poke also done under Matrix at You're invited to talk on Matrix

Thanks for your help as usual!
May freer firmwares be with you.

2 Likes