To upgrade Insurgo Privacybeast to 4.1 or not?

User Support
21

@Demi :

Heads packaged cryptsetup1 initially. To have support for TPM released Disk encryption key, cryptsetup2 needs to be in the firmware, you are right. The user could theoretically still boot their laptop by typing their LUKS disk encryption key passphrase. But they could not install Q4.1 without having done the firmware upgrade first, read below on the whys.

@Sven The firmware update is not only for that LUKS2 support, but to resolve graphical glitches that were not permitting Qubes 4.1 graphic installer to be launched without a coreboot upgrade (coreboot+vbt). Otherwise, users attempting to install Q4.1 would face QubesOS 4.1 installer doesn't boot on sandy/ivy bridge (xx20/xx30) ¡ Issue #789 ¡ linuxboot/heads ¡ GitHub graphical glitches, and Qubes installer just fails after a while, trying to work in text-based version which is not aimed to be functional under Qubes even today.

22

Unfortunately, this is why things are complicated, since how Heads was deployed before Maximized builds existed (legacy boards).

I can try to summarize here again real short. Initially, Heads was aimed to not interfere with ME. Since the Intel Firmware Descriptor is normally locked, and the BIOS region is normally protected, Heads and Skulls required an external initial flash of the 4mb top SPI chip present in xx30 boards (the X230 was the first board to land under Heads).

Neither Heads nor Skulls required the end user to do anything with the 8mb bottom SPI flash. Still today, legacy boards do not expect the user to have neither unlocked the IFD or have neutered ME to take advantage of the freed space there. So Skulls and Heads (x230-flash) prepared 4mb roms to be flashed to the top SPI, and then flash internally the 7mb bios region, respecting the BIOS region defined under IFD.

The reasoning of this comes from the alignment of the BIOS region, which is spanning across the two SPI chips. See the flash layout here: Lenovo Ivy Bridge series — coreboot 4.22-679-gcc2ab49525 documentation

So to unlock IFD and neuter ME, the 8mb chip needed to be backuped and those instructions needed to be followed Cleaning Intel Management Engine | Heads - Wiki. ME was neutered, consequently, but the ME freed space was not made available to the BIOS region. The only way to do that is to also change the IFD with a smaller ME region matching its size, and the BIOS region expended (maximized) to take advantage of the freed ME space (bringing the BIOS region available space from 7mb to 11.5mb). This is what Maximized boards are.

1vyrain project landed, and was able to chainload exploits to bypass BIOS protections, mainly from S3 resume path vulnerabilities to unlock the BIOS region and be able to flash 4mb ROM images (Heads x230-flash and Skulls), permitting the end user to then flash the 7mb BIOS region contained under the invalid 12MB image (x230 legacy rom does not contain a valid IFD and is only flashing the BIOS region defined under the IFD). But it is not possible to unlock IFD nor ME regions internally. An initial external flashing of an unlocked IFD and ME region is required.

The PrivacyBeast shipped with IFD unlocked (as opposed to the NitroPad in the past). The PrivacyBeast permits users to upgrade to maximized boards, internally, but with a manual flashrom invocation, since maximized roms contains a valid IFD, a valid neutered ME and a valid GBE region. And because the ME region and IFD regions are unlocked.

So a PrivacyBeast owner can move away from legacy to maximized roms, manually upgrading once from the recovery shell by doing:

  • mount-usb
  • flashrom --force --noverify-all -p internal -w /media/PathToMaximizedRom.rom

Otherwise, the GUI will try to overwrite the 7mb BIOS region with a 11.5 BIOS region, which of course would result in a brick

All future firmware upgrades will flash the whole 12mb rom internally from the GUI. But the initial upgrade needs to be manual, which floods my support channels.

1 Like
23

@Insurgo
I see that you updated the above posts. Some people, including some core developers, are interacting with this forum via email. They do not receive the edits made after 10 minutes. For this reason, it’s better to make a new post, not update the old one (and/or use quotations like I do below).

Here are your edited posts for them

@Demi :

Heads packaged cryptsetup1 initially. To have support for TPM released Disk encryption key, cryptsetup2 needs to be in the firmware, you are right. The user could theoretically still boot their laptop by typing their LUKS disk encryption key passphrase. But they could not install Q4.1 without having done the firmware upgrade first, read below on the whys.

@Sven The firmware update is not only for that LUKS2 support, but to resolve graphical glitches that were not permitting Qubes 4.1 graphic installer to be launched without a coreboot upgrade (coreboot+vbt). Otherwise, users attempting to install Q4.1 would face QubesOS 4.1 installer doesn't boot on sandy/ivy bridge (xx20/xx30) ¡ Issue #789 ¡ linuxboot/heads ¡ GitHub graphical glitches, and Qubes installer just fails after a while, trying to work in text-based version which is not aimed to be functional under Qubes even today.

Unfortunately, this is why things are complicated, since how Heads was deployed before Maximized builds existed (legacy boards).

I can try to summarize here again real short. Initially, Heads was aimed to not interfere with ME. Since the Intel Firmware Descriptor is normally locked, and the BIOS region is normally protected, Heads and Skulls required an external initial flash of the 4mb top SPI chip present in xx30 boards (the X230 was the first board to land under Heads).

Neither Heads nor Skulls required the end user to do anything with the 8mb bottom SPI flash. Still today, legacy boards do not expect the user to have neither unlocked the IFD or have neutered ME to take advantage of the freed space there. So Skulls and Heads (x230-flash) prepared 4mb roms to be flashed to the top SPI, and then flash internally the 7mb bios region, respecting the BIOS region defined under IFD.

The reasoning of this comes from the alignment of the BIOS region, which is spanning across the two SPI chips. See the flash layout here: Lenovo Ivy Bridge series — coreboot 4.22-679-gcc2ab49525 documentation

So to unlock IFD and neuter ME, the 8mb chip needed to be backuped and those instructions needed to be followed Cleaning Intel Management Engine | Heads - Wiki. ME was neutered, consequently, but the ME freed space was not made available to the BIOS region. The only way to do that is to also change the IFD with a smaller ME region matching its size, and the BIOS region expended (maximized) to take advantage of the freed ME space (bringing the BIOS region available space from 7mb to 11.5mb). This is what Maximized boards are.

1vyrain project landed, and was able to chainload exploits to bypass BIOS protections, mainly from S3 resume path vulnerabilities to unlock the BIOS region and be able to flash 4mb ROM images (Heads x230-flash and Skulls), permitting the end user to then flash the 7mb BIOS region contained under the invalid 12MB image (x230 legacy rom does not contain a valid IFD and is only flashing the BIOS region defined under the IFD). But it is not possible to unlock IFD nor ME regions internally. An initial external flashing of an unlocked IFD and ME region is required.

The PrivacyBeast shipped with IFD unlocked (as opposed to the NitroPad in the past). The PrivacyBeast permits users to upgrade to maximized boards, internally, but with a manual flashrom invocation, since maximized roms contains a valid IFD, a valid neutered ME and a valid GBE region. And because the ME region and IFD regions are unlocked.

So a PrivacyBeast owner can move away from legacy to maximized roms, manually upgrading once from the recovery shell by doing:

  • mount-usb
  • flashrom --force --noverify-all -p internal -w /media/PathToMaximizedRom.rom

Otherwise, the GUI will try to overwrite the 7mb BIOS region with a 11.5 BIOS region, which of course would result in a brick

All future firmware upgrades will flash the whole 12mb rom internally from the GUI. But the initial upgrade needs to be manual, which floods my support channels.

1 Like
24

Otherwise, the GUI will try to overwrite the 7mb BIOS region with a 11.5 BIOS region, which of course would result in a brick

That’s a GUI bug. Is there a reason that the GUI cannot do the steps you mentioned?

25

It’s not a GUI bug. Technically, board configurations declare how flashrom is to be called in the future for the next firmware upgrade. This is why upgrading from legacy to maximized boards needs to be done from recovery shell, once. I feel we are complexifying things (technical speaking mixed with worried users doesn’t seem to help.)

This is defined at build time and exported under initrd under /etc/config (which contains git commit and other exported variables per board configuration). The GUI (actually flash.sh script) will flash the firmware following what is configured to be used.

For legacy boards, only the IFD BIOS region is expected to be touched by an upgrade:

Where for maximized boards, flashrom is expected to flash the whole combined 12mb opaque SPI chips (8mb+4mb):

CONFIG_FLASHROM_OPTIONS is then used in the flash.sh script, called by flash-gui.sh to know how to communicate with SPI chips:

So in this case of legacy to maximized ROM upgrade path, the end-user has to explicitly, and manually, tell flashrom to flash the whole 12mb combined 8mb+4mb SPI chips (not only the BIOS region):

flashrom --force --noverify-all -p internal -w /path/to/12mb.rom

As opposed to using per board declared CONFIG_FLASHROM_OPTIONS in firmware which otherwise would only flash the ifd defined BIOS region with something too big to fit in it, which would happen otherwise since flashrom options are set to do so for legacy boards:

--ifd --image bios

Again, moving from legacy to maximized board is only possible if IFD and ME regions were unlocked on initial external flash, which is the case here with the PrivacyBeast.

So upgrading the firmware to maximized builds is possible and recommended on the PrivacyBeast. It requires the user to download the firmware, verify the hashes, put the rom on a USB dongle, go to recovery shell and call flashrom as written.

Qubes community (@fsflover @golden_jar56 : How can I make the following guides clearer for you to follow:

As said previously, more technical blog posts have been written and followed by customers who contacted me directly, hosted over ZeroNet and accessible through clearnet proxy here: https://zerolink.ml/1DMb3CV66qZPwJqkgm4z12nu8BrAwDoD4g/?Post:57

Let me help you help me :slight_smile:

26

I still think this is something that should be handled by the GUI, even though it requires a special case in the code. Users expect the GUI to Just Work, even if it needs to go through contortions to do so.

27

@Demi you cannot change the ROM flashrom statement magically. And implementing the change you are talking about would also require a firmware upgrade, so that GUI change would deal with the new logic, which would only be applied for the next firmware upgrade. Right? :slight_smile:

28

Ah, I thought that the GUI ran in the OS. I did not know that the GUI was part of the firmware.

1 Like
29

@Insurgo

Appreciate the attempts to try to help. Unfortunately, there is simply no way for me to perform the upgrade without messing something up. I dont feel comfortable using qubes while its outdated, and the likelyhood of compromising the entire machine by messing up the upgrade myself is even greater than that. I am afraid I will simply have to retire my privacybeast completely for the foreseeable future.

If at some point either “selling” my current privacybeast back to you and purchasing a new one with 4.1 already instaed, or simply mailing my current laptop back and paying for the upgrade service on it is an option, please be sure to let me know, I would be very interested in either option.

30

As a newbie to Qubes in general, and looking ahead to solutions such as PrivacyBeast, it seems like documentation is often written as “type this”, “expect this”, but rarely does it cover “but if it didn’t do that”. I don’t know if it’s too detailed to explain what those bad things are and why it’s important. I do know it’s a large documentation undertaking. Linux consoles are full of noisy information. So much output makes it difficult to know what’s really important. Especially when dealing with specialized hardened solutions. Maybe start there?

In your defense, the mechanics involved and the technical details behind them are, in most cases, already over simplified. This is no simple feat and the level of detail required is so unpredictable when a user’s knowledge or exposure is limited to a specific use case/task. Moreover, the technology evolves rather quickly in this arena.

Just because one has access to a precision instrument, doesn’t negate a certain level of knowledge required. On the flip side, “search the docs/forums” for answers isn’t generally a good support response either when one doesn’t know if they are even asking the right question.

Genuinely feel for both sides here.

1 Like
31

@golden_jar56 I’m sorry the steps seem complicated. Unfortunately, upgrading firmware is needed. And as of now, some fixes are still applied to the qubes 4.0->4.1 upgrade script.

Qubes users, as of now, are expected to be able to do backups and restores per How to back up, restore, and migrate | Qubes OS instructions. The “complicated” step here is to upgrade firmware manually a first time, which I can understand being worrying. If this is worrying you, I would gladly receive your concerns and address them one by one with documentation instead of providing a reinstallation service (providing a fishing line instead of unlimited supplies of fishes, if that is an expression also in english.) I would not really help you doing so, and that would be crazily expensive, shipping/importing it back and forth. That would be the worst foreseeable situation.

The steps, in short, are:
0. Backup Qubes over external storage. This is encrypted. Can coexist on unencrypted storage.

  1. Preparing a USB thumb drive (format it with ext3/ext4) having enough space for Qubes ISO and firmware. That can be an external USB hard 2.5" drive where you would also put your Qubes encrypted backups (500Gb+ extendable to your budget) or a simple 8Gb USB thumb drive if you are on a budget. I would still recommend backups. Did I say backups? Not doing backups on external drive is really risky to anyone. Qubes is made to be backuped/restored on reinstalled clean deployment when needed.
  2. Download Qubes from a disposable qube/dedicated qube, directly from https links/through torrent software, directly on the prepared USB thumb drive, that storage devices being passed to that disposable qube, or create qube for that role. Logic behind that is that there is no need, really, to keep the downloaded ISO locally. Torrents is recommended over sys-firewall, since torrent are not illegal (torrenting can be illegal IF used to download illegal content, otherwise the protocol resolves 2 problems in one: integrity validation and parallel downloads; which is why Qubes provide such link). If downloading through a stable internet connection from HTTPS links (ISO, and detached signature download links), that is good enough. Heads will verify the integrity/authenticity of the downloaded ISO+detached signature prior of attempting to boot from it. It will not boot from a corrupted downloaded ISO.
  3. Download firmware to USB storage. Verify checksum prior of manually flashing per instructions.

I have done a quick guide for customers, available on ZeroNet.
Here is a functional clearnet proxy on the article: https://zerolink.ml/1DMb3CV66qZPwJqkgm4z12nu8BrAwDoD4g/?Post:61:Downloading+and+installing+Q4.1

I would love to address the concerns; not feed them.
It is simply not possible to offer upgrade as a service. It would be a really costly reinstall service, which you will probably need to do again later if not learning from this experience.

Unfortunately, Whonix will keep its distinct EOL policies from Qubes, and Whonix will deprecate itself in next releases of Qubes as well. Qubes requires of its users to do in-place Templates upgrades or download new Templates once in a while as well.

When Qubes does a new release, the best way, historically, is still to do a clean install and restore backups on top of it. If you need coaching doing so, please contact me directly. But I restate that clarifying instructions if worrying to you is the best solution.

2 Likes
32

Hi folks,

Firmware download/upgrade/prerequisites cleanup by tlaurion ¡ Pull Request #93 ¡ osresearch/heads-wiki ¡ GitHub has been merged and pertinent additional pages can be found under

Q4.1 install guide was already at

I still need to cleanup

Feedback from the community always welcome, since it is there for all to understand/improve!

33

Also did

I still need to cleanup

34

Just ran through this upgrade last night. Some early thoughts and advisories for those nervous about it like I was.

I printed out the instructions. Easier to markup and line through things I didn’t need or didn’t apply. Unfortunately, I went to the osresearch wiki and went top to bottom. Completely neglected to notice, at the very absolute end of Step 1 - Downloading Heads is a completely separate line for upgrading. So I ended up wiping all the GPG keys and all that. Complete scorched earth. No survivors. Oh well.

**SUGGESTION for @Insurgo - Note that there is a separate path for upgrading HEADS under the section in Step 1 " Migrating from on board configuration to another". Just in case they do like I did and presume to go with ‘Step 1’. **

Point to this upgrade section and note that for users whose machines are compatible for internal flash to new maximized version can follow instructions under “Flashing new firmware under Heads” there for this portion of the upgrade process.

I’d also note this subsection big and bold in a second sentence in the first paragraph of the “Upgrading Heads” introductory section.

Outside of this, I had no issues walking through this from a procedural standpoint. I do have a bit of IT experience, working the field and having worked on other hardware and done some guided reflashing of a similar vein.

My second request regarding the wiki would be to note in clear text commands being run in Step 1.
Specifically, I was a bit nervous about the new flash command -
(flashrom -p internal -w /media/new_heads_rom_version_here.rom)

  • as the image in the installation instructions in ‘Step 1’ section were just blurry/indistinct enough that I wasn’t sure if I was looking at a ‘w’ or some other character.

All-in-all, the experience was not bad, even with my screw ups, as one might make it out to be in advance.

To my fellow users - BACK UP YOUR VMs.
I personally recommend this in three sections.

  • AppVms in one backup

  • templates in another

  • sys/dvm in a third

Time to verify the headers and open these can be long if you have everything crammed in one. If you have standalone VMs (like Windows which I rarely ever used), then back it up with one of the three above according to your usage.

Previously, I did this only because I backed up the actual AppVMs I worked in frequently, with templates and sys/dvm VMs rarely getting backed up since their configurations didn’t change.

Less space consumption per backup on my external device that way, less time spent doing backups, and easier in the event of an upgrade, as I found out today.

It has been easier than it might have been to be able to access the qubes I need as I am rebuilding my environment today. Since I have them broken up, I can unpack the couple I need from a relatively short list quickly instead of sorting through a 25+ qube list in the backup tool for the three I want, unpacking,
having it close out, then decrypting it all again for the next 3 or 5 I want to do, and sorting through that long list again.

MAKE SURE YOU HAVE THAT BACKUP ENCRYPTION PASSWORD NOTED DOWN!
Seriously, its easy to be lazy and always use the same encryption password stored in dom0 and forget whether or not you have it down somewhere. In my case, I noted it down during my first backups, then promptly forgot about it during this whole process until this morning where upon you can imagine I was a bit anxious.

Thankfully, I confirmed that previous-me’s paranoia about losing data/access to data had seen to noting that passphrase down on day one with the first round of backups I ever did more than a year ago.

The upgrade process is not bad. If you are a Privacybeaast owner, this is a relatively painless process. Don’t make it out to be more than it is. And I’ve always found the Insurgo team to be responsive to questions and willing to explain. Print out the (correct) instructions, annotate, ask some questions, make your last backups, and make it happen.

This also has the side benefit of refreshing your user experience. I might wipe Qubes and reinstall every year or at least every new version release. It wasn’t that bad and having walked through it once, doing this again in the future is not so daunting.

Thank you for your work @Insurgo in making the machine and the process accessible.
I’m looking forward to a new Privacybeast based on some newer hardware so I can have a bit bigger screen and little higher resolution.

2 Likes
35

Thanks for the review @justanothervmjockey.

My time to review your review here:

So if you were using Insurgo firmware (unlocked IFD, unlocked ME, but legacy nonetheless) and migrating to osresearch maximized rom (x230-hotp-maximized 12mb image), what you did there was the exact steps expected, minus maybe having instructions to extract your public key prior of flashing to be able to reinject it on freshly flashed firmware?

On first boot into the fresh firmware, Heads complained of not having any public key in ROM, right? So the problem here was that you didn’t have any backup of your public key to provide in a USB thumb drive?

Here your request would be to have that textually written, separately from the screen capture?
Please provide links to subsections and tag me back, I will gladly make it easier for everyone by modifying upstream instructions.

36

“So if you were using Insurgo firmware (unlocked IFD, unlocked ME, but legacy nonetheless) and migrating to osresearch maximized rom”

No, I had a maximized board. Where I went wrong was I followed Step 1 and flashed the board from the shell rather than following instructions under the Update Heads page which would have pointed me to use the Heads GUI for upgrading as specified under " Flashing new firmware under Heads" (bullet point in question copied below)

If you are not migrating from Legacy boards to Maximized board configurations, you can safely upgrade from Heads GUI through Options->Flash/Update BIOS and choose the retain settings option. This will copy your GPG keyring and user configuration into the new ROM prior of flashing it the whole combined 12Mb SPI flash with the ROM.

I instead executed the flashrom -p internal -w /media/heads-hotp-maximized-version-gcommit.rom command found under Step 1, wiping all my keys and such. Which was no particular big deal for me.

“On first boot into the fresh firmware, Heads complained of not having any public key in ROM, right? So the problem here was that you didn’t have any backup of your public key to provide in a USB thumb drive?”
Yep. Missed the chance to back up the keys due to my failure about.

And I had messed up in not having the backup copies with me when I did this. Completely on me. I forgot the storage drive with all the backup stuff. So once I realized I had gone wrong, I shrugged my shoulders and carried on. I was with family hours from home and it didn’t really matter.

But if it matters for someone else, I’d like to spare them the heartache. I’m also guessing the GUI walkthrough is more comforting for people who are nervous about this.

“Here your request would be to have that textually written, separately from the screen capture?”
Correct.
Step 1 Downloading page Step 1 - Downloading Heads | Heads - Wiki
Under " Migrating from on board configuration to another" section.
Specifically, this text:

If you have no warning/error (see above), you can proceed with a manual flashrom command to migrate once and for all to Maximized board:

edited to read:

If you have no warning/error (see above), you can proceed with a manual flashrom command to migrate once and for all to Maximized board:
flashrom -p internal -w /media/heads-hotp-maximized-version-gcommit.rom

OR it can mimic the exact same instructions as seen under the Upgrading Heads page under " Unlocked IFD and ME" section (where the exact same set of pictures is used.

37

@justanothervmjockey :

So. On Step 1 - Downloading Heads | Heads - Wiki, bottom of page:

  • I clicked Edit this page on GitHub.
  • Searched in page for " If you have no warning/error (see above), you can proceed with a manual flashrom command to migrate once and for all to Maximized board:"
  • Implemented your change.

Instead of doing a PR, I committed directly into heads-wiki, which https://osresearch.net page should be updated in a minute.

May I ask for suggestions into heads-wiki to be discussed/proposed directly under heads-wiki to facilitate documentation readability for end users? That would be so much more beneficial for the whole community.

I expect qubes users to be aware even more then Heads community that community driver projects benefit even more directly of community driven participation in documentation.

As of

I am still confused how you were already under a maximized board and followed the migration path instead of directly following the upgrade path, i’m sorry. Can we continue collaboration directly under a PR you would create following above directions? When it come to single page edits, it can be done directly through Github without the user having to deal with commit signing etc and would be so much more easier to track and understand what is going on there.

Thanks!

38

Apologies. Been traveling.

When I get back home, I’ll sit down and address.

39

Update for late upgraders (No. Internal upgrade from 4.0 to 4.1 is still not recommended on my side, unfortunately. Hopefully, next 4.1-> 4.2 upgrade will work out of the box, but this is not the case for 4.0 to 4.1 even today with fixes still happening where 4.2 is now in the works).

So time to practice your backup/restore mental muscle and template specialization skills, reapplying some tweaks manually.

Upstream instructions have been updated and clarified a lot since the previous discussions:

  1. Backup of Qubes OS How to back up, restore, and migrate | Qubes OS
  2. Download Qubes ISO and Qubes ISO detached signature on previously prepared USB thumb drive (EXT3/EXT4 partition required). (right-click, “save link as…” (or equivalent) on mounted USB thumb drive passed to your qube). Verify integrity on actual Heads firmware by attempting Options->Boot Options-> Boot from USB (should verify ISO and propose boot options)
    1. https://ftp.qubes-os.org/iso/Qubes-R4.1.1-x86_64.iso
    2. https://ftp.qubes-os.org/iso/Qubes-R4.1.1-x86_64.iso.asc
  3. Download x230-htop-maximized latest firmware (right-click link in artifacts, “save link as…” or equivalent), verify hashes as instructed putting rom alongside Qubes iso and Qubes iso detached signature on a EXT3/EXT4 USB thumb drive following Step 1 - Downloading Heads - Heads - Wiki
  4. Make sure you have a copy of your GPG public key on that USB thumb drive as well to be able to inject it back after manually flashing to maximized board.
  5. Follow manual flashrom instruction to upgrade once from Recovery Shell from legacy to maximized firmware following Upgrading Heads - Heads - Wiki
  6. Install Qubes 4.1: Step 4 - Installing Qubes and other OSes - Heads - Wiki
  7. Depending of if you inject back your public key or apply Re-Ownership, follow appropriate steps Step 3 - Configuring-Keys - Heads - Wiki

Be cautious, specially with step 3 above, making sure your full 12mb rom image has good checksums. You then have to manually flash once from the revocery whell, to migrate from legacy boars to maximized boards, with mount-usb and flashrom -p internal -w /media/heads-x230-hotp-maximized-version-commit_id.rom as documented. DO NOT USE MENUS TO MIGRATE FROM LEGACY BOARDS TO MAXIMIZED BOARDS. Internal upgrades can then be flashed internally through the menus, if you already are using a maximized board.

Notes

[connection]
ethernet.cloned-mac-address=stable
3 Likes