"Maybe I messed up my qubes installation?" related support questions involving "some" Heads subtleties, aka "How to disable autostarting of service qubes at boot without Grub interface"

[Insurgo]

Here is another customer question that was addressed to me that should have been addressed to the forum for obvious reasons, being Qubes OS related:

Sys-net stopped starting
I checked pci devices and reset.

After that it started to boot but it wasn’t searching for networks so I was unable to connect to the WiFi network I use.

I thought that maybe it was missing a PCI device so I then checked and that didn’t work as I was unable to know which one to add or not.

Some interesting bits here are

• Why would a PCI device would be missing? Why would that have changed?
• Most probable cause here is driver problems linked to kernel upgrade
• Otherwise, Fedora probably having grown in memory footprint and drivers “disappearing” and wifi stack being messed up. Logs would help. Solution here would be to switch sys-net to debian instead of fedora template, or assign more memory to sys-net (Fedora is known to increase in memory footprint at every release, which a lot of other posts on the forum refer to and where I also have replied. The solution is definitely not assigning all unrelated to network PCI devices to sys-net.
• User decided to go wild without asking questions, leading to even more problems with the belief that free support should cover his mistakes on top of mistakes
• What if… the wifi slider was simply slide to off?
• What else could explain behavior, prior of doing not-so-easy-reversible and dramatic changes on the system?

This makes me think of the Qubes mini-summit 2022’s talk “Qubes Support stories” where Certified hardware sellers are considered liable for Qubes OS user related configuation errors.

This sometimes lead to user even reinstalling Qubes by themselves and still request direct free support instead of coming to the forum; even if their system is now Qubes OS, not having anything related to the OEM deployment.

This user report is such report, done on user’s behalf since doing 1:1 support simply doesn’t scale and doesn’t help anybody. Giving a fishing line is one thing, giving unlimited supplies of fish is another and the latter is not possible. But some users learn slower then others. And those users should be more careful doing massive and dramatic changes on their systems but they do not understand the impacts of what they are doing.

Later on the 1:1 discussion evolved a bit to:

I then moved all PCI devices from the left side to the right
side sys-net to see if that would fix the problem. Low and behold all of a sudden screen goes black and I am unable to see screen of qubes when I boot into it.

I have tried restarting the pc several times. Just doesn’t work now.

Why on earth those customers do not contact at first problem being:

• sys-net is not detecting any network anymore?

No. Those users contact us when they assigned all their PCI devices to sys-net thinking this is a logical thing to do.

---

Why we love Heads

Going back to basics. Heads users have the possibility to boot from a trusted measured firmware state into Tails DVD ISO verified integrity/authenticity image using distribution detached signature verification, since Heads contains Tails distribution public signing key in the firmware (same applies to Qubes OS and Archlinux’s).

Users can consequently boot directly from a trusted firmware state to an amnesiac and verified ISO image directly, thanks to Tails doing things right with good privacy defaults, without having to install anything, while insalling things in RAM and through enforced Tor is also possible if Administrative passphrase is defined at boot, see below. This makes Tails ISO a good thing to have on a USB Thumb drive. Do that today if your platform is inisitalized through coreboot’s Heads linux payload.

For the end user, that means having a USB Thumb drive, formatted into ext3/ext4, where Qubes ISO and Tails ISO can be dropped there alongside of their distribution detached signature (basically iso and iso.asc files need to be dumped on the USB thumb drive.) This means that untrusted USB Thumb drive can have latest ISO alongside with your needed untrusted stuff, and your Heads booting platform will be able to boot from it when you need to.

As of today, that means end user can download and put on that USB Thumb drive, from an untrusted system (unfortunately, another Linux/MacOS system, Windows do not support Ext3/Ext4) the following files:

• https://tails.hivane.net/tails/stable/tails-amd64-5.4/tails-amd64-5.4.iso
• https://tails.boum.org/torrents/files/tails-amd64-5.4.iso.sig
  That’s it.

Heads will verify authenticity+integrity prior of permitting to boot into Tails through Options → Boot Options → USB boot.

Heads verify it and propose ISO’s boot options:

Magical. Powerful. Useful.

---

Now the issue at stake.

This issue could also be named “I passed all my PCI devices to sys-net, now what” but could also be named “my network adapter stopped working, is it hardware related or Qubes related?” or again “How can I recovery my Qubes installation” which this post would permit users to boot into Tails and use that as a good base to do all kind of stuff, where shutting down the machine will leave no trace and where Heads will pick up any boot related changes that were previously verified.

Having another way of booting/recovering the system is always needed. So at this point, I consider Heads user being able to boot Qubes installer or Tails from actual Heads documentation.

To go back to this user support request, how to actually follow upstream documentation’s Autostart troubleshooting | Qubes OS under Heads? The only hint there is

For Qubes OS R4.0 in UEFI mode, there is no GRUB, so manual boot from another operating system is needed.

Under Heads, grub files are parsed and used to boot the system, where the guidelines there should be used to modify grub.cfg config from a booted operating system (the easiest solution) otherwise requiring individual hacks to get around protections that would be one-fits-all if you had a recovery system to be able to boot from USB for whatever other problem you might come into in your lifetime using Qubes). You could edit config file from heads, remounting /boot in rw (mount -o remount,rw /boot) and then learning vi to edit grub.cfg, saving changes and remounting in read only to sync changes on the disk. Who has technical knowledge to resolve this since the user who did the initial user error was not knowledgeable enough to understand that passing all PCI devices to sys-net would break the system?

Botting into Tails and have graphical editor fits the bill here. Otherwise, requiring unlimited supplies of fish fixing unlimited number of issues.

---

How to disable autoboot of system qubes (sys-net/sys-usb/sys-gui from another bootable system?

From Autostart troubleshooting | Qubes OS, the solution to the problem is pretty straigthforware: add qubes.skip_autostart as a kernel option.

Ok… But… How?!

1. Boot into verified Tails from Options → Boot Options → Boot from USB

2. Cutomize Tails by adding additional functionality (+)
   

   

3. Add Administrative password to be able to do changes on the system
   

   
   
   
   

4. Start Tails. Here you can also connect to network to see if any hardware faults are present
   

   

5. Launch a Administrative Terminal (Root Terminal)
   

   
   
   
   

6. Mount main disk (sda is USB, so main drive is sdb, boot is sdb1) into /mnt
   

   

7. Launch a graphical text editor, here gedit, to edit /mnt/grub2/grub.cfg. Good additional advice here would be to cp /mnt/grub2/grub.cfg to /mnt/grub2/grub.cfg.bak so that it can easily be moved back when we finish our surgical operation here and everything is working as expected again)
   

   

8. We add qubes.skip_autostart in the first grub entry found, appending to the first module2 line we see.
   

   

9. We save the changes. We just tampered with grub.cfg, which Heads will complain about later on
   

   

10. We reboot
    

    

11. Launching default boot option will have Heads complain. We won’t update checksums now. Say No!!!
    

    

12. We will instead boot in unsafe mode the first option we modified
    

    
    
    
    
    
    
    
    
    
    

13. We can see if we check carefully that our added qubes.skip_autostart is passed to Qubes
    

    

14. We see that no qubes are automatically started
    

    

15. We fix things and make sure that things are as expected. We can then start a normal workflow here. We are under Qubes, we just prevented any qubes from starting automatically. We can start things manually here, which is the goal of the operation.
    

    
    
    
    

16. We Apply, save and reboot, and boot again with second boot option (unmodified)
    

    
    
    
    
    
    
    
    
    
    
    
    
    

17. We see our option not being passed to kernel
    

    

18. Typing ESC keyboard’s key show qubes are starting
    

    

19. We make sure the wifi slider is in ON state
    

    
    
    
    
    
    
    

20. Reboot into Tails to move grub.cfg.bak backup into its original location to be able to boot the normal way into verified Heads detached signed configuration by re-doing steps 1-7 above, and then do from terminal : mv /mnt/grub2/grub.cfg.bak /mnt/grub2/grub.cfg as a replacement to step 8’s command. Reboot default boot Heads’s configuration. ( You just undid all the bold and too wild move of assigning all your PCI devices to sys-net. Congratulations).

---

If there is a different issue, that your network devices are actually working inside of Tails but not under Qubes OS, that you are suffering from a change that happened after upgrading your kernel, or upgrading your Fedora template or whatever else, please search the forums and qubes issues over github and do not hesitate to open an issue/new forum post that doesn’t exist yet or to comment on one that already exists, even tagging (@people) if you need direct involvement.

But please. Do not contact me directly for Qubes OS support. Tagging me in the forum on something that will benefit someone else then yourself alone should be the way. And yes, I might become angry if you do that over and over and do not understand when I say to open an issue on the forum and I make the searches at your place and you ask forever more for free… I love Qubes, as many others, and I rely on the forum myself to have the community think/collaborate on issues I cannot resolve myself. You should do the same so that it benefits all.

Where matters of Heads related discussions here will lead into proper entrees under Heads wiki. I am not a documentation expert. Learned a lot but most of the times, others will be better then myself at being short and spot-on, where I propose sometimes longer solution and become tired of saying the same things all the time. This is why I’m answering to this question here. Fishing line, not unlimited supplies of fish. Aho.

[Insurgo]

An alternative to the precedent would obviously to do the work from Heads recovery shell.

• Options → Recovery shell
• put /boot filesystem in read-write mode:
  • mount -o remount,rw /boot
• create backup of the file we will modify next:
  • cp /boot/grub2/grub.cfg /boot/grub2/grub.cfg.orig

Then, use vi if that is something you are comfortable with. Most won’t, which is why I proposed to go the Tails way in previous post, which would permit to troubleshoot and fix a plethora of other issues that Heads recovery shell cannot do, like offering a GUI based text editor.

Otherwise, we have a backup here now. We could use sed savagely to do textual search and replace, which is less evolving for most then learning to use vi (repeated nightmare to get users into insert mode (Escape, :i) and saving (Escape, :wq!). Yes this is the challenge of explaining easy things for one to someone that thinks the hadrware is cursed for one, and is ready to blame everything and everyone but himself for things not working. Support is not an easy daily day job, it is a vocation, it is a job in itself, and a really repetitive one which a community does a way better job).

Let’s fix our grub.cfg file so Qubes does not autostart any qube here so we can fix our previous mess:

• Replace quiet with quiet qubes.skip_autostart everywhere in the file:
  • sed "s/quiet/quiet qubes.skip_autostart/g" /boot/grub2/grub.cfg -i
• save the changes (commit them) under heads:
  • mount -o remount,ro /boot
• reboot

Test Qubes without it autostarting any Qubes:

• Options → Boot Options → Unsafe boot option : Select first boot option
• Do your thing as per prior post, test your changes starting qubes manually.
• Reboot when done and satisfied. Otherwise open issues/other forum posts.

Then revert changes so Heads doesn’t complain about any tampering:

• Options → Recovery shell
• mount -o remount,rw /boot
• mv /boot/grub2/grub.cfg.orig /boot/grub2/grub.cfg
• mount -o remount,ro /boot
• reboot

[unman]

This makes me think of the Qubes mini-summit 2022’s talk “Qubes Support stories” where Certified hardware sellers are considered liable for Qubes OS user related configuation errors. which sometimes lead to user even reinstalling Qubes by themselves and still request direct free support instead of coming to the forum. This is such report done on user’s behalf since doing 1:1 support simply doesn’t scale and doesn’t help anybody. Giving a fishing line is one thing, giving unlimited supplies of fish is another.

Certified sellers should be liable for configuration errors. That’s
part of what “certified” should stand for. (At one time certification
seemed to represent hardware, but now it clearly applies to providers
of hardware.)
They should also provide support to customers. Any one paying the
premium to buy from a certified vendor has a reasonable expectation of
support - pointing to the Forum isn’t good enough.

I never presume to speak for the Qubes team.
When I comment in the Forum or in the mailing lists I speak for myself.

[adw]

@unman

(At one time certification seemed to represent hardware, but now it clearly applies to providers of hardware .)

Just to clarify, “Qubes-certified hardware” still applies to specific hardware models. It is not the case, for example, that a vendor can become a “Qubes-certified hardware provider” and introduce new “Qubes-certified” models without going through the Qubes team. The Qubes team still has to physically receive two units of a computer model, test them, and approve them before that model is officially Qubes-certified.

I believe this is (still) clear from the way we describe hardware certification on the Qubes website, but if anything is unclear, please let me know.

Certified hardware

[Insurgo]

@unman:

@unman

Certified sellers should be liable for configuration errors.

And they should (and I am responsible for what I deploy, which I call OEM disk image, and provide with wyng-restorable state and scripts). My sentence was:

@unman

Certified hardware sellers are considered liable for Qubes OS user related configuation errors.

After that, no: Certified hardware sellers cannot be held responsible for user related configuration errors. In the above example, the person passed all PCI devices to sys-net. How can the Certified hardware seller be responsible for that?

@unman

They should also provide support to customers. Any one paying the
premium to buy from a certified vendor has a reasonable expectation of
support - pointing to the Forum isn’t good enough.

This is one of the reason i’m getting away of selling certified hardware myself to go back more into R&D. There is, to my opinion, way too much unresolved upstream issues that needs after-install customizations. I maintain my OEM disk image to deploy such fixes, by default, on every install, but this is not the case for any other Qubes Certified hardware sellers I am aware of. The reason I do this is to not have to repeat myself to each customer having the same behavior, and also why I collaborate upstream into trying to change those defaults. But those defaults, most of the time, are not changed and issues are kept open. So I fix those issues downstream in my OEM image. I learned the hard way that issues needs to be fixed upstream, including configuration defaults.

Those deploy a customized ISO which as of today, modifies the kickstart to have automatic installation, which facilitates OEM install with encryption of the install deploying the same encryption passphrase for all installation and do not customize anything from the default settings.

They do not fix nor attempt to fix problems at install. I do not see a lot of participation from those into Qubes issues nor the forum to try to address the community issues. And they also have a questionable reputation on this forum in terms of support they offer to their customers, from their customers who come here, since they understand it will benefit others as well. At least this is my understanding. My point here is that those things are sometimes difficult to seperate. A bug is a bug. If under Heads, it should be under Heads issues to be tracked and addressed. If under Qubes, same thing. When it comes to configuration, yes greed could apply and if rentability calculation wins, make the user pay for support. But that would not help the Qubes project. Issues should be fixed upstream and supporting users one at a time is not a desirable path, nor has a desirable outcome. Fixes should either happen in code, or in documentation. I suppose we all agree on that.

I offer free 1 hour support to customers and paid support after that. At the beginning of my adventure, my changes were not merged into Heads (too many: merge was refused). I took total responsibility of the firmware differences on user experience side of things, and worked my way into slowly merging the functionnalities into Heads so that the user experience was the same across Heads supported platforms so that Insurgo was not the unique Heads support channel. I still struggle with that, and that is the project I try to maintain the best I can myself.

A lot of customers discover Qubes issues. Of course, some report them to me. I then channel them to Qubes issues or poke the developers.

When it comes to user experience like this story sharing here, no @unman. Certified reseller cannot be responsible for Qubes user experience. Certified reseller cannot be liable outisde of their deployed OEM image, and as said before, and historically, Qubes resellers have historically got away of cusomizing in any way possible but last resort the defaults, so that they are not liable for user’s/Qubes defaults behaviors, and can point to already opened issues/discussions.

The points you are bringing should be part of a bigger discussions. If any new Qubes certified hardware provider also is required to provide extended Qubes OS support, for free, and is gonna be liable for fixing Qubes configuration issues (to name only one massive issue: Fedora being the default template for sys-net/sys-usb not having enough assigned memory causing camera/wifi disconnection issues; affecting Librems/Nitropads but not PrivacyBeast. Or tsc issues having affected sys-whonix, causing system hangs…)

I agree that Qubes certified hardware should be expected to collaborate with Qubes OS to resolve issues that they are experiencing/issues being reported to them by their customers.

But making them liable for Qubes issues, some of which defaults are refused to be changed, is another subject then here.

The scope and intent of this post is to invite users into being considerate and also becoming autonomous at some point. I think there should be paid support channels. I discovered recently through another post that @adrelanos was offering such paid support and I think it should become a revenue market for Qubes certified hardware sellers and part of their marketing offer, if this is part of their business model.

Mine, since the beginning, was to try to have those issues reported where they should, if possible by the users themselves, and collaborating on those issues with the whole ecosystem.

Taking Qubes OS users configuration errors liability would be suicidal.

Thank you for your interaction to this post @unman, hopefully it will permit to clarify further what is expected by a Qubes OS certified hardware reseller in terms of included support, participation into Qubes OS development, release testing, customer support.

From what I read here, it feels like OEM should be expected to deal with customer support when deploying Linux on their products. If this is the case, this largely explains why so little are doing so… and sticks with deploying Windows on their machines to not suffer from liability issues.

[Insurgo]

@adw

The Qubes team still has to physically receive two units of a computer model, test them, and approve them before that model is officially Qubes-certified.

@adw @unman: absolutely. And then Qubes OS is responsible to do regression tests so that Xen/Kernels are working out of the box, even from unstable repos, without customizations, so that when they land in stable repos, users that would want to reinstall Qubes on those machines themselves are not becoming beta-testers themselves.

Just to be clear here, its not a 100% bug-free process. And a lot of collaboration happened in the past to have Qubes 4.1 work on those machines. Qubes 4.0-> 4.1 in place upgrade was also another place of immense collaboration on my side, and the same applies for numerous other examples.

Maybe it would be nice to open a forum section for Qubes OS certified resellers. And maybe have a section there so that there is more collaboration happening there, if liability is expected on that level, cause it is something that sometimes feels really lonely to do where benefits are literally for all, not just the resellers.

[unman]

Just to clarify, “Qubes-certified hardware” still applies to specific hardware models. It is not the case, for example, that a vendor can become a “Qubes-certified hardware provider” and introduce new “Qubes-certified” models without going through the Qubes team. The Qubes team still has to physically receive two units of a computer model, test them, and approve them before that model is officially Qubes-certified.

My point is that it isn’t “certified hardware” - that could have been
done. I recall Michael proposing that the x220 be the second certified
after Purism, and going through some available hardware configurations.
That is hardware certification: a specification that is endorsed to run
properly under Qubes.
What we have now is vendor certification, although strangely Qubes
takes " no responsibility for any vendor’s manufacturing, shipping,
payment, or other practices;"

If this were certified hardware then there would be no point in Qubes
accumulating multiple copies of the same models, two from each
potential vendor. It is obvious too from the announcements that Qubes is
certifying configurations that go beyond the base hardware.
This has limited the market of potential machines and vendors, and to
some extent limited the reach of Qubes.

x230 used to be a Qubes workhorse: that hardware could have been
certified and anyone could have sold it as such. Some vendors would have
pursued vendor certification, based on what they did with that hardware,
and their business practices. That’s a different approach. I suspect it
would be better.

I never presume to speak for the Qubes team.
When I comment in the Forum or in the mailing lists I speak for myself.

[Insurgo]

@unman

My point is that it isn’t “certified hardware” - that could have been
done. I recall Michael proposing that the x220 be the second certified
after Purism, and going through some available hardware configurations.
That is hardware certification: a specification that is endorsed to run
properly under Qubes.
What we have now is vendor certification, although strangely Qubes
takes " no responsibility for any vendor’s manufacturing, shipping,
payment, or other practices;"

If this were certified hardware then there would be no point in Qubes
accumulating multiple copies of the same models, two from each
potential vendor. It is obvious too from the announcements that Qubes is
certifying configurations that go beyond the base hardware.
This has limited the market of potential machines and vendors, and to
some extent limited the reach of Qubes.

x230 used to be a Qubes workhorse: that hardware could have been
certified and anyone could have sold it as such. Some vendors would have
pursued vendor certification, based on what they did with that hardware,
and their business practices. That’s a different approach. I suspect it
would be better.

I totally agree with the above. The problem here is not the hardware itself, but the requirement of deploying a constant, and supported, hardware configuration (same specs) from the certified hardware reseller for one year. That would probably also have had encouraged firmware development and collaboration to some level, but that would also not be guaranteed.

As per certification requirements, those included hardware lock down and firmware lockdown. this changed since then. This evolved with participants collaborations (the certified hardware resellers and feedback, collaboration) and collateral learnings.

The devil is in the details here, unfortunately. I can only speak for what I know here and not want to blame any party, just trying to recall the facts so that the certification process can continue to evolve, being aware of a lot of new certification hardware that will come, which I really welcome. Diversity is needed. ewer hardware is needed. Others participants, new blood and new participants in the process will make things better. Compliance and collaboration of all partieswill make things better. If collaboration is enforced.

Historically to this point:

• Purism had difficulties deploying Qubes on their machines. Qubes OS+Purism tried to develop an OEM installation media and maintain it conjointly with Qubes to ease unattended installation and failed to do so in the long term. I also recall the certification fees to be too high at that time, being the first certified partners and all of that limited the sustainability and profitability of the partnership, but I am not aware of all the details here.

• Nitrokey took hat unattended OEM installation ball recently, and Purism joined in the effort. This is a separate thing altogether from having some hardware certified. This has to do with facilitating Qubes installation on certified hardware. Without any Qubes customization outside of kickstart selected options. Basically permitting to install Qubes with a hardcoded encryption key passphrase that the user is encouraged to change at hardware reception, relying on external, physical tampering mechanisms (read tamper evident bags) to protect Qubes installation integrity since the encryption key passphrase is known and exposed on their respective github repositories for their kicstart files qubes-oem/ks_template.cfg at 48a2d9dbc2fe792fc5d3fc33dedef470ac8bbd0f · Nitrokey/qubes-oem · GitHub and discharging liability in their Readme: qubes-oem/README.md at main · Nitrokey/qubes-oem · GitHub. But this is a good start: it permits Qubes unattended installation. This is first step. As discussed before, Qubes installation media would benefit having external customization possible (external kickstart source, absorbing salt recipes to be deployed on first boot with network being configured etc, but this is still not existing). Meanwhile, Qubes OS media is modified by resellers.
• Then, the certification process and requirements changed. When Purism first certified their hardware Librem13 which then diverged into Librem13v1/Librem13v2) it was first an Ivy bridge based laptop, this means it could have its ME neutered, which was not the case on Librem13v2, that Librem13 marketing speech was not consistent, made users roar and Qubes support complicated and Heads support request and discussions complicated. Nothing existed in that time saying they were supposed to support a specific hardware certified model for at least a year, and changed their components. Librem 13v1 became v2, and a lot of ink spilled on that matter everywhere on the internet, where some posts can be read on this forum on the matter, even related to Qubes being confortable certifying them again in the future. Have not followed that discussion thread myself. On a firmware perspective, this meant a lot of differences. Its not the same chipsets, not the same ME, not the same anything. This is where the requirement to have a hardware model, with same components and same firmware, was locked to be produced and supported for a year, minimally. This lead to a lot of problems with locking the firmware for next certified model (PrivacyBeast) which needed to be extensively tested, and required a lot of documentation to be produced instead of fixing the firmware itself and providing firmware upgrades to mitigate issues). This requirement on firmware lockdown vanished since then. But the hardware produced (same specs) support of a year (logical) survived. This eased the path for Nitropad certification, which used already existing hardware and existing firmware. Basically making them exactly what you are saying here @unman: Nitropad is basically selling a certified hardware, reusing existing firmware. Just so you know, they basically use the x230 and t430 Heads boards and rebranded them to be Nitropads in their fork.

Some learnings on firmware side of the certification (part I am aware of being firmware maintainer and in Heads development loop):

• Some pureboot (heads fork) firmware update release was produced without measured boot enabled, of course fixed on later release PureBoot Security Flaw for Librem 14 Patched - Hardware - Purism community
• Firmware was flashed without firmware regions being unlocked, limiting user control over their firmware and requiring sellers to offer firmware upgrade as a service Firmware Update v1.4+ - Nitrokey Documentation . This was a side effect of not taking seriously testing of firmware prior of accepting merge upstream under Heads. users/collaborators/contributers are all different participation levels.
• Firmware testing from certified sellers (all Heads based) is not easy to coordinate testing, even today when it comes to regression testing of some features, definitely impacting coreboot/linux version bumps and speed of release. Too many forks, not enough upstream collaboration.

From all this learning, my take is to be an upstream contributor as much as possible and I encourage others to take that path as much as possible.

Same applying to cacher, whonix, Heads, coreboot, flashrom, kexe, cryptsetup etc, obtaining/pushing others to get grants applications, obtaining funds to further develop tools we all need, (coreboot, linux, heads, wyng, cacher, remote support tooling) and promoting those projects so they evolve with the passion of their contributors.

For the sole Qubes scope, extending to all its dependencies, it means collaborating and contributing with Qubes developers as much as possible.

On Heads, this means exactly the same. It all depends if we are simply users, collaborators or contributors of such projects, if we want the whole ecosystems we are using to evolve.

Or… If making profits on such ecosystems is enough. Of course profit is needed for each of those projects to survive. But for such projects to not only survive but thrive, this needs knowledgeable people to contribute back. All of those are open source projects. And can evolve only if such contributions back exist.

In the current case: this means support related to Qubes to be under Qubes if beneficial to the whole ecosystem. That is my opinion, and this is why the OP post was created.

Assigning all PCI devices to sys-net will always be a bad idea. Having a way to boot the system into a live system is needed. And this is why I created this post.

Aho.

[adw]

@Insurgo

Maybe it would be nice to open a forum section for Qubes OS certified resellers. And maybe have a section there so that there is more collaboration happening there, if liability is expected on that level, cause it is something that sometimes feels really lonely to do where benefits are literally for all, not just the resellers.

@moderators, thoughts?

@unman

My point is that it isn’t “certified hardware” - that could have been
done. I recall Michael proposing that the x220 be the second certified
after Purism, and going through some available hardware configurations.
That is hardware certification: a specification that is endorsed to run
properly under Qubes.

A stock X230, for example, is different from an Insurgo PrivacyBeast X230 or a NitroPad X230. The latter are configured or modified versions of the stock X230, but that doesn’t make it any less accurate to describe them as “computer hardware models,” and that doesn’t it make any less accurate to call them “certified hardware.” Just because we didn’t also certify the stock X230 doesn’t mean that the what we’re currently calling “certified hardware” isn’t certified hardware.

I don’t remember why we never certified any base/stock models. I vaguely recall that we looked into it and ran into some roadblocks. Maybe @marmarek can refresh my memory on that.

@unman

What we have now is vendor certification

I’m still not sure if that’s entirely accurate. It seems to imply that we certify vendors, which would be misleading. We still certify only specific hardware models. A vendor successfully getting one particular model certified does not guarantee that the vendor will be successful in getting any other models certified.

@unman

although strangely Qubes takes " no responsibility for any vendor’s manufacturing, shipping, payment, or other practices;"

Why is that strange? How could we possibly take responsibility for any of that?

@unman

If this were certified hardware then there would be no point in Qubes accumulating multiple copies of the same models, two from each potential vendor.

How else are the Qubes devs supposed to test those exact models to ensure adherence to the hardware certification requirements and compatibility with Qubes OS software as it gets updated, which requires repeated testing over time? We can’t really afford to pay for that much hardware out of our own pockets, especially when vendors often mistakenly think that their hardware satisfies our certification requirements when it actually does not.

@unman

It is obvious too from the announcements that Qubes is certifying configurations that go beyond the base hardware.

Correct. Is that a bad thing?

@unman

This has limited the market of potential machines and vendors, and to some extent limited the reach of Qubes.

But how could we do otherwise? There would have to be a base model that satisfies all of our hardware certification requirements without being configured to do so (which, AFAIK, doesn’t exist).

@unman

x230 used to be a Qubes workhorse: that hardware could have been
certified and anyone could have sold it as such.

Again, I can’t remember why that never happened with the X230 specifically (@marmarek might remember), but with the new certification requirements (e.g., open-source boot firmware), I don’t see how the X230 or any other ThinkPad would qualify.

[Insurgo]

The issue for x220/x230 got closed when x230 got certified.

@adw @unman : proposal: second qubes-certified laptop is Lenovo Thinkpad x220/x230 · Issue #1771 · QubesOS/qubes-issues · GitHub

Houla. Things changed a lot in the process described there and what happened under Heads and OEM process that evolved since then.

To take with a grain of salt.

@michael closed the issue with

hey all, given the Insurgo PrivacyBeast came out, there is now a certified laptop based on globally accessible/source-able hardware should someone want to make it themselves, or buy from Insurgo. so I am closing this ticket.

---

The idea was for people to replicate the work, knowledgeable people to collaborate, and contribute back.

On my side, i did the work, waited for good partners to come…

I never expected to do one laptop at a time for so long, so much user support, the r&d, the maintenance of Heads, getting funding for others, the bug reports etc. What an interesting journey!

I expected to train organizations that would have applied and customized my OEM process to their own needs and maintained their own images while making enough money to continue R&D the rest of the time, but as said everywhere else: everybody being afraid of liabilities, they preferred sending their employees/contractors/contacts my way.

EDIT For completeness: This is where collaboration to Heads started, which led to supporting the x230, the legacy Heads roms (two stage flashing for xx30) and later on support on the x220, dismissing libreboot for explained reasons here: Research support for libreboot/coreboot-based systems · Issue #1594 · QubesOS/qubes-issues · GitHub

[unman]

A stock X230, for example, is different from an Insurgo PrivacyBeast X230 or a NitroPad X230.

Indeed.
If I configure a stock x230 in the same way as Insurgo, can I sell it as
certified hardware?
Would I have to send two boxes to Qubes? To what purpose?

The explosion in PCs came because IBM had an open standard and used
commodity hardware. It was easy for manufacturers to produce compatible
“clones”.

I’m still not sure if that’s entirely accurate. It seems to imply that we certify vendors, which would be misleading. We still certify only specific hardware models. A vendor successfully getting one particular model certified does not guarantee that the vendor will be successful in getting any other models certified.

Again, I agree. But it’s Insurgo selling that model that is certified.
Any one can produce that model but they will not be certified. (I do not
underestimate the work that goes in to the Privacy Beast or the
Nitropad.)

@unman

although strangely Qubes takes " no responsibility for any vendor???s manufacturing, shipping, payment, or other practices;"

Why is that strange? How could we possibly take responsibility for any of that?

In the same way that any organisation deals with subcontractors or 3rd
part suppliers. They draw up an agreement specifying services and
standards to be met.
There would be minimal monitoring required, since any issues would
likely surface on the Forum or mailing lists. Breaches could lead to
removal of certified status.

This is, as I have said, a different model. I think it would work better
for Qubes, users, and vendors who pursue certification.

I never presume to speak for the Qubes team.
When I comment in the Forum or in the mailing lists I speak for myself.

[adw]

@unman

If I configure a stock x230 in the same way as Insurgo, can I sell it as
certified hardware?
Would I have to send two boxes to Qubes? To what purpose?

Good question. I imagine that the Qubes devs would want to be able to verify for themselves that it is, in fact, exactly the same configuration without having to take a new vendor’s word for it. A vendor might think they had configured things exactly the same way, but the Qubes devs’ testing might reveal problems. It would be bad for the Qubes devs to give their certification stamp of approval without doing any actual testing only for users to be burned by incompatibilities that the testing was supposed to catch.

@unman

The explosion in PCs came because IBM had an open standard and used
commodity hardware. It was easy for manufacturers to produce compatible
“clones”.

Yes, I see what you’re saying. It seems to me that there are significant, warranted differences between an open standard and the Qubes hardware certification program as it currently stands. It would be great if our thing eventually became an open standard, but it seems like we’re a long way off from that.

@unman

But it’s Insurgo selling that model that is certified.
Any one can produce that model but they will not be certified. (I do not
underestimate the work that goes in to the Privacy Beast or the
Nitropad.)

Right, I see what you mean. I think right now the concern is that new vendors (especially small ones) might not get the requirements right on the first try, so the Qubes devs want to test “new” models themselves. But that’s partially speculation on my part. @marmarek would be able to provide a better answer.

@unman

In the same way that any organisation deals with subcontractors or 3rd
part suppliers. They draw up an agreement specifying services and
standards to be met.
There would be minimal monitoring required, since any issues would
likely surface on the Forum or mailing lists. Breaches could lead to
removal of certified status.

This is, as I have said, a different model. I think it would work better
for Qubes, users, and vendors who pursue certification.

Ah, I see what you mean now. Yes, this would be a very different model from the status quo. I think perhaps MiCh (no forum account, notified manually), @marmarek, and @michael would be in the best position to weigh in on this.

[Insurgo]

@Insurgo

To go back to this user support request, how to actually follow upstream documentation’s Autostart troubleshooting | Qubes OS under Heads?

Where to put:

@Insurgo

qubes.skip_autostart

The easiest way to pass this to Heads without modifying grub config is from Heads recovery shell by doing the following on with Heads with normal verifying /boot:
kexec-select-boot -b /boot -a qubes.skip_autostart

• b: boot dir /boot
• a: add kernel boot option
  Select normal boot option, do not try to pass TPM disk unlock key passphrase (that won’t work from recovery shell). Qubes will ask for the Disk Recovery Disk passphrase, you will boot into Qubes without any qubes automatically starting!