QubesOS test media always fails

Whenever I try to install QubesOS from the USB stick and run “Test this media and install QubesOS …” it says that the image is corrupted. However, this only happens on one PC, which is most surprising. I test after downloading the iso with sha256sum and the sha256 hash I pull from several sources over different internet connections and the iso is not corrupt. What I noticed is that in the list of events, the white areas of the texts look corrupt until shortly before the start of the check, as in the case of RAM damage (marked in the screenshot).

So I have done/tested the following:

  • RAM test with MemTest86 & Memtest86+
  • CPU test with Intel® Prozessor Diagnose-Tool
  • USB test with random data which I checked with sha256sum
  • the iso written from different PC to different USB sticks
  • out of desperation I wrote the iso to an SSD and connected it via sata and booted it
  • since the installation of fedora is very similar I wrote the fedora iso on the USB stick and booted and tested it without any problems
  • i have connected only the minimun to the PC and used other input devices

All tests have shown no error and everything I have tried has not helped. I am not the biggest newbie in this area and have already installed QubesOS countless times and on this PC. I can’t understand why it doesn’t work on this PC but it does on all the others.

I hope someone can help me and thanks in advance also for reading through.

I have been looking for the problem on the internet for some time but only found posts indicating that the iso was corrupt. however, what I could have thought of earlier is to look for the same problem on fedora.
I found the following:

which is a very identical error I have also noticed that from the moment I booted the PC with the USB stick this error also occurs on other PCs so something is changed on the USB stick.

I am currently trying to work out what is being changed but does anyone have any ideas?

**Edit
i have checked every file on the USB stick from the accessible partitions with sha256sum before and after booting and there is no difference and no new files have been added or anything like that… so what is being changed?

Is it possible that some hardware component, be it the UEFI/BIOS, is changing data outside of accessible partitions?

But then why is only QubesOS affected and not fedora and other systems?

to be honest, it’s driving me insanely slowly

I don’t use the test option, I just straight install. Is there a reason you don’t do that ? How are you writing the iso to USB ? I use DD and have never had any issues

If possible i always use the test option because something can always go wrong when writing the usb stick and in the past it has happened once.

I could skip the test but that doesn’t change the fact that for whatever reason something is modified the usb stick that shouldn’t be.

When i write the iso to the USB stick, i always use the recommended dd command from the QubesOS documentation sudo dd if=Qubes-RX-x86_64.iso of=/dev/sdY status=progress bs=1048576 conv=fsync

I have never had any problems with it so far too and on other PC’s there are no problems only on the one.

I still had an older QubesOS iso and i just tried to boot it and did the self test without any problems, it’s just the latest iso that doesn’t work for me. Which probably indicates a software problem.

If I understand you run the test on one PC and it fails. You have run
the test *with the same medium on other PCs, and it has passed.
Can you confirm:

  1. How you prepared the install medium.
  2. the specs of the “fail” PC
  3. the specs of other PCs you have used where it has passed.
I never presume to speak for the Qubes team. When I comment in the Forum I speak for myself.

Yes
1:
i have used 2 USB sticks from different manufacturers for the current problem treatment. i have also written to the USB sticks from another QubesOS from a fedora vm and also from another pc with debian.

  • before every write operation i test the iso with the hash sum via sha256sum
  • then I run “sudo dd if=Qubes-R4.2.2-x86_64.iso of=/dev/sdc status=progress bs=1048576 conv=fsync”
  • to be sure i run sync an unplug the USB stick

2: (PC-A)

  • Mainboard: Gigabyte Z490 GAMING X
  • CPU: intel i5-10600
  • RAM: 2x 16GB 2666.66MHz

– Qubes-R4.2.0-rc1-x86_64.iso - passes the media test
– Qubes-R4.2.1-x86_64.iso - fails the media test
– Qubes-R4.2.2-x86_64.iso - fails the media test
– Fedora-Workstation-Live-x86_64-40-1.14.iso - passes the media test

3: (PC-B)

  • Mainboard: Asus P8B75-M
  • CPU: intel i7-3770
  • RAM: 2x 8GB Random ones

– Qubes-R4.2.0-rc1-x86_64.iso - passes the media test
– Qubes-R4.2.1-x86_64.iso - passes the media test
– Qubes-R4.2.2-x86_64.iso - passes the media test
– Fedora-Workstation-Live-x86_64-40-1.14.iso - passes the media test

I used the same USB stick for both PCs each time and booted it first into PC-B and then into PC-A and then the other way round after a rewrite. After it was booted in PC-A, it could no longer run the test on PC-B without failing too BUT only with Qubes-R4.2.2-x86_64.iso.

**edit
Something that has also crossed my mind:
What if it is a Bootkit?

well the PC has only ever had QubesOS running on it since it came out of the box except now when testing Windows for the intel diagnostic tool. The PC has also not had an internet connection for at least half a year. And apart from that, I am very far away from the Bootkit target group.

**edit
I just tested the Qubes-R4.2.1-x86_64.iso where i have the same effects.

Do you have Windows installed on PC-A?
Did you plug your USB in PC-A while working in Windows?
Windows can automatically write some files in EFI System partition on the USB disk that will cause the disk check to fail.

1 Like

I only installed windows on an HDD for the CPU test, the HDD is no longer in the PC-A. Otherwise I only have GNU/Linux systems in use (QubesOS, Debian 12, Fedora 40).

If all systems would fail I would say that something in the PC changes / corrupts the data but it really only happens with the mentioned QubesOS iso and only when I boot from it.

Try to check it like this:
Write Qubes-R4.2.2-x86_64.iso on media using PC-B.
Check that it passes the media test on PC-B (try it twice).
Check if it passes the media test on PC-A.
If it fails then connect the media to PC-B and run diff between the files in EFI System partition between the media and ISO.

I also have a question

Have you tested different combination of UEFI & legacy boot on PC-A & PC-B? To see if the media verification fails on all possible combinations? (there should be 4 possible cases).

I have already done this.
I wrote the iso to the USB stick and did run “find /media/user/sdc1 -type f -print0 | xargs -0 sha256sum >> sha256sum.txt” and the same for the sdc2, so for both mountable partitions.

Then i booted the USB stick from PC-B and it passed the test, then i used “sha256sum -c sha256sum.txt” to check the data after mounting the partitions and all were ok. then i booted again on PC-B with success and then on PC-A where it failed again.

Then I tested it again with sha256 and again all data was ok … . That was the moment when I thought I was getting crazy so I booted the USB stick again on PC-B but again failed. What I am testing right now is what I should have done before to test the partitions directly with sha256sum so “sha256sum /dev/sdc1” I am doing that right now it just takes time as rewriting the whole USB stick does need time.

I also think that this is exactly what the media tester does, isn’t it? i.e. testing the parttion directly

I have found the problem.

It has something to do with the CSM support of my mainboard.

I have already tested if it works with UEFI and legacy and both failed. But I have not yet tested what happens if I deactivate CSM support, i.e. legacy boot support, completely. After I started it in UEFI without CSM support activated it works without problems !

CSM Support Activated:

  • UEFI Qubes-R4.2.2-x86_64.iso : Fail
  • Legacy Qubes-R4.2.2-x86_64.iso : Fail
  • UEFI Fedora-Workstation-Live-x86_64-40-1.14.iso : Pass
  • Legacy Fedora-Workstation-Live-x86_64-40-1.14.iso : Pass

CSM Support Deactivated:

  • UEFI Qubes-R4.2.2-x86_64.iso : Pass
  • UEFI Fedora-Workstation-Live-x86_64-40-1.14.iso : Pass

But, for whatever reason other systems do not have the problem like fedora. I have tested it again fedora which also checks the medium does not fail neither in UEFI, Legacy and with and without CSM support.

Even if the error is permernent from the moment it has booted with CSM Support Activated, the partitions are not changed, which still confuses me.

  • PC-B: Write iso to the USB Stick

  • PC-B: hash sha256
    sdc1: ac0bdf60fac61d14ad81c2484306209bc29302a82a5af3db8a2bfe86156765a1
    sdc2: 20fbece84f07cb0783bec3ad7f09631347fcb12961ffe14fae648510ad991db5
    sdc3: 7818f5542a0404157573be6cffc0e0c8e68ce3c0f5d17d07ccdd9313fb700baf

  • PC-B: Boot USB stick, media test passed

  • PC-B: check the hash: ok

  • PC-A: Boot USB stick, media test failed with CSM support in UEFI mode

  • PC-B: Boot USB stick, media test failed in UEFI mode (this mainboard is too old it has no CSM support)

  • PC-B: hash sha256
    sdc1: ac0bdf60fac61d14ad81c2484306209bc29302a82a5af3db8a2bfe86156765a1
    sdc2: 20fbece84f07cb0783bec3ad7f09631347fcb12961ffe14fae648510ad991db5
    sdc3: 7818f5542a0404157573be6cffc0e0c8e68ce3c0f5d17d07ccdd9313fb700baf

Because the error persists despite the fact that the partitions have not been changed, i now hash the entire usb stick but this takes some time.

Is this a bug I should report ? I mean the CSM support also does something that it is not supposed to do when booting in UEFI mode, but other systems can handle it except Qubes-R4.2.1-x86_64.iso, Qubes-R4.2.2-x86_64.iso.

2 Likes

I would investigate the content of EFI partition after booting with CSM enabled. To see if it adds anything suspicious to it. Specially if the Fast Boot or Ultra Fast Boot options are enabled in motherboard setup.

You have to find out what it does to the EFI partition (and why). Then it is up to you to decided to report (and complain) about it or not. I am not sure if Gigabyte would take any action about it. Their Microsoft Windows users love their Ultra Fast Boot tweak.

Regarding the Qubes R4.2.1+ complains, it has more strict verification compared to most other Distros. It is imaginable that its verification would fail while Fedora would pass

Related:

1 Like

As I said the partitions are not changed (not even the efi) the hash remains the same, something outside is changed.

From now on I don’t know much about it but what I can say is that the sha256 checksum of each partition remains the same but the whole hash of the USB stick is changed.

Before:

aa25ae7537a1a61ad8d5a29bda7e75cef5ee7e4513dfbbbcec3712c96e5e9450  /dev/sdc

ac0bdf60fac61d14ad81c2484306209bc29302a82a5af3db8a2bfe86156765a1  /dev/sdc1
20fbece84f07cb0783bec3ad7f09631347fcb12961ffe14fae648510ad991db5  /dev/sdc2
7818f5542a0404157573be6cffc0e0c8e68ce3c0f5d17d07ccdd9313fb700baf  /dev/sdc3

Afterwards:

979b59250f6fb4496cac718f858e5f9881fe2d7f88a1b975224286ba3be50a51  /dev/sdc

ac0bdf60fac61d14ad81c2484306209bc29302a82a5af3db8a2bfe86156765a1  /dev/sdc1
20fbece84f07cb0783bec3ad7f09631347fcb12961ffe14fae648510ad991db5  /dev/sdc2
7818f5542a0404157573be6cffc0e0c8e68ce3c0f5d17d07ccdd9313fb700baf  /dev/sdc3

Fastboot is deactivated

1 Like

Check the /dev/sdc partition table info before and after:

sudo fdisk -x /dev/sdc
1 Like

Could be beyond that. It could have modified anywhere within ISO9660 hybrid table. This is bizarre. It is the 1st time I have encountered a moderboard which modifies a bootable USB stick. This is truly worth of investigation.

Yes, I think so too, it’s really bizarre.

i built a little script which gives me the info from fdisk, creates hashes and creates an .img of the drive via dd so i used an 8GB one that the .img does not get too big.


  • VM = Fedora 40 on an another PC
  • PC-A = is the one with the CSM Support problems
  • PC-B = is the test PC

  • VM: write Qubes-R4.2.2-x86_64.iso to the USB stick
    execute script to collect data
    before.txt:
=====================
fdisk output:
=====================
Disk /dev/sda: 7,31 GiB, 7849115648 bytes, 15330304 sectors
Disk model: Cruzer Blade    
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F06F80A4-8A9A-47F4-9311-95828702C3A1
First usable LBA: 64
Last usable LBA: 13471132
Alternative LBA: 13471195
Partition entries starting LBA: 2
Allocated partition entries: 248
Partition entries ending LBA: 63

Device        Start      End  Sectors Type-UUID                            UUID                                 Name      Attrs
/dev/sda1        64 13460703 13460640 EBD0A0A2-B9E5-4433-87C0-68B6B72699C7 F06F80A4-8A9A-47F4-9310-95828702C3A1 ISO9660   RequiredPartition LegacyBIOSBootable GUID:60
/dev/sda2  13460704 13470531     9828 C12A7328-F81F-11D2-BA4B-00A0C93EC93B F06F80A4-8A9A-47F4-9313-95828702C3A1 Appended2 
/dev/sda3  13470532 13471131      600 EBD0A0A2-B9E5-4433-87C0-68B6B72699C7 F06F80A4-8A9A-47F4-9312-95828702C3A1 Gap1      RequiredPartition GUID:60
=====================
hash values:
=====================
460e5289ebfa4ac3463101bc68d0d21e41a209f2e197629efcf7a3d11164dd7d  /dev/sda
ac0bdf60fac61d14ad81c2484306209bc29302a82a5af3db8a2bfe86156765a1  /dev/sda1
20fbece84f07cb0783bec3ad7f09631347fcb12961ffe14fae648510ad991db5  /dev/sda2
7818f5542a0404157573be6cffc0e0c8e68ce3c0f5d17d07ccdd9313fb700baf  /dev/sda3
=====================
generating .img
=====================
  • PC-B: boot and test media passed

  • VM: check hash values ok

  • PC-A: boot and test media in UEFI with CSM Support activated failed

  • VM: execute script to collect data
    afterwards.txt:

=====================
fdisk output:
=====================
Disk /dev/sda: 7,31 GiB, 7849115648 bytes, 15330304 sectors
Disk model: Cruzer Blade    
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F06F80A4-8A9A-47F4-9311-95828702C3A1
First usable LBA: 64
Last usable LBA: 13471132
Alternative LBA: 13471195
Partition entries starting LBA: 2
Allocated partition entries: 248
Partition entries ending LBA: 63

Device        Start      End  Sectors Type-UUID                            UUID                                 Name      Attrs
/dev/sda1        64 13460703 13460640 EBD0A0A2-B9E5-4433-87C0-68B6B72699C7 F06F80A4-8A9A-47F4-9310-95828702C3A1 ISO9660   RequiredPartition LegacyBIOSBootable GUID:60
/dev/sda2  13460704 13470531     9828 C12A7328-F81F-11D2-BA4B-00A0C93EC93B F06F80A4-8A9A-47F4-9313-95828702C3A1 Appended2 
/dev/sda3  13470532 13471131      600 EBD0A0A2-B9E5-4433-87C0-68B6B72699C7 F06F80A4-8A9A-47F4-9312-95828702C3A1 Gap1      RequiredPartition GUID:60
=====================
hash values:
=====================
efd6e70ad288c49ff521e94a29d37368dbc506b346e88cb33d16dbdded8a1400  /dev/sda
ac0bdf60fac61d14ad81c2484306209bc29302a82a5af3db8a2bfe86156765a1  /dev/sda1
20fbece84f07cb0783bec3ad7f09631347fcb12961ffe14fae648510ad991db5  /dev/sda2
7818f5542a0404157573be6cffc0e0c8e68ce3c0f5d17d07ccdd9313fb700baf  /dev/sda3
=====================
generating .img
=====================

Unfortunately, I don’t know what to look for in the .img data or directly on the USB stick, that’s now outside the fields I know.
I would also be willing to provide the .img files, but they are 14,6GiB and I don’t know where I can upload them anonymously.

Check the binary changes between the images:

cmp -l file1.bin file2.bin | gawk '{printf "%08X %02X %02X\n", $1-1, strtonum(0$2), strtonum(0$3)}'
1 Like
user@fedora:/run/media/user/data$ cmp -l before.img afterwards.img | gawk '{printf "%08X %02X %02X\n", $1-1, strtonum(0$2), strtonum(0$3)}'
000001BE 00 80
000001CE 80 00