Did the name change hack.
uname -a
shows I’m in Linux 6.1.75 instead of 6.6.5
But sys-net still not booting.
Check the sys-net-dm.log, maybe there will be some errors there. And sys-net.log as well.
You can see then using Qube Manager or in dom0:
/var/log/xen/console/guest-sys-net.log
/var/log/xen/console/guest-sys-net-dm.log
/var/log/xen/console/guest-sys-net.log
hasn’t logged anything since failure to boot, except log file being opened
/var/log/xen/console/guest-sys-net-dm.log
has lots in it, but don’t see errors or warnings
I’d try to downgrade the xen-hvm-stubdom-linux/xen-hvm-stubdom-linux-full packages first to the versions that were previously installed and worked.
You can find the old working versions like this:
Get the History of an Installed Package with DNF | Thomas Stringer
If it won’t help then I’d try to downgrade Xen packages.
I think I’d be better to install Qubes OS on a separate disk and debug this issue there.
oh, can’t do this, no network
You can download the packages on another system and then move them to dom0 to install.
But it’s not really recommended from security standpoint.
I’ve got no sys-usb either though.
Then you can boot from some LiveCD and mount the dom0 there to copy the files.
But it’s better to debug the issue on Qubes OS installed on another disk first.
Thanks for this, figured out what versions of packages were there before.
Are these qubes specific packages? Trying to get them in a Fedora LiveCD but they don’t exist. Where can I download them?
Yes, they are Qubes-specific.
You can download them from the repository:
Index of /r4.2/current/dom0/fc37/rpm/
You can check the signature of downloaded package like this:
Anyone got an AppVM or Debian standalone solution for Mullvad VPN? - #16 by unman
Tried:
- boot into Fedora live disk
- from here: Index of /r4.2/current/dom0/fc37/rpm/
I downloaded: xen-hvm-stubdom-linux-4.2.9-1.fc37.x86_64.rpm AND xen-hvm-stubdom-linux-full-4.2.9-1.fc37.x86_64.rpm - mount my Qubes OS encrypted harddrive
- copy downloaded packages to home directory
- reboot into Qubes
- ran
rpm -K <each package name>
to verify signatures - ran
sudo dnf localinstall <each package name> - qvm-start sys-net
Got same results.
Tried downgrading xen, xen-hypervisor, xen-libs, xen-licenses and xen runtime to 4.17.3-4, but it gave me error:
The operation would result in removing the following protected packages: qubes-core-dom0
So not sure how to proceed.
I guess last thing to try is to downgrade the Xen packages:
[user@dom0 ~]$ dnf list installed | grep 4.17.3-5
python3-xen.x86_64 2001:4.17.3-5.fc37 @qubes-dom0-cached
xen.x86_64 2001:4.17.3-5.fc37 @qubes-dom0-cached
xen-hypervisor.x86_64 2001:4.17.3-5.fc37 @qubes-dom0-cached
xen-libs.x86_64 2001:4.17.3-5.fc37 @qubes-dom0-cached
xen-licenses.x86_64 2001:4.17.3-5.fc37 @qubes-dom0-cached
xen-runtime.x86_64 2001:4.17.3-5.fc37 @qubes-dom0-cached
But there could be some problems with dependencies which could require you to downgrade some other packages as well.
Tbh it’d be much easier if you could install test Qubes OS on a separate drive and update the packages there without updating just Xen packages (install a specific Xen version that worked for you and exclude the Xen packages from the updates).
Yes, qubes-core-dom0 (see above)
Well, that’d be a pain to go through all the package dependencies and downgrade them as well.
You’ll try to downgrade qubes-core-dom0 and it could require another 10 packages to downgrade and these 10 packages could require another 10 packages each to downgrade.
Maybe there is a way to list all the dependencies that you need to downgrade as well to downgrade specific package, but I don’t know of it.
Maybe you can get the list of installed packages since the last working ones from dnf history and just downgrade them all.
But this list could be quite big.
But you can try to downgrade
python3-xen.x86_64 2001:4.17.3-5.fc37 @qubes-dom0-cached
That you’ve missed as marmarek suggested, maybe that’d be enough.
1 Like
That did it. Including python3-xen in the list of packages I was downgrading for xen allowed it to work. Now everything is booting again.
Guess I just won’t update dom0 for a bit.
Thanks so much for walking me through this process today!
You can exclude the Xen packages from dom0 updates by editing the /etc/dnf/dnf.conf in dom0 and adding exclude=pkg1 pkg2 ... there:
Got the solution from Tonux599 on github:
I added
GRUB_CMDLINE_XEN_DEFAULT="$GRUB_CMDLINE_XEN_DEFAULT spec-ctrl=ibpb-entry=no-pv"
to the end of /etc/default/grub and updated with grub2-mkconfig -o /boot/grub2/grub.cfg and then reboot. Then updated Xen from 4.17.3-4 to 4.17.4-2 and other xen-hvm-stubdom* packages from 4.2.9 to 4.2.12, rebooted again, and now all my Qubes boot with latest Xen.
NOTE This is opening up a big security hole as marmarek explains:
So, I’m afraid there is not much hope for this old-ish system… The only way to make the system kinda-usable has a tradeoff with security here, by disabling the mitigation for PV domains (which should mean just stubdomains, make sure you don’t have any really untrusted PV qubes) with
spec-ctrl=ibpb-entry=no-pv. It does mean that stubdomain will be able to mount the attack, potentially leaking memory of any other VM (so isolation of sys-net/sys-usb and any other HVM becomes weaker). If that is not an acceptable risk, blame AMD for making buggy CPU, and replace with something newer…
And the Qubes team has declined to attempt to address this bug in the future.