Hello Qubes OS community,
I finally have Qubes OS R4.0.3 installed and somewhat functional, after the long custom installation saga I have been on (updates on that will come after I have solved this), though I have ran into many snags that I suspect are artifacts of that custom installation and the mistakes I made along the way. I have solved or hacked my way around most of them, but I am now faced with a problem for which no attempted fix has worked and which blocks my ability to use Qubes OS all together. Reinstalling Qubes OS may prove to be a “solution” of last resort, but that will requiring undoing and redoing much of my custom setup, so I want to avoid that if at all possible.
I am unable to open the terminal or any other app on almost any VM that was generated during initialization. This includes the
debian-10 TemplateVMs, as well as
personal, and all the others based on them. The only exception to this are the Whonix VMs, including
whonix-ws-15-dvm, strangely enough.
When I create a new qube based on one of the dysfunctional TemplateVMs, it works without a problem. I am able to start any of the apps through any of the available methods. So while the TemplateVMs themselves (and all the VMs based on them during initialization) do not properly work, strangely a qube manually created and based on them does. Dom0 works without any problems, as well. Cloning the dysfunctional qube does not solve the problem; I would have to manually recreate all of them, which I would not mind if it were not for the fact that the TemplateVMs are themselves broken and I am not aware of any way I can recreate them without either reinstalling Qubes OS or reinstalling them via download.
Since I have not gotten this installation online yet, I have not updated anything beyond what was available in the Qubes R4.0.3 ISO. I want to keep it this way before I connect to the Internet, at least until I have finished customizing my networking. The last steps of that involve editing some configuration files on the TemplateVMs, but I am unable to do so for the reasons stated above.
I am finding no one else reporting issues similar to this, except for this [qubes-users] mailing list thread, which unfortunately went nowhere because the mailer never responded. I did encounter some generally similar issues, at least to parts of it, which are enumerated in the list at the bottom.
If it is at all relevant, I installed these VMs by manually starting the initialization process with the following command (source):
This was done because the autostart initialization was skipped on the USB intermediate install, since I had to reinitialize once the system was transferred to my internal SSD in order to allow the
sys-usb qube to set up (which successfully proxies USB connections, by the way, but whose terminal also cannot be launched).
When I did so, the only step that failed was updating GRUB through
grub2-mkconfig, which was because the
/boot partition was located on my USB drive and it was not mounted. Since I do not want the USB controllers being hidden from dom0 due to my intention of using Anti Evil Maid, anyway, I did not manually complete this step.
When I launch an application within one of the initialized TemplateVMs or AppVMs, whether through the GUI or through a
qvm-run command, the application is successfully launched.
When I attempt to launch any application from any TemplateVM or AppVM other than a manually created one or, for some reason, Whonix VMs, nothing happens. No application is launched through the GUI, even though the VM appears to start without any problem, and
qvm-run commands simply hang or error (see notes below).
(Whonix VMs are unaffected.)
What I have tried
- Attempted to open a terminal through the Start Menu, through the “Run Terminal” option from the Qubes Domains manager, and via
qvm-run. None work for the affected qubes; all work for the manually created ones, and for the Whonix qubes.
qubesctl state.highstateas described here and here, which came back without any errors and reported a fully successful configuration. This did not fix the problem.
- Manually installed
qubes-start.desktopmissing on dom0 from
/usr/share/qubes-appmenususing the file described here, which fixed the “FileNotFoundError” described in #4684. This successfully allowed me to refresh the appmenus from either within the Qubes settings or from the terminal, but still did not allow any of the apps to launch.
~/.xsession-errorsin dom0 as suggested here, which revealed no relevant log information.
- Verified and ensured that
qrexecare set to
qvm-featuresfor all affected VMs, as suggested here, to no effect. There was no problem with
qvm-run --service -p -a <affected qube> qubes.StartApp+<any application>from dom0, as suggested here, which simply hangs.
- Attempted to run
qvm-revert-template-changesas suggested here, which errors because this command does not exist. Either this is a pre-R4.0 command, despite still being mentioned here, or my installation for some reason does not have it.
qvm-run <affected qube> "xterm" --pass-iofrom dom0, as suggested here, which also hangs.
qvm-run -v <affected qube> "xterm" --pass-io --noguifrom dom0, as also suggested here, fails but results in the following error:
Running 'xterm' on <affected qube> qrexec-agent: chdir(/home/user): No such file or directory xterm: Xt error: Can't open display: :0
qvm-console-dispvm <affected qube>, which resulted in the same error as above. The DisposableVM does successfully launch, just not the console.
qvm-run -v <affected qube> "ls" --pass-io --nogui, which gave the same error as before, except
lssuccessfully ran and displayed the directory contents.
/var/log/xen/console/guest-<affected-qube>, which (in
debian-10) stated the following:
-- [XXXXXXXXXX] EXT4-fs (xvda3): re-mounted. Opts: (null) switch_root: failed to mount moving /dev to /sysroot/dev: Invalid argument switch_root: forcing unmount of /dev switch_root: failed to mount moving /proc to /sysroot/proc: Invalid argument switch_root: forcing unmount of /proc switch_root: failed to mount moving /sys to /sysroot/sys: Invalid argument switch_root: forcing unmount of /sys switch_root: failed to mount moving /run to /sysroot/run: Invalid argument switch_root: forcing unmount of /run [XXXXXXXXXX] random: crng init done -- Welcome to Debian GNU/Linux 10 (buster)! [XXXXXXXXXX] systemd: No hostname configured. [XXXXXXXXXX] systemd: Set hostname to <localhost>. [XXXXXXXXXX] systemd: Failed to bump fs.file-max, ignoring: Invalid argument. -- Starting Initialize and mount /rw and /home... Starting Adjust root filesystem size... [XXXXXXXXXX] Started udev Kernel Device Manager. [FAILED] Failed to start Initialize and mount /rw and /home See 'systemctl status qubes-mount-dirs.service' for details. [ OK ] Started Adjust root filesystem size. -- [ OK ] Started File System Check on /dev/xvdb. [FAILED] Failed to mount /rw. See 'systemctl status rw.mount' for details. [DEPEND] Dependency failed for /home. [DEPEND] Dependency failed for Session c2 of user user. [DEPEND] Dependency failed for Session c1 of user user. [DEPEND] Dependency failed for Session c3 of user user. --
- Entered a started qube’s console via Xen with
sudo xl console <affected qube>(in this case
debian-10, but the same on the others) as suggested here and here, logging in as passwordless
root, and ran
systemctl status qubes-mount-dirs.service, which indicated that
/usr/lib/qubes/init/mount-dirs.shfailed and exited with status 32 with the following error logs from
Private device management: checking /dev/xvdb Private device management: fsck.ext4 /dev/xvdb failed: fsck.ext4 Bad magic number in superblock while trying to open /dev/xvdb /dev/xvdb: The superblock could not be read or does not describe a valid ext2/ext3/ext4 filesystem. If the device is valid and it really contains an ext2/ext3/ext4 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 8193 <device> or e2fsck -b 32768 <device> mount: /rw: wrong fs type, bad option, bad superblock on /dev/xvdb, missing codepage or helper program, or other error.
- Also ran
systemctl status rw.mountfrom within the console, which also showed that it failed and exited with status 32.
systemctl status qubes-gui-agent, as well, which succeeded but also complained it failed to create
'session-c1.scope'and cannot create
/home/user/.xsession-errors, and so closed the session for
- Manually running the
e2fsckcommands above within the console failed without effect, and
mkdir /home/userand other missing directories then
mount /home(as suggested here) and so on did not accomplish anything, even though
/rwdid not mount because of a missing “special device” and I decided not to continue further so as to not break anything.
- Attempting to
systemctl restart qubes-mount-dirs.servicefailed as well.
I am at a loss of what I can do about this. I hope the wizards here can point out some other arcane invocations I can use to either fix this problem, with a hacky workaround if necessary just so I can update and hopefully resolve all further problems, or at least better diagnose it. Why are the VMs not mounting properly? What controls this and how can I debug it? Is there a file I can edit to fix this? According to this, the
e2fsck errors may have to do with inconsistent system time. Is there any way I can determine whether this is a factor and how to fix it if so? As far as I can tell, my system time is accurate and consistent with network time, though I am not sure about the VMs. If it has to do with modification times, might
touching some file fix it?
Alternatively, is there a back-end way of getting into the TemplateVM to modify its configurations, which should work because the VM otherwise does despite being unable to launch any apps? Might manually mounting it be one such way? If so, then I can finish my configuration and connect to the Internet, at which point updating might fix it.
Any instruction on this complicated matter will be greatly appreciated.
- FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/qubes-appmenus-qubes-start.desktop' · Issue #4684 · QubesOS/qubes-issues · GitHub
- Apps do not open when starting VM from launcher · Issue #3533 · QubesOS/qubes-issues · GitHub
- Update causes qubes-mount-dirs.service to not start on Whonix (Debian8) template / Missing libxen dependency · Issue #2484 · QubesOS/qubes-issues · GitHub
- Nothing happens when trying to start application in whonix-14 · Issue #4067 · QubesOS/qubes-issues · GitHub
- Debian-8 /home not mounted resulting in no display :0 · Issue #3398 · QubesOS/qubes-issues · GitHub
- Standard fsck isn't run on VM startup · Issue #979 · QubesOS/qubes-issues · GitHub
- [qubes-users] Long boot time for "Initialize and mount /rw and /home" unit
- [qubes-users] AppVM won't start any application
- [qubes-users] Debian standalonevm always enters emergency mode