Cannot launch applications in most pre-installed VMs (including TemplateVMs)

User Support
1

Hello Qubes OS community,

I finally have Qubes OS R4.0.3 installed and somewhat functional, after the long custom installation saga I have been on (updates on that will come after I have solved this), though I have ran into many snags that I suspect are artifacts of that custom installation and the mistakes I made along the way. I have solved or hacked my way around most of them, but I am now faced with a problem for which no attempted fix has worked and which blocks my ability to use Qubes OS all together. Reinstalling Qubes OS may prove to be a “solution” of last resort, but that will requiring undoing and redoing much of my custom setup, so I want to avoid that if at all possible.

Problem

I am unable to open the terminal or any other app on almost any VM that was generated during initialization. This includes the fedora-30 and debian-10 TemplateVMs, as well as sys-firewall, sys-net, personal, and all the others based on them. The only exception to this are the Whonix VMs, including whonix-gw-15, whonix-ws-15, sys-whonix, anon-whonix, and whonix-ws-15-dvm, strangely enough.

When I create a new qube based on one of the dysfunctional TemplateVMs, it works without a problem. I am able to start any of the apps through any of the available methods. So while the TemplateVMs themselves (and all the VMs based on them during initialization) do not properly work, strangely a qube manually created and based on them does. Dom0 works without any problems, as well. Cloning the dysfunctional qube does not solve the problem; I would have to manually recreate all of them, which I would not mind if it were not for the fact that the TemplateVMs are themselves broken and I am not aware of any way I can recreate them without either reinstalling Qubes OS or reinstalling them via download.

Since I have not gotten this installation online yet, I have not updated anything beyond what was available in the Qubes R4.0.3 ISO. I want to keep it this way before I connect to the Internet, at least until I have finished customizing my networking. The last steps of that involve editing some configuration files on the TemplateVMs, but I am unable to do so for the reasons stated above.

I am finding no one else reporting issues similar to this, except for this [qubes-users] mailing list thread, which unfortunately went nowhere because the mailer never responded. I did encounter some generally similar issues, at least to parts of it, which are enumerated in the list at the bottom.

If it is at all relevant, I installed these VMs by manually starting the initialization process with the following command (source):

sudo /usr/libexec/initial-setup/initial-setup-graphical

This was done because the autostart initialization was skipped on the USB intermediate install, since I had to reinitialize once the system was transferred to my internal SSD in order to allow the sys-usb qube to set up (which successfully proxies USB connections, by the way, but whose terminal also cannot be launched).

When I did so, the only step that failed was updating GRUB through grub2-mkconfig, which was because the /boot partition was located on my USB drive and it was not mounted. Since I do not want the USB controllers being hidden from dom0 due to my intention of using Anti Evil Maid, anyway, I did not manually complete this step.

Expected behavior

When I launch an application within one of the initialized TemplateVMs or AppVMs, whether through the GUI or through a qvm-run command, the application is successfully launched.

Actual behavior

When I attempt to launch any application from any TemplateVM or AppVM other than a manually created one or, for some reason, Whonix VMs, nothing happens. No application is launched through the GUI, even though the VM appears to start without any problem, and qvm-run commands simply hang or error (see notes below).

Affected TemplateVMs

debian-10 and fedora-30.

(Whonix VMs are unaffected.)

What I have tried

  • Attempted to open a terminal through the Start Menu, through the “Run Terminal” option from the Qubes Domains manager, and via qvm-run. None work for the affected qubes; all work for the manually created ones, and for the Whonix qubes.
  • Ran qubesctl state.highstate as described here and here, which came back without any errors and reported a fully successful configuration. This did not fix the problem.
  • Manually installed qubes-start.desktop missing on dom0 from /usr/share/qubes-appmenus using the file described here, which fixed the “FileNotFoundError” described in #4684. This successfully allowed me to refresh the appmenus from either within the Qubes settings or from the terminal, but still did not allow any of the apps to launch.
  • Checked ~/.xsession-errors in dom0 as suggested here, which revealed no relevant log information.
  • Verified and ensured that gui and qrexec are set to 1 in qvm-features for all affected VMs, as suggested here, to no effect. There was no problem with qvm-features misconfiguration.
  • Ran qvm-run --service -p -a <affected qube> qubes.StartApp+<any application> from dom0, as suggested here, which simply hangs.
  • Attempted to run qvm-revert-template-changes as suggested here, which errors because this command does not exist. Either this is a pre-R4.0 command, despite still being mentioned here, or my installation for some reason does not have it.
  • Ran qvm-run <affected qube> "xterm" --pass-io from dom0, as suggested here, which also hangs.
  • Ran qvm-run -v <affected qube> "xterm" --pass-io --nogui from dom0, as also suggested here, fails but results in the following error:
Running 'xterm' on <affected qube>
qrexec-agent: chdir(/home/user): No such file or directory
xterm: Xt error: Can't open display: :0
  • Ran qvm-console-dispvm <affected qube>, which resulted in the same error as above. The DisposableVM does successfully launch, just not the console.
  • Ran qvm-run -v <affected qube> "ls" --pass-io --nogui, which gave the same error as before, except ls successfully ran and displayed the directory contents.
  • Checked /var/log/xen/console/guest-<affected-qube>, which (in debian-10) stated the following:
--
[XXXXXXXXXX] EXT4-fs (xvda3): re-mounted. Opts: (null)
switch_root: failed to mount moving /dev to /sysroot/dev: Invalid argument
switch_root: forcing unmount of /dev
switch_root: failed to mount moving /proc to /sysroot/proc: Invalid argument
switch_root: forcing unmount of /proc
switch_root: failed to mount moving /sys to /sysroot/sys: Invalid argument
switch_root: forcing unmount of /sys
switch_root: failed to mount moving /run to /sysroot/run: Invalid argument
switch_root: forcing unmount of /run
[XXXXXXXXXX] random: crng init done
--
Welcome to Debian GNU/Linux 10 (buster)!

[XXXXXXXXXX] systemd[1]: No hostname configured.
[XXXXXXXXXX] systemd[1]: Set hostname to <localhost>.
[XXXXXXXXXX] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid argument.
--
Starting Initialize and mount /rw and /home...
Starting Adjust root filesystem size...
[XXXXXXXXXX] Started udev Kernel Device Manager.
[FAILED] Failed to start Initialize and mount /rw and /home
See 'systemctl status qubes-mount-dirs.service' for details.
[ OK ] Started Adjust root filesystem size.
--
[ OK ] Started File System Check on /dev/xvdb.
[FAILED] Failed to mount /rw.
See 'systemctl status rw.mount' for details.
[DEPEND] Dependency failed for /home.
[DEPEND] Dependency failed for Session c2 of user user.
[DEPEND] Dependency failed for Session c1 of user user.
[DEPEND] Dependency failed for Session c3 of user user.
--
  • Entered a started qube’s console via Xen with sudo xl console <affected qube> (in this case debian-10, but the same on the others) as suggested here and here, logging in as passwordless root, and ran systemctl status qubes-mount-dirs.service, which indicated that /usr/lib/qubes/init/mount-dirs.sh failed and exited with status 32 with the following error logs from {localhost/debian-10} mount-dirs.sh:
Private device management: checking /dev/xvdb
Private device management: fsck.ext4 /dev/xvdb failed:
fsck.ext4 Bad magic number in superblock while trying to open /dev/xvdb
/dev/xvdb:
The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem. If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
or
    e2fsck -b 32768 <device>
mount: /rw: wrong fs type, bad option, bad superblock on /dev/xvdb, missing codepage or helper program, or other error.
  • Also ran systemctl status rw.mount from within the console, which also showed that it failed and exited with status 32.
  • Ran systemctl status qubes-gui-agent, as well, which succeeded but also complained it failed to create 'session-c1.scope' and cannot create /home/user/.xsession-errors, and so closed the session for userui-runuser.
  • Manually running the e2fsck commands above within the console failed without effect, and mkdir /home/user and other missing directories then mount /home (as suggested here) and so on did not accomplish anything, even though /home was mounted. /rw did not mount because of a missing “special device” and I decided not to continue further so as to not break anything.
  • Attempting to systemctl restart qubes-mount-dirs.service failed as well.

Final remarks

I am at a loss of what I can do about this. I hope the wizards here can point out some other arcane invocations I can use to either fix this problem, with a hacky workaround if necessary just so I can update and hopefully resolve all further problems, or at least better diagnose it. Why are the VMs not mounting properly? What controls this and how can I debug it? Is there a file I can edit to fix this? According to this, the e2fsck errors may have to do with inconsistent system time. Is there any way I can determine whether this is a factor and how to fix it if so? As far as I can tell, my system time is accurate and consistent with network time, though I am not sure about the VMs. If it has to do with modification times, might touching some file fix it?

Alternatively, is there a back-end way of getting into the TemplateVM to modify its configurations, which should work because the VM otherwise does despite being unable to launch any apps? Might manually mounting it be one such way? If so, then I can finish my configuration and connect to the Internet, at which point updating might fix it.

Any instruction on this complicated matter will be greatly appreciated.

Best regards,
John

Sources

Related issues

Documentation

1 Like
2

Have you tried reinstalling the TempalteVMs?

3

Hey @deeplow,

Thanks for the suggestion. However, I understand those instructions to assume access to the Internet. Is that correct? If so, then that conflicts with the current state of my installation, as described above:

In fact, I do not even know if I can connect to the Internet in this state. I have not yet tried because I have not finished my configuration, but there is a good chance that these VMs are not actually as functional as I assume, especially if they are not even mounting properly. If that is the case, then any changes I will need to do will have to be offline.

If I am able to reinstall the TemplateVMs without connecting to the Internet, perhaps with the preinstalled packages that may still be on my computer, then I can try that. I do not know how, though, and the documentation you linked does not make that clear. Last time I uninstalled the TemplateVMs (I really need to post an update to that, the instructions I gave need correction), I had to reinstall Qubes OS on a separate USB drive and rsync over the missing files because I removed the template images and had no Internet connection or VMs set up to fix it.

If you know of a way that I can do an offline reinstallation of the TemplateVMs, I will gladly give it a shot. If not, then maybe my best chance remains with mounting the TemplateVMs manually and reconfiguring them from the terminal.

Appreciating any help,
John

4

Sorry, I missed the no network part.

I don’t know either, but I bet it’s possible.

Or you can download it elsewhere and add it with a USB stick and then transfer it to dom0. Not sure if that a risk you find acceptable or not. Find the RPM here:
https://yum.qubes-os.org/r4.0/templates-itl/rpm/

1 Like
5

Last time I brought TemplateVM packages over to my Qubes OS installation, through a second intermediate install (itself installed from the DVD live disk), I used a USB drive purchased locally in-person that has never been used for any other purpose and never touched any other device (while in my possession); and I rsynced the files over by mounting both drives from within the same Qubes R4.0.3 live disk that I used to install them.

I don’t know if that sufficiently mitigates most risks of transferring files from a USB into dom0, but that was the most secure I could think of without extracting the files and burning them onto a read-only DVD. Naturally, risk still exists here (mainly from indiscriminate supply-chain attacks affecting in-store USB sticks in my area), but I am already using a USB stick (from the same pack) for my detached /boot and LUKS headers and will also be using one for AEM, which apparently must be attached directly to dom0 anyway.

But if I need the files, I can just extract them from one of the Qubes installations I still have, which should be fine. Given that I rsynced from one and did not otherwise modify it, though, it is strange that they are malfunctioning like this. Surely, a fresh installation of Qubes OS is not this broken out of the box, so either Qubes OS is too fragile for intermediate installation and rsyncing or somewhere along the way I broke something. I just don’t know what.

After filtering out differences based only on timestamps, my copies keep reporting as equivalent except for some cache files and salt configurations that are likely the oneshots triggered and deleted during initialization. (There are also the custom configuration files I have, such as /etc/crypttab, but I am ignoring those too.) So I am not sure what I am missing.

For now, unless anyone has a better suggestion (I really want one!), I will try to see how I can manually mount and access the TemplateVMs. If that does not work, or I am unable to connect to the Internet with my configurations and update the system, or if that does not fix it, then I may just need to either reinitialize with the command above or reinstall Qubes OS altogether. :frowning:

Might emailing the [qubes-users] mailing list be more successful than here, given the complicated technical nature of my problem? I have been trying to avoid doing so, but maybe I will have better luck there.

Regards,
John

1 Like