Broken Qubes after changing keyboard layout with `localectl` - cannot run anything in AppVMs

User Support
1

After rebooting (everything was working before that), dom0 loads fine and I can start a terminal window, issue commands etc. qvm-ls shows that various AppVMs start fine, and remain in Running mode.

However, qvm-run --pass-io --verbose AppVM echo f hangs forever and never returns anything. This is true for all AppVMs, even newly created ones.

Obviously this makes my QubesOS device unusable, so I’m quite desperately hoping for suggestions on how to troubleshoot it. Many thanks in advance.

1 Like
2

What is the content of:


/var/log/xen/console/guest-${AppVM}.log

… where ${AppVM} is the name of your AppVM where you tried to run “echo f”?

EDIT: another idea… what kind of template do you use? Debian, Fedora, or a mix? If you rely heavily on fedora-41* templates, you may want to read this thread and maybe apply these steps (from the same thread) to fix your template(s).

1 Like
3

thanks, @barto for your quick answer.

In a newly created AppVM, the log you pointed at shows a pretty standard bootup sequence. The last line is: (AppVM) login: . So, there doesn’t seem to be anything useful in the logs. Do you have any suggestions for how to troubleshoot this further?

In fact, I was able to fix it. And in the process, I believe I found a qubesOS bug. Here is how to reproduce it:

  1. In dom0, change the keyboard layout using: localectl set-x11-keymap (layout) "" "" 'keycode 94 = grave'. This command should normally use the systemd-localed service, to change your keyboard layout to (layout) (the value of (layout) can be anything from /usr/share/X11/xkb/symbols/; for example, a typical value could be us). The part which reproduces the bug seems to be the keycode instruction. This is because it’s an invalid instruction (although localectl doesn’t seem to complain). Acceptable instructions can be found using localectl list-x11-keymap-options.
  2. Reboot.

That’s it, on my system, this triggers the bug I described.

To fix it, simply issue the localectl command, without the problematic keycode instruction: localectl (layout). Then check that the instruction is no longer active, using localectl status.

2 Likes
4

Ah, you figured out what you changed before the error! Best way to debug! :upside_down_face:

2 Likes
5

I think I was extremely lucky that I remembered what I changed, so I could undo it. Especially since it’s a very inconspicuous command; plus, as I mentioned, the logs didn’t provide any clue at all for what was going on.

2 Likes
6

This is an excellent reason to adjust the bash history setup, for BASH_ETERNAL_HISTORY & for appending the latest history at every prompt (PROMPT_COMMAND='history -a, methinks).

Then there is:

  • Up and down arrows to look at the latest few commands
  • history | less for manual searching by page
  • history | grep haddock for finding all entries containing the string “haddock”
  • and for recalling. Type a few characters to find a recent entry. Then to rerun it, or left/right arrows to modify it. Repeated looks further back in history.

Being cursed with bad luck, fat fingers, and a poor memory, bash history has saved me numerous times… but it does require occasional cleaning of the history file, because the prompt gets slow once the history file gets above a few 100000 entries.

1 Like