According to the debugging section of the official Qubes Salt documentation, I should be able to run qubesctl, Ctrl+Z when the disp-mgmt VM is in a transient, starting state. Then launch a terminal in the disp-mgmt VM and view logs from there. I’m following this process because (despite setting --log-file-level=all) the logs present in dom0 have not been detailed enough for debugging/troubleshooting purposes (/var/log/qubes/mgmt-* and /var/log/salt/minion). In most cases, /var/log/salt/minion is completely void of logs despite running several commands using qubesctl. Attempting to spawn a terminal in the relevant disp-mgmt VM fails, both using qvm-run and via the Qubes managment tray icon. I tried checking journalctl, but couldn’t find anything useful in there.
I maybe can’t help, but what are you trying to figure? Is there any problem occur? And what are you trying to do beside debugging?
My guess is that you are blocking too soon - you want the mgmtVM to be up
and running, but the target not yet started.
It takes a little practice, but you’ll get used to it.
Also, try using qvm-console-dispvm to connect to a console, and log in
as root. That will work in almost every case.
The issue is lack of insight into what is executing at any given point in time (/var/log/salt/minion is empty unless passing --skip-dom0 and /var/log/qubes/mgmt-* are very bare). I’m attempting to troubleshoot why some portions of my state functions are failing.
Thanks for the direction, @unman! I was able to establish a terminal within the disp-mgmt-vm using qvm-console-dispvm (after the disp-mgmt-vm started and while the target VM was in a “starting” state). I then executed the last two commands within /etc/qubes-rpc/qubes.SaltLinuxVM (export PATH and salt-ssh), subbing in the target VM name where needed. Now, the salt-ssh command seems to hang, with no output. Should I be tailing a log file somewhere in the dispvm-console?
You’re hoping for feedback while the state is running? You can add -l debug to the ssh command.
As to tracing the state while it is running that can be difficult.
Post the (anonymised) state file and I’ll take a look. Much quicker.