Feedback Qubes OS alpha with sys-gui

it was /srv/formulas/base/virtual-machines-formula/qvm/ (sorry i was trying to remember the path)

This is happen when you try to run qubesctl --all state.highstate command and you have ‘enabled’ top file of sys-gui, to resolve the issue you should remove first.

1 Like

Thanks! I disabled the sys-gui and now the above mentioned error is gone and there is no error message regarding the state files (10 succeeded, failed 0) but I get an exit code 20 nonetheless. The mgmt-sys-gui-gpu.log is like this:

2021-06-30 17:00:50,263 output: sys-gui-gpu:
2021-06-30 17:00:50,265 output:     ----------
2021-06-30 17:00:50,266 output:     _error:
2021-06-30 17:00:50,266 output:         Failed to return clean data
2021-06-30 17:00:50,266 output:     retcode:
2021-06-30 17:00:50,266 output:         126
2021-06-30 17:00:50,266 output:     stderr:
2021-06-30 17:00:50,267 output:         Request refused
2021-06-30 17:00:50,267 output:     stdout:
2021-06-30 17:00:50,267 exit code: 20

When sys-gui-gpu is starting there is this error:
No such file or directory 'usr/bin/qubes-input-trigger --all --dom0'

The most basic way is creating a new .conf file under /etc/X11/xorg.conf.d and specifying:

Section "Device"
   Identifer "intel graphics"
   Driver "intel"
EndSection

See man xorg.conf in case there are additional options you might want to specify.

I might have gotten closer to the promised land.

I had an older install of Qubes 4.1 on another SSD, pre Xen 4.14.1 to be precise. I recalled that at some point I could start sys-gui-gpu and so I tried again yesterday. The template was still fedora-32-xfce, everything like in the first thread opened by @panati last year:

Thanks to the information trickeling in bits and pieces from here and there I think I know a little more (not much) on how this is supposed to work.

I disabled sys-gui and enabled sys-gui-gpu.
I tried attaching my 2nd GPU via qubesctl state.sls qvm.sys-gui-gpu-attach-gpu like explained in the documentation draft. This selected 00:02.0 PCI (...) Processor root port which I think isn’t the right choice. I got an error message regarding a “non-ending pci device (…)”. I don’t know what to make of it but I read that the whole sys-gui-gpu is meant to be working out of the box for Intel rather than AMD graphics. So I looked at the list of pci-devices in my sys-gui-gpu app.

My 2nd GPU is listed as a display controller for my Radeon HD 8570.
01:00.0 Display Controller: Advanced Micro Devices, Inc. [AMD/ATI] Sun PRO [Radeon HD 8570A/8570M]
I selected this one and my laptop froze immediately after applying and restarting sys-gui-gpu. I read that this is expected and a reboot might be needed.

I restarted and my autostart-apps are booting up including sys-gui-gpu.

Now I am asking myself how I can check if things are working as they should?
First of all, might this be the right choice for my 2nd GPU. (It looked right to me but what do I know?)
Do I have to specifically assign selected or all apps to the sys-gui-gpu as guiVM like with sys-gui?
Something like qvm-prefs work guivm sys-gui-gpu ?

Thanks in advance!

1 Like

Just a little addition to my monologue:
So it looked like it was working the past few days until an hour ago.
When I booted up everything looked normal but then the task bar with all system icons disappeared and sys-gui-gpu doesn’t start anymore. The same libxenlight error I had with every other test system before prevents sys-gui-gpu from starting. Bummer!

1 Like

After clicking a few of the system tools and most of the stuff working perfectly there was an error window regarding freedesktop.org.
After rebooting everything (=taskbar & symbols) is working again, including sys-gui-gpu. Looking at the service entry says it’s loaded and working.

I noticed that my root volume is almost full, so the problems might be connected to that.

1 Like

There are quite several threads mentionning those problems with sys-gui-gpu, but I can’t really find an “official statement” about what we should do to get it running. Here is my attempt to put everything in one place at the same time I’m doing the steps. Hopefully a dev in-the-know will tell if this is what we’re expected to do, and fill in the blanks :wink:
@fepitre or @marmarek maybe ?

I went directly to sys-gpu-gui without going the sys-gui path first, as that hybrid approach is not I’m after.

The steps go like:

  • official instructions, failing with failure to install dummy-psu-dom0
    qubesctl top.enable qvm.sys-gui-gpu
    qubesctl top.enable qvm.sys-gui-gpu pillar=True
    qubesctl --all state.highstate
    
  • fix the Salt recipe by editing /srv/formulas/base/virtual-machines-formula/qvm/sys-gui-gpu.sls to replace dummy-psu-dom0 with dummy-psu-sender
  • rerun the 3 qubesctl calls (though I guess the 3rd one could be sufficient):
    • the 10 Salt steps all pass this time
    • sys-gui-gpu is reported as ERROR, mgmt logfile say “request refused”, qrexec-mgmt logfile show something I did not find in any other post:
WARNING:root:warning: !compat-4.0 directive in file /etc/qubes/policy.d/35-compat.policy line 16 is transitional and will be deprecated
INFO:policy:qrexec: qubes.VMRootShell+: disp-mgmt-sys-gui-gpu -> sys-gui-gpu: allowed to sys-gui-gpu
Traceback (most recent call last):
  File "/usr/bin/qrexec-policy-exec", line 5, in <module>
    sys.exit(main())
  File "/usr/lib/python3.8/site-packages/qrexec/tools/qrexec_policy_exec.py", line 221, in main
    return asyncio.run(handle_request(
  File "/usr/lib64/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/lib/python3.8/site-packages/qrexec/tools/qrexec_policy_exec.py", line 274, in handle_request
    await resolution.execute(caller_ident)
  File "/usr/lib/python3.8/site-packages/qrexec/tools/qrexec_policy_exec.py", line 169, in execute
    await super().execute(caller_ident)
  File "/usr/lib/python3.8/site-packages/qrexec/tools/qrexec_policy_exec.py", line 133, in execute
    await super().execute(caller_ident)
  File "/usr/lib/python3.8/site-packages/qrexec/policy/parser.py", line 538, in execute
    self.ensure_target_running()
  File "/usr/lib/python3.8/site-packages/qrexec/policy/parser.py", line 579, in ensure_target_running
    utils.qubesd_call(self.target, 'admin.vm.Start')
  File "/usr/lib/python3.8/site-packages/qrexec/utils.py", line 95, in qubesd_call
    raise AssertionError(
AssertionError: invalid qubesd response: b''
2021-07-30 13:57:01.566 qrexec-daemon[47761]: qrexec-daemon.c:1212:main: qrexec-agent has disconnected
domain dead
2021-07-30 13:57:01.567 qrexec-daemon[47761]: qrexec-daemon.c:1104:handle_agent_restart: cannot connect to qrexec agent: No such process
2021-07-30 13:57:01.567 qrexec-daemon[47761]: qrexec-daemon.c:1214:main: Failed to reconnect to qrexec-agent, terminating

That kinda looks like a protocol mismatch between qubesd and qrexec-agent ?

  • fedora-33-xfce is reported as ERROR, logfile say as in this post
  • I see complaints about qubes-input-trigger in popups (which Discourse Search strangely locates in no other post, but which Duckduckgo luckily confirms occurs in other posts)
  • change the template for default-mgmt-dvm from fedora-32 to fedora-33-xfce
  • rerun the 3 qubesctl --all state.highstate
    • fedora-33-xfce now OK
    • sys-gui-gpu still reported in ERROR, with popup telling it still cannot '/usr/bin/qubes-input-trigger --all --dom0'.
  • something obviously missing here

inconclusive attempts to tackle the input-trigger --all --dom0 issue

  • applied this patch to /usr/lib/python3.8/site-packages/qubes/ext/gui.py as mentionned here
  • systemctl restart qubesd.service
    • now it’s /usr/bin/qubes-input-trigger not being found (though it is included in the template), and the qrexec_policy_exec exception persists

USB mouse

At this point when I plug a USB mouse, Qubes tries to start sys-gui-gpu, which fails because of qubes-input-trigger, and my mouse is not usable. The following step probably has to be added somewhere in the recipe:

  • qubesctl state.sls qvm.sys-usb

operations to do later when that step has been fixed

  • qubesctl top.disable qvm.sys-gui-gpu
  • attach AMD GPU to sys-gui-gpu since only Intel ones are handled today
  • qvm-prefs work guivm sys-gui-gpu ?
  • other ?
1 Like

This is likely just a workaround for the failure to configure sys-gui-gpu: while this does fix the loss of USB mouse, it what it does is undoing one thing that qubesctl --all state.highstate does:

----------
          ID: sys-usb-input-proxy
    Function: file.prepend
        Name: /etc/qubes-rpc/policy/qubes.InputMouse
      Result: True
     Comment: Prepended 1 lines
     Started: 00:21:49.493232
    Duration: 1.833 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1,2 +1,3 @@
                  +sys-usb dom0 allow,user=root
                   sys-usb dom0 allow,user=root,target=sys-gui-gpu
                   $anyvm $anyvm deny

Aside from this, after updating to current 4.1 snapshot I still have those same symptoms: @marmarek @fepitre qrexec-agent seems to crash in sys-gui-gpu, what kind of information could I collect ? Wouldn’t this script benefit from more logging ?

… so let’s fill in the blanks.

Rerunning on nowadays 4.1beta, the last problem blocking sys-gui-gpu installation was https://github.com/QubesOS/qubes-issues/issues/6893, which can be worked around by installing qubes-input-proxy-sender by hand in dom0.

Then comes the time to attach the GPU and reboot. Be ready to have you QubesOS loop-rebooting: you may need to add qubes.skip_autostart on kernel command-line to be able to boot again, until you find the proper way to attach your GPU.

I tried to test sys-gui on latest kernel 4th Sep 2021 weekly build and I got the following errors at the last command. Where should I go from here. Please guide. I am clueless.

[Anirban@dom0 ~]$ sudo qubesctl top.enable qvm.sys-gui
local:
----------
qvm.sys-gui.top:
----------
status:
enabled
[Anirban@dom0 ~]$ sudo qubesctl top.enable qvm.sys-gui pillar=True
local:
----------
qvm.sys-gui.top:
----------
status:
enabled
[Anirban@dom0 ~]$ sudo qubesctl --all state.highstate
[ERROR ] Command ‘systemd-run’ failed with return code: 1
[ERROR ] stdout: Running scope as unit: run-rc28f9fd9937b4a92af56de82f7460ead.scope
Using sys-firewall as UpdateVM to download updates for Dom0; this may take some time…
Running ‘/usr/lib/qubes/qubes-download-dom0-updates.sh --doit --nogui ‘–exclude=qubes-template-whonix-gw-15,qubes-template-debian-10,qubes-template-fedora-34,qubes-template-whonix-ws-15’ ‘-y’ ‘–best’ ‘–allowerasing’ ‘–clean’ ‘–action=install’ ‘qubes-template-fedora-34-xfce’’ on sys-firewall
sys-firewall: command failed with code: 1
[ERROR ] retcode: 1
[ERROR ] Error occurred installing package(s). Additional info follows:

errors:
- Running scope as unit: run-rc28f9fd9937b4a92af56de82f7460ead.scope
Using sys-firewall as UpdateVM to download updates for Dom0; this may take some time…
Running ‘/usr/lib/qubes/qubes-download-dom0-updates.sh --doit --nogui ‘–exclude=qubes-template-whonix-gw-15,qubes-template-debian-10,qubes-template-fedora-34,qubes-template-whonix-ws-15’ ‘-y’ ‘–best’ ‘–allowerasing’ ‘–clean’ ‘–action=install’ ‘qubes-template-fedora-34-xfce’’ on sys-firewall
sys-firewall: command failed with code: 1
[ERROR ] ====== [‘present’] ======
====== stderr ======
/usr/bin/qvm-create sys-gui --class=AppVM --template=fedora-34-xfce --label=black
app: Error creating VM: Got empty response from qubesd. See journalctl in dom0 for details.

====== [‘prefs’] ======
Virtual Machine does not exist!

====== [‘service’] ======
[SKIP] Skipping due to previous failure!

====== [‘features’] ======
[SKIP] Skipping due to previous failure!
local:

      ID: topd-always-passes
Function: test.succeed_without_changes
    Name: foo
  Result: True
 Comment: Success!
 Started: 20:01:57.951700
Duration: 0.984 ms
 Changes:   

      ID: qubes-template-fedora-34-xfce
Function: pkg.installed
  Result: False
 Comment: Error occurred installing package(s). Additional info follows:
          
          errors:
              - Running scope as unit: run-rc28f9fd9937b4a92af56de82f7460ead.scope
                Using sys-firewall as UpdateVM to download updates for Dom0; this may take some time...
                Running '/usr/lib/qubes/qubes-download-dom0-updates.sh --doit --nogui '--exclude=qubes-template-whonix-gw-15,qubes-template-debian-10,qubes-template-fedora-34,qubes-template-whonix-ws-15' '-y' '--best' '--allowerasing' '--clean' '--action=install' 'qubes-template-fedora-34-xfce'' on sys-firewall
                sys-firewall: command failed with code: 1
 Started: 20:02:02.712592
Duration: 292687.428 ms
 Changes:   

      ID: dummy-psu-sender
Function: pkg.installed
  Result: True
 Comment: The following packages were installed/updated: dummy-psu-sender
 Started: 20:06:55.400359
Duration: 26406.807 ms
 Changes:   
          ----------
          dummy-psu-sender:
              ----------
              new:
                  1.0.0-2.fc32
              old:

      ID: dummy-backlight-dom0
Function: pkg.installed
  Result: True
 Comment: 1 targeted package was installed/updated.
 Started: 20:07:21.849464
Duration: 24626.707 ms
 Changes:   
          ----------
          dummy-backlight-dom0:
              ----------
              new:
                  1.0.0-2
              old:

      ID: /usr/share/xsessions/sys-gui.desktop
Function: file.managed
  Result: True
 Comment: File /usr/share/xsessions/sys-gui.desktop updated
 Started: 20:07:46.507336
Duration: 82.714 ms
 Changes:   
          ----------
          diff:
              New file

      ID: sys-gui
Function: qvm.vm
  Result: False
 Comment: ====== ['present'] ======
          ====== stderr ======
          /usr/bin/qvm-create sys-gui --class=AppVM --template=fedora-34-xfce --label=black 
          app: Error creating VM: Got empty response from qubesd. See journalctl in dom0 for details.
          
          ====== ['prefs'] ======
          Virtual Machine does not exist!
          
          ====== ['service'] ======
          [SKIP] Skipping due to previous failure!
          
          ====== ['features'] ======
          [SKIP] Skipping due to previous failure!
 Started: 20:07:46.603628
Duration: 818.917 ms
 Changes:   

      ID: sys-gui-rpc
Function: file.managed
    Name: /etc/qubes/policy.d/50-gui-sys-gui.policy
  Result: True
 Comment: File /etc/qubes/policy.d/50-gui-sys-gui.policy updated
 Started: 20:07:47.422933
Duration: 21.854 ms
 Changes:   
          ----------
          diff:
              New file

      ID: sys-gui-admin-local-rwx
Function: file.append
    Name: /etc/qubes/policy.d/include/admin-local-rwx
  Result: True
 Comment: Appended 2 lines
 Started: 20:07:47.445158
Duration: 21.312 ms
 Changes:   
          ----------
          diff:
              --- 
              
              +++ 
              
              @@ -8,3 +8,5 @@
              
               
               ## Add your entries here, make sure to append "target=dom0" to all allow/ask actions
               
              +sys-gui @tag:guivm-sys-gui allow target=dom0
              +sys-gui sys-gui allow target=dom0

      ID: sys-gui-admin-global-rwx
Function: file.append
    Name: /etc/qubes/policy.d/include/admin-global-rwx
  Result: True
 Comment: Appended 3 lines
 Started: 20:07:47.466618
Duration: 3.329 ms
 Changes:   
          ----------
          diff:
              --- 
              
              +++ 
              
              @@ -8,3 +8,6 @@
              
               
               ## Add your entries here, make sure to append "target=dom0" to all allow/ask actions
               
              +sys-gui @adminvm allow target=dom0
              +sys-gui @tag:guivm-sys-gui allow target=dom0
              +sys-gui sys-gui allow target=dom0

Summary for local

Succeeded: 7 (changed=6)
Failed: 2

Total states run: 9
Total run time: 344.670 s
DOM0 configuration failed, not continuing
[Anirban@dom0 ~]$

My initial GPU-attaching attempts were “just attach the Renoir iGPU device”, and I could observe that:

  • starting sys-gpu-gui, even with iGPU not attached, even manually after starting with qubes.skip_autostart, would make the system unusable (I’m being voluntarily vague here)
  • with sys-gpu-gui freshly installed, even with iGPU not attached, and with no apparent autostart flag set, it gets nevertheless autostarted. Could not tell precisely why, but it seems to have something to do with sys-usb: whenever I start this one, sys-gui-gpu will attempt to start (though qvm-prefs sys-usb does not reveal to sys-gui-gpu).

Trying now to avoid possibly unsupported configuration by attaching the whole set of functions on the same device (which includes all USB controllers, which I detached from sys-usb for a first test)…

  • sys-gpu-gui this time does not get autostarted

But then How to enable the (new) GUI VM? - #5 by marmarek talking about rollback does tell we have to unset autostart and default_guivm… so let’s set that and see… I see the system booting, but suddenly reseting, supposedly when trying to start sys-gui-gpu.

Naturally once you have set default_guivm to sys-gui-gpu, if you need to start with autostart disabled and need to start any app in a vm, you need to either default_guivm or the guivm pref for that vm back to dom0. Failing that, you can see the vm consuming CPU cycles… but no window on your screen.

Now I’m pretty sure I recently read in a post some xen and linux boot parameters to avoid the reboot and properly capture the dying words of the system, from @marmarek probably, but can’t find it, and can’t find the info in any “troubleshooting” page either – and just removing rhgb in the hope a cam will get anything useful is not much more than wishful thinking. Sure we could have the info in such a doc :slight_smile:

Edit: despite the difficulty, my cam can still capture part of a message from amdgpu claiming “Hotplug removal is not supported”, followed by a “finishing device” line.

Following this, the logical step seems to avoid attaching the iGPU to dom0 in the first place. Thus I’m extending the existing pci-stub config to include it (in addition to the dGPU): pci-stub.ids=1002:7340,1002:1636. To avoid any bad interaction, as above I detached the xHCI controllers from sys-usb and attached all functions to sys-gui-gpu.

Console on ttyUSB0 shows the following (pretty much obviously it gets interrupted when attaching to sys-gui-gpu):

[   27.602043] pciback 0000:07:00.2: xen_pciback: seizing device
[   27.602534] Already setup the GSI :36
[   27.692618] pciback 0000:07:00.6: xen_pciback: seizing device
[   27.693123] Already setup the GSI :36
[   27.695599] xhci_hcd 0000:07:00.4: remove, state 4
[   27.696021] usb usb4: USB disconnect, device number 1
[   27.697030] xhci_hcd 0000:07:00.4: USB bus 4 deregistered
[   27.697769] xhci_hcd 0000:07:00.4: remove, state 1
[   27.698193] usb usb3: USB disconnect, device number 1
[   27.698586] usb 3-3: USB disconnect, device number 2
[   27.716608] xhci_hcd 0000:07:00.4: USB bus 3 deregistered
[   27.718182] pciback 0000:07:00.4: xen_pciback: seizing device
[   27.718680] Already setup the GSI :38
[   27.764558] pciback 0000:07:00.5: xen_pciback: seizing device
[   27.764697] Already setup the

… at which point I’m left with the lightdm screen showing but no possible interaction from USB devices or laptop keyboard (whose CAPSLOCK LED still responds) or touchpad (both usually still work with eg. sys-usb down).

If now I don’t forward the xHCI devices to sys-gui-gpu (but keep them in dom0, with the console going through them), I get called out for doing precisely this:

Sep 21 02:10:08 dom0 qvm-start[2427]: Start failed: internal error: Unable to reset PCI device 0000:07:00.5: internal error: Active 0000:07:00.4 devices on bus with 0000:07:00.5, not doing bus reset, see /var/log/libvirt/libxl/libxl-driver.log for details

… but said logfile does not have anything about the event, the above journalctl trace seems to be the only line about the event.

If I just stop using the USB console, a similar error happens. If I attach the xHCI controllers to sys-gui with all other functions of the iGPU PCI device, I’m back to a black screen (but the keyboard’s LED still answers), but don’t get any log on disk.

And using pci-stub to blacklist all ids for the functions in the device… leads to a similar-looking blackscreen, which this time gets a reboot after a few seconds.

Tough, not having a single usable debug channel :slight_smile: