Problems with sys-usb, affecting usb keys/ssd in appVM on a fresh updated 4.3rc3 install

User Support Testing Release
1

Hi,

I have a fresh Qubes 4.3rc3 install, and as soon as I wanted to restore my Qubes 4.2.4 qubes, I’m facing a USB problem.

Already done :

  • looking for other previous posts about something like this → nothing found
  • Qubes Update on the fresh install → OK

I have problems : with USB SSDs, USB keys, and running thunar in sys-usb.

  • problems with USB SSDs:
    • one is not appearing in the USB devices widget, in the task bar. When opening thunar in sys-usb, there’s a device, in grey : “removable”. Clicking on it gives an error

failed to mount “removable” - mount : /mnt/removable must be superuser to use mount. dmseg(1) may have more info…

(Ok, I think this item is always present in sys-usb thunar ? I restarted sys-usb with no device plugged, and “removable” is still here)
- other one : appears in the USB menu, can affect to a AppVM. There, when running Thunar via a terminal, many warnings appear :

thunar-volman: unknown block device type “disk” (many time), and often but not always "thunar-volman: unsupported USB device type "usb-storage.

When clicking to mount and browse the SSD, an error occurs

“Error mounting /dev/sda1 at /media/user/ssdname : wrong fs type, bad option, bad superblock on /dev/sda1 missing codepage or helper program, or other error”

Notes :

  1. this is port-independant. I’ve tried many USB ports, same results.
  2. the 2 SSDs are Luks encrypted
  • With USB keys :
    • I plugged a USB Key in the same port I unplugged the USB SSDs : many on-screen notifications appear : among them, some are for the USB key, and in a same time, other are concerning the unplugged SSDs. Affecting a USB key on the same AppVM, the same thunar-volman errors arise ; but finally I can open the USB key and browse.
    • when unmounting the USB key in the AppVM Thunar, on-screen notifications reappear, including the USB key itself and the former SSD. In the task bar USB widget, the SSD appears as a Block Storage :

sub-device of disp1111:2-1 CTnameofssd NEW

(the “NEW” mention is the blue one, as when inserting a new device.)

Notes :

  1. Among tested USB keys, one is fully Luks-encrypted (an OS is installed on it), but which I can successfully open in the AppVM thunar.
  2. Other non encrypted usb keys seem to be ok
  • All things dealing with sys-usb are very long :
    • usb ssd detection : when inserting such a device, if I click on the “Q” menu, nothing happens for long time, I have to click on it several times. USB keys seem to be faster to be recognized.
    • opening thunar in sys-usb : needs something like 1minute to open, and sometimes I need to click twice on it before it shows up (then, the two instances in a same time).
    • in contrary, a USB mouse is recognized very fast

What’s happening, how could I solve theses problems ?

I can send any needed info, logs,… As I’m a neebie, I don’t really know which to send for the moment,
Many thanks for any help,

2

I have nothing of such.
And I don’t use Thunar in sys-usb but specially for testing run it once.

dom0:

qvm-pci ls sys-usb
qvm-usb ls sys-usb
qvm-block ls sys-usb
3

Hi again,

I just checked it, the “removable” item appears in sys-usb thunar at Qubes start up (I didn’t plug anything in usb ports, except the mouse).

Here’s the outputs:

1. with nothing plugged except the mouse :

user@dom0:~/Desk$ qvm-pci ls sys-usb
BACKEND:DEVID         DESCRIPTION                                                         USED BY
dom0:00_08.1-00_00.3  PCI_USB: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1  sys-usb (attached: no-strict-reset=True)
dom0:00_08.1-00_00.4  PCI_USB: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1  sys-usb (attached: no-strict-reset=True)

user@dom0:~/Desk$ qvm-usb ls sys-usb
BACKEND:DEVID  DESCRIPTION                           USED BY
sys-usb:2-3    Camera:  Quanta 0001                  
sys-usb:2-4    Bluetooth: MediaTek Inc. 000000000    
sys-usb:4-1    Mouse: Logitech, Inc. TrackMan Wheel 
BACKEND:DEVID  DESCRIPTION                           USED BY
sys-usb:2-3    Camera:  Quanta 0001                  
sys-usb:2-4    Bluetooth: MediaTek Inc. 000000000    
sys-usb:4-1    Mouse: Logitech, Inc. TrackMan Wheel

user@dom0:~/Desk$ qvm-block ls sys-usb
BACKEND:DEVID  DESCRIPTION  USED BY

(nothing)

2. After pluging as USB Key (without attaching it to any qube)
(The USB key I used for installing Qubes)

qvm-pci ls sys-usb gave the same result as above


user@dom0:~/Desk$ qvm-usb ls sys-usb
BACKEND:DEVID  DESCRIPTION                           USED BY
sys-usb:2-2    USB_Storage: USB 9284621397903297968  
sys-usb:2-3    Camera:  Quanta 0001                  
sys-usb:2-4    Bluetooth: MediaTek Inc. 000000000    
sys-usb:4-1    Mouse: Logitech, Inc. TrackMan Wheel

user@dom0:~/Desk$ qvm-block ls sys-usb
BACKEND:DEVID  DESCRIPTION                                                       USED BY
sys-usb:sda    Block_Storage: sub-device of sys-usb:2-2 QUBES-R4-3-0-RC3-X86-64  
sys-usb:sda1   Block_Storage: sub-device of sys-usb:sda QUBES-R4-3-0-RC3-X86-64  
sys-usb:sda2   Block_Storage: sub-device of sys-usb:sda ANACONDA                 
sys-usb:sda3   Block_Storage: sub-device of sys-usb:sda ProductCode              
user@dom0:~/Bureau$

3. After attaching the key to vault

user@dom0:~/Desk$ qvm-block ls sys-usb
BACKEND:DEVID  DESCRIPTION                                                       USED BY
sys-usb:sda    Block_Storage: sub-device of sys-usb:2-2 QUBES-R4-3-0-RC3-X86-64  vault (attached: read-only=no, frontend-dev=xvdi)

4. With the Luks SSD (not attached)
(same port)

qvm-pci ls sys-usb gave the same result as above

user@dom0:~/Desk$ qvm-usb ls sys-usb
BACKEND:DEVID  DESCRIPTION                           USED BY
sys-usb:2-2    USB_Storage: Micron 2521E8D8FF7E      
sys-usb:2-3    Camera:  Quanta 0001                  
sys-usb:2-4    Bluetooth: MediaTek Inc. 000000000    
sys-usb:4-1    Mouse: Logitech, Inc. TrackMan Wheel

user@dom0:~/Desk$ qvm-block ls sys-usb
BACKEND:DEVID  DESCRIPTION                                            USED BY
sys-usb:sda    Block_Storage: sub-device of sys-usb:2-2 CT2000X9SSD9  
sys-usb:sda1   Block_Storage: sub-device of sys-usb:sda CT2000X9SSD9

5. When trying to attach the SSD to vault
This time I can’t : it is even not appearing in the device list (and no on-screen notification)

6. Attaching the SSD on another port (USB-C)
now appearing on

user@dom0:~/Desk$ qvm-usb ls sys-usb
...
sys-usb:3-1    USB_Storage: Micron 2521E8D8FF7E
...

(qvm-block and qvm-pci results are the same as on the previous port)

7. trying to attach the SSD to vault
(as now it appears as USB storage in the widget)

qvm-pci : unchanged
qvm-usb now sees it is attached :

sys-usb:3-1    USB_Storage: Micron 2521E8D8FF7E      vault (attached)

qvm-block : nothing anymore

user@dom0:~/Desk$ qvm-block ls sys-usb
BACKEND:DEVID  DESCRIPTION  USED BY

And this time I can open it in vault… I don’t understand nothing, yesterday I couldn’t…
SSDs seem not to work on standard USB ports (non type-C)

8. Other SSD on the same “working” USB-C port
So, I tried the other USB SSD : it is not working, even on the USB-C where the first SSD was working. (pluging again the first ssd here works again)

Extra notes

  • I had to click several times on “File manager” in the Service>Sys-usb tab before one instance show up
  • When plugging the usb key, a long time passes before notifcations show up ; during this time the USB widget doesn’t react when clicking on it.

I guess all these is not normal neither…?

  • sorry if I sent too much useless tests… I prefered trying as much things as possible to get sure there is at least usefull things
4

in dom0:

xl info |grep total_memory
xl list sys-usb
qvm-run -p sys-usb 'free -h'
5

Here’s the outputs :

user@dom0:~/Desk$ xl info |grep total_memory
total_memory           : 15712

user@dom0:~/Desk$ xl list sys-usb
Name                                        ID   Mem VCPUs	State	Time(s)
sys-usb                                      3   284     2     -b----     103.9

user@dom0:~/Desk$ qvm-run -p sys-usb 'free -h'
               total        used        free      shared  buff/cache   available
Mem:           212Mi       140Mi        16Mi        92Ki        63Mi        71Mi
Swap:          1.0Gi        95Mi       928Mi
6

you know, you can’t expect to run anything with gui at 284MiB of ram

7

Yes, sys-usb and sys-net are in minimal state in Q4.3rc3, as I saw afterwards stated here Qubes OS 4.3 release notes — Qubes OS Documentation

Minimal state qubes to make NetVM and USBVM to consume as little RAM as possible.

Previous Qubes 4.2 installs seem to have the same amount of memory for sys-usb.
I didn’t note my 4.2 settings, but here I found some indications (dating 2022) :
https://forum.qubes-os.org/t/questions-on-memory-allocations-for-sys-qubes/13045/2

Sys-usb was already 300 Mo, and not included in memory balacing. With these settings, I didn’t have problems with Q4.2.

Should I increase the minimal memory in Qubes 4.3 to have sys-usb attach properly any usb devices to AppVMs (and opening Thunar, as it is AFIK the single way of browsing an android phone, with mtp, in sys-usb, and I like being able to browse pictures to transfer only the ones I want) ?

Thanks for your help !

8

I tried increasing memory, and all the problems remain with even 400Mo. EDIT : and even 800Mo.

Then, I think memory is not the single problem.

  • usb ssds are still not always appearing in the usb widget
  • the “remote” item in sys-usb is still here.
  • thunar is always very long to start (and even opening a terminal)

Thunar run from a terminal says (with no usb ssd/key connected)
1. at start up:

user@sys-usb:~$ thunar
Error creating proxy: Error calling StartServiceByName for org.gtk.vfs.UDisks2VolumeMonitor: Timeout was reached (g-io-error-quark, 24)

(thunar:1487): thunar-WARNING **: 11:39:34.601: ThunarThumbnailer: failed to create proxy: Error calling StartServiceByName for org.freedesktop.thumbnails.Thumbnailer1: Timeout was reached

2. Then
something dealing with remote i wonder:

user@sys-usb:~$ thunar

(thunar:1592): GVFS-RemoteVolumeMonitor-WARNING **: 11:28:10.937: remote volume monitor with dbus name org.gtk.vfs.UDisks2VolumeMonitor is not supported

3. And when I connect a not-recognized usb ssd:

(thunar-volman:1787): GVFS-RemoteVolumeMonitor-WARNING **: 11:32:57.951: remote volume monitor with dbus name org.gtk.vfs.UDisks2VolumeMonitor is not supported
thunar-volman: Unsupported USB device type "uas".

(thunar-volman:1886): GVFS-RemoteVolumeMonitor-WARNING **: 11:33:23.003: remote volume monitor with dbus name org.gtk.vfs.UDisks2VolumeMonitor is not supported

(thunar-volman:1893): GVFS-RemoteVolumeMonitor-WARNING **: 11:33:48.053: remote volume monitor with dbus name org.gtk.vfs.UDisks2VolumeMonitor is not supported
thunar-volman: Unknown block device type "disk".

Feel free to tell me what you need to identify the problem,
Thanks for your help

9

Hi,

I ask my question again, as I’ve still no solution. I can’t use well Qubes without accessing USB devices (usb keys and ssds).

USB devices are still randomly recognized. Some USB keys are recognized well, some other never, or depending on the USB port (USB A → never, USB-C → sometimes). The USB widget sometimes shows only one partition (sda3), nothing else. USB key (and partitions) even have been reported in the USB widget as “block_storage: hosted by disp9119” when attached to that disp9119 (I never saw something like this in 2-3 years of using Qubes). Pluging a USB key often makes my computer not responding (Q menu, qubes widgets… not showing when clicking, until the USB-key gets recognized)

I had no problem like theses with previous 4.2.4 version. Actual 4.3rc3 install showed usb problems as soon as the first boot after install (from ISO).

I’m nota very skilled user. I had a look to creating a minimal sys-usb, but it seems it is very unrecommended to noobs like me. I have no idea to what to do, any help will be much appreciated !

How could I do to get sys-usb working fine ?
All the best,

10

Cant help but feeling the pain.

Had maybe simillar issues with my wifi usb mouse dongle… usb vm will crash and refuse to start if the dongle was inserted at power on time. Only way to get it working eventually was to unolug dongle, wait for usb to atart fine,then plug the dongle and accept the warning from dom0.
Lately,i had to “accept” usb mouse connection,automatically.

In your case,try power on with no usb peripherals attached. Then, attach one by one and see which one works.

Also, create few more usb-net qubes/clones and add devices,one by one from the settings tab,hardware pass_through…again, process of elimination.

11

I do not know the answer, but is this the exact same hardware where you were using 4.2.4 before? (The HP victus?

There is nothing on HCL for it, I think.

I did not use 4.3 for quite a long time, but I would maybe try changing sys-usb to a full-size template, with enough memory, for a test. If that is possible…?

Something is different in 4.3, and that sounds like a quite big thing… (but also new xen, new dom0, new everything else!!!)

Edit: changed category to “testing release”, because it almost sounds like a regression.

12

Hi,
Many thanks for your interest and indications.

Yes I changed nothing in the hardware between the fine working 4.2.4 version, and the actual 4.3rc3.

I tried many things. Here’s some overall observations :

  • Increasing the sys-usb’s initial memory to a « normal » qube’s 4000 Mo changed nothing. Delays for recognizing devices, when they are recognized, freezing the Q menu and QubeDomain widget…
  • I always run sys-usb with no USB storage plugged. Restarting sys-usb with usb storage plugged changes nothing. Only restarting the computer with storage already plugged allowed me to access one SSD, whereas I couldn’t if plugging when computer is running.
  • It seems that as soo as a problem occurs, it may cause other problems later. I can usually acces the 2To SSD on USB-C ; but if I had a trouble before with another USB storage, it often doesn’t work afterwards ; needing me to restart completely the computer
  • some problems are usb port-independant, some depends on the port (USB-C working sometime better than the two other USB-A)
  • there’s always a « removable » item listed in sys-usb thunar : what is it ?
  • Something probably interesting : one problem is debian version-dependant. I have a 1To SSd which is working fine with disp VM from debian12 templates (from restored qubes 4.3.4) ; but if I try to attach it to a dispVM from the debian13 template, they are very weird problems (see below).
  • A very few times I have in dispVM’s thunar a picto for « ejecting » the storage, but generally I don’t. I don’t know if it is normal or not (It was already the case in 4.2.4 version).
  • restarting sys-usb doesn’t change nothing
  • USB storages are always listed in sys-usb « lsusb » command, whatever the problems.
  • I made tests with various usb keys, ssds, and a hdd. I wrote the results I summarize here, but problems occur with other usb storage too…
  • not all usb storage have the same problems… depends on the format ? the luks encryption ? the partionning ?…. I’m lost.

Some precisions :

  • the 1To SSD is well recognized and appears in the widget. Problem then is template dependant :
    • on debian 12 based dispVM no problem
    • on debian 13 based dispVM : it is unmountable. Always « error mounting /dev/xvdi1 at /media/user/usbkey : wrong fs type, bad option, bas superblock on /dev/xvdi, missing codepage or helper program or other error ».
  3. This error can be solved by gnome-disks filesystem repair (even if it tells there is no problem, i have to repair to get it working). BUT, if I unmount then remount it, the same problem appears again, needing me to do again the filesystem repair.

very weird things appear in the widget, when attaching or detaching storages when there are problems.

  • a usb key is working fine in my dispVM (say disp123). I unmount it in thunar (right click, unmount, as I never have the « eject » picto). Then in the widget : sda and sda1 appears as NEW « subdevice of disp123 ». There’s no « detach from disp123 ». I have to « power off » in gnome-disks to get the normal situation with a device ready to be attached in the widget (just ejecting is not enough).
  • a partitionned usb HDD is partialy recognized. Only sda2, sda3, sda4 appear in the widget, nothing else. Unable to easily attach the drive to a qube. Moreover, all these partitions appear as « subdevice of sys-usb ».
  • the 2to SSD often works, but sometimes not : I noticed it probably works if plugged before any other usb storage ; and not working if a usb storage bugged before (the previous usb key, or hdd for example). When there are troubles, maybe there are more often on the usb-A port.
  • once, the 2to SSD wasn’t recognized on none of the ports. I plugged in the samed time the USB key and the SSD : both are recognized (SSD was working, but the usb key was a subdevice of sys-usb).
  • once, when unmounting the usb key from the a debian12 based disp345, it appeared as « NEW block_storage hosted by disp345 » (and not « subdevice of disp345 » as when unmounted from a debian13 based one).

Here I am with my tests, hope it is not too confused, and it can be helpful to understand what’s happening here.

Let me know what your are thinking of all of this, and what I can try to solve all of this,
Many thanks !

13

Hi,

Updated observations here :

  • some disks which are not recognized when qubes is running are sometimes recognized when plugged before start up
  • when attaching another USB HDD in an dispVM, it is not mountable (except if "repairing it in gnomes-disks, after every mounting). sda1 → sda4 partitions appear as “sub-device” of dispVM. Sometime even not visible in gnome-disks…
  • detaching it then allows it to be normally recognized : in the widget, there’s finally just one storage recognized, and not every sda1->sda4. I can attach it and mount in an AppVM, but even then, the others sdaX partition appear in the widget as “hosted by dispXXX”.
  • May the storage be recognized or not, appearing or not in the widget… they are always listed in sys-usb “lsusb”, and when attached in a qube, they are listed too in this qube (even if not visible in thunar nor in gnome-disks).

Please tell me your feeling, what should I try to get it works, as it is very difficult to work without usb keys or ssds…
thanks,

alle the best

14

Hi there,
You have provided a lot of information of different tests. It is good.

I notice that you have seen storage that needs “repair” - it means it did not get disconnected cleanly.

For me, this could mean either a bad electrical connection , or a bad power supply, or you unplug the disks before you “eject” or umount them.
Because you also say thing go slow, it could also indicate that the devices appear/disappear many times, which can happen with bad power or bad connection.

Maybe it is useful to open a terminal in both sys-usb and target qube. In each one, run
sudo journalctl -f
This will show (maybe) some log messages when the USB devices appear. Maybe it will give a clue…

(It is important to know- I often work with old hardware, which makes me often see broken or defective hardware. Often, software gets the blame when there are hardware problems! Here, I am suspecting hardware problems.
BUT, sometimes software problems can look like hardware problems. )