Not sure how to correctly attach USB external hard drives to disposable VM

leni1 · May 30, 2024, 2:50pm

As the title says, I am unable to figure out how to correctly attach two USB external hard drives to my laptop that uses Qubes OS 4.2 (with the latest updates installed)

I created a disposable VM and then did the following:

[user@dom0 ~] qvm-usb attach disp4158 sys-usb:2-2
[user@dom0 ~] qvm-usb attach disp4158 sys-usb:3-1

after which they appear in the qvm-block output like so:

[user@dom0 ~] qvm-block
BACKEND:DEVID      DESCRIPTION         USED BY
disp4158:sda           <device_name>       disp4158
disp4158:sda1         <device_name>       
disp4158:sdb           <device_name>       
disp4158:sdb1         <device_name>       disp4158

When I try to detach and re-attach, I see the message:

Device attach failed: 2024-05-30 qrexec-client[8669]: qrexec-client.c:497:wait_for_vchan_client_with_timeout: vchan connection timeout

The documentation I read is here and here

The end result I’d like to have is the drives being attached as /dev/sdb and /dev/sda, which as of now I can’t do reliably.

barto · May 30, 2024, 3:11pm

Hmm, weird. It works for me in R4.1.2, I guess you have 4.2

I don’t get anything under “USED BY” in the qvm-block command output, even if I mount the attached usb device (sda1).



[user@dom0 ~]$ qvm-usb
BACKEND:DEVID  DESCRIPTION                                USED BY
sys-usb:2-1    SanDisk_Cruzer_Blade_NNNNNN 

[user@dom0 ~]$ qvm-usb attach disp6477 sys-usb:2-1
[user@dom0 ~]$ qvm-usb
BACKEND:DEVID  DESCRIPTION                                USED BY
sys-usb:2-1    SanDisk_Cruzer_Blade_NNNNNN  disp6477

[user@dom0 ~]$ qvm-block
BACKEND:DEVID  DESCRIPTION               USED BY
disp6477:sda   Cruzer_Blade ()
disp6477:sda1  Cruzer_Blade (R421)

[user@dom0 ~]$ qvm-usb detach disp6477 sys-usb:2-1
[user@dom0 ~]$ qvm-usb attach disp6477 sys-usb:2-1

[user@dom0 ~]$ qvm-block
BACKEND:DEVID  DESCRIPTION               USED BY
disp6477:sda   Cruzer_Blade ()
disp6477:sda1  Cruzer_Blade (R421)

leni1 · May 30, 2024, 3:14pm

Yes I am using 4.2. Should update the post to reflect that

Yeah the output doesn’t seem consistent; I am not able to copy and paste from the dom0 terminal so have to write out the output by hand.

Let me try again with a new disposable VM and see if that works.

apparatus · May 30, 2024, 4:34pm

It works for me on Qubes OS 4.2 correctly.

leni1 · May 30, 2024, 8:00pm

Forgot that guide because I didn’t think I would ever need it

Here’s fresh output from when I tried this afresh using dom0 and a new disposable VM as the target to attach the drives to:

[user@dom0 ~]$ qvm-usb attach disp1254 sys-usb:2-2
[user@dom0 ~]$ qvm-usb attach disp1254 sys-usb:3-1                           
[user@dom0 ~$] qvm-usb 
BACKEND:DEVID  DESCRIPTION                                                             USED BY
sys-usb:2-2    Generic_USB_Mass_Storage_Device_116AC2101219                            disp1254
sys-usb:3-1    Western_Digital_My_Passport_2626_575837324441334331354834               disp1254

Output of qvm-block

[user@dom0 ~]$ qvm-block
BACKEND:DEVID   DESCRIPTION                             USED BY
disp1254:sda    WDC_WD50NDZW-11BCSS0 ()                 
disp1254:sda1   WDC_WD50NDZW-11BCSS0 ()                 
disp1254:sdb    TOSHIBA_MK2565GSX ()                    
[user@dom0 ~]$ qvm-block attach disp1254 disp1254:sda
[user@dom0 ~]$ qvm-block attach disp1254 disp1254:sdb
Got empty response from qubesd. See journalctl in dom0 for details.
[user@dom0 ~]$ qvm-block 
BACKEND:DEVID   DESCRIPTION                             USED BY
disp1254:sda    WDC_WD50NDZW-11BCSS0 ()                 
disp1254:sdb    TOSHIBA_MK2565GSX ()                    
disp1254:sdb1   TOSHIBA_MK2565GSX ()                    
[user@dom0 ~]$ qvm-block detach disp1254 disp1254:sda
Got empty response from qubesd. See journalctl in dom0 for details.
[user@dom0 ~]$ qvm-usb detach disp1254 sys-usb:2-2
[user@dom0 ~]$ qvm-usb detach disp1254 sys-usb:3-1

Output of journalctl:

May 30 22:32:37 dom0 libvirtd[2847]: internal error: libxenlight failed to attach disk 'xvdj'
May 30 22:32:37 dom0 qubesd[2819]: unhandled exception while calling src=b'dom0' meth=b'admin.vm.device.block.Attach' dest=b'disp1254' arg=b'disp1254+sdb' len(untrusted_payload)=0
May 30 22:32:37 dom0 qubesd[2819]: Traceback (most recent call last):
May 30 22:32:37 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/api/__init__.py", line 297, in respond
May 30 22:32:37 dom0 qubesd[2819]:     response = await self.mgmt.execute(
May 30 22:32:37 dom0 qubesd[2819]:                ^^^^^^^^^^^^^^^^^^^^^^^^
May 30 22:32:37 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/api/admin.py", line 1301, in vm_device_attach
May 30 22:32:37 dom0 qubesd[2819]:     await self.dest.devices[devclass].attach(assignment)
May 30 22:32:37 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/devices.py", line 251, in attach
May 30 22:32:37 dom0 qubesd[2819]:     await self._vm.fire_event_async('device-pre-attach:' + self._bus,
May 30 22:32:37 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/events.py", line 227, in fire_event_async
May 30 22:32:37 dom0 qubesd[2819]:     sync_effects, async_effects = self._fire_event(event,
May 30 22:32:37 dom0 qubesd[2819]:                                   ^^^^^^^^^^^^^^^^^^^^^^^
May 30 22:32:37 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/events.py", line 164, in _fire_event
May 30 22:32:37 dom0 qubesd[2819]:     effect = func(self, event, **kwargs)
May 30 22:32:37 dom0 qubesd[2819]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 30 22:32:37 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/ext/block.py", line 282, in on_device_pre_attached_block
May 30 22:32:37 dom0 qubesd[2819]:     vm.libvirt_domain.attachDevice(
May 30 22:32:37 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/app.py", line 103, in wrapper
May 30 22:32:37 dom0 qubesd[2819]:     return attr(*args, **kwargs)
May 30 22:32:37 dom0 qubesd[2819]:            ^^^^^^^^^^^^^^^^^^^^^
May 30 22:32:37 dom0 qubesd[2819]:   File "/usr/lib64/python3.11/site-packages/libvirt.py", line 691, in attachDevice
May 30 22:32:37 dom0 qubesd[2819]:     raise libvirtError('virDomainAttachDevice() failed')
May 30 22:32:37 dom0 qubesd[2819]: libvirt.libvirtError: internal error: libxenlight failed to attach disk 'xvdj'
May 30 22:36:31 dom0 libvirtd[2847]: internal error: libxenlight failed to attach disk 'xvdj'
May 30 22:36:31 dom0 qubesd[2819]: unhandled exception while calling src=b'dom0' meth=b'admin.vm.device.block.Attach' dest=b'disp1254' arg=b'disp1254+sdb' len(untrusted_payload)=0
May 30 22:36:31 dom0 qubesd[2819]: Traceback (most recent call last):
May 30 22:36:31 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/api/__init__.py", line 297, in respond
May 30 22:36:31 dom0 qubesd[2819]:     response = await self.mgmt.execute(
May 30 22:36:31 dom0 qubesd[2819]:                ^^^^^^^^^^^^^^^^^^^^^^^^
May 30 22:36:31 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/api/admin.py", line 1301, in vm_device_attach
May 30 22:36:31 dom0 qubesd[2819]:     await self.dest.devices[devclass].attach(assignment)
May 30 22:36:31 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/devices.py", line 251, in attach
May 30 22:36:31 dom0 qubesd[2819]:     await self._vm.fire_event_async('device-pre-attach:' + self._bus,
May 30 22:36:31 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/events.py", line 227, in fire_event_async
May 30 22:36:31 dom0 qubesd[2819]:     sync_effects, async_effects = self._fire_event(event,
May 30 22:36:31 dom0 qubesd[2819]:                                   ^^^^^^^^^^^^^^^^^^^^^^^
May 30 22:36:31 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/events.py", line 164, in _fire_event
May 30 22:36:31 dom0 qubesd[2819]:     effect = func(self, event, **kwargs)
May 30 22:36:31 dom0 qubesd[2819]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 30 22:36:31 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/ext/block.py", line 282, in on_device_pre_attached_block
May 30 22:36:31 dom0 qubesd[2819]:     vm.libvirt_domain.attachDevice(
May 30 22:36:31 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/app.py", line 103, in wrapper
May 30 22:36:31 dom0 qubesd[2819]:     return attr(*args, **kwargs)
May 30 22:36:31 dom0 qubesd[2819]:            ^^^^^^^^^^^^^^^^^^^^^
May 30 22:36:31 dom0 qubesd[2819]:   File "/usr/lib64/python3.11/site-packages/libvirt.py", line 691, in attachDevice
May 30 22:36:31 dom0 qubesd[2819]:     raise libvirtError('virDomainAttachDevice() failed')
May 30 22:36:31 dom0 qubesd[2819]: libvirt.libvirtError: internal error: libxenlight failed to attach disk 'xvdj'



May 30 22:37:16 dom0 libvirtd[2847]: internal error: libxenlight failed to detach disk 'xvdi'
May 30 22:37:16 dom0 qubesd[2819]: unhandled exception while calling src=b'dom0' meth=b'admin.vm.device.block.Detach' dest=b'disp1254' arg=b'disp1254+sda' len(untrusted_payload)=0
May 30 22:37:16 dom0 qubesd[2819]: Traceback (most recent call last):
May 30 22:37:16 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/api/__init__.py", line 297, in respond
May 30 22:37:16 dom0 qubesd[2819]:     response = await self.mgmt.execute(
May 30 22:37:16 dom0 qubesd[2819]:                ^^^^^^^^^^^^^^^^^^^^^^^^
May 30 22:37:16 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/api/admin.py", line 1326, in vm_device_detach
May 30 22:37:16 dom0 qubesd[2819]:     await self.dest.devices[devclass].detach(assignment)
May 30 22:37:16 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/devices.py", line 312, in detach
May 30 22:37:16 dom0 qubesd[2819]:     await self._vm.fire_event_async('device-pre-detach:' + self._bus,
May 30 22:37:16 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/events.py", line 227, in fire_event_async
May 30 22:37:16 dom0 qubesd[2819]:     sync_effects, async_effects = self._fire_event(event,
May 30 22:37:16 dom0 qubesd[2819]:                                   ^^^^^^^^^^^^^^^^^^^^^^^
May 30 22:37:16 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/events.py", line 164, in _fire_event
May 30 22:37:16 dom0 qubesd[2819]:     effect = func(self, event, **kwargs)
May 30 22:37:16 dom0 qubesd[2819]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 30 22:37:16 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/ext/block.py", line 296, in on_device_pre_detached_block
May 30 22:37:16 dom0 qubesd[2819]:     vm.libvirt_domain.detachDevice(
May 30 22:37:16 dom0 qubesd[2819]:   File "/usr/lib/python3.11/site-packages/qubes/app.py", line 103, in wrapper
May 30 22:37:16 dom0 qubesd[2819]:     return attr(*args, **kwargs)
May 30 22:37:16 dom0 qubesd[2819]:            ^^^^^^^^^^^^^^^^^^^^^
May 30 22:37:16 dom0 qubesd[2819]:   File "/usr/lib64/python3.11/site-packages/libvirt.py", line 1497, in detachDevice
May 30 22:37:16 dom0 qubesd[2819]:     raise libvirtError('virDomainDetachDevice() failed')
May 30 22:37:16 dom0 qubesd[2819]: libvirt.libvirtError: internal error: libxenlight failed to detach disk 'xvdi'

It’s late where I am so will step away and come back to read the responses tomorrow.

tempmail · May 30, 2024, 8:24pm

You can do that by using “Copy dom0 clipboard” widget.

What happens when you try to attach them via widget?

Looking your qvm-block output I see inconsistencies (sda, sda1, sdb, then sda, sdb, sdb1: then it looks you are trying to attach already attached disks - qvm-block shows disks are attached to disp 1254, and after that you are applying attaching command again, after which qvm-block shows first stated inconsistency).

leni1 · May 31, 2024, 10:47am

Thanks. Was able to do this via the documentation link provided

Can’t use the widget since I only have two USB ports which are being used by both the external drives and my mouse uses a USB wireless receiver. If there is a command to run it via the Application Finder available via Alt+F2, I can try that as well.

Yes. Thus the opening of this thread

My question is: should I attach using only qvm-usb attach to the disposable VM directly and any device operations that I perform will be able to find the devices under /dev/sda and /dev/sdb or do I also need to run qvm-block attach?

It is not clear what I should do to get this to work.

apparatus · May 31, 2024, 11:12am

You can attach the device as USB device or as block device
In case of attaching the device as USB device, the device will be initialized as USB device in the disposable qube using USB protocol.
In case of attaching the device as block device, the device will be initialized as USB device in the sys-usb qube and then the whole disk or just a single partition of the disk will be attached to the disposable qube as a virtual disk that won’t use USB protocol inside disposable qube.
Don’t attach the same disk as both USB device and block device at the same time.
Also note:

For example, attaching a full USB-device offers more attack surface than attaching a single block device, while attaching a full block device (e.g. sda) again offers more attack surface than attaching a single partition (e.g. sda1), since the targetVM doesn’t have to parse the partition-table. (Attaching a full block device offers the advantage that most file-managers will mount and display them correctly, whereas they don’t expect single partitions to be added and therefore don’t handle them correctly.)

leni1 · May 31, 2024, 11:22am

Thanks. That clears it up for me.

Since I need the USB protocol for interacting with the disks, I guess that I should attach the disks using the qvm-usb approach. I assume that if I wanted to load an operating system that was installed on an external drive, then I would use qvm-block. Is this a correct understanding of when to use qvm-usb vs when to use qvm-block?

These are disks I trust so all good here

apparatus · May 31, 2024, 11:37am

No, you can access the data on your USB disk both when attaching it as USB device with qvm-usb and when attaching it as block device with qvm-block.
When attaching USB device as block device the USB protocol still will be used but not in the disposable qube to which you’re attaching the device but in sys-usb qube.
About using qvm-usb vs qvm-block when attaching USB storage device - except for security reasons stated in the quote in my previous post, I can’t give you another use case when using qvm-usb or qvm-block will be preferable.

leni1 · May 31, 2024, 11:43am

So to clarify if I want to get the disks under /dev/sdX format e.g. /dev/sda as an example, then which should I use: qvm-block or qvm-usb?

Sorry that I am taking so long to understand. I just want to be sure of what I am doing going forward.

apparatus · May 31, 2024, 11:44am

qvm-usb

rustybird · May 31, 2024, 1:06pm

Are you sure that’s necessary for your use case? Unless you’re planning on using lower-level tools like S.M.A.R.T. clients directly inside the destination VM, it’s generally preferable to attach storage as a simple virtual block device (showing up as /dev/xvdi, /dev/xvdj, …) rather than as a relatively complex virtual USB device wrapping a block device. In other words, prefer qvm-block to qvm-usb if you can. It’s more robust and more secure, even if you still attach the full disk and not only one partition.

More like, qvm-usb is for when you need arbitrary USB device functionality beyond what can be provided by qvm-block (which specializes in storage and doesn’t really care if that storage is physically attached over USB or otherwise).

For most things, /dev/xvdY can be used exactly like /dev/sdX and will result in the same data being written to (or read from) the storage device.

As far as the security aspect goes, it’s not just about trusting the disk. Whenever you connect two VMs - e.g. sys-usb and a DisposableVM - they can attempt to attack each other, so it’s better if the protocol that’s connecting them is as simple as possible.

leni1 · May 31, 2024, 1:57pm

Yes it is. I need to resume a data recovery process and at the time I started it, the disks were /dev/sdX. I would like to minimize the interaction with the damaged drive so starting from scratch again just to do it with the best practices isn’t worth it but I appreciate the feedback.

rustybird · May 31, 2024, 2:21pm

If it’s something high-level like ddrescue that doesn’t use hardware specific functionality, the recovery process that was started at /dev/sdX can be resumed from /dev/xvdY.

Although if the storage hardware is failing, I’d run the data recovery tool directly in sys-usb. Forwarding to another VM (whether as a virtual block device or as a virtual USB device) could introduce another source of problems if there are transient errors - considering caching, device resets, etc.

leni1 · May 31, 2024, 4:21pm

Yes it is ddrescue. Since I am using the mapfile to resume this process, I feel it’s a bit risky to switch to the /dev/xvdY format instead.

have already started with a disposable VM since that’s what the wikis recommended so will continue with that

But will see if I can copy the log file over to sys-usb and see the drives that way directly. If so will switch to sys-usb instead.

rustybird · May 31, 2024, 4:35pm

No need to switch if it’s working for you, but it should be totally fine because the virtual block device preserves the original block size that’s (apparently?) used for addressing in the mapfile.

In the context of rescuing an optical disc, it’s pretty common to move the disc into a different drive at some point, which seems kind of similar.