Slow transfer (Read/Write) speeds to USB flash drives

Hello,

I couldn’t really find any relevant or similar issues in these forums, on the QubesOS Reddit or the GitHub issues register so thought to ask the question here before creating a GitHub issue.

Machine is a Lenovo Thinkpad P1 (Gen 4) running QubesOS 4.1.0-rc2.

“sys-usb” is running in HVM mode, has 1GB - 2GB of memory with the “memory balancing” option enabled and has the USB controllers assigned to it.

dom0:00_0d.0   USB controller: Intel Corporation Tiger Lake-H Thunderbolt 4 USB Controller                        sys-usb (no-strict-reset=True)
dom0:00_0d.2   USB controller: Intel Corporation Tiger Lake-H Thunderbolt 4 NHI #0                                sys-usb (no-strict-reset=True)
dom0:00_0d.3   USB controller: Intel Corporation Tiger Lake-H Thunderbolt 4 NHI #1                                sys-usb (no-strict-reset=True)
dom0:00_14.0   USB controller: Intel Corporation Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller                sys-usb (no-strict-reset=True)

I’ve been experiencing really slow USB read/write speeds. It’s been happening since day #1 and I first noticed it just copying files to and from USB drives (directly accessed via sys-usb). I’ve been using DD as a benchmark to test read/write and here’s what I’m seeing:

user@sys-usb ~]$ sudo dd if=/dev/sda of=/dev/null bs=1M count=1024 conv=fsync status=progress
1071644672 bytes (1.1 GB, 1022 MiB) copied, 96 s, 11.2 MB/s
dd: fsync failed for '/dev/null': Invalid argument
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 96.2545 s, 11.2 MB/s

[user@sys-usb ~]$ sudo dd if=/dev/zero of=/dev/sda bs=1M count=1024 conv=fsync status=progress
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 104 s, 10.3 MB/s 
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 114.937 s, 9.3 MB/s

If I change sys-usb to be a PV, the results improve a bit:

[user@sys-usb ~]$ sudo dd if=/dev/sda of=/dev/null bs=1M count=1024 conv=fsync status=progress
1067450368 bytes (1.1 GB, 1018 MiB) copied, 13 s, 82.1 MB/s
dd: fsync failed for '/dev/null': Invalid argument
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.156 s, 81.6 MB/s

[user@sys-usb ~]$ sudo dd if=/dev/zero of=/dev/sda bs=1M count=1024 conv=fsync status=progress
1065353216 bytes (1.1 GB, 1016 MiB) copied, 21 s, 50.7 MB/s
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 46.0618 s, 23.3 MB/s

Booting into a Linux LiveUSB environment (SystemRescueCD), gives the results I would normally expect to see and suggests that it’s unlikely to be a hardware issue:

dd if=/dev/sdb of=/dev/null bs=1M count=1024 conv=fsync status=progress
..
<ommitted> 105 MB/s

dd if=/dev/zero of=/dev/sdb bs=1M count=1024 conv=fsync status=progress
...
<ommitted> 29.7 MB/s

Is there a tweak or a hack that I’m missing here?

Thanks for your help.

Did you try increase sys-usb memory from 300 default to 1000? For testing purpose I mean.
found in post, but no memory balancing available with device assiging :slight_smile:

I’ve tried changing it to 2GB (initial and max) and turned memory balancing off.

No difference; a dd read operation stayed at (11.1 MB/s) and memory usage didn’t go past 400MB.

[user@sys-usb ~]$ free -m
               total        used        free      shared  buff/cache   available
Mem:            1899         377         417           2        1104        1375
Swap:           1023           0        1023

Nothing too concerning that I can see in the dmesg output, but here’s what it’s recognized the drive as.

[user@sys-usb ~]$ sudo dmesg | grep -i 'scsi\|sda'
[    0.435302] SCSI subsystem initialized
[    0.652269] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
[    0.707089] scsi host0: ata_piix
[    0.707421] scsi host1: ata_piix
[    5.000756] scsi host2: usb-storage 2-1:1.0
[    6.298105] scsi 2:0:0:0: Direct-Access     Samsung  Flash Drive      1100 PQ: 0 ANSI: 6
[    6.306325] sd 2:0:0:0: Attached scsi generic sg0 type 0
[    6.312386] sd 2:0:0:0: [sda] 125304832 512-byte logical blocks: (64.2 GB/59.8 GiB)
[    6.313936] sd 2:0:0:0: [sda] Write Protect is off
[    6.313962] sd 2:0:0:0: [sda] Mode Sense: 43 00 00 00
[    6.318865] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    6.355780]  sda: sda1
[    6.408947] sd 2:0:0:0: [sda] Attached SCSI removable disk

Also changed the underlying TemplateVM for sys-usb from “fedora-34-dvm” to “debian-11-dvm”, but no difference… 11.1 MB/s.

What kernel version are you running in the VM?

uname -a

For my sys-usb-3, an HVM running kernel 5.15.5-1 (dom0 provided), which has 1 VCPU and a static assignment of 400MB of RAM … with a low performing Verbatim Store-N-Go USB 3.0 flash drive attached:

[user@sys-usb-3 ~]$ uname -a
Linux sys-usb-3 5.15.5-1.fc25.qubes.x86_64 #1 SMP Sat Nov 27 14:17:00 CET 2021 x86_64 x86_64 x86_64 GNU/Linux

[user@sys-usb-3 ~]$ sudo dmesg
...
[   40.850090] usb 3-1: new SuperSpeed USB device number 2 using xhci_hcd
[   40.878521] usb 3-1: New USB device found, idVendor=18a5, idProduct=0243, bcdDevice= 0.02
[   40.878560] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   40.878589] usb 3-1: Product: STORE N GO
[   40.878605] usb 3-1: Manufacturer: Verbatim
[   40.878620] usb 3-1: SerialNumber: xxxxxxxxxxxxxxxx
[   41.070333] usb-storage 3-1:1.0: USB Mass Storage device detected
[   41.075031] scsi host2: usb-storage 3-1:1.0
[   41.075860] usbcore: registered new interface driver usb-storage
[   41.093936] usbcore: registered new interface driver uas
[   42.312377] scsi 2:0:0:0: Direct-Access     Verbatim STORE N GO       8.01 PQ: 0 ANSI: 6
[   42.316029] sd 2:0:0:0: Attached scsi generic sg0 type 0
[   42.317451] sd 2:0:0:0: [sda] 245760000 512-byte logical blocks: (126 GB/117 GiB)
[   42.317803] sd 2:0:0:0: [sda] Write Protect is off
[   42.317828] sd 2:0:0:0: [sda] Mode Sense: 23 00 00 00
[   42.318018] sd 2:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[   42.324064]  sda: sda1
[   42.325161] sd 2:0:0:0: [sda] Attached SCSI removable disk
...

[user@sys-usb-3 ~]$ sudo dd if=/dev/sda of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 11.9004 s, 90.2 MB/s

[user@sys-usb-3 ~]$ sudo dd if=/dev/zero of=/dev/sda bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 58.5915 s, 18.3 MB/s


Brendan

The speeds you’re seeing are much more reasonable and inline with what I’d expect. What version of QubesOS are you running?

I’m on the following kernel version on sys-usb:

[user@sys-usb ~]$ uname -a
Linux sys-usb 5.10.76-1.fc32.qubes.x86_64 #1 SMP Fri Oct 29 03:58:58 CEST 2021 x86_64 x86_64 x86_64 GNU/Linux

In the QubesManager settings for sys-usb, I don’t have any other options in the Kernel drop down list other than the one I’m currently running. I’ll look into this a bit more and see if I can work out how to get newer kernels as an option to see if there’s any improvement for this USB speed issue.

R4.0

Installed a newer kernel into Dom0 with:

sudo qubes-dom0-update kernel-latest-qubes-vm

Changed sys-usb to use the newer kernel, but still no improvement:

[user@sys-usb ~]$ uname -a
Linux sys-usb 5.14.15-1.fc32.qubes.x86_64 #1 SMP Fri Oct 29 07:05:20 CEST 2021 x86_64 x86_64 x86_64 GNU/Linux

[user@sys-usb ~]$ sudo dd if=/dev/sda of=/dev/null bs=1M count=1024 conv=fsync status=progress
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 97 s, 11.1 MB/s 
dd: fsync failed for '/dev/null': Invalid argument
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 97.1453 s, 11.1 MB/s

[user@sys-usb ~]$ sudo dd if=/dev/zero of=/dev/sda bs=1M count=1024 conv=fsync status=progress
1072693248 bytes (1.1 GB, 1023 MiB) copied, 88 s, 12.2 MB/s
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 128.985 s, 8.3 MB/s

Is the USB drive attached to the USB 3.2 Gen 2x1 xHCI controller or to the Thunderbolt 4 USB Controller?

Can you try moving the device from port to port to see if moving it from one controller to another has an impact?

On latest spec Thinkpad P15 Gen 2 which is almost exact same as yours using a USB 3.1 san disk extreme thumb drive (not 3.2!) and testing with fedora disk app benchmark feature I am seeing about average ready rate of 130 MB/s and average write rate of 40-45 MB/s

if i decrease the sample size from 100MB to 10MB average read speed goes to 90MB/s and average write rate goes to 60MB/s

hope that helps

All my settings I am running are default out the box fresh install of Qubes 4.1rc2

no other changes

SanDisk Extreme GO USB 3.1 thumb drive

The drive is connected to the USB-A ports which I’m guessing map back to the USB 3.2 Gen 2.1 xHCI controller.

[user@sys-usb ~]$ lsusb | grep Flash
Bus 004 Device 017: ID 090c:1000 Silicon Motion, Inc. - Taiwan (formerly Feiya Technology Corp.) Flash Drive

[user@sys-usb ~]$ lsusb -t
/:  Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 20000M/x2
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M
    |__ Port 1: Dev 17, If 0, Class=Mass Storage, Driver=usb-storage, 480M
    |__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/5p, 480M
        |__ Port 3: Dev 8, If 0, Class=, Driver=, 12M
        |__ Port 1: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 12M
        |__ Port 4: Dev 9, If 0, Class=Hub, Driver=hub/4p, 480M
            |__ Port 3: Dev 10, If 0, Class=Hub, Driver=hub/4p, 12M
<output ommitted>

Transfer speeds when using native USB-A ports on the Thinkpad P1:

[user@sys-usb ~]$ lsusb | grep Flash
Bus 004 Device 017: ID 090c:1000 Silicon Motion, Inc. - Taiwan (formerly Feiya Technology Corp.) Flash Drive

[user@sys-usb ~]$ sudo dd if=/dev/sda of=/dev/null bs=1M count=1024 conv=fsync status=progress
1069547520 bytes (1.1 GB, 1020 MiB) copied, 98 s, 10.9 MB/s
dd: fsync failed for '/dev/null': Invalid argument
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 98.5441 s, 10.9 MB/s

[user@sys-usb ~]$ sudo dd if=/dev/zero of=/dev/sda bs=1M count=1024 conv=fsync status=progress
1070596096 bytes (1.1 GB, 1021 MiB) copied, 124 s, 8.6 MB/s
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 135.68 s, 7.9 MB/s

Transfer speeds when using the native TB4/USB-C ports on the Thinkpad P1:

[user@sys-usb ~]$ lsusb | grep Flash
Bus 003 Device 005: ID 090c:1000 Silicon Motion, Inc. - Taiwan (formerly Feiya Technology Corp.) Flash Drive

[user@sys-usb ~]$ sudo dd if=/dev/sda of=/dev/null bs=1M count=1024 conv=fsync status=progress
1071644672 bytes (1.1 GB, 1022 MiB) copied, 97 s, 11.0 MB/s
dd: fsync failed for '/dev/null': Invalid argument
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 97.3092 s, 11.0 MB/s

[user@sys-usb ~]$ sudo dd if=/dev/zero of=/dev/sda bs=1M count=1024 conv=fsync status=progress
1065353216 bytes (1.1 GB, 1016 MiB) copied, 91 s, 11.7 MB/s
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 99.3344 s, 10.8 MB/s

Transfer speeds when using USB-A ports on the TB4 connected Lenovo Workstation Dock:

[user@sys-usb ~]$ lsusb | grep Flash
Bus 004 Device 018: ID 090c:1000 Silicon Motion, Inc. - Taiwan (formerly Feiya Technology Corp.) Flash Drive

[user@sys-usb ~]$ sudo dd if=/dev/sda of=/dev/null bs=1M count=1024 conv=fsync status=progress
1072693248 bytes (1.1 GB, 1023 MiB) copied, 98 s, 10.9 MB/s
dd: fsync failed for '/dev/null': Invalid argument
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 98.1956 s, 10.9 MB/s

[user@sys-usb ~]$ sudo dd if=/dev/zero of=/dev/sda bs=1M count=1024 conv=fsync status=progress
1071644672 bytes (1.1 GB, 1022 MiB) copied, 124 s, 8.6 MB/s
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 134.757 s, 8.0 MB/s

Flash drive used for these most recent tests is a Samsung Duo Plus USB Type-C Flash Drive 64GB:

I also ran the disk app benchmark as mentioned by ewokky and got poor performance results.

It feels like there’s something here which is artificially limiting USB I/O speeds. I wonder if temporarily connecting the USB controllers into Dom0 and re-running the tests as a troubleshooting exercise would provide any additional insight.

Very strange our computers are almost identical. Same 4.1rc2.

I did my tests on the USB-A port as well. I tried both ports on the right hand side both gave similar speed results.

Those results are super slow. Please keep us posted on this. Sorry I am not clever enough to help you troubleshoot.