I am using a HP G2 thunderbolt dock and the displayport’s work. The only issue with those is I have to reconfigure the display layout every time I connect the dock. My main issue is that I cannot use any of the USB-A or USB-C ports on the dock. I have looked around the documentation and the forums. I haven’t been able to figure anything out and I was wondering if someone can point me to what I need to do.
The only issue with those is I have to reconfigure the display layout every time I connect the dock.
That’s by design. I’ll be the first to say that Qubes doesn’t do multiple displays very well (…yet, but it’s only a matter of time).
My main issue is that I cannot use any of the USB-A or USB-C ports on the dock.
Have you managed to find which PCI device the ports are located on?
If you can, check the output of dom0
lsusb -t. That will tell you which devices to pass through.
Post the output too, if you feel comfortable doing so, but feel free to redact any info
Update: this is probably a bad idea, I just learned that, if you do this, then the USB qube will only run if you boot the computer with the dock connected. I’ll keep the original post so you don’t have to repeat my mistake.
I haven’t used it enough to call this a solution, but early results seem promising if I remove the USB qube and then re-create it with my Caltech dock connected. The the very least, the dock’s USB ports work the same way as the computer’s built-in ports. The next step is to try various tests around both hot and cold-plugging the dock. I also need to try out its audio, video, and network ports.
I followed the instructions at USB Qubes | Removing a USB qube and then the command to create one from the top of that document:
sudo qubesctl state.sls qvm.sys-usb
Maybe this might help?
udev would be the place to start to “reset” your ports when you plug in and remove the dock. That way, you wouldn’t need to cold-boot (unless of course, the BIOS took care of it, and I’d be horrified if it did…),
Thank you, @alzer89. I’ll have to review your article a bit more carefully. It could be part of a solution for Qubes. Am I correct in understanding this would let me hot-plug the dock into dom0? After that, I’ll need to figure out how to handle the USB controllers that are only present when the dock is connected.
Does Qubes support multiple USB qubes? I’m wondering if it would help to have a sys-usb-dock that I would only run if the dock is connected. I’m thinking of adding the dock’s USB controllers to this new qube, while the laptop’s built-in controllers would remain with sys-usb.
Alternately, can we programmatically add and remove devices from existing qubes?
Well, almost everything goes into dom0 at some point, because dom0 has the passthrough kernel modules and drivers.
The reason I brought this up is that I find it weird (and incredibly un-Linuxy) that the dock only works from a cold boot. That would imply that the dom0 kernel (or potentially the BIOS, but hopefully not) is essentially telling the kernel the system tree (with the dock either connected or unconnected).
(I say BIOS because I recently discovered how proprietary the rackmount server industry is, and how big a role BIOSes actually play in server hardware, when I got a ProLiant DL380p Gen8; so I honestly wouldn’t be surprised if HP’s BIOSes “micromanage” your machine, instead of the OS…)
But all that information should be able to be modified with
udev rules and hooks. For example, you could set up scripts for your machine to run when ports have things plugged into them, etc.
That system tree can be modified and remapped without a reboot. Pretty much, the only thing in -nix OSes that actually requires a reboot is the kernel itself (and even then, live patching is possible, albeit it so painfully annoying to do that it’s often easier to just reboot).
The issue that we’ll have to solve is how to update your machine’s system tree to what it looks like from cold boot with the dock connected. My guess is that
udev will be the answer.
Check to see what your system tree looks like with the dock connected (and functional) and without the dock connected. Once we know that information, we can then figure out a way to remap it whenever the dock is connected/disconnected.
See that’s the thing. Am I right in assuming that this dock contains things other than USB ports? Like an Ethernet port, a DisplayPort, HDMI port, etc.?
If they do, and you pass the entire dock into a VM, then pretty much only that VM will be able to use the dock. I’m guessing that’s not what you want… That’s why it’s important to take a “snapshot” of your system functioning with it and without it.
You can as many VMs with as many PCI devices passed through to them as you like, as long as each PCI device is only passed through to one VM at a time (they don’t like sharing…).
If you’ve got multiple USB buses on your machine, you could definitely create a separate VM for each bus, if you wanted.
I have laptops with an xbox 360 gamepad built into the chassis, on its own USB bus, and I pass that bus into a VM for playing Nintendo 64 ROMs occasionally (I will play GameCube ROMs as soon as integrated graphics performance in Qubes OS gets better), so it’s definitely possible.
(bear in mind that you’re passing through the entire bus, and not individual ports; so be aware of what else is on that bus. You might just lose your internal keyboard/mouse to a VM )
The way you’ve described it to me, this wouldn’t fix your “cold boot” problem. It seems like your machine tells the kernel “I’ve got this many USB ports, this many HDMI ports, this many Ethernet ports, etc.” when the kernel is loaded.
Sure, you could create a separate qube for your dock, but if it wasn’t attached when you cold-booted your machine, I don’t think it would make any difference…
You can do this too. It can cause kernel panic inside VMs sometimes depending on what devices you do it to, and how they’re done.
(If a real machine suddenly got a GPU shoved into one of its PCI slots, it probably wouldn’t react very well…)
Basically, what you need is a solution that, when you plug in this dock, will:
- Tell dom0 that a dock is plugged into one of your thunderbolt ports
- Tell dom0 to treat the individual ports/buses on the dock as separate buses (so you can have the multiple monitors in
sys-gui, the Ethernet port in
sys-net, etc.), and remap your system tree.
- Pass through the appropriate ports/buses to the appropriate VMs (similar to the way
sys-usbwill pass through a USB keyboard/mouse to dom0, but in reverse).
I have a USB-C “dock” that came with my PinePhone that has a similar layout to your dock, but I haven’t been able to achieve what I’m describing yet. If I find a solution, I will post it. (Sorry!)
Thank you for the detailed answer! That will help a great deal. The way I see it, I have two, mostly independent problems.
Let’s set hot-plugging aside for the moment (that’s problem #2 for me).
The first problem is that the dock presents itself as several PCI devices to the computer:
|02:00.0||PCI bridge||Intel Corporation||JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]|
|03:00.0||PCI bridge||Intel Corporation||JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]|
|03:01.0||PCI bridge||Intel Corporation||JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]|
|03:02.0||PCI bridge||Intel Corporation||JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]|
|03:03.0||PCI bridge||Intel Corporation||JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]|
|03:04.0||PCI bridge||Intel Corporation||JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016]|
|04:00.0||USB controller||Fresco Logic||FL1100 USB 3.0 Host Controller|
|05:00.0||USB controller||Fresco Logic||FL1100 USB 3.0 Host Controller|
|06:00.0||USB controller||ASMedia Technology Inc.||ASM1142 USB 3.1 Host Controller|
|07:00.0||Network controller||Intel Corporation||I210 Gigabit Network Connection|
Several of these devices need to be mapped to various qubes to be useful. USB controllers to sys-usb and network controller to sys-net in this case. However, if I map a device to a certain qube, and the device isn’t present, such as when the dock isn’t connected, then that qube cannot boot.
That’s why I asked about multiple sys-usb qubes. If it works as I hope, then sys-usb will always be bootable because the laptop’s built-in USB ports will always be there, and sys-usb-dock will be bootable only if the dock is present.
I still need to figure out what, if anything, can be done for the network controller. I can certainly make a sys-net-dock qube for that controller, but how can I integrate that with the rest of the system? I’m kinda thinking I’ll need some way to add and remove the Ethernet controller to sys-net on bootup.
As for hotplugging, that’s another can of worms that I should probably save for after I get cold-plugging working. Basically, it sounds like I need to set up udev to detect when the dock is plugged or unplugged and then rescan the PCI bus and make whatever changes it needs to make in response:
- Start or stop qubes.
- Restart qubes while slipping in changes to their device configuration (hopefully without needing to restart other qubes, like how stopping sys-net requires that sys-firewall stop, which in turn requires that all network-connected appvms also stop).
- Adjust screen layout.
- Adjust audio outputs.
By the way, if it makes any difference, there’s no HP in my system. I have a Framework laptop with a CalDigit TS3Plus+ dock. I originally hoped to contribute to the original question before seeing for myself how big a dumpster fire Thunderbolt docks are.
That’s not a problem. That’s by design. That’s how Thunderbolt works
Yeah. That makes sense a lot more now.
This is an example of a command to attach PCI devices to VMs in dom0:
sudo xl pci-attach sys-net '03:00.0'
Maybe that would work when linked to a
It might also help you avoid separate qubes entirely
Oh, fun! I just finished testing the separate qube for the dock’s USB ports (still cold-plugging only). It was a partial success. Everything works fine if I start the laptop with the dock disconnected or with the dock connected to the same Thunderbolt port as when I configured it. However, if I plug the dock into a different Thunderbolt port, then the dock’s PCI identifiers all change.
I guess, in my case, I can just make a rule for myself that the dock must be connected to a specific port. That’s probably not true in a general case though.
Just throwing my two cents in here. Accidentally, I figured out the other day that you can shut down sys-net without shutting down sys-firewall (and thus don’t need to shut down workVM, personalVM etc.)
If you go into sys-firewall settings, you can set the network qubes to none, shut down sys-net, do whatever changes you like, reboot sys-net and in sys-firewall settings set networking to sys-net again.
Did you make any progress on this topic?
Interesting. I’m very interested to learn that upstream network qubes can be changed on the fly.
Unfortunately, I hit a brick wall with hot-plugging because I can’t get it to rescan the PCI bus after boot, and dom0 can’t see the Thunderbolt bus at all, even when it’s actively using the dock. My best guess is that Xen hides the Thunderbolt bus, even if it treats Thunderbolt-connected PCI devices as normal PCI devices. Actually, I wonder if that could be an attack vector…
So far, I’ve only gotten my dock to work as a cold-pluggable USB hub and hot-pluggable USB-PD device.
Although for cold plugging, I wonder if I can make a sys-net-dock qube that has the dock’s Ethernet port and sys-net as an upstream port. Then, activating the dock’s Ethernet port would involve starting that qube and reassigning the firewall’s upstream. Worth a try anyway.
The thunderbolt dock just like usb dock but for things that required some fast connection. Take my thunderbolt 3 dock as an example. I connected the thunderbolt 3 display adapter for my laptop (which is I’m currently using on) to a display that I have only VGA cable. So, I just plug the adapter to a designated port for usb extension thingy that I don’t know why Lenovo put it there, and plugin the VGA male to both of my display and the adapter that attached with my laptop. Then choose your display configuration and, bam! I now have a second monitor for multi tasking. Although your thunderbolt port might be different, you still need another cable.
With that been said, if you are using MacBook Air 2017 model, you still need mini display port cable for that to been able to work. But, that still outdated because we have a new thing called USB-C that is introduced in 2018 by big techs.
it usb + dp + pcie 4x (that why it hard to make it work)
Unfortunately, although USB and Thunderbolt look similar (especially Thunderbolt 3/4 which use USB-C plugs) they are very different electrically. USB is a peripheral bus, but Thunderbolt is a PCI-PCI bridge.
Why is that important? USB is a well-known bus with excellent support by the sys-usb qube. Thunderbolt devices, however, appear as separate PCI devices.
When you connect a Thunderbolt dock, you actually have to go into qube settings and assign its PCI devices to the appropriate qubes, and those qubes will now only boot if the dock is connected (because a qube can’t boot if its PCI devices are missing).
As an extra bit of fun, Thunderbolt devices connect to dom0 by default just like any other PCI device. I wonder if a Thunderbolt rubber duck might bypass the sys-usb protection. Such a device would look like a PCI-USB controller, which will probably be connected to dom0 and not sys-usb because it wasn’t present during Qubes installation. And whatever peripherals the manufacturer wants. It’s on dom0, so a keyboard will work.
I know it’s been a while, but I thought I would give an update to this.
My Thunderbolt dock’s USB ports have been working well with a separate USB qube for them. In addition to assigning the dock’s USB devices to the new qube. I also needed to update the mouse policy to allow USB mice connected to the dock to work. I decided to block USB keyboards (the default behavior) because the dock emulates a USB keyboard, and I can think of no sane reason for that.
Networking works with some babysitting. I created a sys-net-dock qube for the dock’s ethernet port, to keep the device off dom0 if nothing else. I can manually move the firewall between the 2 net qubes, but I haven’t figured out how to automate it. Good enough.
The dock’s DisplayPort, audio, and card reader work with no tinkering needed. The DisplayPort goes to dom0 and extends the desktop. The audio and card reader get managed like any other USB device.
Hotplugging remains a non-starter. I need to plug it in before booting Qubes. Other than that, everything seems to work.
I am facing a the same issue and my primary objective here is to enable my external display port when hot-pluggin it (via hdmi) to my thunderbolt 4 dock.
@alzer89, I was intrigued by your idea to address the issue via udev, but I was wondering where within the steps proposed above you would handle the display port issue.