Getting a serial console on USB-only Qubes laptop

My goal here is to get boot printk’s to understand a hardware compat issue, so first I’m looking for info on how to setup a boottime serial console with qubes.

From what I saw:

  • console= kernel parameters are ignored, and we see through dmesg that printk get directed to tty0 and hvc0
  • some information exist on qubes website (Safe Remote Dom0 Terminals | Qubes OS), but they are only applicable to a running system, and won’t help getting early kernel logs

Then, as is not uncommon those days on a laptop, I do not have an RS232 interface. The kernel does support console=ttyUSB to cope with this, but how we can make it work in the qubes model, where we cannot even attach a USB/serial device do dom0, is not yet clear to me either.

Any hint to get those early printk’s to the outside world ?

You can, of course, attach a USB device to dom0, by removing the
“rd.qubes.hide_all_usb” from the boot parameters.
If you have a USB3 debug port, DbC is designed to log printk messages -
you’ll need a debugging kernel, of course, but it will give you
something before the console comes onstream.

Let’s try to document how it goes.


does it support DbC ?

In sys-usb:

  • does the xhci driver expose DbC functionnality ?
[root@sys-usb ~]# ls -1 /sys/bus/pci/drivers/xhci_hcd/0*/dbc
[root@sys-usb ~]# cat /sys/bus/pci/drivers/xhci_hcd/0*/dbc
  • does it work ?

    For each xHCI controller:

    • enable it, check it was enabled:
      [root@sys-usb ~]# echo enabled > /sys/bus/pci/drivers/xhci_hcd/0000:00:06.0/dbc
      [root@sys-usb ~]# cat /sys/bus/pci/drivers/xhci_hcd/0000:00:06.0/dbc
    • plug a USB-A/USB-A cable to a port on this controller (you may need to go trial-and-error, or just enable on all controllers and see) with the other end to another computer, check kernel logs – if all goes well you should see:
      [ 2462.569824] xhci_hcd 0000:00:06.0: DbC connected
      [ 2462.834314] xhci_hcd 0000:00:06.0: DbC configured
      … and on the other end of the cable you should see a new USB device appear, if if you have the proper driver available it should bind to /dev/ttyUSB0: the xHCI host controller was really replaced with a device.

activate on Qubes

QubesOS kernels have CONFIG_EARLY_PRINTK_USB_XDBC=y defined already, ready for the job.


We need to:

  • allow dom0 do get the xHCI controller (remove rd.qubes.hide_all_usb)
  • prevent sys-usb from stealing it (add qubes.skip_autostart)
  • direct kernel messages to DbC (add earlyprintk=xdbc, or =xdbc1 to select the second controller – the core also looks for the undocumented keep string)

current status

  • for now all I could get from earlyprintk is as follows (here for xdbc1, on AMD Renoir 4800H) :
[    0.000000] xhci_dbc:early_xdbc_parse_parameter: dbgp_num: 1
[    0.000000] ------------[ cut here ]------------
[    0.000000] WARNING: CPU: 0 PID: 0 at mm/early_ioremap.c:145 __early_ioremap+0x129/0x2c0
[    0.000000] Modules linked in:
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.13.13-1.fc32.qubes.x86_64 #1
[    0.000000] RIP: e030:__early_ioremap+0x129/0x2c0
[    0.000000] Code: c4 4a 89 0c ed 20 5c 2b 83 4c 89 e6 48 89 55 c8 48 c1 ee 0c 4c 89 65 c0 83 fe 40 76 14 48 b8 d9 90 a9 9c 50 42 c9 16 48 31 c3 <0f> 0b e9 30 01 00 00 48 b8 3b 3a 2f 8a 7c f1 85 e1 41 6b cf c0 48
[    0.000000] RSP: e02b:ffffffff82803c40 EFLAGS: 00010086 ORIG_RAX: 0000000000000000
[    0.000000] RAX: 16c942509ca990d9 RBX: de10a3bc57bf5e9a RCX: 0000000000100000
[    0.000000] RDX: 0000000000000000 RSI: 0000000000000100 RDI: 93e922a75e8a18c7
[    0.000000] RBP: ffffffff82803c88 R08: 65fcb0e9ce5cfcb4 R09: 0000000000000000
[    0.000000] R10: fffffffffff00000 R11: 95a6ac2ad0bed3c4 R12: 0000000000100000
[    0.000000] R13: 0000000000000000 R14: 8000000000000063 R15: 0000000000000000
[    0.000000] FS:  0000000000000000(0000) GS:ffffffff830a0000(0000) knlGS:0000000000000000
[    0.000000] CS:  10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.000000] CR2: 0000000000000000 CR3: 0000000002810000 CR4: 0000000000040660
[    0.000000] Call Trace:
[    0.000000]  ? xdbc_map_pci_mmio+0x1d7/0x27c
[    0.000000]  ? early_xdbc_parse_parameter+0x27c/0x397
[    0.000000]  ? setup_early_printk+0x3bd/0x406
[    0.000000]  ? do_early_param+0xe8/0x19e
[    0.000000]  ? parse_args+0x108/0x1b0
[    0.000000]  ? bootconfig_params+0x6e/0x6e
[    0.000000]  ? bootconfig_params+0x6e/0x6e
[    0.000000]  ? parse_early_options+0x51/0x55
[    0.000000]  ? bootconfig_params+0x6e/0x6e
[    0.000000]  ? parse_early_param+0xaf/0x137
[    0.000000]  ? setup_arch+0x434/0xa42
[    0.000000]  ? early_security_init+0xa2/0xcf
[    0.000000]  ? start_kernel+0x8e/0x6f4
[    0.000000]  ? xen_start_kernel+0x683/0x6a2
[    0.000000]  ? startup_xen+0x3e/0x3e
[    0.000000] random: get_random_bytes called from __warn+0xab/0x110 with crng_init=0
[    0.000000] ---[ end trace 599f71c9036487cb ]---
  • I hoped to be activate the device as console afterwards, like we can do with /sys/class/tty/ttyS0/console, but even as /sys/class/tty/ttyDBC0/ appears, in gets no console control.

  • late console

All I’m able to get for now is a late dom0 console through an external USB/serial adapter and console=ttyUSB0,115200.

What’s your experience with DbC ?

Addendum: Xen and USB console

While getting logs on serial line for dom0 can be useful, when investigating early boot failures it can also be valuable to extract the logs from the hypervisor itself.

Xen has support for EHCI DBGP (USB2) but as of 4.14/4.16 no support yet for XHCI DBC. So recent laptops without an EHCI controller are out here.

For those with EHCI support, I could find no HOWTO (only the Xen commandline doc) so let’s summarize here: you will need to:

  1. activate the DBGP driver on the controller you mean to use
  2. specificy dbgp as console

The commandline options for Xen (not for the dom0 linux kernel) would be, once you have identified the BDF address of your PCI EHCI, something like:

dbgp=ehci@pci7:0.3 console=dbgp,vga

Disclaimer: could not test it yet, only have a XHCI laptop available, for now :cry:

In addition even if you have a USB2 laptop with debug support on one of the ports (which is supposedly supported by Xen), there are no currently on the market products that support connecting a USB2 debug port to a monitoring host. I’ve looked. The last (high-end) one on the market is mentioned here in 2015, but does not appear to be sold any more: Xen project Mailing List

The only way I know of how to use the USB2 debug port is via a homebrew cable using this approach: GitHub - avivgr/teensy_debugdev: USB 2.0 Debug cable using teensy

To discover whether a USB2 controller supports such a debug port, see output of lspci -v|less and search for a line with ‘ebug’ in one of the USB controller blocks.

Other ideas:

If the device has an externally facing PCIE port (e.g. expresscard, or a desktop with PCIE slots), a PCIE serial port card (as long as it’s real and not a PCIE->USB->serial combo device) may work, depending upon Xen (and possibly BIOS/UEFI). This is supposedly supported by Xen.

Another approach which I haven’t seen attempted, would be to set up a thunderbolt machine with the bios thunderbolt settings set to the least secure settings, connected to a PCIE enclosure with a PCIE serial port adapter. Again, this may depend on BIOS/UEFI support.

IIRC, Marek did not have luck with an M.2 NVME → PCIE serial port adapter, though unclear if that was a BIOS limitation.


1 Like


Also, if you have a server (or a business laptop) with Intel AMT or V-PRO, you can always use the Serial Over LAN functionality for Xen or Dom0 debugging.

For obvious reasons, you want to have AMT disabled if it’s a feature of your hardware build. I keep AMT disabled in the BIOS of the Lenovo W520s and W530s I have…but I did a test for you, dear reader.

If I set it up in BIOS, a new ttyS port appears in dom0. I then shut down and disabled it.

Xen docs say you can use the port info available in dom0 to construct the Xen command line (to start with this port as the debug console). You will need another machine on the same ethernet LAN segment and will need to dig out the information you need from this guide: Xen Serial Console - Xen