Windows 10 Questions

[zithro]

I’ll answer quickly and “messily” as I’ve no time now, but I’ll develop more tomorrow if you need/want.

My working Xen setup is :

• BIOS boot on a MSI B350 PC-Mate, Ryzen 1700x 8c/16t
• AMD RX580 on the primary/x16 PCIe slot → PT to my gaming win7 HVM domU. I modprobe-blacklist the amdgpu driver, and tell in xorg.conf to only use the nvidia card. The GPU does not support FLReset, but I found ways to make it work, ie. I can reboot w7 w/o rebooting dom0
• nvidia GT710 on a PCIe x1 slot (the small ones) → for dom0 GUI
• the 2nd “GPU/PCIe x8 slot” (the one meant for SLI/crossfire) is occupied by a 10GbE NIC to connect to my backup dom0
• both GPUs are connected to my 3 screens
• Xen on debian stable, no libvirt. Using XFCE as dom0 GUI
• working HVM domUs with Seabios, I never used OVMF/UEFI : Debians, pfsense (as a global dom0/domU fw, with MoBo NIC PT), TrueNAS core (freeBSD-based, with a NVMe optane PT), w7 (GPU+audio+USB controller: all PT), openBSD, Solaris 11.3, a nested Xen-on-Debian, and even … a nested Qubes \o/

Everything is flawless, maybe I’m just lucky with HW choices, or HW+SW combo.
On boot, the AMD GPU displays the BIOS, then GRUB, then some part of VT1 logs, then it automagically swaps the display to the nVidia GPU for the rest of VT1 logs (hence the display gets frozen on the AMD-connected screens from now on, till I PT it to w7 or another U), then it displays X.

Considering the “from Xen to Qubes” doc, I was talking about the Qubes documentation only.
For the Xen docs: I recently joined the Debian-Xen team because I wanted to help them - and thank them - and from there I learnt there’s a new guy responsible for the rework of the Xen docs (he’s working at Vates, creators of XCP-ng/XenOrchestra). I contacted him and I’m heavily pushing for the docs to be up-to-date really quickly. But will I succeed, it’s another story !

But I feel you, my first experiences with Xen were exactly as you said, painful : “googling and reading documentation, wikis, reddit posts, forum posts, random blog posts, and other sites, Virtualization doc, specifically for Xen”. There’s no easy way.
For Xen, I find the best ressource is the man pages on “Xen Documentation”. You read the wiki pages for general pointers/guidance, then check the last edit date on the bottom of the page, then read the man pages to adapt to the recent way of doing stuff.
Also, I found the documentation from XCP-ng may be of help sometimes, for anything not too related to XAPI stuff. Also the virt guide from Suse.
You can also mimick KVM/libvirt config stuff to Xen.
More later ^^ Don’t give up !

[TalarusLuan]

Nice! Thanks for the details on your setup!

I just tried what I mentioned above, moving Qubes back to the primary 6950XT gpu (was a pain, because it requires nomodeset=0 to not hang Qubes, and no xorg.conf in /etc/X11 so I can get the GUI to come up).

I pciback’d the RX550 pci ID, and assigned it to the Win10 qube, then started it. Same thing, window comes up, trying to start, black window. 30 seconds pass, and qrexec_daemon complains about not being able to connect, and it kills the qube. Checking the dm log, I see no errors whatsoever. None on the PCI setup for the video card (unlike the 6950XT, which has the errors mentioned above), and no errors from anything else, but it still hangs and dies.

sigh I seem to be chasing an untamed ornithoid without cause. >.<

[disp6252]

How much RAM did you assign for this VM? Check this:
Contents/gaming-hvm.md at master · Qubes-Community/Contents · GitHub

[disp6252]

Regarding the problem with passthrough of your primary GPU check this issue:
rd.qubes.hide_pci doesn't work anymore after upgrade to Qubes 4.1 · Issue #7976 · QubesOS/qubes-issues · GitHub

[TalarusLuan]

It’s currently set at 8GB, but I’ve tried 16GB up to 64GB (system has 128GB memory) in it.

I did not do the TOLUD mod, because further reading into the github issue seemed to indicate that R4.1 versions were already patched (both xen-hvm-stubdom-linux and xen-hvm-stubdom-linux-full are at v1.2.5-1).

That said, I checked the rootfs images and they were indeed NOT patched, so I went ahead and patched them. I started the VM, and it actually booted into Windows. It also loaded the Microsoft AMD driver for the RX550, and the display is working! Woo! Progress!

I have a desktop displayed on the RX550, but of course I can’t move the mouse into it because it’s just using the Qubes Display/mouse. I have to figure out how to get the mouse/keyboard passed through directly into Windows, but before I do that, I need to swap GPUs, and I think the RX6950 is going to be a bit more of a problem child.

Thanks for the reminder to try that.

[disp6252]

@TalarusLuan

I have to figure out how to get the mouse/keyboard passed through directly into Windows

You may be interested in this:

Mouse and Keyboard passthrough to Windows HVM

I am trying to pass a keyboard and mouse to a windows hvm (or any hvm) via libvirt xml config. For reference, I do not want to use a USB kvm or use the USB proxy. I do not want the hvm to have control of the usb device. I want to pass this mouse/keyboard inputs via software only. I cannot seem to find a lot of information about passing mouse/keyboard in Xen or qubes specifically. Could the stubdom-linux may be getting in the way? I have added a devices block to extend my hvm’s xml libvirt confi…

[TalarusLuan]

Quick update, I swapped back to the RX550 as the Qubes dedicated GPU and the RX6950 as the passthrough for win10. With the 3.5G patch, it also works!

Next tasks I have are:
1.Mouse/KB passthrough (going through that post above; sounds like what I want to do, thanks!). I would like Windows to recognize the device as the actual device (so I can load drivers for it, like the Razer Naga Mouse, and the Logitech G910 keyboard so the Razer/Logitech drivers can be installed and the devices managed with the utility programs).
2. Custom Volume settings.
3. Getting sound to work fully (I’d like to have the full support for my audio codecs and channels, if possible).
4. Possibly dropping the Qubes GUI for it, just in case things get weird with desktop/graphics settings.
5. Hotplug PCI passthrough – not high on the priority list, but would be nice.

One other thing about the GPU passthrough, when the VM boots, the windows logo and spinning dots appear only in the Qubes window. Only once it gets to or past the login screen does the GPU activate and display anything. If I disable the Qubes VGA adapter, will it display the boot up logo on the PT GPU?

[zithro]

1: I dunno if that changed, but I found USB passthrough kinda sluggish for games (your HW suppose gaming). My solution was to PT one of the USB controllers to the Win domU, and plug the dedicated KB/mouse to it, but that’s a bit cumbersome …

2+3 : As you’re on a desktop, there are some HW tricks to make audio work w/o touching software, but it depends on your setup AND requirements !

Your audio card has Audio Out and Line In jacks.
Your GPUs only have Audio Out via HDMI/DP, but sound is passed to your monitor(s).
On your monitors, you have an Audio Out jack.
Plug one of the monitors’ Audio Out into Line In of the soundcard with a jack-jack, and config the audio card to play input from the Line in.
As you may game and use vocal comms, note this won’t allow the mic to work.
So like in (1), I PT the audio card to the gaming machine, and use the same method but reversed (sound for dom0 and other qubes go through HDMI to monitor to Line In).
Another solution would be to use a cheap USB audio device.

5: I think you’re saying the Qubes manager won’t allow that, but it works using Xen commands.
xl pci-attach , xl block-attach, etc (xl help | grep attach).
I don’t know if it still works (last try was last year), if it’s recommended, and the implications concerning Qubes.

@TalarusLuan

One other thing about the GPU passthrough, when the VM boots, the windows logo and spinning dots appear only in the Qubes window. Only once it gets to or past the login screen does the GPU activate and display anything. If I disable the Qubes VGA adapter, will it display the boot up logo on the PT GPU?

Just try it ^^ But for info, it works like that on my system too. Except it seems you also get a desktop from the VGA adapter ? I don’t, the VGA adapter gets deactivated, but I’m on w7 so YMMV (IIRC the Xen wiki talks about that).
I’ve never tried to remove this “feature”, because at least on my setup, if there’s a PT problem, the VGA adapter would still allow me to login to the VM and check what the problem is.

[TalarusLuan]

Thanks for the feedback in info!

I’m currently making an override libvirt xml template for my win10 vm using the Customization doc from a few posts above and had a few questions.

First, the doc mentions putting user templates and overrides in the /etc/qubes/templates/libvirt (or the xen/by-name dir inside of it). However, that dir (from templates/ forward) doesn’t exist. I presume it is the correct folder, but has to be created first?

Next, when I look at the default templates in /usr/share/qubes/templates/libvirt(/devices) look like they are missing something. For example, the xen.xml and devices/block.xml in the sections create a line that is missing the type=‘raw’ attribute, and a line that is missing the bus=‘xen’ attribute. Are these the actual templates used, or are there others somewhere else?

(What I mean is that when the actual final libvirt vm xml files are generated, they have these attributes, but they are nowhere to be found in the templates, which would seem to point to these templates not being the ones used to generate the final vm xml files)

In fact, it is even weirder that the template xml files use double-quotes (") as delimiters, but the generated <vm_name>.xml files use single quotes (') instead.

I think I can get the changes to work, but I am concerned that I am not editing the correct template files because of these weird little quirks.

[disp6252]

@TalarusLuan

However, that dir (from templates/ forward) doesn’t exist. I presume it is the correct folder, but has to be created first?

Yes.

@TalarusLuan

I think I can get the changes to work, but I am concerned that I am not editing the correct template files because of these weird little quirks.

These are correct templates that are used to generate configs.

[TalarusLuan]

OK. Then I guess I am misunderstanding something about how it goes from the xen.xml template to the final generated version. As an example:

xen.xml:

        {% for device in vm.block_devices %}
            <disk type="block" device="{{ device.devtype }}">
                <driver name="phy" />
                <source dev="{{ device.path }}" />
                {% if device.name == 'root' %}
                    <target dev="xvda" />
                {% elif device.name == 'private' %}
                    <target dev="xvdb" />
                {% elif device.name == 'volatile' %}
                    <target dev="xvdc" />
                {% elif device.name == 'kernel' %}
                    <target dev="xvdd" />
                {% else %}
                    <target dev="xvd{{dd[counter.i]}}" />
                    {% if counter.update({'i': counter.i + 1}) %}{% endif %}
                {% endif %}

                {% if not device.rw %}
                    <readonly />
                {% endif %}

                {% if device.domain %}
                    <backenddomain name="{{ device.domain }}" />
                {% endif %}
            </disk>
        {% endfor %}

gpu_base-win10.xml (final):

<disk type='block' device='disk'>
  <driver name='phy' **type='raw'**/>
  <source dev='/dev/qubes_dom0/vm-gpu_base-win10-root-snap'/>
  <target dev='xvda' **bus='xen'**/>
</disk>

Where do those two specified attributes come from? Is there another config override xml that I am missing?

The reason I am concerned is if I make an override xml and put it in the /etc/qubes/templates/… folder, I am basing my edits/overrides on the existing templates I know about. If there’s one I am missing, it will likely muff something (with my luck, anyway :P). Is there any way to get it to test generate the final vm xml file without launching the vm?

Thanks!

[disp6252]

They are templates that will be used by Qubes tools to generate final config and not parts of config that will be put together as is.

[TalarusLuan]

Yeah, I was just using the devices section as an example to point out the discrepancy I am seeing.

I think I understand the basics of the templating system, I’m just having a bit of difficulty tracing the generation process going from the various templates to the end product. The way I understand it, is it passes the vm object to the jinja parser, and starts with the xen.xml template in /usr/share/qubes/templates/libvirt, which includes the three templates in the devices folder (block.xml, pci.xml, net.xml) if they are required, then goes to the /etc/qubes/templates/libvirt folders to pick up the xen-user.xml and the by-name/<vm_name>.xml templates as overrides.

None of the templates seem to add the type=‘raw’ and bus=‘xen’ attributes, so where do they come from? Are they added by another post-processing function after the templates have been processed or from another template I am unaware of? That’s what is concerning me that I am missing something.

Also, to do what I want to do, it would seem I need to define a “disks” block as a replaceable sub-block nested inside of the “devices” block. I don’t want to have to regen the entire “devices” block just to change out the disk sections. It would seem to be easier to just add some extra block definitions in the devices section of the base xen.xml template (as opposed to copy/pasting most of the devices block in an override template). The downside to doing that would be that the changes would get clobbered in a qubes update which updated the xen.xml template. That’s not horrible, though; I could always re-add them.

Adding them to a xen-user.xml override template is an alternative, but it would be a bit of copypasta that would still need to be examined after a qubes update which touched xen.xml.

I apologize if I am seem a bit dense. I’ve been looking at this the entire evening, and it’s 3AM, so I likely am missing the bleeding obvious from fatigue.

[disp6252]

@TalarusLuan

None of the templates seem to add the type=‘raw’ and bus=‘xen’ attributes, so where do they come from? Are they added by another post-processing function after the templates have been processed or from another template I am unaware of? That’s what is concerning me that I am missing something.

It’s better to check the source code to see what’s happening.

@TalarusLuan

Adding them to a xen-user.xml override template is an alternative, but it would be a bit of copypasta that would still need to be examined after a qubes update which touched xen.xml.

Did you read this?
Custom libvirt config — core-admin mm_203ee458-0-g203ee45-dirty documentation

[TalarusLuan]

@disp6252

It’s better to check the source code to see what’s happening.

Oof. Have any more light reading you can recommend?

Yes, I read the custom libvirt domain config doc, and the accompanying Jinja template documentation.

[disp6252]

@TalarusLuan

Oof. Have any more light reading you can recommend?

Well either devs come here and tell you how it’s done (wich is unlikely) or you find that out by yourself in sources (which I think will be faster).

@TalarusLuan

Yes, I read the custom libvirt domain config doc, and the accompanying Jinja template documentation.

So why don’t you just inherit from the original template and override the devices block as stated there?

The better way is to inherit from the original template and override any number of blocks. This is the point when we need the name of the original template.

{% extends 'libvirt/xen.xml' %}
{% block devices %}
    {{ super() }}
    <serial type='pty'>
        <target port='0'/>
    </serial>
{% endblock %}

{% extends %} specifies which template we inherit from. Then you may put any block by putting new content inside {% block %}{% endblock %}. {{ super() }} is substituted with original content of the block as specified in the parent template. Untouched blocks remain as they were.

[disp6252]

Like this:

/etc/qubes/templates/libvirt/xen/by-name/YourWindowsVM.xml

{% extends 'libvirt/xen.xml' %}
{% block devices %}
    {{ super() }}
    <disk type='block' device='disk'>
      <driver name='phy' type='raw'/>
      <source dev='/some/device/path'/>
      <target dev='xvde' bus='xen'/>
    </disk>
{% endblock %}

[TalarusLuan]

@disp6252

Well either devs come here and tell you how it’s done (wich is unlikely) or you find that out by yourself in sources (which I think will be faster).

Yeah, I know. Just adding a little levity. Though I don’t know if it would be any faster… there’s a LOT of code to read. XD

@disp6252

So why don’t you just inherit from the original template and override the devices block as stated there?

Yeah, I know. That’s what I am planning on doing.

The way I work on these things is to try and understand how it works the best I can by reading the docs first, and anything that sticks out as a potential problem I ask questions about first. I am a bit leery of just “Try it and see!” when it comes to VMs because I don’t want to break something that will cause many more hours of work to get back to a known good current state. Plus, I want to try and reduce the problem space a bit up-front.

@disp6252

Like this:

Yes, that’s sorta what I am doing, but 1) I don’t want to do {{ super() }} because I don’t want the pregenerated disk volumes on this VM, and 2) based on my current readings, will it do something like this:

>      <driver name='phy' type='raw' type='raw'/>
>      <source dev='/some/device/path'/>
>      <target dev='xvde' bus='xen' bus='xen'/>

or something stranger because I haven’t grokked the entirety of the xml generation process.

Basically, I don’t want the VM to launch with a bad config, since Windows is notoriously bad about clobbering stuff if it isn’t what it expects (it tries to ‘make it right’). So, I either would like to do “dry runs” where it generates the final xml, but doesn’t launch the vm, or I need to do “config proofing”, which is doing due diligence up-front to make sure the configs are as correct as possible before launch. Part of the process of doing the latter is making sure I understand the best I can what is happening by reducing/correcting issues or gaps in my understanding I see beforehand.

If the only way to do that is to keep studying the process with increasing depth until I get to that point, so be it. I am asking here in the hopes that someone might have a better understanding of what I am missing so that I can narrow the time window a little bit.

Anyway, I will keep on studying it. I DO appreciate your help, though.

[disp6252]

You can just create test AppVM and use it to generate test configs before you copy verified config to your Windows VM config.

[TalarusLuan]

If I pass it the real disks, will it futz with them? I don’t want it to write over anything.