MacOS VM on qubes

ckN6QwSZ · June 8, 2022, 1:46pm

qvm-features hvm-mac uefi 1

(credits: Enabling UEFI boot in qube - #2 by kommuni )

qvm-start hvm-mac --hddisk=dispXXXX:/home/user/OpenCore.img

aaaand OpenCore works!

ckN6QwSZ · June 7, 2022, 9:58pm

Hi there, I know this could be a huge waste of time, but it still might be fun and I couldn’t resist. As for all Hackintosh projects this is for educational purposes only.

In a fedora-35-dvm and after increasing private storage max size to 10GB for dispXXXX:

wget https://github.com/kholia/OSX-KVM/raw/master/fetch-macOS-v2.py
chmod +x fetch-macOS-v2.py
./fetch-macOS-v2.py
[..]
Choose a product to download (1-5): 4
[..]
qemu-img convert BaseSystem.dmg -O raw BaseSystem.img

In dom0:

qvm-create -P poolhd0_qubes --property=virt_mode=hvm --property=kernel='' --label blue --standalone hvm-mac
qvm-start hvm-mac --cdrom=dispXXXX:/home/user/BaseSystem.img

So far, so easy.

Now follows the more difficult part. One has to boot via OpenCore and to attach OVMF_CODE (firmware) and OVMF_VARS (nvram) to the HVM. This works for Qemu/KVM (plus libvirt) and should be doable for Qemu/Xen as well.

Anybody interested in joining me in this ride?

Kholia’s github repo (kholia) contains an opencore.qcow2 and a script which sets up the environment for Qemu/KVM: OpenCore-Boot-CD.sh.

PS: worst part might be a few cpu flags which Xen does not pass through as far as I can tell.

ckN6QwSZ · June 8, 2022, 10:55am

First problem I encountered is that neither BaseSystem.img, BaseSystem.iso, OpenCore.img or OpenCore.iso are recognized as bootable devices (passed with --cdrom or --hddisk alike).

So, let me rephrase my above post into a few questions:

how can I pass arguments (an environment) to an Xen-HVM like it is done in OpenCore-Boot-CD.sh (1) for Qemu/KVM?
what is Qubes equivalent to qemu-system-x86_64 "${args[@]}"?

I have taken a look into /usr/lib/python3.8/site-packages/qubesadmin/ in dom0 and tried to understand how qubes starts Qemu/Xen VMs under the hood. Did grep -Ri qemu and grep -Ri xen and was vim-ing through some of the files, but did not really get a grip on how HVMs are started.

anybody out there who would like to share his or her knowledge about the starting process of HVMs?

These are the parameters which attach the devices to Qemu/KVM:

[user@dispXXXX ~]$ grep -i drive Downloads/OSX-KVM/OpenCore-Boot-CD.sh
-drive if=pflash,format=raw,readonly,file=“$REPO_PATH/$OVMF_DIR/OVMF_CODE.fd”
-drive if=pflash,format=raw,file=“$REPO_PATH/$OVMF_DIR/OVMF_VARS-1024x768.fd”
-drive id=OpenCoreBoot,if=none,snapshot=on,format=qcow2,file=“$REPO_PATH/OpenCore/OpenCore.qcow2”
-device ide-hd,bus=sata.2,drive=OpenCoreBoot
-device ide-hd,bus=sata.3,drive=InstallMedia
-drive id=InstallMedia,if=none,file=“$REPO_PATH/os.dmg”,format=raw
-drive id=MacHDD,if=none,file=“$REPO_PATH/mac_hdd_ng.img”,format=qcow2
-device ide-hd,bus=sata.4,drive=MacHDD

An important step I believe is to exchange the SeaBIOS to OVMF_CODE.fd which seems to be a tianocore firmware port (2) for Qemu.

ckN6QwSZ · June 8, 2022, 11:32am

This is a start:

https://wiki.osx86project.org/wiki/index.php/Snow_Leopard_Server_on_Xen

In qubes-os the config-files for the VMs are not located in /etc/xen/, however, I found them in /etc/libvirt/libxl/ and that are libvirt-files which can even be edited with virsh edit hvm-mac.

alzer89 · June 8, 2022, 4:32pm

As per the macOS EULA, I have my Apple hardware on standby ready to try this

ckN6QwSZ · June 8, 2022, 4:44pm

Topics have been merged and my last three posts are older then my first three posts.

Anyway, I got OpenCore to run and BaseSystem to boot like this

qvm-features hvm-mac uefi 1
qvm-block a --persistent hvm-mac dispXXXX:loop0
qvm-block a --persistent hvm-mac dispXXXX:loop1
qvm-start hvm-mac

after making the images available in dispXXXX like this

sudo losetup -f /home/user/git/OpenCore.img
sudo losetup -f /home/user/git/BaseSystem.img

Kudos and Thanks to @tzwcfq who also suggested to create a seperate template like this

which might suite the usecase better, since we have to add a lot more then opencore and the installer medium to get the HVM Hackintosh to work.

Next step is to convert as much args from OpenCore-Boot(-CD).sh into /etc/qubes/templates/libvirt/xen/by-name/hvm-mac.xml as possible.

ckN6QwSZ · June 8, 2022, 6:08pm

And that is clearly a pain in the a… - might be easier to take another machine, install qemu, kvm, libvirt, setup a macOS-VM and transfer the macOS.xml to qubes to see which parts of the kvm xml can be used.

For instance to transfer

-cpu Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check

to the existing

  <cpu mode="host-passthrough">
    <feature policy="disable" name="vmx"/>
    <feature policy="disable" name="svm"/>
    <feature policy="require" name="invtsc"/>
  </cpu>

one probably has to take a big dive into qemu’s and libvirt’s manuals.

https://qemu-project.gitlab.io/qemu/system/qemu-cpu-models.html

At least

  <cpu mode="custom">
    <model name="Penryn"/>
    <feature policy="disable" name="vmx"/>
    <feature policy="disable" name="svm"/>
    <feature policy="require" name="invtsc"/>
    <feature policy="require" name="ssse3"/>
    <feature policy="require" name="sse4.2"/>
    <feature policy="require" name="popcnt"/>
    <feature policy="require" name="avx"/>
    <feature policy="require" name="aes"/>
    <feature policy="require" name="xsave"/>
    <feature policy="require" name="xsaveopt"/>
    <feature policy="require" name="check"/>
  </cpu>

doesn’t work out of the box.

edit: this cpu-config is supported though…

  <cpu mode="host-passthrough">
    <feature policy="disable" name="vmx"/>
    <feature policy="disable" name="svm"/>
    <feature policy="require" name="invtsc"/>
    <feature policy="require" name="ssse3"/>
    <feature policy="require" name="sse4.2"/>
    <feature policy="require" name="popcnt"/>
    <feature policy="require" name="avx"/>
    <feature policy="require" name="aes"/>
    <feature policy="require" name="xsave"/>
  </cpu>

tzwcfq · June 8, 2022, 6:26pm

Try:

  <cpu mode="host-model">
    <model>Penryn</model>

ckN6QwSZ · June 8, 2022, 6:32pm

Works.

And writing the config to /etc/qubes/templates/libvirt/xen/by-name/hvm-mac.xml to make changes permanent works, too.

tzwcfq · June 8, 2022, 6:45pm

Also you can add vendor id if it’ll make a difference:

  <cpu mode='host-model'>
    <model vendor_id='GenuineIntel'>Penryn</model>

ckN6QwSZ · June 8, 2022, 6:54pm

I’ll do that. And

are accepted now, too.

Do you have an idea how to get

  -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
  -drive if=pflash,format=raw,readonly=on,file="some_dom0_path/OVMF_CODE.fd"
  -drive if=pflash,format=raw,file="some_dom0_path/OVMF_VARS-1024x768.fd"

inside of the xml? I tried a few times and that just got erased by virt-manager without a failure notice.

ckN6QwSZ · June 8, 2022, 6:59pm

Hold on… kholia has got a .xml file in his repo…

github.com

kholia/OSX-KVM/blob/master/macOS-libvirt-Catalina.xml

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<!--
	macOS libvirt XML configuration.

	Run "virt-xml-validate macOS-libvirt-Catalina.xml" to validate this file.

	To install this file, you may place it at ~/.config/libvirt/qemu/
	and run: virsh define macOS-libvirt.xml.

	This configuration has been tested in Ubuntu 20.04 with stock QEMU-KVM.

	Move/rename images and loader/nvmram files and paths as you wish.

	!!! Don't forget to replace CHANGEME with your values !!!

	Adjust memory and currentMemory to 3145728 if you want only 3 GiB.

	Consider removing some cpu features if your hardware lacks support.

This file has been truncated. show original

tzwcfq · June 8, 2022, 7:00pm

libvirt: Domain capabilities XML format

ckN6QwSZ · June 8, 2022, 9:01pm

Strangely enough… kholia’s macOS-libvirt-Catalina.xml gets validated

[user@dom0 osx]$ sudo virt-xml-validate libvirt.xml 
libvirt.xml validates

and mine .xml fails to validate

[user@dom0 by-name]# virt-xml-validate /etc/qubes/templates/libvirt/xen/by-name/hvm-mac.xml
/etc/qubes/templates/libvirt/xen/by-name/hvm-mac.xml:92: namespace error : Namespace prefix qemu on commandline is not defined
  <qemu:commandline>
                   ^
/etc/qubes/templates/libvirt/xen/by-name/hvm-mac.xml:93: namespace error : Namespace prefix qemu on arg is not defined
    <qemu:arg value='-device'/>
                             ^
/etc/qubes/templates/libvirt/xen/by-name/hvm-mac.xml:94: namespace error : Namespace prefix qemu on arg is not defined
a-applesmc,osk=ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc'
                                                                               ^
/etc/qubes/templates/libvirt/xen/by-name/hvm-mac.xml:95: namespace error : Namespace prefix qemu on arg is not defined
    <qemu:arg value='-smbios'/>
                             ^
/etc/qubes/templates/libvirt/xen/by-name/hvm-mac.xml:96: namespace error : Namespace prefix qemu on arg is not defined
    <qemu:arg value='type=2'/>
                            ^
Relax-NG validity error : Extra element devices in interleave
/etc/qubes/templates/libvirt/xen/by-name/hvm-mac.xml:44: element devices: Relax-NG validity error : Element domain failed to validate content
/etc/qubes/templates/libvirt/xen/by-name/hvm-mac.xml fails to validate

I copied&pasted those 6 lines. I don’t get it.

tzwcfq · June 8, 2022, 9:15pm

Try to change:
<domain type='xen'>
to
<domain type='xen' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

libvirt: QEMU command-line passthrough

ckN6QwSZ · June 8, 2022, 9:52pm

works! thanks again!

I still got

Relax-NG validity error : Extra element devices in interleave
/etc/qubes/templates/libvirt/xen/by-name/hvm-mac.xml:44: element devices: Relax-NG validity

Google hints to a typo. I have got that kind of stuff in python-scripts with indendation or forgetting a closing charakter or something like that, gave me an error in one of the following lines after the actual mistake.

tzwcfq · June 8, 2022, 9:53pm

Paste the whole file, I don’t know what’s there on the line 44.

ckN6QwSZ · June 9, 2022, 2:53pm

Just a visual progress report… this is the EFI loader:

This is OpenCore:

And this is BaseSystem.img starting and getting stuck:

ckN6QwSZ · June 9, 2022, 4:29pm

I’m going to get help from the Hackintosh-Community. Before I continue I need to know which components of the qemu-config are essential and which are not.

Some features are not supported by qubes’ qemu (i.e. the clock settings and the usb-model which kholia’s libvirt.xml uses), but it might be possible to install them in dom0 in case that is a must.

ckN6QwSZ · June 9, 2022, 6:59pm

https://dortania.github.io/OpenCore-Install-Guide/config.plist/penryn.html

This looks like the best source for OpenCore-Hackintoshes I have found so far.