How to install Qubes OS on block device without a partition table?

User Support
1

Hello again Qubes OS community,

This topic is very difficult to title because I am not sure how to describe it. Alternative titles:

  • How to get Anaconda/Blivet to recognize existing LVM volumes?
  • How to prevent Anaconda/Blivet from deactivating volume group during manual partitioning?
  • How to force Qubes OS installation after manually assigning mountpoints?
  • chroot installation of Qubes OS?

If anyone can suggest a better title to change this to, please do. Unfortunately, this difficulty is because of how complex this problem is, and with that complexity comes a lot of text. Before I fully explain my situation, I will try to summarize as briefly as I can:

  • sda, its mapped LUKS volume, and its internal LVM logical volumes are not showing up as “Unknown” existing partitions in the “Manual Partitioning” section, even when completely unlocked and opened as confirmed with lsblk. Only sdb is seen as an encrypted LUKS partition at sdb1, which can be unlocked and assigned a mountpoint. Both drives are shown as available and are otherwise recognized.
  • The entire device tree is being recognized by Anaconda/Blivet, however, even though it just wants to destroy all that in favor of an MSDOS partition table. The GUI’s “Summary of Changes” and backend logs both confirm this.
  • In those logs, it is reported that the volume group is being deactivated without explanation and Accordion unselects all items. This is done every time the volume group is (re)activated and the disks are rescanned in the GUI. Activating the volume group without rescanning does nothing.
  • Please advise.

Now, let me explain my situation:

I am trying to install Qubes OS R4.0.3 from the ISO burned onto a DVD+R DL. I am attempting to implement a custom installation which includes a detached encrypted /boot and detached LUKS headers, with Qubes OS being installed in LVM on LUKS within a triple cipher cascade. I also want the main drive to appear as nothing but random data to outsiders, a form of deniable encryption, and this requires both detaching the /boot and LUKS headers and not formatting the raw block device or writing any cleartext partition table to the beginning of the disk. If possible, I want to accomplish this entirely from within the live OS, without any additional tools or intermediary systems.

So far, I have accomplished every single step of the process up to the actual installation, thanks to the input of many sources, including the wizards here. That means I have successfully prepared both drives for use by checking for bad sectors and filling them with random data; encrypted and formatted the USB drive (here-after sdb) and generated the detached headers and keyfiles on it; encrypted and formatted the main drive (here-after sda), including all the LVM partitioning described in the “custom installation” documentation; and even manually formatted the logical volumes, as well, all from the terminal (CTRL+ALT+F2) in the Anaconda installer. In other words, I have everything set up and prepared for installation.

In the GUI, I go to “Installation Destination”, select both drives, choose “I will configure partitioning.” and deselect “Encrypt my data.” In the bottom left, I click “Full disk summary and boot loader…” and set sdb as the Boot Device, then close. Then, I click “Done” and proceed to the “Manual Partitioning” section.

Here be dragons: After doing all this, I have been unable to get the Anaconda installer to accept sda and expose the existing LVM partitions on it for configuration (it does not appear under “Unknown” existing partitions). I know Anaconda can see it, since when I attempt to continue with “Done” twice, it displays a summary of changes that plans to destroy all the formatting, device partitioning, and encryption that I have done just so it can install an MSDOS partition table.

The /tmp/storage.log documents that everything I have done on sda is recognized, that the volume group (qubes_dom0) is scanned, and that each logical volume is enumerated, only for it to then bail and remove them all from the device tree. lvm.log seems to be processing the volumes and group just fine, though I am no expert in reading these logs. But then, curiously, /tmp/anaconda.log reports that the following command is also ran:

--
INFO program: Running [XX] lvm vgchange -an qubes_dom0 --config= devices { preferred_names=["^/dev/mapper/", "^/dev/md/", "^/dev/sd"] } log {level=7 file=/tmp/lvm.log syslog=0} ...
INFO program: stdout[XX]:  0 logical volume(s) in volume group "qubes_dom0" now active
INFO program: stderr[XX]:  /usr/sbin/dmeventd: stat failed: No such file or directory
--

Only when I mount a logical volume does the stderr change to complaining that the group “contains a filesystem in use.” Returning to /tmp/lvm.log, I see the following:

--
lvmcmdline.c:1611   DEGRADED MODE. Incomplete RAID LVs will be processed.
libdm-config.c:1064   activation/monitoring not found in config: defaulting to 1
lvmcmdline.c:1617   Processing: vgchange -an qubes_dom0 '--config= devices { preferred_names=["^/dev/mapper/", "^/dev/md/", "^/dev/sd"] } log {level=7 file=/tmp/lvm.log syslog=0}
lvmcmdline.c:1618   Command pid: 2730
lvmcmdline.c:1619   system ID:
lvmcmdline.c:1622   O_DIRECT will be used
libdm-config.c:992   global/locking_type not found in config: defaulting to 1
--
libdm-common.c:1475   qubes_dom0-pool00: Skipping NODE_DEL [trust_udev]
libdm-common.c:1475   qubes_dom0-root: Skipping NODE_DEL [trust_udev]
libdm-common.c:1475   qubes_dom0-pool00-tpool: Skipping NODE_DEL [trust_udev]
libdm-common.c:1475   qubes_dom0-pool00_tdata: Skipping NODE_DEL [trust_udev]
libdm-common.c:1475   qubes_dom0-pool00_tmeta: Skipping NODE_DEL [trust_udev]
vgchange.c:160   Deactivated 3 logical volumes in volume group qubes_dom0
activate/dev_manager.c:755   Getting device info for qubes_dom0-swap [LVM-string1]
ioctl/libdm-iface.c:1838   dm info  LVM-string1 [ noopencount flush ]   [16384] (*1)
activate/dev_manager.c:755   Getting device info for qubes_dom0-pool00 [LVM-string2-pool]
ioctl/libdm-iface.c:1838   dm info  LVM-string2-pool [ noopencount flush ]   [16384] (*1)
ioctl/libdm-iface.c:1838   dm info  LVM-string2 [ noopencount flush ]   [16384] (*1)
activate/dev_manager.c:755   Getting device info for qubes_dom0-root [LVM-string3]
ioctl/libdm-iface.c:1838   dm info  LVM-string3 [ noopencount flush ]   [16384] (*1)
activate/activate.c:1835   Counted 0 active LVs in VG qubes_dom0
vgchange.c:259   0 logical volume(s) in volume group "qubes_dom0" now active
activate/dev_manager.c:755   Getting device info for qubes_dom0-swap [LVM-string1]
ioctl/libdm-iface.c:1838   dm info  LVM-string1 [ noopencount flush ]   [16384] (*1)
activate/dev_manager.c:755   Getting device info for qubes_dom0-pool00 [LVM-string2-pool]
ioctl/libdm-iface.c:1838   dm info  LVM-string2-pool [ noopencount flush ]   [16384] (*1)
ioctl/libdm-iface.c:1838   dm info  LVM-string2 [ noopencount flush ]   [16384] (*1)
activate/dev_manager.c:755   Getting device info for qubes_dom0-root [LVM-string3]
ioctl/libdm-iface.c:1838   dm info  LVM-string3 [ noopencount flush ]   [16384] (*1)
activate/activate.c:1835   Counted 0 active LVs in VG qubes_dom0
mm/memlock.c:562   Unlock: Memlock counters: locked:0 critical:0 daemon:0 suspended:0
activate/fs.c:489   Syncing device names
cache/lvmcache.c:157   Metadata cache: VG qubes_dom0 wiped.
misc/lvm-flock.c:71   Unlocking /run/lock/lvm/V_qubes_dom0
misc/lvm-flock.c:48   _undo_flock /run/lock/lvm/V_qubes_dom0
device/dev-io.c:625   Closed /dev/mapper/luks3
metadata/vg.c:89   Freeing VG qubes_dom0 at 0xXXXXXXXXXXXX
metadata/vg.c:89   Freeing VG qubes_dom0 at 0xYYYYYYYYYYYY
--

Separately (though maybe not), Accordion states in /tmp/anaconda.log that is is “unselecting all items” when the disks are rescanned.

Back to the “Manual Partitioning” GUI: sdb does show up under “Unknown” existing partitions, and sdb1 can be decrypted either from the GUI (though it sometimes errors) or from the terminal (after rescanning the disks). But every time, neither sda or any of its mappers or logical volumes show up, even when the device is manually unlocked from the terminal and lsblk shows that the entire tree is open. Upon disk rescan, Accordion unselects all items, vgchange -an qubes_dom0 is ran, and lsblk only reports the mapped LUKS volume (where the physical volume is) as still open. Even reactivating the volume group with vgchange -ay qubes_dom0 does nothing in the GUI, and is reversed upon disk rescanning.

What I have inferred from all this is that, unlike Debian and Arch installers, Anaconda’s top-down approach is not friendly to custom installation setups like this and Blivet does not make it much better, at least in F25. (The Red Hat bugzilla has many bug reports since F9 and including F25 regarding similar LVM on LUKS issues, though they all seem to be with an otherwise standard partitioning setup. They were all either closed upon release EOL or were given a temporary updates.img to include, or otherwise declared “fixed”.) It is unclear to me what the problem is, whether it is because Anaconda/Blivet insists on a cleartext partition table for every disk, despite LVM needing none; or because something is causing the volume group to deactivate upon rescan, thus preventing the logical volumes from appearing for assignment; or both, or something else entirely.

What solutions might there be to this conundrum? Is it possible to manually mount all the volumes and directories and force an installation? Is a chroot path possible? Is there a way to trick Anaconda/Blivet into seeing the logical volumes or otherwise recognizing an MBR table on sdb as describing the “partitioning” of sda? Or are the limitations of Anaconda/Blivet so great that it is simply not possible to implement the setup I want, despite it being doable on Debian, Arch, and others?

If necessary, I can re-generate the full logs and copy them over to a separate USB for transfer, but that may have to wait until I am back with the drives again. If there is something specific I should look for in them, I can extract the relevant output and post it here, as well. Regardless, maybe a sanity check is in order and one of you can point out what’s wrong with this old novice’s implementation. Am I just trying to make Anaconda do something it cannot do? Is there any way I can make Anaconda do so anyway, even if it means delving into debugging or modifying files? Or will I have to do some restructuring after installation to get the setup I want?

Any advice here will be greatly appreciated. Qubes/Fedora/Anaconda wizards needed and welcome. Thank you for your time.

With many words,
John

1 Like
2

ok not so much my filed of expertise but…
take a look at this

idk how the setrup detects/deals in general with this stuff but it seems like when the partitions weren’t active there was a issue with the recognition

maybe you can’t just fill them with random data or they won’t mount onto the dir tree

try formatting everything and see if it gets ride o the not recognizing issue…
go at it 1 issue at a time and if this is where the issue is then well maybe someone else more familiar with the system could help you or maybe there’s a bug/serious issue tht’s not easy to resolve

if after formatting and… you’re still getting the same issue wll then it has a different reason
not muh for me to say here

from personal experience let me tell you that usually when there are 2 partitions or devices or whatever
that interact with the system in such a way where one “method of verifying it” at the lack of a better articulation
gives you a positive result but another system which should reach the same conclusion and recognize the volume/device the same exact way but doesn’t it usually means there’s a issue with the volume/device/whatever
because it is unluckily the “recognition” system has a flaw in it

u know what i’m not explaining my self well… there’s a pretty old but somewhat famous of a issue with url parsers maybe you herd about that
u can google it

maybe the issue with 1 system recognizing is that it check for validity differently then the other
see my point

so maybe just try regular formatting and activation

again not really my field

1 Like
3

Hello @rjo74618,

Thank you for taking your time to offer your input, and for trying to make sense of this confusing problem.

I actually came across that same topic during my searching here for potentially related threads. While it has some similarities to this, the solution found there appears to be to format the raw device. That is exactly what I am trying to avoid because formatting sda prior to encryption and LVM partitioning introduces visible structure to the raw device, including a cleartext partition table.

I understand that I am most likely able to proceed with installation if only I format sda with something like ext4 and add an MBR table at the beginning of the disk, since that is what Anaconda is complaining about (the lack of any MBR). That precludes my goal of deniable full-disk encryption on the main drive, however, and renders it as a mainly standard installation. At that point, I have much less incentive to store /boot on a separate USB drive (sdb) as well, since the partition table will already be a dead give-away that the disk is likely more than just random junk data.

I suspect you may be right that Anaconda just does not know what to do with a raw device other than to insert a partition table, despite being able to see the well-defined structure and partitioning inside the unlocked LUKS volume. This is very unfortunate, and seems to be a clear limitation of the installer if not altogether a bug (though the Red Hat/Fedora developers will of course claim it is a “feature”). Other installers, particularly “bottom-up” ones like those used by Debian and Ubuntu and especially minimal bootstrappers like pacstrap and debootstrap, do not limit the user’s freedom in this way. And it seems that I am not the only one who has experienced this, either, as this Fedora 16 bug report from 2011 shows (closed as WONTFIX). It was perhaps put best on the [qubes-users] mailing list: Anaconda is “ugly wood”.

I previously attempted to explore the possibility of a “partitionless” or “superfloppy” installation (more specifically, a raw data disk drive without any VBR/MBR/GPT partition table) in my first post on this forum, very early on in this saga, because I anticipated that the Anaconda installer might not be friendly to such custom setups. That unfortunately went nowhere, so I am now circling back to where I started; only this time, I am afraid I may have to conclude that this whole month-long journey was toward a dead end.

At this point, I am more than happy to just manually install Qubes OS from scratch if at all possible, through a series of terminal commands that implements what the Anaconda installer would otherwise do for me, like pacstrap in Arch or debootstrap in Debian. If necessary, I am willing to consider writing my own Kickstart file if that can achieve what I want. I am completely comfortable with the command line and would love nothing more than to bypass Anaconda altogether, but the chances of that look bleak.

I am still hoping that this is just a bug that I can hotfix in the installer or somehow circumvent by manually assigning mountpoints from the terminal and forcing an installation, but I do not know how I can go about doing that from here. The “Begin Installation” button remains grayed out because Anaconda still insists that I configure the “Installation Destination”, despite having a fully configured system ready and mounted from tty2. If only I could skip that step, rather than continue to be treated like an idiot by Anaconda, I would be running Qubes OS right now.

I left Windows years ago because I felt myself pushing up against the walls of the OS environment, handcuffed and prevented from tinkering any further with a system that treats me like it knows best better than I do. I did not feel free. I am beginning to feel that way about Fedora now, right out of before the gate, with this Anaconda installer being my first exposure to it. I hope a solution can be found for this problem of mine that does not involve reverting back to a more standard installation. If not, then I may have to proceed with one and restructure the system post-installation, but that should not have to be done and I would not know where to start anyway.

If you have any further ideas or advice, I am willing to entertain just about any creative solution you can imagine. I know this is not your field of expertise, as you said, but your willingness to help is admirable. Thanks again for your time.

Kindest regards,
John

4

not much more to say man… happy to help
but even open source has it’s limitations

idk how one can leave windows… nothing is supported
i like linux for it’s security and well just as u said the control over the system making it as customizable and secure and… as possible

but u know everything has it’s limitations
hope a fedora/qubes professional will be able to help you
but other then that… to idk the brute force solution if you really need a deniable encryption will be to edit and recompile the entire thing and that’s gonna be a big mess
maybe it’s something you can do fairly easily but i can’t suggest you waste you’re time on it

since you can run vm’s on qubes it’s fairly easy to implement any deniable encryption on that level
not ideal but u know…

personally i just use such tools as veracrypt simple and ready to go of any flashdrive/…

what can i say man good luck

1 Like
5

I have cross-posted on the Ask Fedora Discourse forum, seeing how this appears to ultimately be a Fedora/Anaconda issue and not a Qubes OS one. I doubt any of the patches that Qubes OS did to Anaconda would be the cause of this and I suspect that this has more to do with a hardcoded limitation in Anaconda.

You can find the Ask Fedora topic thread here. I think I did a better job at compressing my post and clarifying my situation there than I did here, so I recommend reading my post there if any of the above is unclear. I can re-post it here, but I do not want to clutter the thread with any more redundancy than it already has.

Some additional remarks:

I have discovered that this problem may in fact be intentional, if the following is any indication (emphasis mine):

The ability to install to a hard drive is available releases since Fedora 7. After the live media boots, click on the install icon on your desktop to start the installation. Installation from live image requires that GRUB and the /boot directory be installed onto a drive with an MSDOS partition label, or that the current machine supports EFI booting. If a pc-clone machine has only GPT hard drives, then you may need to use something such as a USB2.0 flash memory device (with an MSDOS partition label) as an intermediate destination.

(Source: Fedora Project Wiki: Fedora Live CD § Hard Drive Installation)

I am not the only one to experience this problem when trying to install Fedora through Anaconda onto a raw device containing LVM on LUKS, either, given that this seems to be a very similar situation. Others have run into similar problems with Anaconda’s limitations more generally, as well, as shown in this reddit thread. Such problems and complaints seem to be exceedingly rare however, and those are the only two instances I have found that bare any close similarity to what I am experiencing (three if including this aforementioned bug report). That probably should not come as much of a surprise, since what we are all trying to implement are highly customized and unusual setups, but I would expect more than that given how pointlessly restrictive Anaconda appears to be and how many Arch and Debian users hop to Fedora.

I am pessimistic about the chances that my setup can be implemented entirely through Anaconda or otherwise from the live disk. So, I am now exploring the possibility of creating an intermediate installation of Qubes OS on a USB drive (sdc), from which I will then mount sda and copy over the data. Does anyone here anticipate Qubes-specific complications involved with this approach? Maybe I should make a new thread about it, since that would be a different topic.

Wearily,
John

6

@rjo74618,

I just saw your reply. Once more, thank you for your willingness to help me here. Simply discussing this with you has helped clarify the situation, even though no satisfactory solution has yet been found.

These apparent limitations in Anaconda do not seem necessary at all. They resemble the freedom-hostile behavior I identify with Windows, not what I have come to expect from Linux distributions and software, especially from one of the big ones. This is my first direct experience with anything Fedora or Red Hat related, however, since I have always stuck with Debian, Ubuntu, Arch, Alpine, etc. So while I agree that all software has limitations (even open-source and Linux software), I do not think that explains or excuses the apparent limitations I am experiencing with Anaconda, especially when installers for all the other distributions I have tried do not have them.

I understand that I can encrypt on a per-VM basis, but the threat model I am mitigating involves inspection of the raw drive when locked and not remote surveillance or local inspection while unlocked; in that sense, my threat model is more similar to the ones full-disk encryption protects against rather than ones that involve encrypting VMs within an already encrypted LUKS container.

I am fully aware that what I am trying to do may seem like madness, or at least overkill. I will not argue that I am not a madman, only that there is a method to my madness.

With respect to leaving Windows and the claimed “security” of Linux (and its arguable status as a myth), we can discuss that further in private if you wish. That is strictly off-topic and although I have some opinions about that, this is not the place to voice them.

Have a great day.

Best wishes,
John

1 Like
7

what can i say man i hear you’re pain (btw when you set you’re profile to be hidden people can’t pm u)
one of reason i can understand some limitation is that when you’re building a system focused and based on security you (at least in theory) attempt not to bloat that system

if you don’t focus on something as a feature then you’re trying to usually simplify it to avide the most common cause of errors stupid human interaction
and if not that then well…

‘it’s better to keep something completely sealed then to do a halfass job make a hole in the water tank that’s patched with duct tape to maybe later on add a valve
-because eventually water will leak and some nasty snit crowl in’ sorts of mentality

that’s not to say not this was intentional probably just a mistake

but i can understand the security focused mindset where you focus on and get used to

what’s the word… “close” everything right away

when you focus on not making security mistakes then you might not allways notice the other issues you’re causing… like user experience issues
the os isn’t really as complicated as advertised
i’m not a linux expert as well played around with ubuntu arch kali an so on
but not a expert/complicated
non the less the pure amount of errors and issues i’ve been having

the time i’ve spend looking for solutions (the forum is nothing for the most part i had no issue finding information online because those are such common issues u get the point)

old ancient bugs not yet solved

and that’s because when you focus on one aspect you neglect another man

i know i’m spamming this thread but you can see my point

some times you gotta comprmise

and oh yes i never argued that linux on it’s own is more secure
but you have more control over it’s security
updates are much easier and common much simpler and less of a headache to give an example
great antivirus’s and so on open source often is more secure
(also given that this os is designed to be used in vms u know there are a few lines of defance first you ave to compromise the vm and then the entire system which isn’t exactly easy when you secure them well and keep up to date)

8

Although I agree that feature creep can be a problem, including for the security of a system, this particular Anaconda issue smells to me like more of a hardcoded check for which there is no override, ie something coded in rather than left out, than it is something that a feature implementation would address. I may be wrong though (I have not inspected the relevant code yet), or at least a Kickstart file may be able to bypass it. I am still awaiting elaboration from a regular on the Ask Fedora thread about that very possibility.

I understand that a private profile cannot be viewed and so no “Message” button is available, but I believe you can still message a private user by going to https://forum.qubes-os.org/u/YOURUSERNAME/messages, selecting “New Message”, and composing it directly to them. If I am mistaken, I apologize for the confusion. I will send you a message to open up the line, just in case.

Best,
John

1 Like
9

good day @jteller3. very interesting concept. i would like to know since it has been some time since your last comment here if you have solved this situation or made progress?