Partitioning and Install on 3 NVMe disks

User Support General
1

Hello,
Back to it …
Computer didn’t change, 3x NVMe + 1x usb key for the boot
NVMe #1 = 512GB, will have a 64GB part for swap+tmp
NVMe #2 = 2Tb, LVM
NVMe #3 = 2Tb, LVM

I have 4.2, going through init, everything works fine, I select part1 to be /boot (format ext4), part2 to be /boot/efi (format efi)
Then select the 2 big 2Tb in LVM

And now I cant validate, I have error msg saying my efi can’t be ext4 (it is not) and that I need at least a / partition, but when i try to access/mod/create a partition in the LVM, i have no option to do so.

What am I missing ?

1 Like
2

which version of qubes are you trying to install exactly?

you say 4.2 but is that 4.2.0 or 4.2.1 or 4.2.2 or 4.2.3 or any of the rc?

i have had issues with some of the 4.2 variants installers when i was doing my installs in the past

but i do not use lvm for installs, i set it up as real partitions.

what is your drive configuration?
what partitions are you putting where?
what are you using for partitioning?

i have qubes on a usb with multiple drives

i have qubes with usb boot and 3 drives in raid

i used to have qubes set up as an lvm but all the guests became lvm partitions and that caused issues so that is why i went away from lvm. this way i have all virtuals in their own images that i can backup instead of trying to manipulate partitions for backups.

do you have the 2tb nvme drives in a raid?
should ask (if is raid, is it software or hardware based?)

what is the layout/setup that you are wanting?

2 Likes
3

Hi, thank you for your quick reply.
I don’t necessarily want to use LVM, it just seemed easier to have 1part across, meaning only 1 LUKS password
The 2x 2TB are not raid.
I have 4.2.2
But what I want, is to -finally- install and use Qubes.

What I’m afraid of, by experience, is to lose everything after reinstall. I reinstall a lot, as i have a very limited knowledge in debuging and fixing, so … yeah, i just reinstall.
This is not a problem with / separated from /home, but in Qubes, IDK how to make a /home which would get all qubes, while the templates and system would stay on /
To answer my (unfortunately justified) fear, I received this answer a while back, so I was relieved, but life went on and I didn’t come back to try until now.

I have a 6TB usb disk for backups
Partitions would be
/boot 640MB; /boot/efi 1GB, and several small part for LUKS keys

/ to whatever size it needs, 64GB, 128GB, …
/Qube1 128GB (work Fedora)
/Qube2 128GB (vault Debian)
/Qubes3 512GB (perso Debian)
/Qubes4 680GB (win10)
/Qubes5 128GB (WinXP)

Nothing too exotic, at least not for now, but after a few weeks I might change my mind (I ofyen do) and do a complete repartitionning and reinstall

Even though I like the idea, I’m even wondering if I should keep the /boot on usb or just get ride and leave the /boot on native disk

1 Like
4

You can have 1 password and 50 LUKS drives if you want.
You don’t have to use LVM at all.

I have 65 guests on my PC at the moment.
Bu having normal partitioning I do keep my guests backed up regularly so I won’t ever lose everything.

I even have a backup clone of my drive every week along side the daily backups of my work machines.

I’m only running a 500 GB drive at the moment though unfortunately.
I don’t use EFI either, I stick to standard.

I have reinstalled this machine about 40 times recently after doing stupid things with Qubes tryign to update it and the qubes updates breakign the isntallation.

as I have my guests on a separate drive, all I do is reinstall qubes on another drive, then go to my alternate 500 GB paired drive, create the guests on there, then copy across the images, and done, I’m up and running within 2 hours.

If I had NVME drives instead of slow SATA I’d have then set up as mirrored.

I would tell you to go normal partioning instead of LVM.

keep the boot and root system on one drive. Don’t really worry about a USB unless you need it for booting security.

And then you would have the boot, efi and root filesystem on that USB.

I would tell you just set up the 2TB as mirrored. Faster reads.
But since it’s NVME, depending on the generation and drive speeds you may not need to.

They can just be configured as a JBOD in the BIOS, then in Qubes installation, set up the JBOD as one partition mounted at “/var/lib/qubes”
for the 512 set it up as /boot 512M , EFI 1GB, / (remainder) You can always set up as / (64GB), /home (remainder)
if you want the home directory to be separate.
Do the same thing on the usb is you want to use the usb instead.

As for the backups, I have backups, and I don’t use the qubes built-in backup system, I just copy off my image files.

I rarely take a backup of my qubes (os) drive because I jsut run the installer as I think it’s faster (since I have SATA). You can have multiple pools for the systems as well. You could set one up for /var/lib/qubes and one for /qubesvms/ and have guests set up in both. there isn’t any real restriction on it.

I have a full qubes install on one of my USBs that I can run properly, not very fast, but allows me to do most things when on the run, as long as it’s on USB3, since it is slow. But without running guests, if it’s just the operating system, it runs fast enough on USB 3 with the guests on the local drive(s).

So there are many ways to do things, it’s all a matter of how you want to work it.

I don’t use LUKS because I run SATA, and since I generally have 20-30 guests running at most times, the SATA gets a bit hot and heavy.
With your NVME, you should be fine to have it with LUKS.

But if using LUKS, make sure you back up enoguh, as if it doesn’t encrypt properly and the power goes out before writing finishes, you lose it all…

I can help with a lot regarding installation and setup if you need further assistance.

2 Likes
5

P.S. I never use LVM or EFI, even on normal installs. Rarely do I ever use LUKS, no real need for it for the things I do as I have plenty of security.

2 Likes
6

Hello,
Thank you for your message.
wow, that’s a lot of user management :slight_smile:
I have … well, myself to manage, and that’s already a lot for me :-p

Right. I opted for the Qube internal BU because I’ve been lead towards it, but I’m open to any other solution, I have a dedicated 6TB disk only for that purpose (saving my Qubes, leaving the root to reinstall)

What I would like is for the laptop (bought specifically for Qube, P15 maxed to 128GB RAM, 3 screens) to boot-up to Win10 (because Catia) when USB key not inserted, boot-up to Qube when USB key inserted
USB key with /boot and /boot/efi and some smaller partition for encryption keys

Mirroring the 2x 2TB (1800GB) would prevent me from adding qubes, and would makes the BU more tricky, no ?

I’m mentioning /home as if the place in Qube for the qubes, but if they are in a different place, then I don’t need /home, I just go with the limited names I know
As for JBOD or pools and such, I would rather keep it simple, no mirroring, and for /var/lib/qubes, should I understand that qubes (not the templates, the actual running qubes) are created there ? so I would need a /var of at least (640+128+128+128+512) = 1,5TB, meaning the entire NVMe#2 for /var; while the NVMe#1 would have root and a partition for the templates ?

I’m considering LUKS because it’s standard in Qube install, and I want to make full use (and learn) at this point, I don’t really need full encryption, but I don’t want to have to do it later, so I’ll stick to it for now

That’s very kind, TY, and knowing that I’ve had this laptop since late 2021 only for Qube but never really came around to install it, I’m so discouraged that I’m really open to any help !
To be fair, I’ve had great help from @51lieal who did a nice script (formatting, creating subs, creating partition with each its own encryption, etc…) but never got it to work by myself (The bug between the keyboard and the chair)

Sumup:
Boot with usb key. No key = boot local, key-in = boot Qube, with encryption key
NVMe#1 = 1 partition root, 1 partition X, 1 partition Y,
NVMe#2 = 1 partition for each qubes (/var ?)
Preferably with same LUKS for all

1 Like
7

Not a lot of user management at al, I’m the only user.
Anyone I invite to assist with coding just log in to the guest by SSH.

I feel that having images is actually more beneficial, because I can assign the ones I want to be thick assignment to be think and have all sectors assigned, rathe than being LVM and always being thin.
If I use my normal backup method, copying off the guests, I can jsut restore them myself whenever I want, and if there are issues with the images I can recover the data.
If there is an issue with the backup drive and I lose one sector of the backup image, I can’t get anything back.
And whenver I back up or restore, I use “qemu-img convert” to do so, this means that the backed up images ae cleaner than what may be on my machine.
but I do clean up my drives images every now and again too though. Means I keep the space more available, and the images stay cleaner and faster.

I can get you my backup script if you would like, and I can send it to you via private message.
If you want any specific functionality in the backup script let me know, I keep advancing it all the time with things that I want to get done. Recently I removed about 50% of the code as it had things that were just way over the top.

Mirroring doesn’t affect anything like that.
Just means if one SSD dies, you can plug in another one, then the good one mirrors to the new one, and you actually only have te downtime of the copying of the drive from one to the other on boot.

Mirroring may mean slower write speeds, but not by much at all, maybe 1%, but read speeds are doubled in some cases.

For further information…

Having WinBlows boot up locally like that would just void the point of having Qubes for security reasons.
It can be done, but I would not recommend it.

If you wan tto have that work,
install Windows to the 512 MB drive.

In BIOS, make sure to turn off the other drives COMPLETELY when you want to boot to WinBlows.
When booting to Qubes, you should do viseversa and turn off the Windows drive port.

Personally, I have one set of drives for Qubes, and one set for WinBlows.
I change them out every time I want to switch between them.

/var/lib/qubes (ref as root) contains all your qubes.
contains…
/vm-templates (templates directory)
/appvms (your normal use guests)
/servicevms (Where your net vms and all are stored)
/backups (Where it stores all your changes to the XML file.)
/updates (I think the qubes (Dom0) updates?)
/vm-kernels (The kernels that are available to use for the virtuals in PV/PVM mode)

LUKS isn’t the standard in the Qubes Install that I know of. Only in 3.2 did I find that LUKS was the default for install. And that was only in one variant.
Might be different in the latest itteration of the ISO thoguh, I don’t know, I will have to look at that.

Make sure the 2TB Mirror is done in the BIOS. Should not be a software mirror.

USB
/boot (512MB) No LUKS here
/ ( > 24 GB) Optional encryption. I would advise against it for the USB.

NVME 2TB Mirror
/var/lib/qubes (Fill) If you want LUKS, go for it.

Just use the partition manager to greate the partitions and make it work.
That is how you really should set up the layout.
Just have the virtuals on the 2TB.

Hope this helps. If you have any questions, don’t be aftraid to ask.

2 Likes
8

I don’t think so, since Qube’ boot is on a separate medium, and / is FDE, there is no way for windaube to access it in any way

As for the BU, I will first try with the embedded tool, and if it doesn’t fit, I will look for other options, but first …

OK, so let’s make a partition/install plan:

USB 2GB:
SDD1: /boot (640Mb) exFAT, non-encrypted
SDD2: /boot/efi (1024Mb) exFAT, non-encrypted
SDD3: /key1 (16Mb) to hold encryption key
SDD4: /key2 (16Mb) to hold encryption key (now useless if all are with a unique key)
SDD5: /key3 (16Mb) to hold encryption key (now useless if all are with a unique key)
=> Boot sequence set to USB before drives, so USB-Key-in = boot to Qubes (SDD1), USB-key-out = boot to win10 (SDA1)

NVMe #1 512GB (478GB)

SDA1: /boot (768MB; GRUB2 from a previous Suse install, now only Win10 entry)
SDA2: /boot/efi (510MB; Grub2, idem)
SDA3: 372GB for C:
SDA4: /LVM (48GB) (Since this NVMe is smaller=cheaper than the other big 2, I set the heavy R/W to here)
… Swap (16Gb) for Qube
… TMP (32GB) for Qube
SDA5: /W.Swap (16GB) for Win
SDA6: /W.TMP (32GB) for Win

  • total 1.5+372+48+48 = 470GB

NVMe#2 1863GB

SDB1: 128GB for Win D: VeraCrypt
SDB2: / (128GB for root)
SDB3: /vm-templates (128GB templates directory)
SDB4: /appvms (128GB for normal use guests)
SDB5: /servicevms (64GB for stores NET vms and all such)
SDB6: /backups (64GB for stores all your changes to the XML file.)
SDB7: /updates (64GB for qubes (Dom0) updates? TBC)
SDB8: /vm-kernels (128GB The kernels that are available to use for the virtuals in PV/PVM mode)

  • total 128+128+128+128+64+64+64+128=832GB

NVMe#3 1863GB

SDC1: /var/lib/qubes (BTRFS? EXT4? XFS?) 1863GB encrypted.
… qubes#1 Perso (Debian 128GB)
… qubes#2 Work (Fedora 128GB)
… qubes#3 Unsecure (Fedora 64GB)
… qubes#3 Vault (Debian 64GB)
… qubes#4 WinXP (64GB)
… qubes#5 Win10 (420GB image from existing)
… qubes#6

  • Total = 1863GB

or:

Mirroring NVMe#2 1863GB+NVMe#3 1863GB in RAID1
Total capacity by half = 1800GB
SDB1: 128GB for Win D: VeraCrypt
SDB2: / (64GB for root)
SDB3: /vm-templates (64GB templates directory)
SDB4: /appvms (128GB for normal use guests)
SDB5: /servicevms (64GB for stores NET vms and all such)
SDB6: /backups (64GB for stores all your changes to the XML file.)
SDB7: /updates (64GB for qubes (Dom0) updates? TBC)
SDB8: /vm-kernels (64GB The kernels that are available to use for the virtuals in PV/PVM mode)
SDB9: /var/lib/qubes (BTRFS? EXT4? XFS?)
… qubes#1 Perso (Debian 128GB)
… qubes#2 Work (Fedora 128GB)
… qubes#3 Unsecure (Fedora 64GB)
… qubes#3 Vault (Debian 64GB)
… qubes#4 WinXP (64GB)
… qubes#5 Win10 (420GB image from existing)
… qubes#6

  • total 128+64+64+128+64+64+64+64+128+128+64+64+64+420=1444GB, it fits and I still have 6,5x 64GB (or 128+128+96+64) for other qubes

But something feels not right, having to pre-determine qubes, hence me wondering about BTRFS, so I would have 1 large partition in which I can create qubes as will ?

1 Like
9

What on earth are you trying to do?

USB looks fine. If you want encryption keys on there, that’s fine too.

If you want to have an insecure and unsafe system, then you can go with that structure.
Yes, windows can still access and see the drives because they are enabled.
I can access the drives using Windows.

So you can choose either the secure and safe way, or the insecure way. Up to you. But honestly, I wouldn’t have the PC dual booting like that at all.

To use Windows, I just run it under Qubes for things, or else I just run a variant of WINE or Crossover or any of the others out there that are based on WINE to run Windows applications under Linux.

I have attached a lit of my guests to show you how the /var/lib/qubes folder looks.

Okay, I can’t upload a txt file, will have to just paste it.

And yes I have had to annonomise the data to protect some information, so there are some removed from the list.

So, in the end the decision of what you want, secure system or not is up to you.
But people always say, dual booting for Qubes is rhediculous.

You could always just have a boot sector to load the Windows virtual image to run on the machine and not have access to the local system as it would be a virtual system. (When the machine doesn’t have the USB key in it.)

This is what the file sizes look like…

Here is my structure for /var/lib/qubes (most of it)

2 Likes
10

Having the bootable Windows does mean that it should have Qubes bootable always, but will boot to a windows virtual that runs in a virtual environment.
This isn’t too hard to do if you want to have Qubes booting on the local drive.

2 Likes
11

I’ll do an installation to build something like this so I can provide you the accurate details and configurations to make something like this happen for you.

Will maintain your security of using Qubes.
Will allow you to have Windows running fully.

I mean you could use whatever actual system you wanted. Doesn’t have to be Windows. Could even be another Linux variant.
Would have to configure everything within the properly booted Dom0 though.
And only can change configurations within the real Dom0.

But this system would be workable and manageable completely.

And Dom0 would also set up what devices get passed to the system when it’s booted.
Essentially, all USB Ports would pass to it.

I don’t know if you would want to have a Sys-USB type system or not?
I think that would be the deciding factor of multiple ways to go with things in relation to the USB side.

Would you like this as an option?

2 Likes
12

Trying to finally get to install and use Qube. But with constraints, and wishes.
i.e: I have to keep Win native. I bought a USB-C NVMe to transfer Win to it (the current image), but never achieved :frowning:
But that is way beyond this topic (Maybe I should open a new one : “How to install and use Qube with frustrating constraints”)
My previous msg with partitioning sugestion was just my autistic brain trying to make use of the information you kindly gave me, not at all a “ok, I know now, I will do it that way” on the contrary, it is to be criticized and hopefully corrected

Thank you.
Yes, that setup is from the previous tentative

That’s obviously not why I want to use Qube. And I’m sorry if I sound like I do.

Yes, obviously any OS have access to the drives, and whatever is on it, but Win won’t be able to access the data itself, nor write anything in it, since FDE.

I will have a Win10 qubes, but it won’t run Catia
Now that I’m writing this, I’m questioning having a Win10 qubes as the only use (after transferring everything, files and daily activity to Qube) for it, is Catia, occasionally, when I have to finish work transferred from the office (and some games I confess)

Thank you, but that doesn’t … hum … I don’t know yet what to do with all that information

Yes, that is the reason why they both have a their own boot (partition, file, etc …)
So it is not a “dual boot” per-se

That is indeed interesting !
But I would still have to have a native install to run Catia (and games, I confess)

A file of what ? a Back-up ? oh wow, only 2Gb ?!! I sure have plenty room in my 6TB :slight_smile:

So what I have and need:
a 2GB USB key for Qube’s boot
a 512GB USB-C NVMe where I wanted to transfer Win10 “native”
a 512GB NVMe where I currently have Win, and since the 512GB is cheaper than the 2TB, I use it to hold the swap+tmp (in an encrypted LVM)
and a pair of 2TB which, after some math, would be indeed able to hold my wished setup even if in RAID1, meaning half it size

By experience, I have to add that I want/need a sure and easy way to backup my qubes, as I often mess up and have to reinstall. That where the idea to have separate partitions comes from, based on the /home separate partition.
So I would say that I would need a partition of TBD.GB for /; and a partition of REMAINING.GB for the qubes
But that is me not knowing the details, and maybe I should have a separate partition for the templates ? I sure will need a separate one for the backups, so I can transfer the entire content to the external disk, but there again …

1 Like
13

Since my first try in 2018, every further times I tried to install Qube, I always opted for a joint sys-net & sys-usb, as separated it ddin’t work properly, I think I didn’t have Internet or something.

I’m open to any suggestion ! :slight_smile:

1 Like
14

First steps done,
Removed any previous Linux entries from SDA2 /boot/EFI, back to Win10 only,
As I no longer need SDA1 /boot so formatted empty.
And I gave another try at transferring it all to the USB NVMe but still can’t boot Win, the splash screen goes up, and it spin until it freezes :frowning:

BIOS RAID1 the two 2TB NVMe = done !

Ready for next step, partitioning for Qube :slight_smile:

1 Like
15

New try (suggestion to be criticized):

USB 2GB:
SDD1: /boot (640Mb) exFAT, non-encrypted
SDD2: /boot/efi (1024Mb) exFAT, non-encrypted
SDD3: /key1 (16Mb) to hold encryption key
SDD4: /key2 (16Mb) to hold encryption key (now useless if all are with a unique key)
SDD5: /key3 (16Mb) to hold encryption key (now useless if all are with a unique key)
=> Boot sequence set to USB before drives, so USB-Key-in = boot to Qubes (SDD1), USB-key-out = boot to win10 (SDA1)

NVMe #1 512GB (478GB)

SDA1: /boot (768MB erased, empty)
SDA2: /boot/efi (510MB; Win10 only, any Linux entry removed)
SDA3: 372GB for C:
SDA4: 1GB WinRE
SDA5: /LVM (48GB) (Since this NVMe is smaller=cheaper than the other big 2, I set the heavy R/W to here)
… L.Swap (16Gb) for Qube (knowing I have 128GB RAM)
… L.TMP (32GB) for Qube
SDA6: /W.Swap (16GB) for Win
SDA7: /W.TMP (32GB) for Win

  • total 1.5+372+48+48 = 470GB

NVMe#2+NVMe#3 (RAID1) 1863GB

SDB1: 64GB for Win D (Document kept local, not transferred to NAS)
SDB2: / => BTRFS ? 256GB ? => TBC
Includes: /; /servicevms; /appvms; /updates; /vm-kernels; /backups
SDB3: /vm-templates (128GB templates directory)
SDB4: /appvms (768GB for normal use guests)

==> total 64+256+128+768=1216GB (out of 1863GB)

  • Win boot and Qubes boot have no interaction, on two separate media
  • Qubes is entirely encrypted,
  • Only /appvms and /vm-templates will be backed-up to external 6TB usb drive, in case (very probable) of crash, / will be re-installed, these two will be re-attached

Would that work ?

1 Like
16

I guess my main question is:

If I isolate

  • /vm-templates (what size would be good ?)
  • /appvms (in a biiiig one)

Would that be enough ?
If I format & reinstall / and then re-attach these two partitions, then everything will be back on running ?
or do I need something else in either of these two separate partitions ? (to be “safe” from re-formatting /, and to be easily copied to backup)

1 Like
17

My original thoughts, based on what you were after won’t change.

I have provided the optimal solution to best benefit you and what you are after.
It provides ease of use for Windows and Qubes, so the 2 systems are separate and everything remains reasonably secure.

If you want to alter things and make things harder and more complicated for yourself, that is fine.

2 Likes
18

I haven’t read most of this thread but I’d suggest /boot be 1GiB (that’s what the standard install does) and it should be ext4.

You might want to check this guide out since parts of it (/boot and /boot/EFI on a USB) match

Using blivet advanced install option might also help

2 Likes
19

TY !
I will check this out :slight_smile:

Edit: Yes, Blivet is already what I’m using, and the USB key is already set as suggested.
Useful link for the LUKS header transfer still !

1 Like
20

I’m sorry, but I don’t see where you provided a sollution, an optimal one at that ?
You have provided me with a lot of information about your setup, and I’ve tryied to make something out of it, using what I could understand, and that’s why I’m focusing on /vm-templates and /appvms
If I just blindly follow (copy-past) your set-up, I don’t fully understand, I will need help everytime I have even a little glitch, but I would like to fully understand and know what I’m doing, all the while sticking to my “view” plan and desire of a partitionning.
You suggested RAID, I was against, but then once I did the math, I changed my mind and agreed with you
I’m hopping for the same for the partitionning

…

1 Like