Custom installation with aes256 and sha512 fs btrfs and luks

How can I use custom installation to have aes256 and sha512 but filesystem to be btrfs?
Tried but I was able to do so only if I reformat to ext4

look my featured post, you may get hint in there for custom install.

if you don’t care about hash, i think you could just do custom configuration then change lvm thin provisioning to btrfs.