To answer my own question: at least in Qubes 4.0 both U2F proxy and keepass-yubikey-login in Fedora 32 are working.
The problem I had with U2F proxy was due to a simple mistake. I forgot to change my template my sys-usb was based on to my template-clone that I use for each template (and where I install software).
I imported the keypassXC database with yubikey login from my working setup to my keypassXC in Qubes 4.0 and it worked (with attaching the yubikey).
Edit
I haven’t tested all that on my 4.1 test setup.
Both U2Fproxy and keepass-yubikey-login are working fine with Qubes 4.1 as well. 