For most VMs, you can lower it considerably. For example, most sys-* VMs are fine with ~600MB, if you’re just browsing with firefox in a VM 1.5-2GB is fine, etc. Technically, Qubes should manage the memory but in my 6 years of experience it isn’t the smartest at managing memory, especially when you have tons of VMs open.
sys-usb may need substantially more, depending on the kinds of devices and data that are being routed through sys-usb.
If you don’t really use sys-usb, except for occasional usb-pen-drive usage, it can probably be set lower than the default, but just remember you may need to adjust it later if your usage changes.