If I were to use the main qubes encryption mechanism in the install
Which is just the standard LUKS included in Fedora.
And then maybe have an external usb drive which is encrypted. Within
that usb drive could I have a windows hvm?
Well, your entire QubesOS installation could be on a USB drive, so sure.
There may also be a way to install only certain VMs to a different disk, but I haven't tried it. There has been discussion about it on the list. Do a search.
If so, if an attacker were to get my main sys encryption password could
that I had been mounting a windows hvm from that usb drive by analyzing
the main disk?
I take it you mean they would have not just your password, but also access to the system disk (or a copy of it or something), in which case they would have access to dom0, which means they would have access to everything you would have access to.
And, of course, <my-super-secret-windows-hvm> will be sitting right there in the list in Qubes VM Manager, but I'm sure that indications of its existence would also be scattered all around dom0 in more obscure locations, as well.
I'm assuming they wouldn't be able to get my key to the secret usb based
vm. As long
as I of course clear my memory, etc.
If they have physical access to the machine AND your LUKS passphrase, then it's trivially easy to install a keylogger in dom0 that could be used to collect the disk encryption password for the USB drive. I suppose even without the passphrase (but still assuming physical access), it is only marginally less trivial, if you don't have any sort of AEM.
It sounds like what you're looking for is a "deniable VM" of some sort. There was a discussion on here about using DispVMs with hidden TrueCrypt containers for plausible deniability. Assuming that works, you might be able to use the hidden container as the "deniable VM" storage backend, but we're now talking about nested virtualization (a VM within a VM). Xen actually supports this (according to their wiki), but this whole discussion is extremely speculative.
Why would you want to do this, anyway? What are you trying to accomplish?