While I believe you have the competences to build your own system, it needs to be said, that this is hard. It’s easy to make a mistake. It should be noted that you’ll need to trust yourself too that no mistakes have been made.
And when it comes to security, it might be better to use a ready-made product that many other people use and gets updates regularly rather than do it all manually.
Also, Qubes is more than just Xen. It has it’s own advantages, on which you can read in this thread.
To sum it all up: