If the adversary was powerful enough to coerce an ISP, then I’d say you’d need more than open source router firmware to protect yourself.
But downsides to desktop QubesOS are still real.
Such as some modern desktops don’t include PS/2 connections, forcing you to use a keyboard with sys-usb (if you’re not careful during setup, or if QubesOS has a bug, you might get locked out). Also you need to give up a USB controller to untrusted keyboard firmware (unless you have $300 to spend on the Launch keyboard by System76 with open firmware).