Did you configure some firewall rules in the AppVM qube? This would explain why it works with whonix as a netvm, as it does not support holding firewall rules. You might have a rule allowing port 443 to everywhere, this would explain why it’s working for git over https or websites in general.
Maybe a specific ssh configuratoin in ~/.ssh/config?
Does it work if you run ssh from sys-net, I’m not saying to login from there, but at least, does it prompt for the public key? If so, could you try the same from sys-firewall?