Is sys-net the weak link in our systems' security?

General Discussion
1

An XSA in the most recent QSB* reminded me to ask about the vulnerability of sys-net and whether there should be increased efforts to make this particular sys-VM run from a unikernel, microkernel, or something else with a minimal attack surface, especially since it really only does one simple job (I think).

In brief: My gut feeling since before this QSB has been that sys-net presents the most accessible target for a remote attacker who might want to compromise the whole system as it isn’t (can’t be) firewalled and runs as HVM with hardware access. This QSB supports this. However, since I’m not a technical person and my understanding of sys-net might be wrong, my intuition might be completely off.

 

*XSA-337: A malicious HVM with a PCI device (such as sys-net or sys-usb in the default Qubes OS configuration) can potentially compromise the whole system.

2 Likes
2

An XSA in the most recent QSB* reminded me to ask about the vulnerability of sys-net and whether there should be increased efforts to make this particular sys-VM run from a unikernel, microkernel, or something else with a minimal attack surface, especially since it really only does one simple job (I think).

It’s a “kind of” simple job. If you are just providing Ethernet in and
network out, the only real complex bit is the Ethernet adapter driver
(still not trivial).

But if you are expecting to be able to use WPA3-Enterprise secured WiFi,
there is a lot more work which would need to go into a uni-kernel sys-net.

This isn’t to say it wouldn’t be great to see, just that it is more
complex than (for example) the mirage firewall because you are now
working with real hardware.

In brief: My gut feeling since before this QSB has been that sys-net presents the most accessible target for a remote attacker who might want to compromise the whole system as it isn’t (can’t be) firewalled. This QSB supports this. However, since I’m not a technical person and my understanding of sys-net might be wrong, my intuition might be completely off.

The question of firewalling is not really a useful one. You shouldn’t be
running services within sys-net, meaning that any packets sent to it
that are not responses will be rejected by the kernel anyway. Responses
will be forwarded to sys-firewall. This basically leaves kernel-space as
the main place to attack, and finding exploitable bugs in the Linux
kernel TCP/IP stack is somewhat rare. The Ethernet driver might be worse
in this respect though.

I think the short answer is that your intuition is right, but it’s a
harder problem to solve that it seems.

2 Likes
3

IMO since WPA3 is not significantly more secure than WPA2, Qubes users should be encouraged to avoid using wifi anyways. Just improving the security of ethernet input would go a long way.

Quick question about that: I ran systemctl in my disposable sys-net based on Debian minimal with only qubes-core-agent-networking and qubes-core-agent-network-manager installed. It printed out a long, long list of services, with a lot of them being network related. Is this normal?

If I were a wealthy/powerful attacker, this would be the first place I would pour resources into sniffing out exploits–what I find would of course be kept under wraps.

4

IMO since WPA3 is not significantly more secure than WPA2, Qubes users should be encouraged to avoid using wifi anyways. Just improving the security of ethernet input would go a long way.
That was meant as an example. Any WiFi would be difficult. As would
802.1x secured wired networks to a lesser extent. But yes, I would also
like to see even the simplest version of this.
Quick question about that: I ran systemctl in my disposable sys-net based on Debian minimal with only qubes-core-agent-networking and qubes-core-agent-network-manager installed. It printed out a long, long list of services, with a lot of them being network related. Is this normal?
Seems pretty normal to me, there will be network management daemons, xen
drivers, qubes agents and a whole bunch of other default stuff running.
If I were a wealthy/powerful attacker, this would be the first place I would pour resources into sniffing out exploits–what I find would of course be kept under wraps.

This has happened before. See EternalBlue. Worth noting that the premise
of Qubes is that you would need more than this one, expensive exploit to
get to any useful data. To get to a default vault VM for example, you
would need both this and a weaponized Xen exploit. The theory being that
even a nation state actor is unlikely to be able to find and hold two
exceptionally valuable exploits in unrelated code bases.

The TCP/IP code in Linux is one of the few places where this theory
falls down. All VMs by default use the same kernel, meaning that the
same kernel TCP/IP exploit is likely to affect every VM with networking.

There are two questions that are worth asking about this:

  1. Is there anything you can do right now (aside from write the proposed
    unikernel)?

Disposable sys-net with a unikernel sys-firewall would be a big step up.
This removes the “same kernel” attack above and makes persistence harder.

  1. Threat Modeling: Are you really such a valuable target that a
    nationstate actor would potentially waste two highly valuable zero day
    exploits on you? This is likely not the case for most users of QubesOS,
    though it is the target audience to some extent.
2 Likes
5

Thinking about what a sys-net (router) ip stack encounters, I have to
wonder if there is much attack surface there. What is the most complex
part, the connection tracker?

Qubes was created more as a response to insecure hardware interfaces and
their drivers, and that’s why sys-net exists and uses pci passthrough on
network cards… The major risk is from ‘remote’ attackers in your radio
(or wired LAN) range that can use exploits below the IP layer to gain
control of the NIC’s microcontroller (which has DMA access to the
system). This type of attack has been demonstrated effective against
wifi under normal OSes.

IOMMU containment of NICs in sys-net prevents attackers from using the
NICs to directly access the rest of your computer. However, indirect
access may still be possible via side-channel attacks which is a whole
topic in itself.

1 Like
6

A question I have about exploits is how they’re considered ‘spent’ by those who wield them. The threat of ‘wasting’ exploits come from detection and reverse engineering, with the latter being the fatal blow (detection alone might provide pieces of the puzzle, sometimes intentionally misleading). As far as I can tell, many sophisticated modern malware have ways to obfuscate and self-destruct to protect the value of the investment(s) contained within.

However, with less savvy targets like journalists and whistleblowers, this isn’t even a concern unless they have experts analyze the compromised VMs after the fact (which requires them to have detected the intrusion in the first place). If you’re a target who’s been driven to use Qubes, the value of the potential payoff is worth the risk, as you’ve noted. However, even when dealing with cybersecurity veterans, I think sophisticated malware can avoid reverse engineering most of the time. Then again, I’m not incredibly well-informed on this.

 

I don’t quite understand this–are you referring to ‘below ring zero’ access (probably using the wrong term) facilitated by Intel ME and AMD PSP?

 

And this is why I’m worried about sys-net–because side channel attacks have been the talk of town since the reveal of Meltdown and Spectre (I held off on learning Qubes for two years since this seemed to be pretty serious for the core security concept of the OS) as well as Rowhammer.

Anyways, it’s hard to articulate my gut feeling, but the gist of it is that the Linux kernel is a highly sought-after target for obvious reasons and having sys-net based on it is risky, despite what the technical limitations for an attacker might seem to be–‘unknown unknowns’ and all that. This beachhead, disposable or not, can be used as the staging ground for a system-wide attack that may or may not use side-channel attacks. Because of this, there should be an ethernet unikernel for those who feel they need it, and those users really shouldn’t be using wifi anyways, obviating the need for a wifi unikernel that would significantly complicate it. I hope I don’t come across as nitpicky and demanding, especially since I can’t really contribute to the development of such a unikernel (maybe I can help with non-technical tasks?)

However, maybe I’m overestimating the amount of protection firewalling provides, as firewalls shouldn’t be much of an obstacle for an actor in possession of side channel attacks, Xen zero days, and a kernel TCP/IP exploit.

7

No, I mean the network layer below IP packets: Ethernet frames that are
designed to exploit a vulnerability in the NIC’s firmware and thus take
over the NIC’s microcontroller. If this happens, sys-net will protect
against direct DMA access since the NIC is contained by the IOMMU. This
leaves side-channel attacks as the only option, although they are harder
to pull off.

But with side-channel attacks the threat can come from just any browser
(or other net app) in your system being exploited for that purpose.
IIRC, just having data separated in different VMs can mitigate some
side-channels, and Qubes also takes the precaution of turning off
hyper-threading which is a bad source of side-channels especially on
Intel processors.

8

I have always considered sys-net to be the weakest link, and I think this
is in line with Qubes philosophy.
It’s reasonable to treat sys-net as part of the untrusted
infrastructure. Nothing valuable should be passing through sys-net in
the clear.

1 Like
9

A question I have about exploits is how they’re considered ‘spent’ by those who wield them. The threat of ‘wasting’ exploits come from detection and reverse engineering, with the latter being the fatal blow (detection alone might provide pieces of the puzzle, sometimes intentionally misleading). As far as I can tell, many sophisticated modern malware have ways to obfuscate and self-destruct to protect the value of the investment(s) contained within.

However, with less savvy targets like journalists and whistleblowers, this isn’t even a concern unless they have experts analyze the compromised VMs after the fact (which requires them to have detected the intrusion in the first place). If you’re a target who’s been driven to use Qubes, the value of the potential payoff is worth the risk, as you’ve noted. However, even when dealing with cybersecurity veterans, I think sophisticated malware can avoid reverse engineering most of the time. Then again, I’m not incredibly well-informed on this.

You’re right that plenty of people who use Qubes do not have the skill
or tools to do this analysis. Even still, we are talking about exploits
which would provide kernel mode access to literally any Linux system,
just by sending a packet through the system. This is an exceptionally
valuable exploit. Even if you are not expected to be able to detect it,
questions would be asked about the risk of such detection.

The problem with exploits is that unlike malware, they have to conduct
their attack mostly in the open. It is very difficult to obfuscate the
initial exploit because it must be interpreted and run in code which is
designed not to run it. Any obfuscation just makes the exploit less
likely to work, or worse, more likely to miss-behave and cause obvious
system instability.

Anyways, it’s hard to articulate my gut feeling, but the gist of it is that the Linux kernel is a highly sought-after target for obvious reasons and having sys-net based on it is risky, despite what the technical limitations for an attacker might seem to be–‘unknown unknowns’ and all that. This beachhead, disposable or not, can be used as the staging ground for a system-wide attack that may or may not use side-channel attacks. Because of this, there should be an ethernet unikernel for those who feel they need it, and those users really shouldn’t be using wifi anyways, obviating the need for a wifi unikernel that would significantly complicate it. I hope I don’t come across as nitpicky and demanding, especially since I can’t really contribute to the development of such a unikernel (maybe I can help with non-technical tasks?)

It absolutely is a highly sought-after target. But this provides some
protections too. There are plenty of organisations trying to secure the
kernel as well.

This goes back to the most secure computer on the planet. It’s powered
off, encrypted, no-one knows the key and the vault it’s locked in is
lead lined. But no beachheads. At some point, if you want a useful
computer, you need to expose something. While sys-net seems quite
exposed, it’s still one of the more secure options out there.

So far as helping with development, there are plenty of non-technical
options. Things such as threat modelling (which is basically this thread
with a few extras), and community development are probably the most
useful at this early stage.

1 Like
10

@unman, I have seen in your notes that you run openbsd netvm, Do you still do the same?
If yes Why not to integrate it in default Qubes installation?

1 Like
11

I do still use this configuration.

There are two reasons not to integrate as default:

  1. hardware support
  2. user support - users (many new to Linux) seem to have enough trouble
    coping with Fedora/Debian netvm without throwing BSD in to the mix.
12

Have you got a link to these notes? I’d be quite interested in how this
is configured.

13

Problem is I was not able to run openbsd as HVM at all yet. I haven’t figured out networking part in simple Openbsd hvm although I was able to run Windows as HVM.
So no option to progress on this guide.
Make a try and if possible create some guide for persons who are not very competent in tech stuff.

1 Like
14

I was thinking more along the lines of being much more easily targetable via your ISP IP address (since that’s usually directly linked to you) than via VPN/Tor, since attacking you through those channels requires a lot more observation and planning (e.g. figuring out where the traffic exits and sending things your way that you might or might not open or placing things on sites that you trust). I think the term for the ISP one is ‘directed drive-by attacks’.

This is where my lack of formal training in infosec begins to show–I know that vulnerabilities are exploited by malware via exploit(s), but have never thought of exploits as single items akin to malware in independence. Are they what scripts are to programs? Why would sophisticated actors use exploits instead of malware that includes them (aside from cost of development), especially if standalone exploits are more easily detectable, which puts the investment at risk?

This reminds me of ‘the most secure computer on the planet’ argument later in your post–a powerful exploit is worth a lot less if you can’t even use it on a bunch of muggles.

By community development I presume you mean being a constructive poster in forums? Since I think I’m already doing that, is there anything else I can help with?

This sounds like something that’d tide me over until a unikernel hopefully comes. Maybe this will spawn a thread on how to use openBSD as netVM. Thanks for bringing this up.

@unman are there any critical pitfalls to avoid when configuring an openBSD netVM? Like, what might a newbie to openBSD misconfigure that might lead to them being even more vulnerable than using Linux?

15

“Community development” usually refers to writing code and submitting patches/PRs to the Qubes codebase. However, there are a also a lot of other ways to help. :slightly_smiling_face:

16

This is where my lack of formal training in infosec begins to show–I know that vulnerabilities are exploited by malware via exploit(s), but have never thought of exploits as single items akin to malware in independence. Are they what scripts are to programs? Why would sophisticated actors use exploits instead of malware that includes them (aside from cost of development), especially if standalone exploits are more easily detectable, which puts the investment at risk?

That’s not a bad way to think about it, but there is some more depth.
Malware has to be delivered somehow. Complex malware which uses
spohisticated obfuscation and hiding techniques is usually a secondary
or tirtary payload. Something has to get onto the system in the first
place to download and execute it. This “something” is the exploit and
it’s shellcode. By virtue of the fact that it is first and the target
computer does not want to run it, the exploit often cannot contain any
sophisticated anti-detection techniques. Most of the time, it is an
otherwise legitimate packet or file, which happens to contain something
which will trigger the invalid program state.

While it is possible to do some obfuscation, you are limited by what the
machine on the other hand can interpret.

This reminds me of ‘the most secure computer on the planet’ argument later in your post–a powerful exploit is worth a lot less if you can’t even use it on a bunch of muggles.
This is exactly the case, but the other question to ask is why target
the “muggles”?

By community development I presume you mean being a constructive poster in forums? Since I think I’m already doing that, is there anything else I can help with?
Andrew’s comment (and example) answer this, but I was meaning helping to
bring people together around the issue, motivating those with technical
skills to solve the problem.

1 Like
17

Ah so it’s the spearhead/point-man/vanguard who has to lower the castle drawbridge to let the actual raiding party in, so he has to stay lightweight. Thanks for taking the time to educate me on this.

By ‘muggle’ I’m referring to people who are not trained in infosec and not protected by those who are trained. There are plenty of these people with valuable information worth stealing (business executives, politicians, journalists, even software engineers).

Sounds good. So I’ll just keep doing what I’ve been doing. I might work on overhauling (or just editing) the documentation simply because I think good documentation and education provides the most bang for the buck (in my case, zero bucks) in terms of accessibility.

18

My solution WAS anonsurf in sys-net. It uses pandora to clean memory and other apps. You start it and stop ip.
I did not bother to switch to the new version:

You can even use the old Kali anonsurf on github!

19

I use sys-net in PV mode, which increases the attack surface even more. Not willingly, but because my components force me to.

20

Qubes is designed under the assumption that sys-net is already compromised, which is why it’s untrusted from the start. However, there are things you can do right now to improve security further:

  • Make it a DisposableVM.
  • Pause it or shut it down when performing sensitive operations in other VMs (to mitigate speculative execution zero day risk).
  • As unman said, make sure nothing it doesn’t have access to any important plaintext, for example, by always using HTTPS when connecting to important websites.
7 Likes