Forensics on sys-net

I am currently having an issue with intruders hacking my sys-net on a passwordless debian-11-minimal template.

The intruders are somehow able to change the host name, that implies they are able to elevate to root.

Can anybody suggest any tools or methods that I can use to improve my defenses; or obtain further information.

My network setting is as follows:

VPN → sys-firewall → sys-net

Much appreciated…

Did you stop to wonder why an intruder that could do anything that they want as root and, especially, do those things undetected, stops to change your hostname so that you can find out that they are there and stop them? sys-net is just a packet router, with a fairly small surface of exposure. Unless they compromised the hardware somehow and used the PCIe threat vector to attack it, or somehow it’s grossly misconfigured (ssh exposed, etc.), there are not that many things to attack from the outside. A skilled attacker that manages to compromise a properly configured sys-net without compromising anything else first (hardware, hypervisor, etc.) is unlikely to change the hostname so that the user finds out and stops them.

Did you think of a more trivial cause of a hostname change? Perhaps IP address change due to DHCP refresh? Dynamic DNS? Misconfiguration?

1 Like

Thanks for your response Flavio.

I should have probably gone into more depth.

I have been under cyber-attack for almost 2 years by a group of israeli private security contractors. Whilst attempting to create defense in depth I discovered Qubes os.

In my part of the world internet is mostly commonly provided by 4G mobile broadband solutions; the contractors have been utilizing signalling system 7 to attack my mobile devices including my broadband internet.

I only discovered my hostname had been changed, whilst attempting to run vnstats in the sys-net terminal. Bash showed me my hostname was now “mint”. Incidently I have another machine loaded with linux mint using a different broadband connection with a different carrier. So I assumed I was being told something…

At the time I was using debian-11-minimal for sys-net in passwordless mode. I have since reverted back to the default fedora-dvm and the issue continued to arise.

There appears to be a 15 minute interval between connecting sys-net to the internet and the hostname changing however; as soon as I disconnect from the internet the hostname reverts back to sys-net.

Hardware compromise is unlikely, despite the fact they have been successful in the past at obtaining physical access to my devices.

In terms of configuration , I used the debian-11-minimal template out of the box , only adding qubes-core-agent-networking / manager and a few net-tools. The same is the case for my fedora sys-net. In both cases the only qube that connects to the sys-net is the VPN qube.

DHCP refresh? Dymanic DNS ? I will look into this.

In terms of Hypervisor compromise are they any resources I could look at?

Finally just to add, since I posted on this forum yesterday; the hostname changes on sys-net have mysteriously stopped. This is actually more worrying…

I still think that there is something benign behind the hostname change. Perhaps the dynamic IP address change when you connect to the internet makes the name to be updated. In any case, a compromised sys-net instance doesn’t necessarily break the overall security model in qubes (but could allow someone to sniff your traffic, if not encrypted). A few recommendations:

  1. Change your sys-net qube to make it disposable, so that any changes are volatile (restarting sys-net reverts all changes automagically);
  2. Use a VPN qube behind your sys-net (qubes → VPN → sys-net) or just sys-whonix, so that a compromised sys-net doesn’t expose your traffic;
  3. Test with a different internet connection (perhaps a 5G router separate from your qubes machine);
  4. If you are ultra-paranoid, you could change that sys-net qube by an HVM running a different operating system like openBSD, which should be relatively painless if you are familiar with BSD

However, as I said above, I suspect a benign cause and no foul play, particularly if this happens always within 15 minutes of connecting to the internet.

1 Like

As I mentioned above the issue has subsided since posting here , i now suspect that they may have accessed the qube i was using to browse to the qubes forum to make that initial post.

Your suggestions are answered below.

  1. sys-net is already disposable and volatile.

  2. I made an error when i originally described my setup. My actual setup exactly what you suggested.

  3. Unfortunately we do not have the luxury of 5G here , I had used different providers when the issue was at hand. As mentioned above the attackers are using signalling system 7, all they require is your telephone number.

  4. I will research into using bsd , I am aware it does provide a hardened sys-net.

Whilst I was replying to your initial post response , I had a firefox crash.
Unfortunately the logs for that session are missing, in addition also missing are the logs for sys-net and my vpn-qube and another unrelated qube used for mail; all were in use at the time of the crash.

I assume the logs for the individual qubes are located in the dom?

All logs are usually local to the corresponding qube. You can create a “logging qube” if you wish, and tell syslog to forward them there but that should not be Dom0. Dom0 does not have connectivity by design and putting it on the network, even to receive logs, is a bad idea.

We haven’t had “Have I been hacked?” posts around here for a few weeks now.

I always wonder if that is just some guys trolling people who dedicate their time for Qubes and/or this forum.

Basically stealing other people’s time. And wasting their own.

I think Flavio has been of immense help , and i am really grateful for his time.
If you have anything to add , i will appreciate it very much.

thank you

1 Like

In regards to the logging qube, i will proceed to set it up.

In all I think the qubes security is holding up very well.If the dom has been already compromised, they would not be wasting time performing these hit and run attacks.

Earlier today, I had a crash in my mail qube running Thunderbird.

journalctl --list-boot showed the session logs yet again to be missing, all the other previous and current logs were visible.

Ok, fair enough.

Let’s say you have serious reasons or even prove that

is preforming attacks against your devices or has successfully attacked and infiltrated one or more of your devices. That has happend tens of thousands of times so you certainly could be a target too for various reasons which you do not have to tell us. Independently thereof I certainly hope NSO Group and similiar criminals gets their asses sued off by Facebook and Apple and that there is no involvement of the US government. The later I doubt.

Anyway, you decided to use QubesOS hoping to keep at least your laptop spyware free. Makes sense to me.

A state sponsered attacker would have to use a zero-day-exploit against the network stack of your sys-net (NetVM). That is basically the only attack surface sys-net has.

Why would a sophisticated attacker change your sys-net’s hostname?

1 Like

That is the question. To change or not to change.

I am in search of answers, I could be an error on their code , which they have since corrected.

However; during these past few years of dealing with these faceless people, I have realized that its not only about surveillance; half the equation is also psychological.

They want you to know your being hacked, in order to cause panic ,paranoia; forcing you to make errors which work in their favour.

These are the only reasons, I can think of.

I am here to share my experiences which may be of use to others and acquire more knowledge , because I think Qubes is excellent.

I know someone from former East Germany who used to be under surveillance because he dared to file an “Ausreiseantrag” for his family (Peaceful Revolution - Wikipedia) back in the 80s. When he left his flat his “Verbindungsoffizier” thought it is funny to use his restroom leaving traces. Like leaving the toilet unflushed or leaving a squeezed cigarette in the rest room.

You can look upon such behaviour in different ways, i.e. you could tell yourself “how nice, they let me know they still care”. I assume that is difficult but still worth a try. By the way things got worse for his wife after they were allowed to leave to West Germany. They let her know that they have a long arm. There are a couple of good movies about that period of time.

Anyway, for startes I would like to suggest that in your case the challenge lies in setting up an OpenBSD Net-HVM as you suggested yourself:

I know there is a discussion around that topic in this forum and if you haven’t joined or read that you are most welcome to do so.

yes indeed , this was a typical tactic of the stazi.

I have read the article you mentioned. I will reread and attempt to implement along with the logging qube.

Thank you so much.

If you further want forensics (spelled prove) you need to setup another machine (separate, freshly bought hardware) and your router to wireshark all the traffic your devices create.

Since your mobile phone is your router, hmmm, you might want to use NetHunter as operating system: Get Kali | Kali Linux

I have no experience with wireshark on NetHunter. This reported issue might be a real problem, it might be fixed, I don’t know. Other issues might exist. You might be able to fix issues easily or it might take you weeks. So no guarantees from my side. Not even sure if NetHunter’s wireshark can listen to all network devices of the choosen mobile phone’s hardware. There might be better ideas how to examine your traffic.

Finding the needle in the haystack might be challenging, too.

For research on all security related topics use a dispVM with whonix-uplink on your qubes machine.

That being said… read @flavio’s estimate and advice again.

sys-net recieves it’s hostname from it’s uplink.

[user@xxx]~% hostnamectl
   Static hostname: n/a                              
Transient hostname: xxxx.localnet
         Icon name: computer-vm
           Chassis: vm 🖴
        Machine ID: xxxx
           Boot ID: xxxx
    Virtualization: xen
  Operating System: Fedora Linux 36 (Thirty Six)     
       CPE OS Name: cpe:/o:fedoraproject:fedora:36
            Kernel: Linux 5.15.63-1.fc32.qubes.x86_64
      Architecture: x86-64
   Hardware Vendor: Xen
    Hardware Model: HVM domU

I assume @expinose had a Linux Mint running on his hardware before switching to QubesOS and his router is assigning the old hostname to the recognized network adapter’s mac address.

1 Like

First of all no panic: I had the same symptom, with a sys-net where random hostname was enabled: for some reason, the computer would randomly choose the names of machines on my local network.
I was surprised at first, but it’s actually a very useful feature.
In addition, I have had no connectivity problems with this random hostname setup.
So no panic. It’s just your sys-net that is fit for the game.
As well: the others are right on something.

  1. When you will get hacked, you won’t know it.
  2. If you get hacked by Israeli hackers: you won’t know it either.

You are on the correct track to eliminate passwordless sudo and use a minimal template. I do have GUFW set at deny/deny and I do use a guix pack for network manager. Obviously I’m not there with you to know what is going on. Do you have a GPS clock or equivalent for time sync and use the time offset for RNG? Can you go behind a server? Do you have access to Beagle Boards X15 export restricted (ecrypted processor) to use as USB Armory? Do you use BIOS hardware ecryption? Do you have secure boot for your VMs? There are lots of countermeasures if you want to play this game. I would give up and go to POTS if available.

1 Like

Can you please elaborate on some of these measures? I’ve got BIOS hardware encryption (HEADS), but how does secure boot for individual VM’s work?

What does it mean to ‘go behind a server’.

Also what is POTS? Guix pack for network manager?

I don’t expect you to invest a lot of time elaborating, but some additional information to help me further research would be greatly appreciated, as this is the first time I’ve read of this.

Secure boot for VMs is done by the Ventoy developer. He sells his machines on Ebay (US) and there is an old link on this forum under i2p handle posting. I do have something…
Go behind a server means: you have a good server to protect you. BSD is legendary.
POTS: Plain Old Telephone Service (modem and is supported).

Beagle Board X15 available only inside i2p:

1 Like

Thanks for the information. Forgive me for a very rudimentary question, but by ‘having a good server to protect you’, does this mean you are connecting to the internet via - in this case BSD - a server as a form of intrusion protection?

Most of my threat model has to deal with the assumption that the network I’m operating on is actively compromised.

I tried to research furhter in to the secure boot for VM’s by the Ventoy developer and couldn’t find anything unfortunately, perhaps this is to esoteric for me at this stage. By secure boot for VM’s I was thinking of a further mechanism to verify the authenticity of each AppVM, but on reflection that would be unmanageable and not really a robust defense.

I would think HEADS would serve as the only real measurable attestation of state for the system, and anything upstream of that would be subject to too much change within each session.