Salting your Qubes

unman · February 10, 2021, 3:02pm

Hi,

I'm in the process of uploading various salt configurations, which you
may find interesting.
The files are in GitHub - unman/shaker.

There's configuration for salting various useful stuff:
A caching proxy,
A version of split-ssh,
Multiple sys-usb using various controllers (specific to x230 model),
Kali template,
A qube for building Qubes,
A qube for building/flashing coreboot,
A multimedia qube,
A file sharing qube,
Adding support for Office or Windows filesystems to templates,

with more to come.

This isn't sophisticated salt - most of these are examples from training,
and are deliberately simple. Some are old, and may need tweaking.
They are also almost all based on a debian-10-minimal template
(naturally).
Even if you have no experience with salt you should be able to read the
files and understand what they are going to do.

I hope you use them to get started with using salt in Qubes.

I'm in the process of packaging these, and will host them at
https://qubes.3isec.org
Although it's simple to produce packages that will actually implement
the states, (as they do in initial Qubes setup), I prefer not to do
this - mainly because I think it better for users to see what's been
installed and what actions will be taken. So the packages will provide
the salt formula, for you to review, change and implement as you will.
(Change my mind if you like.)

Happy to take suggestions for other configurations, or features.

unman

pillule · May 25, 2021, 7:05pm

Hi unman.

I am learning to "salting my Qubes", what decided me is your recipe for
apt-cacher-ng (sometimes my bandwith really matters).

So far I get some things working such as an automated
template/templateDispVM/staticDispVM for firefox and its configuration with some
extensions.

So with a little more confidence, right now I am installing your apt-cacher-ng
recipe.

I read from the docs of apt-cacher-ng :
"6.3 Fedora Core
Attempts to add apt-cacher-ng support ended up in pain and the author lost any
motivation in further research on this subject. "

Do you get it working ?

In all cases, can you explicit how you use apt-cacher-ng only for debian
derivatives? The RPC policy you provided will apply to all templates, and calls
to https from fedora will be denied, preventing them to update ; or I am
missing something ?

Happy to take suggestions for other configurations, or features.

unman

Sometimes ago you mentioned on the mailing list to use snort and tripwire.
I don't know if it is trivial to deploy but that is surely one of the next
things I will look at.

pillule · May 25, 2021, 8:25pm

pillule <pillule@riseup.net> writes:

Hi unman.

I am learning to "salting my Qubes", what decided me is your recipe for
apt-cacher-ng (sometimes my bandwith really matters).

So far I get some things working such as an automated
template/templateDispVM/staticDispVM for firefox and its configuration with some
extensions.

So with a little more confidence, right now I am installing your apt-cacher-ng
recipe.

I read from the docs of apt-cacher-ng :
"6.3 Fedora Core
Attempts to add apt-cacher-ng support ended up in pain and the author lost any
motivation in further research on this subject. "

Do you get it working ?

In all cases, can you explicit how you use apt-cacher-ng only for debian
derivatives? The RPC policy you provided will apply to all templates, and calls
to https from fedora will be denied, preventing them to update ; or I am
missing something ?

Happy to take suggestions for other configurations, or features.

unman

Sometimes ago you mentioned on the mailing list to use snort and tripwire.
I don't know if it is trivial to deploy but that is surely one of the next
things I will look at.

I have been fooled by :

github.com/QubesOS/qubes-issues

Updating via Salt falsely claims to succeed when it actually fails

opened 01:33PM - 07 May 21 UTC

closed 01:37AM - 10 Jul 22 UTC

andrewdavidwong

T: bug P: critical C: mgmt security r4.1-dom0-stable diagnosed C: updates

**Qubes OS version** `R4.0` **Affected component(s) or functionality** … Updating via Salt (Qubes Update tool / `qubesctl`) **Brief summary** When updating via Salt (Qubes Update tool / `qubesctl`) fails, it falsely reports that it was successful. A user could go months or longer without receiving any security updates, while Qubes claims that every update is completing successfully. This can lull users into a false sense of security and constitutes a potential attack vector. **How Reproducible** 100% **To Reproduce** Steps to reproduce the behavior: **Note:** I'll use dom0 in these examples, but the same things happens with templates. 1. Unplug your ethernet cable, disconnect from Wi-Fi, or intentionally set up your system so that the update process will encounter an error. One way to do the latter is by adding `repo_gpgcheck=1` to all enabled repos in `/etc/yum.repos.d/qubes-dom0.repo`, which will currently cause cause a curl error when trying to update, unless you have implemented a specific workaround (see https://github.com/QubesOS/qubes-issues/issues/6275). 2. Try updating via Salt. Since we know (from Step 1) that the update cannot possibly succeed, we should expect some indication that the update encountered a problem. At the very least, the update should not claim to have succeeded, since it couldn't have done so. ``` $ sudo qubesctl --show-output state.sls update.qubes-dom0 local: ---------- ID: /etc/yum.repos.d/qubes-dom0.repo Function: file.replace Result: True Comment: No changes needed to be made Started: 05:21:34.989474 Duration: 5.357 ms Changes: ---------- ID: /etc/yum.repos.d/qubes-templates.repo Function: file.replace Result: True Comment: No changes needed to be made Started: 05:21:34.994933 Duration: 1.083 ms Changes: ---------- ID: update Function: pkg.uptodate Result: True Comment: System is already up-to-date Started: 05:21:35.509286 Duration: 245241.606 ms Changes: Summary for local ------------ Succeeded: 3 Failed: 0 ------------ Total states run: 3 Total run time: 245.248 s ``` The update falsely claims to have succeeded (`System is already up-to-date`, `Succeeded: 3`), even though we know that's impossible. 3. Now, let's see whether `qubes-dom0-update` fares any better: ``` $ sudo qubes-dom0-update Using sys-update as UpdateVM to download updates for Dom0; this may take some time... Warning: Enforcing GPG signature check globally as per active RPM security policy (see 'gpgcheck' in dnf.conf(5) for how to squelch this message) Fedora 25 - x86_64 - Updates 0.0 B/s | 0 B 04:00 Errors during downloading metadata for repository 'updates': - Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f25&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org] [Errno 2] No such file or directory: '/var/lib/qubes/dom0-updates/var/cache/yum/x86_64/4.0/metadata_lock.pid' ``` `qubes-dom0-update` accurately reports that it encountered an error -- exactly the sort of error we would expect from lack of internet access. **Expected behavior** When updating dom0 via Salt (Qubes Update tool / `qubesctl`), any underlying errors should be reported to the user so that the user is alerted that the update was not successful. This could prompt the user to re-attempt the update and try to fix the problem. This is important, because missing out on security updates can be fatal to the security of any operating system, including Qubes. **Actual behavior** Updating via Salt reports success regardless of whether it actually did anything. Even when we physically cut off internet access, the update blithely reports `System is already up-to-date`. It has no way to know this, since it can't even reach the update servers. It's no exaggeration to call this a bald-faced lie. Users could go for months without getting any security updates while Qubes lies that updates are completing successfully the whole time. An attacker who manages to block the user's updates (e.g., connection to update servers) needn't bother to do anything else, since the user will notice no difference in the system, because there is no change to notice. **Additional context** Unfortunately, the answer is not as simple as just exclusively using direct commands like `qubes-dom0-update`, `dnf update`, `apt update`, and their variants. As explained in https://github.com/QubesOS/qubes-posts/pull/79, there are strong reasons to shift the status quo away from using those direct commands and instead to recommend updating via Salt. I wrote that post before discovering this bug. This certainly changes things. Our one saving grace is that we've established a system for reporting QSBs and made it clear to the userbase that it's important to keep up with them. In practice, however, it's likely that some not insignificant portion of the userbase fails to heed QSBs (either at all or in a timely fashion). Moreover, this doesn't help with any upstream security updates for TemplateVMs and StandaloneVMs for which we don't issue QSBs. In addition, we don't want to remain reliant on external messaging like QSBs forever. Qubes OS should be self-sufficient, and our ultimate goal is not to require the user to monitor any external channels (see https://github.com/QubesOS/qubes-issues/issues/3430). **Relevant [documentation](https://www.qubes-os.org/doc/) you've consulted** https://www.qubes-os.org/doc/salt/ **Related, [non-duplicate](https://www.qubes-os.org/doc/reporting-bugs/#new-issues-should-not-be-duplicates-of-existing-issues) issues** #6275 -- I stumbled upon this while testing the `repo_gpgcheck=1` thing mentioned in Step 1 above. When I noticed that updating via Salt was unchanged by the curl error (https://github.com/QubesOS/qubes-issues/issues/6275#issuecomment-834315348), I investigated further, which led me to discover and report this bug. #5186 -- This issue appears quite similar, but it was actually reporting something much narrower: It was reporting the case in which the `qubesctl` output actually prints an error message (`stderr: Failed to synchronize cache for repo 'qubes-dom0-cached', disabling.`) but went on to add that case to the "succeeded" counter. That case was less worrisome, since the user was alerted to the fact that there was a cache synchronization error (but still a bit worrisome, since it was counted as a success). By contrast, this case is extremely worrisome, since there is *no indication whatsoever* that we were not even able to contact the update server, much less actually confirm that we have all updates, yet the output explicitly reports `System is already up-to-date`.

Actually my apt-cacher-ng is also denying qubes updates :

Hit:1 http://HTTPS///deb.debian.org/debian buster InRelease
Hit:2 http://HTTPS///deb.debian.org/debian-security buster/updates InRelease
Err:3 http://HTTPS///deb.qubes-os.org/r4.0/vm buster InRelease
Connection failed [IP: 127.0.0.1 8082]
Err:4 http://HTTPS///deb.qubes-os.org/r4.0/vm buster-testing InRelease
Connection failed [IP: 127.0.0.1 8082]
Reading package lists...
Building dependency tree...
Reading state information...
All packages are up to date.
W: Failed to fetch http://HTTPS///deb.qubes-os.org/r4.0/vm/dists/buster/InRelease Connection failed [IP: 127.0.0.1 8082]
W: Failed to fetch http://HTTPS///deb.qubes-os.org/r4.0/vm/dists/buster-testing/InRelease Connection failed [IP: 127.0.0.1 8082]
W: Some index files failed to download. They have been ignored, or old ones used instead.

unman · May 26, 2021, 1:29pm

Yes, apt-cacher-ng works for Fedora updates.

You have to make some changes -
First, on the client side, comment out "metalink" lines, and uncomment
"baseurl" lines. This is because the metalink will keep loading new
https:// repositories, and apt-cacher-ng cant cache those requests,
as you know.
Second, watch the caches in /var/cache/apt-cacher-ng , and add any new
ones to the fedora_mirrors file - this is because that file doesn't
contain all Fedora repositories.

After a while you will have almost all your Fedora updates cached, and
will see the speed increase.

unman · May 26, 2021, 1:30pm

The repository was unavailable for a while. Was that the issue?

pillule · July 28, 2021, 8:18pm

unman <unman@thirdeyesecurity.org> writes:

The repository was unavailable for a while. Was that the issue?

Yes. I panicked.

Yes, apt-cacher-ng works for Fedora updates.

Thanks for the details. I finally took the time to look at it.

You have to make some changes -
First, on the client side, comment out "metalink" lines, and uncomment
"baseurl" lines.

The cisco repository of the codec openh264 does not have a baseurl, I
found that I could use
http://HTTPS///codecs.fedoraproject.org/openh264/$releasever/$basearch
in place, I assume this can be safely added to
/etc/apt-cacher-ng/fedora_mirrors

Also fedora ships with
#baseurl=https://download.example/\[\.\.\.\]
in /etc/yum.repos.d conf files, I assume I had to replace them with
baseurl=http://HTTPS///downloads.fedoraproject.org/\[\.\.\.\]

Then don't forget to
$ dnf clean all

This is because the metalink will keep loading new https://
repositories, and apt-cacher-ng cant cache those requests, as you
know.

I think we could also specify &protocol=http on metalinks as explained in

I have not tested it thought.

Second, watch the caches in /var/cache/apt-cacher-ng , and add
any new ones to the fedora_mirrors file - this is because that file
doesn't contain all Fedora repositories.

It is maybe too soon to see, I don't know yet if having manipulated the
url to use downloads.fedoraproject.org will effectively lead to mirrors
to manage. What I know is, it was creating a directory named
  downloads.fedoraproject.org
before I add
  https://downloads.fedoraproject.org/pub/fedora/linux/
to
  /etc/apt-cacher-ng/fedora_mirrors

And that downloads.fedoraproject.org is supposed to redirect to mirrors...

In the doubt I run a script to duplicate all http url of fedora_mirror
to https.

I put a systemd timer to watch new directories on /var/cache/apt-cacher-ng/

I also put a timer to run /etc/cron.daily/apt-cacher-ng that manage
expired files and make the html report.

Interestingly enough debian ships with scripts in
/usr/share/doc/apt-cacher-ng/examples/dbgenerators.gz
that may take care to update the mirrors files list at the cost of a
lengthy cycle of queries ... That could be triggered weekly.

Do you known about it?

Your instruction didn't said anything for the AppvM so I figured out
that I could put an instruction in /rw/config/rc.local to switch back
the repositories files to their initial values so I can still test out
packages there before really installing them in a template.

Lastly, whonix-* will fail to update with, in dom0:/etc/qubes-rpc/policy/qubes.UpdatesProxy

$type:TemplateVM $default allow,target=cacher

Because whonix ensure updates comes from the tor network. I didn't
figured yet if it is desirable to search to do something here.

unman · August 6, 2021, 2:08pm

unman <unman@thirdeyesecurity.org> writes:

> The repository was unavailable for a while. Was that the issue?

Yes. I panicked.

> Yes, apt-cacher-ng works for Fedora updates.

Thanks for the details. I finally took the time to look at it.

> You have to make some changes -
> First, on the client side, comment out "metalink" lines, and uncomment
> "baseurl" lines.

The cisco repository of the codec openh264 does not have a baseurl, I
found that I could use
http://HTTPS///codecs.fedoraproject.org/openh264/$releasever/$basearch
in place, I assume this can be safely added to
/etc/apt-cacher-ng/fedora_mirrors

Also fedora ships with
#baseurl=https://download.example/\[\.\.\.\]
in /etc/yum.repos.d conf files, I assume I had to replace them with
baseurl=http://HTTPS///downloads.fedoraproject.org/\[\.\.\.\]

Then don't forget to
$ dnf clean all

> This is because the metalink will keep loading new https://
> repositories, and apt-cacher-ng cant cache those requests, as you
> know.

I think we could also specify &protocol=http on metalinks as explained in
fedora - How to create an on-demand RPM mirror - Unix & Linux Stack Exchange
I have not tested it thought.

I have seen that, but generally I dont want clear traffic, so its not a
good option for me.

> Second, watch the caches in /var/cache/apt-cacher-ng , and add
> any new ones to the fedora_mirrors file - this is because that file
> doesn't contain all Fedora repositories.

It is maybe too soon to see, I don't know yet if having manipulated the
url to use downloads.fedoraproject.org will effectively lead to mirrors
to manage. What I know is, it was creating a directory named
  downloads.fedoraproject.org
before I add
  https://downloads.fedoraproject.org/pub/fedora/linux/
to
  /etc/apt-cacher-ng/fedora_mirrors

And that downloads.fedoraproject.org is supposed to redirect to mirrors...

In the doubt I run a script to duplicate all http url of fedora_mirror
to https.

I put a systemd timer to watch new directories on /var/cache/apt-cacher-ng/

I also put a timer to run /etc/cron.daily/apt-cacher-ng that manage
expired files and make the html report.

Interestingly enough debian ships with scripts in
/usr/share/doc/apt-cacher-ng/examples/dbgenerators.gz
that may take care to update the mirrors files list at the cost of a
lengthy cycle of queries ... That could be triggered weekly.

Do you known about it?

Yes, but I've never(?) used it - the default lists are pretty good, and
it takes nothing to check if there are any rogue mirrors being fetched.

Your instruction didn't said anything for the AppvM so I figured out
that I could put an instruction in /rw/config/rc.local to switch back
the repositories files to their initial values so I can still test out
packages there before really installing them in a template.

Lastly, whonix-* will fail to update with, in dom0:/etc/qubes-rpc/policy/qubes.UpdatesProxy

$type:TemplateVM $default allow,target=cacher

Because whonix ensure updates comes from the tor network. I didn't
figured yet if it is desirable to search to do something here.

I dont use Whonix.
Since you can configure cacher to fetch across the Tor network, this
looks brain dead to me. I think you must mean that Whonix ensures that
updates run through Whonix.

pillule · August 18, 2021, 1:36am

unman <unman@thirdeyesecurity.org> writes:

Because whonix ensure updates comes from the tor network. I didn't
figured yet if it is desirable to search to do something here.

I dont use Whonix.
Since you can configure cacher to fetch across the Tor network, this
looks brain dead to me. I think you must mean that Whonix ensures that
updates run through Whonix.

Yes. That's it.

In another thread you spoke about not indexing for each template (so
eventually reducing our fingerprint by reducing the request we made,
right?) ; and potential drawbacks, do you mind to share what you find
about that? I know there is this this checkbox in acng-report.html but
don't know what option exactly it correspond in acng.conf nor the
drawbacks and eventual mitigations.

unman · August 20, 2021, 3:17pm

The checkbox there is only used in admin operations.

You could look at FreshIndexMaxAge - this is used to "freeze" the index
files if clients are updating at nearly the same time.
In Qubes, this happens a lot.
Set that to a large value, and you can restrict the repeated calls to
fetch the indexes.
This is good - it means that (e.g.) there would be only 1 call to fetch
the Debian indexes while updating 15 templates.
This may be bad - if new packages are released during the "freeze", the
clients will only have the old versions in index and cache. They could
miss crucial security updates.
As always, it's a trade off.