Cannot install packages to VM using saltstack

saltyforsalt · May 7, 2021, 1:08am

Saltstack Issues

Have been trying use saltstack to make a template for a Qubes Building VM, but have constantly run into this issue. While VM creation works fine, every time I try to set packages to be installed in the VM after creation, I get the same error in the management vm log (/var/log/qubes/mgmt-xxx.log), which looks like this:

2021-05-06 20:00:32,983 calling ‘state.highstate’…
2021-05-06 20:00:56,327 output: template-build-f32:
2021-05-06 20:00:56,327 output: An Exception occurred while executing state.highstate: Unable to render top file. No targets found.
2021-05-06 20:00:56,327 exit code: 0

Even though at the end of the output after calling state.highstate it says:

template-build-f32: OK

The template vm does start after the management vm but it only stays on for ~10 seconds until it shuts down, and it doesn’t install any packages.

Things I have tried:

Using a StandaloneVM instead of an AppVM
Using the base dirs instead of the user-dirs
Using a separate top file for the packages.sls file
Using only one top file (with all relevant info)
Creating the buildvm manually and then enabling only the packages topfile and applying it then

The relevant .sls and .top files (yaml syntax doesn’t show in formatted post)

packages.sls

Install:
pkg.installed:
- packages:
- qubes-core-agent-networking
- qubes-core-agent-passwordless-root
- aspell
- aspell-en
- createrepo_c
- createrepo_c-libs
- debootstrap
- devscripts
- dialog
- dpkg-dev
- g++
- gcc
- git
- gitg
- gnupg
- m4
- make
- perl-Digest-MD5
- perl-Digest-SHA
- python3-pyyaml
- python3-sh
- re2c
- reprepro
- rpm-build
- rpm-sign
- rpmdevtools
- texinfo
- wget
- zlib-devel
- fedora-packager
- fedora-review

packages.top

user:
template-build-f32:
- configs.buildvm.packages

If anyone knows how to fix this, please let me know.

gonzalo-bulnes · May 7, 2021, 8:54am

Hi @saltyforsalt,

I assume that your files indentation is correct, despite the formatting issues in your post (YAML is space-sensitive).

From the content of your top file, it seems you’re using the user directories?

Can you provide the full oath of both files, as well as the command you’re using to apply the state?

unman · May 7, 2021, 11:39am

This is a known bug, particularly if you are using Fedora 33 as the
template for the Default-mgmt-dvm - it’s possible that you can work
round it by switching to a Debian template for default-mgmt-dvm.
A Qubes band aid is in the works.

In the meantime you can circumvent the issue by not issuing highstate.
Just run state like this
qubesctl --skip-dom0 --targets=<target> state.apply <name of state>

That will work fine even with Fedora 33 mgmt-dvm and targets.

unman · May 7, 2021, 11:42am

Incidentally, you might like to look at GitHub - unman/shaker
where I have states to create and provision a builder qube, just as you
are doing.

gonzalo-bulnes · May 7, 2021, 11:49am

@unman Is there a bug report or description that I can look at?

saltyforsalt · May 8, 2021, 3:30am

Thanks for letting me know, I was using fedora-33 for the mgmt-dvm - although interestingly when I switched it to the Debian 10 template I got a new error.

2021-05-07 21:32:35,322 output: template-build-f32:
2021-05-07 21:32:35,323 output:     ----------
2021-05-07 21:32:35,323 output:     _error:
2021-05-07 21:32:35,323 output:         Failed to return clean data
2021-05-07 21:32:35,323 output:     retcode:
2021-05-07 21:32:35,324 output:         1
2021-05-07 21:32:35,324 output:     stderr:
2021-05-07 21:32:35,324 output:         Traceback (most recent call last):
2021-05-07 21:32:35,324 output:           File "/var/tmp/.root_dd8a91_salt/salt-call", line 15, in <module>
2021-05-07 21:32:35,324 output:             salt_call()
2021-05-07 21:32:35,324 output:           File "/var/tmp/.root_dd8a91_salt/py3/salt/scripts.py", line 405, in salt_call
2021-05-07 21:32:35,324 output:             import salt.cli.call
2021-05-07 21:32:35,324 output:           File "/var/tmp/.root_dd8a91_salt/py3/salt/cli/call.py", line 5, in <module>
2021-05-07 21:32:35,324 output:             import salt.utils.parsers
2021-05-07 21:32:35,324 output:           File "/var/tmp/.root_dd8a91_salt/py3/salt/utils/parsers.py", line 27, in <module>
2021-05-07 21:32:35,324 output:             import salt.config as config
2021-05-07 21:32:35,324 output:           File "/var/tmp/.root_dd8a91_salt/py3/salt/config/__init__.py", line 101, in <module>
2021-05-07 21:32:35,324 output:             _DFLT_IPC_WBUFFER = _gather_buffer_space() * .5
2021-05-07 21:32:35,324 output:           File "/var/tmp/.root_dd8a91_salt/py3/salt/config/__init__.py", line 90, in _gather_buffer_space
2021-05-07 21:32:35,324 output:             import salt.grains.core
2021-05-07 21:32:35,324 output:           File "/var/tmp/.root_dd8a91_salt/py3/salt/grains/core.py", line 32, in <module>
2021-05-07 21:32:35,325 output:             from platform import _supported_dists
2021-05-07 21:32:35,325 output:         ImportError: cannot import name '_supported_dists' from 'platform' (/usr/lib64/python3.8/platform.py)
2021-05-07 21:32:35,325 output:     stdout:
2021-05-07 21:32:35,325 exit code: 20

At the end of the output after calling state.highstate:

template-build-f32: ERROR (exit code 20, details in /var/log/qubes/mgmt-template-build-f32.log)

I then decided to switch to the fedora-32 template just to see if it would work, and that worked!

Using fedora-32 as the mgmt-dvm template, I tried running this:
qubesctl --skip-dom0 --targets=template-build-f32 state.apply configs.buildvm.packages saltenv=user
and it worked! It also worked with fedora-33 set as the mgmt-dvm template.

Did same thing with mgmt-dvm set to debian 10 and got a very similar error to the one that I got when calling state.highstate using the mgmt-dvm set to debian 10:

2021-05-07 22:08:55,687 calling 'state.apply configs.buildvm.packages saltenv=user'...
2021-05-07 22:09:18,460 output: template-build-f32:
2021-05-07 22:09:18,461 output:     ----------
2021-05-07 22:09:18,461 output:     _error:
2021-05-07 22:09:18,461 output:         Failed to return clean data
2021-05-07 22:09:18,461 output:     retcode:
2021-05-07 22:09:18,461 output:         1
2021-05-07 22:09:18,461 output:     stderr:
2021-05-07 22:09:18,461 output:         Traceback (most recent call last):
2021-05-07 22:09:18,461 output:           File "/var/tmp/.root_dd8a91_salt/salt-call", line 15, in <module>
2021-05-07 22:09:18,461 output:             salt_call()
2021-05-07 22:09:18,461 output:           File "/var/tmp/.root_dd8a91_salt/py3/salt/scripts.py", line 405, in salt_call
2021-05-07 22:09:18,461 output:             import salt.cli.call
2021-05-07 22:09:18,461 output:           File "/var/tmp/.root_dd8a91_salt/py3/salt/cli/call.py", line 5, in <module>
2021-05-07 22:09:18,462 output:             import salt.utils.parsers
2021-05-07 22:09:18,462 output:           File "/var/tmp/.root_dd8a91_salt/py3/salt/utils/parsers.py", line 27, in <module>
2021-05-07 22:09:18,462 output:             import salt.config as config
2021-05-07 22:09:18,462 output:           File "/var/tmp/.root_dd8a91_salt/py3/salt/config/__init__.py", line 101, in <module>
2021-05-07 22:09:18,462 output:             _DFLT_IPC_WBUFFER = _gather_buffer_space() * .5
2021-05-07 22:09:18,462 output:           File "/var/tmp/.root_dd8a91_salt/py3/salt/config/__init__.py", line 90, in _gather_buffer_space
2021-05-07 22:09:18,462 output:             import salt.grains.core
2021-05-07 22:09:18,462 output:           File "/var/tmp/.root_dd8a91_salt/py3/salt/grains/core.py", line 32, in <module>
2021-05-07 22:09:18,462 output:             from platform import _supported_dists
2021-05-07 22:09:18,462 output:         ImportError: cannot import name '_supported_dists' from 'platform' (/usr/lib64/python3.8/platform.py)
2021-05-07 22:09:18,462 output:     stdout:
2021-05-07 22:09:18,462 exit code: 20

Seeing as calling state.highstate works with fedora-32 set as the mgmt-dvm template, it would seem that the issue is somewhere in fedora-33, and that debian 10 may have the same issue or a different bug.

saltyforsalt · May 8, 2021, 3:34am

Solution:

Set mgmt-dvm template to fedora-32 and the issue seems to go away. Fedora-33 and Debian-10 templates each had different errors when calling state.highstate, so probably don’t use those for this - at least until whatever bug(s) involving those two templates get fixed.

saltyforsalt · May 8, 2021, 3:35am

Actually did look at this - my package list is very similar - Also thanks again for letting me know that the issue is an active bug being worked on, and for your assistance in general!

saltyforsalt · May 8, 2021, 3:37am

Fixed the issue, made a post with solution details on it, as well as a more detailed post on what caused errors. Thank you for helping though!