Can not run salt scripts on fedora-39 template

I am trying to set up a fedora template using salt. This involves both creating the correct template as well as installing some packages on it.
Creating the template works as intended, however when salt is supposed to run on the template to install packages it fails with the following log file.

2024-03-17 10:31:29,809 calling 'state.highstate'...
2024-03-17 10:32:10,395 output: fedora-39-less-safe-temp:
2024-03-17 10:32:10,396 output:     ----------
2024-03-17 10:32:10,396 output:     _error:
2024-03-17 10:32:10,396 output:         Failed to return clean data
2024-03-17 10:32:10,396 output:     retcode:
2024-03-17 10:32:10,396 output:         1
2024-03-17 10:32:10,396 output:     stderr:
2024-03-17 10:32:10,397 output:         Traceback (most recent call last):
2024-03-17 10:32:10,397 output:           File "/var/tmp/.root_dd8a91_salt/salt-call", line 27, in <module>
2024-03-17 10:32:10,397 output:             salt_call()
2024-03-17 10:32:10,397 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/scripts.py", line 437, in salt_call
2024-03-17 10:32:10,397 output:             import salt.cli.call
2024-03-17 10:32:10,397 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/cli/call.py", line 6, in <module>
2024-03-17 10:32:10,397 output:             import salt.cli.caller
2024-03-17 10:32:10,397 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/cli/caller.py", line 14, in <module>
2024-03-17 10:32:10,397 output:             import salt.loader
2024-03-17 10:32:10,397 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/loader.py", line 23, in <module>
2024-03-17 10:32:10,397 output:             import salt.config
2024-03-17 10:32:10,398 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/config/__init__.py", line 17, in <module>
2024-03-17 10:32:10,398 output:             import salt.utils.data
2024-03-17 10:32:10,398 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/utils/data.py", line 14, in <module>
2024-03-17 10:32:10,398 output:             import salt.utils.dictupdate
2024-03-17 10:32:10,398 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/utils/dictupdate.py", line 20, in <module>
2024-03-17 10:32:10,398 output:             from salt.utils.decorators.jinja import jinja_filter
2024-03-17 10:32:10,398 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/utils/decorators/__init__.py", line 19, in <module>
2024-03-17 10:32:10,398 output:             import salt.utils.args
2024-03-17 10:32:10,398 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/utils/args.py", line 16, in <module>
2024-03-17 10:32:10,398 output:             import salt.utils.jid
2024-03-17 10:32:10,398 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/utils/jid.py", line 10, in <module>
2024-03-17 10:32:10,398 output:             import salt.utils.stringutils
2024-03-17 10:32:10,398 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/utils/stringutils.py", line 25, in <module>
2024-03-17 10:32:10,398 output:             from salt.utils.decorators.jinja import jinja_filter
2024-03-17 10:32:10,398 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/utils/decorators/jinja.py", line 9, in <module>
2024-03-17 10:32:10,399 output:             import salt.log.setup as logging
2024-03-17 10:32:10,399 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/log/__init__.py", line 16, in <module>
2024-03-17 10:32:10,399 output:             from salt.log.setup import (
2024-03-17 10:32:10,399 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/log/setup.py", line 29, in <module>
2024-03-17 10:32:10,399 output:             from salt._logging import (
2024-03-17 10:32:10,399 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/_logging/__init__.py", line 17, in <module>
2024-03-17 10:32:10,399 output:             from salt._logging.impl import *  # pylint: disable=wildcard-import
2024-03-17 10:32:10,399 output:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-17 10:32:10,399 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/_logging/impl.py", line 28, in <module>
2024-03-17 10:32:10,399 output:             from salt._logging.handlers import StreamHandler  # isort:skip
2024-03-17 10:32:10,399 output:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-17 10:32:10,399 output:           File "/var/tmp/.root_dd8a91_salt/pyall/salt/_logging/handlers.py", line 20, in <module>
2024-03-17 10:32:10,399 output:             from salt.ext.six.moves import queue  # pylint: disable=import-error,no-name-in-module
2024-03-17 10:32:10,399 output:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-17 10:32:10,399 output:         ModuleNotFoundError: No module named 'salt.ext.six.moves'
2024-03-17 10:32:10,399 output:     stdout:
2024-03-17 10:32:10,399 exit code: 20

This does not happen when running the same salt script but for fedora-36 templates.

It does not matter that I do anything specific in my .sls file, it can both be empty or install packages and the failure is the same. It seems like a general failure of salt in the template.

The six.py file is not available in /usr/lib/python3.12/site-packages/salt/ext/ which is the directory imported in python unless specified with a path. six.py is available inside of /var/tmp/.root_dd8a91_salt/pyall/salt/ext/.
I tried copying the six.py file to /usr/lib/python3.12/site-packages/salt/ext/ which made it possible to import salt.ext.six but caused further failure when the salt script tries to import queues from six.

I figure it is the wrong approach to try to move around files and make the python dependencies fit together and asking for help is the better approach.

which Qubes OS version are you using?

Qubes 4.2.0 but I did an in-place upgrade. However the fedora-39 template was installed after the upgrade.

It affects 4.1 also.

What is the problem? The fedora-39 template? Should I use fedora-38 in the meantime? Should I just avoid using salt for my fedora-39 template?

Yes, it’s the template, or perhaps it’s Fedora? Hasn’t been diagnosed
yet.
I suggest you use fedora-38 or Debian. (I’m not using fedora-39 at all,
for this and other reasons.)