How to bring more attention and participation to 4.2 Testing Category

[solene]

@adw

Both. You can see some waiting PRs here, but the reality is that the documentation is mostly a volunteer effort, so it mostly just gets updated as people find the time and inclination to contribute updated content.

I expect many users to struggle with the iptables → nft change when reading the documentation

I suggest to make the documentation of such important changes part of the release cycle.

[balko]

@adw

Both. You can see some waiting PRs here , but the reality is that the documentation is mostly a volunteer effort, so it mostly just gets updated as people find the time and inclination to contribute updated content.

And about mans, do they reflect calling changes in qvm-* and qubes* tools?

[adw]

@solene

I suggest to make the documentation of such important changes part of the release cycle.

Acknowledged. Thank you for the suggestion!

@balko

And about mans, do they reflect calling changes in qvm-* and qubes* tools?

You mean in the OS rather than on the website? (My comment above was about qubes-doc, not in-OS documentation like man pages.) That I’m not sure about, but you can probably check in the appropriate repos on GitHub. If the devs have forgotten to update them, please feel free to open an issue (if one is not already open) and/or submit a PR to update the man pages.

[SteveC]

Oh the tool command line interfaces are changing?

Where can I find out about this? Changes not known about will absolutely bite me (and probably many others) in the butt when I go to upgrade since I use a LOT of scripts to regenerate VMs. (And hopefully salt will know what to do internally, too.)

[51lieal]

Some people enjoy keep documentation up to date (i know some), and some people not (its me).

For me and for now, writing a good documentation still hard, it maybe too technical for novice user.

[adw]

@SteveC

Oh the tool command line interfaces are changing?

Where can I find out about this?

@marmarek?

[balko]

@SteveC

Oh the tool command line interfaces are changing?

Well, I think almost no, it stays the same.

But maybe qubesctl in R4.2 will have some options that are absent in my installation of R4.1. I saw something about this change when I was developing qubes completion (bash completion) for all qvm- and qubes* commands.

Note that qubesctl has no complete documentation anyway, its man page is lacking basic information about arguments. So, maybe I am wrong and nothing changed, I do not remember the details.

[adw]

@deeplow, is Joining the Testing Team still accurate? I’m still linking to it in every RC announcement, so if anything has changed, perhaps that post should be updated to whatever is the current policy.

[deeplow]

Not sure if we got a conclusion out of this discussion, so I guess everything is still the same

[Plum]

Maybe it’s just the slow pace of change. 4.2 will be the first point release in 18 months. As a casual user, that’s too slow a pace to maintain my interest. I’ve not seen much telling me what’s changing and why I should care.

[moonlitOrca]

I’m enjoying reading the discussion here . In reality, things like this conversation thread motivate me to make more time to help with Qubes OS, because I can see other people in a true sense of community working together. I think that goes right along with the suggestion to publish a roadmap of some sort for the next version. That would be another way to show people that things are happening and they can be part of it and excited about it! Honestly, I didn’t know about any of the Qubes OS 4.2 changes that were coming until the announcement by @adw . I see now, from this conversation, that I could probably have dug into github to see what was happening, but I don’t know if most users really know that? Maybe they do and I’m the odd one out .

For me, now that I know, I have two reasons for hesitation:

1. I’ve only been using Qubes OS for a bit over a year now and I have seen some fairly critical problems that even I have dealt with in that short time in updating or running certain things that posting on the forums or digging around the Internet can’t help much with (just not as many users, and the Xen and VM systems absolutely adds another layer of complication for many things, that means I don’t even know where to start looking for solutions sometimes.) and then I have to spend tens of hours trying to figure out how to muddle through or fix things on my daily driver. This is similar to what @balko mentioned above. So testing even less stable things sounds scary… Importantly, I am not blaming anyone or expressing dissatisfaction; I am amazed at this project and with everyone who is part of it, and I think it is incredibly valuable which why I am still learning and using Qubes OS, but it has a pretty steep learning curve if you go off the beaten track (and not as many users who are walking the track ahead of you), with no promise that anyone will be there to help you get back to functioning.

2. As I said in point 1, I am fairly new to this project and so I rely heavily on the very well-written but also somewhat limited in scope documentation that exists. I am concerned with testing that I will not know how to even helpfully assist in reporting problems without some hand-holding and generally being a burden on someone… If there was even slight documentation on the significant changes such as how to use the new fwupd capabilities, for example, or how to set up firewall routes with nftables through Qubes OS, I would be happy to try. This sounds like what @solene mentioned above too. But I don’t know where to begin with some of the changes. Would I be actually useful to the team as a tester? How would I know, how to even try the new features? I am typing this using the 4.2RC btw, so I have been mostly enjoying the new GUI features already.

I actively want to help and would be willing to test things and learn as I go, and I am hugely excited for the fwupd functionality, the new GUI tools, the new “Victoria” menu, and SELinux support. I just glanced at the github for 4.3 and I see s0ix support being worked on, which is yet another thing I really wanted since starting using Qubes OS in the first place. Very exciting features are being worked on. But anyway, these are my thoughts at the moment.

[Sven]

My 2 cents:

• If you only have one computer and you are using it productively (either for your day job or for other tasks that are important to you) then you shouldn’t use it for testing. Things will break in the least opportune moment and you might end up attributing the related stress and frustration to Qubes OS. Just don’t do it.

• If you run Qubes OS on a secondary machine or you don’t use your computer for things that are important to you (e.g. time critical) then by all means … test, experiment, learn. That’s the right way to do it, because this way it won’t cause stress and frustration in your life.

These considerations should be self-evident but might be overlooked in the excitement and promise of new features.

[moonlitOrca]

This is good advice above, by @Sven. It does make me worry that I didn’t express myself clearly though; I actually do enjoy the learning and problem-solving aspect of Qubes OS, otherwise there’s no way I would be using it (and have backups which I can readily restore and secondary computers for use if needed). I’ve learned a ton over this past year using Qubes! It’s fun for me.

Rather, I was expressing thoughts that I think may be more common than just from me about why testing gives me pause, and trying to make 3 implicit suggestions:

• Have more conversations and communications like this one where people can see the wheels turning and can get excited to help or at least to look forward to what’s in the pipeline. Basically, build more community.
• Offer some place for testers to be more certain they can get some level of support while testing, even if it’s just confirmation that their bug is noted and if it is known, or if it’s just user error, and maybe a suggestion from a fellow tester on how to workaround the problem. Even if a user is testing on a backup computer, without some direction, they may end up alone to dig through complex things for many hours to solve problems that may or may not be user-solvable. This may not always be possible, I understand.
• Build very small documentation for the new features before user testing so people know what what to expect and how to test them.

These are just my thoughts of course, and I know that they may not be possible. I do also wonder: Why is testing essentially an invite-only (application) thing? What is if was opened up to more people? Is that just to keep from flooding the forum with problem posts?

[quantum]

I stopped using weekly builds since they usually didn’t have key updates, such as 4.2 SELinux /rw issue on Fedora 38. If bug fixes were quickly pushed to the weekly I would start trying them again. For now, I wait for rc1, rc2, etc.

[Sven]

@moonlitOrca

Why is testing essentially an invite-only (application) thing?

What’s the testing team?

It’s a group of users who help make Qubes better.
They commit to testing updates and releases of Qubes OS, and are willing
to provide test results, confirm issues, and identify particular edge cases.
This is key to detecting issues early on, and stop them affecting more users.

Who should join?

Community members with demonstrated track-record of good quality contributions.

What’s the risk?

For those willing to enable the testing repositories for the current release
the risk is minimal, because the packages will end up in current anyway.

If security and stability are crucial to you, you should use the current stable release,
not enable the testing repositories, and not join the testing team.

How do I join the testing team?

1. Request to join the testing team

• Via forum (preferred): request to join here (requires JavaScript and a forum account)
• Via email: send an email to register-testing-team at forum.qubes-os.org saying you’d like to join (does not require JavaScript or a forum account)

2. Follow the instructions for the kind of testing you want to help with (testing updates / releases)

Testing Type	Instructions	Description
Testing Updates	see here	Enabling the testing repositories (e.g. 4.0 testing)
Testing Releases	part 1 & part 2	Running the upcoming Qubes version (e.g. 4.1) and enabling the testing repositories.

3. Report issues in the appropriate category:

• If testing updates, post in the ‘4.0 Testing’ category, or email testing-updates at forum.qubes-os.org .
• if testing releases, post in the ‘4.1 Testing’ category, or email testing-releases at forum.qubes-os.org .

You can leave the group at any time by following steps similar to the ones in 1.

Packages move through the testing process quite quickly. Ideally you
would be able to update at least once a week.

When 4.1 moves toward release, we will see if we can test some of the
open issues in 4.0, and confirm they have been closed.

[adw]

@moonlitOrca

• Offer some place for testers to be more certain they can get some level of support while testing, even if it’s just confirmation that their bug is noted and if it is known, or if it’s just user error, and maybe a suggestion from a fellow tester on how to workaround the problem. Even if a user is testing on a backup computer, without some direction, they may end up alone to dig through complex things for many hours to solve problems that may or may not be user-solvable. This may not always be possible, I understand.
• Build very small documentation for the new features before user testing so people know what what to expect and how to test them.

I think the main limiting factor for both of these is not having enough knowledgeable people to supply them (or the knowledgeable people we have not having enough time).

@moonlitOrca

Why is testing essentially an invite-only (application) thing?

Just to clarify, anyone on the internet can download any testing ISO (or the source code and builder to build their own) without applying for or being invited to anything. It’s only the testing team that requires applying. You can even test stuff and report issues without joining the testing team.

[moonlitOrca]

Thanks @adw. I totally understand the limiting factor there. Over time, with experience, I can at least add another more knowledgeable person to help, but that’s the tough one.

As for the testing team, I did read the testing team forum post (article Sven posted) before originally posting to this thread in the first place, but I still don’t feel it explains to me why the testing team is exclusive? Again, it’s just a topic I asked about to honestly try to help and contribute, since the question was posed how to get more involvement and attention. It seems making it exclusive automatically limits that?

@adw

Just to clarify, anyone on the internet can download any testing ISO (or the source code and builder to build their own) without applying for or being invited to anything. It’s only the testing team that requires applying. You can even test stuff and report issues without joining the testing team.

I have re-read the description twice, but I don’t see any unique ability afforded by joining the testing team? Is it mostly just an organizational thing to have people who can be counted on to consistently test and who communicate directly? If I am reading correctly, it looks like anyone can enable the testing repositories. Testing the Release candidate for 4.2 has been awesome so far, and I really feel 4.2 makes a meaningful improvement in several key areas. Great work by everyone (I especially don’t understand how marmarek can do everything he does. Incredible )

[adw]

@moonlitOrca

As for the testing team, I did read the testing team forum post (article Sven posted) before originally posting to this thread in the first place, but I still don’t feel it explains to me why the testing team is exclusive? Again, it’s just a topic I asked about to honestly try to help and contribute, since the question was posed how to get more involvement and attention. It seems making it exclusive automatically limits that?

I think this thread contains the answer to your question:

Testing team - how to deal with in Forum

Nice long title to clear the filters. There was a previous thread about feedback on testing which resulted in the “Testing 4.1” category under #user-support. We are revitalising the testing process in Qubes - there’s already a mail group for this purpose. I wondered what people thought about having a separate forum, (possibly closed membership), aimed specifically at testing, or a “testing” category under User-Support. The aim would be to have people commit to using the testing repositor…

@moonlitOrca

I have re-read the description twice, but I don’t see any unique ability afforded by joining the testing team?

This page shows the forum permissions granted to members of the testing team:

https://forum.qubes-os.org/g/testing-team/permissions

@moonlitOrca

Is it mostly just an organizational thing to have people who can be counted on to consistently test and who communicate directly?

I think the general idea is to have a higher-than-usual signal-to-noise ratio, which is enforced by allowing only members to post in the testing forum.

@moonlitOrca

If I am reading correctly, it looks like anyone can enable the testing repositories.

Correct.

@moonlitOrca

Great work by everyone (I especially don’t understand how marmarek can do everything he does. Incredible )

Agreed!

[moonlitOrca]

Yes, thank you @adw for always being so conscientious and caring in your replies. This helps me understand the situation much better, and hopefully will help others too.