There was a previous thread about feedback on testing which resulted in
the “Testing 4.1” category under #user-support.
We are revitalising the testing process in Qubes - there’s already a
mail group for this purpose. I wondered what people thought about having
a separate forum, (possibly closed membership), aimed specifically at
testing, or a “testing” category under User-Support.
The aim would be to have people commit to using the testing
repositories on stable (4.0), identifying any issues that arise:
these can then be tested and identified before hitting GitHub.
When a new version moves closer to release, the testers would aim to
systematically test open issues to ensure that they have been dealt with
in the new release, and can either be closed, or carried forward to the
new release.
What would be the best way to approach this for Forum users?
Anyone not interested can simply hide the category in the preferences. I would say it’s better to show it by default, so that all new users will notice a possibility to help.
There’s a clear conflict of interest here - I don’t particularly
want new users to participate.
There are many reasons for this, but the major ones are that testers
should have significant domain knowledge, and should be able to
effectively communicate with the team.
It’s evident from both this Forum, and GitHub, that new users are
rarely able to fulfil either. Both are littered with cases where the OP
has reported an issue based on lack of understanding, or has failed to
meaningfully contribute to identifying the problem or resolution.
(There are other reasons but those should be enough: they are key
requirements in any effective testing team.)
My preference would be for the categories to be open to read, but
limited in participation. We don’t want “testing” to be a place where
users report issues - we want it for testing.
For the current release this means primarily for testing of candidate
packages in current-testing.
For the next release it is more generalised, but should go ahead of “use
it and say when it doesn’t do what I want”.
At transition to a new release, we need people who will be able to read
GitHub issues, and test them on the new stable release to see if they
have been resolved there. We need this in a somewhat formalised process.
It could be that this isn’t what any one else wants, or what people think
is best for the project. It’s based on my experience.
It’s often said in open source, that many eyes make bugs shallow. I don’t
believe that to be true.
I think your suggestion makes lots of sense! There needs to be a sort of level system to reduce the noise that goes up to the developer level.
The main challenge is that there will always be people that casually contributes with 4.1-related and may not even be aware of the existence of such a team.
Then I suggest the following:
rename the current 4.1 testing to just 4.1 that’s where 4.1 user support will go.
Then we make a testing category with two subcatgories: “4.1 testing” and “4.0 testing”. It will be visible on the forum but posting limited to “testing team” group members.
Sounds good?
Lastly, what should the process for joining the testing team be? Will it be on an apply / invite-basis? if so, what do the users need to do in order to join?
The initial message should be mirrored in the “about” message.
I suggest we say what we want - users who want to join can IM (still
don’t know how to do that from email), and be approved.
I’d be surprised if we see users we don’t know asking to join - they’ll
be people who have some history on the Forum or the lists.
Cooptation, invite-basis, user request, and approved by the forum moderators (based on getting a github account, forum/github/mailing-list history with Qubes-OS skills, …).
I am. I’ll do this this afternoon (EU time) or tomorrow at most.
Joining Message
The following will be the joining instructions. Any suggestions for changes?
What’s the testing team?
It’s a group of people who commit to testing updates or releases of Qubes OS and who are willing to provide test and confirm issues, or identify particular edge cases. This is key to detect issues early on and catch them before they affect more users.
What’s the risk?
For those willing to enable the testing repositories for the current release the risk is minimal to testers because any packages end up in current anyway.
However, if security is crucial in your case, you should strictly use the stable release, not enable the testing repositories and not join the testing team.
How to join the testing team?
Request to join the testing team
Via forum (preferred): request to join here (requires javascript and a forum account)
Via email: send an email to register-testing-team at forum.qubes-os.org saying you’d like to join (does not require javascript nor a forum account)
Follow the instructions for the kind of testing you want to help with (testing updates / releases)
This isn’t right - it’s not sufficient to run 4.1 - you would also have to enable the testing repositories. Otherwise you could
report on something that has already been identified, resolved, and the
fix be in the pipeline.
It’s important that team members update at least weekly. This should be
made explicit. (This is because some packages transition after a week.)
Also, the order is wrong.
The request to join should come first, then enable the testing
repositories.
You have also toned down the warning that I suggested.
We should explicitly say that where security is crucial, users should
use the stable release, and not enable the testing repositories.
It’s important that we use the mailing lists too, including the (soon to
be defunct) testers list.
I agree that this underplays the risk. It’s worth noting that there are (at least) two kinds of risk: security and stability. The stability risk of testing can be quite high. Users who need their systems to be stable (e.g., can’t afford down time or lost work between backups) should not be doing any testing on those systems.
I’m also fine either way. Might get more uptake by publishing a Qubes news post and usual syndication via social media venues, or might not if all would-be participants already see it here and on qubes-testers. If you want to do it, let me know. (We could probably just use an amended version of this post with you as the author.)
@adw@unman@michael I have edited the original post to reflect this feedback. Let me know if there’s anything that needs additional fixing.
@unman since via the email you can’t see the modified post, I’ve PMed you a copy of the current version.
The part that needs the most work probably is now the risk part. Feel free to suggest a new version. I’m out of ideas. The current version is as follows.
@michael thanks for fixing the typos. @adw, if you see typos anywhere, feel free to edit my posts straight away . I’m a typo producing machine
. I’m a typo producing machine 