What to test? Where to get what to test? Where to report testing results?

Testing Team 4.1 Testing
#1

Quoting myself here to open discussion:

So how do we improve this, taking suspend/resume pending PR and vmm-xen fixes for loopback performance issues as a first example to this?

Who are tracking and testing those PRs today?

Some issues with 4.1.1 updates and kernel-latest
#2

Ok. Case study.

Suspend and resume stubdomains by DemiMarie · Pull Request #139 · QubesOS/qubes-vmm-xen · GitHub just got merged.

#3

At the time of writing those lines, i do not see any OpenQA entree either in the PR that was approved above, nor any new issue created under Issues · QubesOS/updates-status · GitHub

Edit: they were there.
Then PR got merged to xmm-xen.

@marmarek/@adw can you detail the flow of a an approved code PR → OpenQA testing output to look at (how are those linked outside of seeing pipelineretry) → signing → issue being created under Issues · QubesOS/updates-status · GitHub → package landing under unstable repositories to be testable ?

Ext4 vs. Btrfs performance on Qubes OS installs
#4

Then packages under testing repositories 21 hours ago and issue created to track what templates it can be tested from: vmm-xen v4.14.5-9 (r4.1) · Issue #3084 · QubesOS/updates-status · GitHub

Addressing as well

Where no reference are done to suspend/resume issue referenced in update-status issue.

And where packages for dom0, bullseye buster and bookworm are available. No clue where/when/if packages for fedora are supposed to show

@marmarek?

Relevant:

#5

The flow is more or less like this:

  1. PR is made
  2. It gets automatically build-tested on gitlab-ci. In few cases, this includes also basic unit tests. This gets reported in the PR status.
  3. PR get review; in case of less than trivial changes, PR get a “openqa-pending” label, to be included in the next openQA run (in the “pull requests” group). Those get scheduled manually, usually just after assigning the label. Results get reported back to PR comments. There is a common run for all the PRs with the label, so a failure listed there may be caused by another PR in the group (IOW, failed tests are just a hint, not a blocker).
  4. PR get merged; at this point, github automatically closes associated issues, but otherwise not much more is happening.
  5. I bump a package version (usually after merging a few PRs in given repository). At this point the package gets built and lands in current-testing repo + an issue in updates-status is created. Relevant issues (qubes-issues repo) get also automatic comment about the packages.
  6. Every Wednesday, automatic openQA run is started for package in current-testing repo (the “updates” group on openqa). Results get reported as a comment in relevant update-status issues.
  7. Every Saturday, a weekly ISO image is built (including packages from current-testing repo) and scheduled on openQA in the “Qubes OS” group.
  8. Depending on testing results, package can be moved to “current” repository after at least a week. For some less trivial changes, I prefer to keep it in current-testing repo a bit longer. I usually do review of “updates-status” issues on Monday or Tuesday.

The last point is where this group can help a lot. Even just giving :+1: or :-1: helps, as an indicator that you actually installed related package and found (or not) it working. Packages with no feedback at all tends to stay longer in current-testing repo.

Those aren’t supposed to be there (vanilla packages from Fedora are enough in VM).

You mean the one about Suspend and resume stubdomains by DemiMarie · Pull Request #139 · QubesOS/qubes-vmm-xen · GitHub ? This one just unblocks one of the points in Properly suspend all VMs, not only those with PCI devices by marmarek · Pull Request #473 · QubesOS/qubes-core-admin · GitHub , so it doesn’t closes any issues on its own - so, it isn’t linked to any.

2 Likes
#6

@marmarek: This is an incomplete example of how this should be run from the testing community?

#7

Yes, thanks! It will be useful to include also the link(s) to updates-status issues (vmm-xen v4.14.5-9 (r4.1) · Issue #3084 · QubesOS/updates-status · GitHub in this case). That’s where :+1: / :-1: are most useful.

#8

@marmarek :
It is not clear to me the link between current-testing notes for bullseye here under vmm-xen v4.14.5-9 (r4.1) · Issue #3084 · QubesOS/updates-status · GitHub
Package for bullseye was built ([build log](https://github.com/QubesOS/build-logs/tree/master/build-debian4/log_2022-10-16_01-33-06)) and uploaded to current-testing repository

And the packages qubes-input-proxy-sender qubes-vm-dependencies qubes-vm-recommended xen-utils-guest that were update candidates with bullseye-testing uncommented, applied on other post for testing.

If it is unclear to me, this is most probably unclear as well for the testing community as well.

Can you elaborate on the meaning of vmm-xen v4.14.5-9 (r4.1) · Issue #3084 · QubesOS/updates-status · GitHub ?

Thanks

Edit: added cross-reference here: vmm-xen v4.14.5-9 (r4.1) · Issue #3084 · QubesOS/updates-status · GitHub

#9

@marmarek, can you explain why some other templates have updates but not others then and how to easily understand the link between vmm-xen to v4.14.5-9 and bullseye current-testing available updates, in this case:

 qubes-input-proxy-sender qubes-vm-dependencies qubes-vm-recommended xen-utils-guest

Or in other terms, what is expected to be tested in this case on

And more importantly, why and what is the relation between dom0 package here and what differenciate fedora templates not needing specifics parallel updates and why here only a subset of supported templates have updates available that do not seem clearly related, nor having specific names of packages that would need to be co-tested for regression/functionality?

Thanks

#10

In case of xen packages for Debian, there is almost nothing qubes specific there, and as such, most most fixes referenced there apply only to the dom0. The only remaining reason we build them (instead of using upstream packages directly, as in case of Fedora) is that Debian packaging insists on having the same version of libraries as the hypervisor. This matters for dom0, but is irrelevant for VM (and indeed Fedora VMs for example use libs from Xen 4.16 and it works just fine). But since the Debian packaging has this versioning requirements, we need to ship the same version (at the moment Debian stable incidentally also ships with Xen 4.14, but it isn’t always the case, and for example Debian testing has Xen 4.16). I hope this will be resolved one day in Debian too…

Long story short: for xen it’s enough to test dom0 packages.

1 Like
#11

Understood for Debian and Xen specifics.
@marmarek : But then, what about centos-stream8?

#12

That’s because CentOS doesn’t have own Xen packages at all.

1 Like