What to test? Where to get what to test? Where to report testing results?

Testing Team
1

Quoting myself here to open discussion:

So how do we improve this, taking suspend/resume pending PR and vmm-xen fixes for loopback performance issues as a first example to this?

Who are tracking and testing those PRs today?

2

Ok. Case study.

Suspend and resume stubdomains by DemiMarie · Pull Request #139 · QubesOS/qubes-vmm-xen · GitHub just got merged.

3

At the time of writing those lines, i do not see any OpenQA entree either in the PR that was approved above, nor any new issue created under Issues · QubesOS/updates-status · GitHub

Edit: they were there.
Then PR got merged to xmm-xen.

@marmarek/@adw can you detail the flow of a an approved code PR → OpenQA testing output to look at (how are those linked outside of seeing pipelineretry) → signing → issue being created under Issues · QubesOS/updates-status · GitHub → package landing under unstable repositories to be testable ?

4

Then packages under testing repositories 21 hours ago and issue created to track what templates it can be tested from: vmm-xen v4.14.5-9 (r4.1) · Issue #3084 · QubesOS/updates-status · GitHub

Addressing as well

Where no reference are done to suspend/resume issue referenced in update-status issue.

And where packages for dom0, bullseye buster and bookworm are available. No clue where/when/if packages for fedora are supposed to show

@marmarek?

Relevant:

5

The flow is more or less like this:

  1. PR is made
  2. It gets automatically build-tested on gitlab-ci. In few cases, this includes also basic unit tests. This gets reported in the PR status.
  3. PR get review; in case of less than trivial changes, PR get a “openqa-pending” label, to be included in the next openQA run (in the “pull requests” group). Those get scheduled manually, usually just after assigning the label. Results get reported back to PR comments. There is a common run for all the PRs with the label, so a failure listed there may be caused by another PR in the group (IOW, failed tests are just a hint, not a blocker).
  4. PR get merged; at this point, github automatically closes associated issues, but otherwise not much more is happening.
  5. I bump a package version (usually after merging a few PRs in given repository). At this point the package gets built and lands in current-testing repo + an issue in updates-status is created. Relevant issues (qubes-issues repo) get also automatic comment about the packages.
  6. Every Wednesday, automatic openQA run is started for package in current-testing repo (the “updates” group on openqa). Results get reported as a comment in relevant update-status issues.
  7. Every Saturday, a weekly ISO image is built (including packages from current-testing repo) and scheduled on openQA in the “Qubes OS” group.
  8. Depending on testing results, package can be moved to “current” repository after at least a week. For some less trivial changes, I prefer to keep it in current-testing repo a bit longer. I usually do review of “updates-status” issues on Monday or Tuesday.

The last point is where this group can help a lot. Even just giving :+1: or :-1: helps, as an indicator that you actually installed related package and found (or not) it working. Packages with no feedback at all tends to stay longer in current-testing repo.

Those aren’t supposed to be there (vanilla packages from Fedora are enough in VM).

You mean the one about Suspend and resume stubdomains by DemiMarie · Pull Request #139 · QubesOS/qubes-vmm-xen · GitHub ? This one just unblocks one of the points in Properly suspend all VMs, not only those with PCI devices by marmarek · Pull Request #473 · QubesOS/qubes-core-admin · GitHub , so it doesn’t closes any issues on its own - so, it isn’t linked to any.

3 Likes
6

@marmarek: This is an incomplete example of how this should be run from the testing community?

7

Yes, thanks! It will be useful to include also the link(s) to updates-status issues (vmm-xen v4.14.5-9 (r4.1) · Issue #3084 · QubesOS/updates-status · GitHub in this case). That’s where :+1: / :-1: are most useful.

8

@marmarek :
It is not clear to me the link between current-testing notes for bullseye here under vmm-xen v4.14.5-9 (r4.1) · Issue #3084 · QubesOS/updates-status · GitHub
Package for bullseye was built ([build log](https://github.com/QubesOS/build-logs/tree/master/build-debian4/log_2022-10-16_01-33-06)) and uploaded to current-testing repository

And the packages qubes-input-proxy-sender qubes-vm-dependencies qubes-vm-recommended xen-utils-guest that were update candidates with bullseye-testing uncommented, applied on other post for testing.

If it is unclear to me, this is most probably unclear as well for the testing community as well.

Can you elaborate on the meaning of vmm-xen v4.14.5-9 (r4.1) · Issue #3084 · QubesOS/updates-status · GitHub ?

Thanks

Edit: added cross-reference here: vmm-xen v4.14.5-9 (r4.1) · Issue #3084 · QubesOS/updates-status · GitHub

9

@marmarek, can you explain why some other templates have updates but not others then and how to easily understand the link between vmm-xen to v4.14.5-9 and bullseye current-testing available updates, in this case:

 qubes-input-proxy-sender qubes-vm-dependencies qubes-vm-recommended xen-utils-guest

Or in other terms, what is expected to be tested in this case on

And more importantly, why and what is the relation between dom0 package here and what differenciate fedora templates not needing specifics parallel updates and why here only a subset of supported templates have updates available that do not seem clearly related, nor having specific names of packages that would need to be co-tested for regression/functionality?

Thanks

10

In case of xen packages for Debian, there is almost nothing qubes specific there, and as such, most most fixes referenced there apply only to the dom0. The only remaining reason we build them (instead of using upstream packages directly, as in case of Fedora) is that Debian packaging insists on having the same version of libraries as the hypervisor. This matters for dom0, but is irrelevant for VM (and indeed Fedora VMs for example use libs from Xen 4.16 and it works just fine). But since the Debian packaging has this versioning requirements, we need to ship the same version (at the moment Debian stable incidentally also ships with Xen 4.14, but it isn’t always the case, and for example Debian testing has Xen 4.16). I hope this will be resolved one day in Debian too…

Long story short: for xen it’s enough to test dom0 packages.

1 Like
11

Understood for Debian and Xen specifics.
@marmarek : But then, what about centos-stream8?

12

That’s because CentOS doesn’t have own Xen packages at all.

1 Like
13

@moderators any idea how to streamline testing?

I would love all of you to read Newly populated private volumes don’t get proper SELinux labels · Issue #8242 · QubesOS/qubes-issues · GitHub

@adw we have a problem. A big one. And we need to find solutions. I’m willing to participate as I can but it needs leadership and transparence and probably documentation improvements.

1 Like
14

I have read your GitHub comment, @Insurgo. To be honest, it is difficult for me to follow your train of thought, but I will try my best.

And weekly ISO’s are also expected to contain the updated templates, which is clearly not the case here. Sidenote, the last ‘weekly’ iso is from August 8th, I do not know what’s up with that.

I don’t know. I think @fepitre is the one to ask about that.

I would suggest that the processes of openqa and weekly ISO’s builds are clarified, as well as when an issue is fixed even though templates are not updated.

I think you would be more likely to get a meaningful response from the Qubes devs if you provide a list of short, direct, concrete questions. A long essay that jumps around between observations, complaints, questions, meta-observations, meta-complaints, and meta-questions is very hard to follow and respond to.

ISO’s templates rpm havents been updated since 2 months. And a bug opened since 3 months, diagnosed confirmed and for which a fix was made made its way in packages but not in templates.

This is an example of a concrete statement that is much easier to understand and respond to. When you put it that way, I agree that it sounds like something has fallen through the cracks. It sounds like everything is working as intended except that templates were not rebuilt to include the package containing the fix, correct? Unfortunately, this is not my area, so I have no idea why that would be the case. @marmarek?

2 Likes
15

@adw marek clarified the situation at Newly populated private volumes don’t get proper SELinux labels · Issue #8242 · QubesOS/qubes-issues · GitHub

I created a separate issue to track possible solutions to real exposed problems of the past that we do not want to recreate at Track installer related bugs and needed packages (templates versions including needed packages) · Issue #8449 · QubesOS/qubes-issues · GitHub

(@adw we are all human, consequently all different with different skill sets. You now understand why I don’t like creating documentation and prefer documentation as code myself. I write walls of text to expose complete thought processes and constantly learn to better eli5/tldr myself. But complex things and dependencies are what they are. Once again nothing personal. I try to help and better the processes and tools/helpers we all depend on everywhere in the ecosystem, first for myself and for others.)

2 Likes
16

Yes, of course, I understand! Thank you for all you do! :slight_smile:

1 Like
17

Note that Qubes-4.2.202308270121-x86_64.iso forward should include the fix to use fedora templates for dispm sys-net.

Haven’t had the chance to fix it myself, but should be there. This iso+detached signature should boot from Heads.

Note the recent builds of Heads now support exfat by default. This means that if you bought a SD card or a USB thumb drive, you can download iso + iso.asc files directly to that drive and boot from USB from Heads to test it directly without having to verify either integrity or authenticity of the ISO. Heads will take care of it (tested in prior weekly iso.)

Happy testing!