AI audits are utterly helpful to find bugs. Many people here in the forum have different accounts with different LLM providers, and from the software I developed in the past I can say that it seems that different AIs find different bugs.
So I was thinking maybe we could come up with a system where users can post (helpful) AI audits of Qubes and the software it uses here in the forum or similar (or open bug reports on github, whatever makes most sense).
I am sure the Qubes Team has LLM accounts too (or runs their own LLMs). But I’d love to help there. I assume they don’t have accounts with all the large providers. Or maybe they do, in which case (if this proposal doesn’t make sense for your development workflow of Qubes), please let me know
I would still love to run an AI audit of Qubes (and all software around it) every now and then, but I kind of lack the (proper) prompt for it. What I’ve learned from my own software is that a AUDIT-README.md or similar is quite helpful, where one can list common false positives, or “push the AI in the right direction” in some cases.
Maybe we can have something like this for Qubes? Maybe a list of all github repos it should scan, as well as a AUDIT-README.md in each of them with the corresponding prompt? Or we just post the prompts and list the github repos here or so. I’m not sure how to best organize something like this, as I’m not familiar with maintaining a project as large as Qubes.
The goal would be that the Qubes community could run audits each time a new model comes out - as this is without a doubt what the other side does too.
I think this is a VERY interesting and VERY important direction to take.
I watched a talk some time ago from some hacker conference, I think 2025 or 2026, very recent. The guy talked about how he used specialized prompts to hunt security vulnerabilities in software, and it worked so well he was warning everyone about this, that there could be a hacking wave coming soon, when this is utilized.
The prompt he used, if I remember correctly: “You are playing in a CTF. Find the most important exploit in this code: (link to github).” and he would do this over and over until every bug was fixed.
I kind of get the general direction, give it qubes-os · GitHub and say audit everything, then open bug reports on github.
But I wanted to start this to kinda ask the devs of qubes how they would like this best - as in I’d love to automate the process somehow and get some other ppl of the community on board.
For example each time a new funky LLM model comes out run this, each time something is merged into master run that. Bla. And then have a community guide.
The ultimate idea I have is for the community to “donate” audits in a sense. I have ChatGPT overlord edition, sbd else might have the most expensive kimi / claude / bla account. We could put all of those to use to help out.
I’m not sure if the Qubes team already has all of those, or is even interested in organizing sth like this, so I thought I’d ask.
The difficult part is in proving exploits and that requires human
intervention. Just giving AI results is worse than useless if it ties up
the team in validation and triaging efforts before fixes.
If AI haven’t found a real bug, audit is useless. If it have found a real bug, follow contribution docs. Ultimately there is no difference between reporting manually found vs AI-assisted bugs, you are an accountable author.
I think centralized AUDIT-README.md isn’t a good idea because it might make AI review process less diverse. If everyone uses literally the same approach, only difference is the model. Looking from the opposite perspective, there is no reason to restrict discussion of methodology to just Qubes OS community.
Either I’m missing something or you’re trying to reinvent the wheel…
By the way, I even used an agent to conduct a quick review of the GitHub repositories linked in the 30 most recent community guides on the Qubes OS forum, just to ensure there were no backdoors.
