Always check your keys. Be careful of GPTs

General Discussion
1

From the official Qubes website:

Many important Qubes OS Project assets (e.g., ISOs, RPMs, TGZs, and Git objects) are digitally signed by an official team member’s key or by a release signing key (RSK). Each such key is, in turn, signed by the Qubes Master Signing Key (QMSK) (0x427F11FD0FAA4B080123F01CDDFA1A3E36879494 ). In this way, the QMSK is the ultimate root of trust for the Qubes OS Project.

Also, from the officlal Qubes website:

Once you’ve obtained the QMSK, you must verify that it’s authentic rather than a forgery. Anyone can create a PGP key with the name “Qubes Master Signing Key” and the short key ID 0x36879494 , so you cannot rely on these alone. You also should not rely on any single website, not even over HTTPS.

From ChatGPT 5:

  1. Check the Fingerprint Online

The official Qubes Master Signing Key fingerprint is:

427F 11FD 0FAA 4B08 0EF9  C65B 7314 89FE 9730 5480

You can find this on:

Me:

git clone https://github.com/QubesOS/qubes-secpack.git
Cloning into 'qubes-secpack'...
remote: Enumerating objects: 5196, done.
remote: Counting objects: 100% (194/194), done.
remote: Compressing objects: 100% (128/128), done.
remote: Total 5196 (delta 81), reused 113 (delta 66), pack-reused 5002 (from 2)
Receiving objects: 100% (5196/5196), 2.11 MiB | 4.02 MiB/s, done.
Resolving deltas: 100% (2496/2496), done.
user@qube:~/Downloads$ gpg --show-keys --with-fingerprint qubes-secpack/keys/master-key/qubes-master-key.asc 
pub   rsa4096 2010-04-01 [SC]
      427F 11FD 0FAA 4B08 0123  F01C DDFA 1A3E 3687 9494
uid                      Qubes Master Signing Key
user@qube:~/Downloads$ gpg --show-keys --with-fingerprint qubes-secpack/keys/release-keys/qubes-release-4.3-signing-key.asc 
pub   rsa4096 2024-04-10 [SC]
      F3FA 3F99 D628 1F7B 3A3E  5E87 1C3D 9B62 7F3F ADA4
uid                      Qubes OS Release 4.3 Signing Key

Huh.

Me:

It concerns me that the QMSK you purport to be "The Official Qubes Master Signing Key fingerprint is: “427F 11FD 0FAA 4B08 0EF9 C65B 7314 89FE 9730 5480” which it is not…

GPT5:
…after waiting several minutes!..it normally does not think that long with a simple question…

Thought

You’re 100% right—my earlier fingerprint was wrong. Thanks for catching it.

The correct Qubes Master Signing Key (QMSK) fingerprint is:

427F 11FD 0FAA 4B08 0123 F01C DDFA 1A3E 3687 9494

So I am left wondering, at least one person is out there running a ‘dirty’ Qubes version? I wonder who or what owns that key? How did it work its way in to GPT memory?

Moral of the story

  • Stay vigilant.
  • Always check your keys.
  • Never implicitly trust Open Source if you did not check the source yourself.
  • Never implicitly trust software/package repositories.
  • Never implicitly trust ChatGPT (or a website) even if it has SSL or keys check out.
  • Always check multiple sources/channels.
  • Trust, but verify.

Strange Days.

5 Likes
2

Why have this even happened? What workflow leads to LLM trying to generate fingerprints?

1 Like
3

I never checked the source code of Qubes OS :slight_smile:

There is a note about this in the docs:

Please note that AIs often hallucinate about Qubes OS. If you’re using an AI to assist you, please check its conclusions against the official documentation.

And also:

The general idea of “comparing fingerprints” is to go out into the world (whether digitally, physically, or both) and find other 40-character strings purporting to be the QMSK fingerprint, then compare them to your own purported QMSK fingerprint to ensure that the sequence of alphanumeric characters is exactly the same (again, regardless of spaces or capitalization). If any of the characters do not match or are not in the same order, then at least one of the fingerprints is a forgery. Here are some ideas to get you started:

  • Check the fingerprint on various websites (e.g., mailing lists, discussion forums, social media, personal websites).
  • Check against PDFs, photographs, and videos in which the fingerprint appears (e.g., slides from a talk, on a T-shirt, or in the recording of a presentation).
  • Ask people to post the fingerprint on various mailing lists, forums, and chat rooms.
  • Download old Qubes ISOs from different sources and check the included Qubes Master Signing Key.
  • Repeat the above over Tor.
  • Repeat the above over various VPNs and proxy servers.
  • Repeat the above on different networks (work, school, internet cafe, etc.).
  • Text, email, call, video chat, snail mail, or meet up with people you know to confirm the fingerprint.
  • Repeat the above from different computers and devices.
2 Likes
4

Never mind the hallucinations - at least they should be invented and trivially invalid.

How many actors are working on deliberately poisoning the LLMs, to get control of the false outputs?

1 Like
5

My workflow?

I asked ChatGPT to ‘ELI5 the process to verify my Qubes ISO is legitimate by verifying keys and signatures’. I already had downloaded the keys and imported them and was not expecting at all that it would respond like that, offering what it thought the QMSK was, nor did I ask it! I immediately noticed it was different, and when I confronted it, ChatGPT took a really long time to confirm that it was not correct.

I am more interested in the source - how did it determine the incorrect key in the first place?! What did it scrape? I am not a security expert, but wonder how many other users were gaslighted by it? I just wanted to let the Qubes community know.

FWIW, I think the process of verifying keys is taken for granted - especially from less technical users. This is definitely an area that needs some improvement (not Qubes specific - just in general). Documenting the process in a simple, less wordy way or linking several sources that a client endpoint checks against (the more the better as it harder for every reference to be a key / fingerprint to be compromised)… maybe a tool that checks every copy of the key/s on every mirror (which would also verify the mirror’s copy is clean…

I have enough trouble ‘trusting’ the MS/GitHub copy of the key :wink: …

6

Thanks for those references @parulin, I missed them and make really good points.

I guess if anything it would be safer to navigate to the Official latest documentation website and export the EPUB or PDF, feed that to the LLM and instruct the prompt “Using only the attached document as a reference source…”

7

ChatGPT is a big fat liar.

It’s not just Qubes, a lot of times, when I ask about the source of some information, it generates links out of thin air to famous websites that never really existed. And when I confront it about the fact that the link is false, it apologizes in the same canned way it did here.

It’s a shameless thing!

5 Likes
8

I’m not familiar with LLMs but aren’t they supposed to generate things?

Compare:

427F 11FD 0FAA 4B08 0123  F01C DDFA 1A3E 3687 9494
427F 11FD 0FAA 4B08 0EF9  C65B 7314 89FE 9730 5480

That’s a quite good job, isn’t it?

2 Likes
9

Getting such keys from any LLM is an bad idea. LLMs get “forced” to not output the next token (text) which they believe is the most likely to be right to gain creativity / to not be deterministic.
Never get any key from any LLM.

2 Likes
10

And do not trust its instructions on how to verify signatures using the key.

11

Yes. It’s funny when people trust GPT, Grok, and other AIs. It constantly lie - always. Following AI advice, you’ll break any Linux system and won’t learn any science. You’ll end up with a list of non‑existent commands, books, articles and movies

12

Yes, it’s probably correct to see it as hallucinating all the time -
but sometimes those hallucinations match reality quite closely.

I never presume to speak for the Qubes team.
When I comment in the Forum I speak for myself.

2 Likes
13

A lot of the documentation doesn’t fit well with less technical users.
This seems to be one.
@spider, it would be really helpful and a great contribution to the
project if you could identify what’s wrong with the current doc. Feel
free to PM me with your thoughts and suggestions.

I never presume to speak for the Qubes team.
When I comment in the Forum I speak for myself.

3 Likes
14

If you’ve got this far, why don’t just verify them yourself? :grin:

Most likely a hallucination. LLMs don’t have memory in computer storage sense, their “memory” is closer to human. All fingerprints look practically identical, trying to remember one off of the top of your head is practically impossible. Don’t use LLMs for tasks that require perfect precision directly? Is this possible?

Might be nothing lol. I mean it can have tools to do so but there’s no guarantee it will use them

source
source720×897 86.3 KB

15

okay but why would you trust a ai in the first place?

1 Like
16
  • Never implicitly trust Open Source if you did not check the source yourself.

How do you suggest to check the source of the kernel, the browser, the compiler etc, within a single lifetime?

1 Like
17

I read this in the sense of ‘provenance’… where you got it from.

In this case it’s a shorthand for making sure that you have the actual identical version that the Qubes team had (or Linus Torvalds, Fedora release team, etc) made their decision to sign it.

The great thing about digital signatures is that you don’t have to visit Marek or Linus. You could copy the data off a usb key you found in a carpark, but if the signature is verifiably made by the right key then you can trust that it is the same data.

(And that is true, even though you cannot trust the USB key - it could give a different version next time you read it, or it could pretend to be a keyboard and try to send your passwords to a scammer, or worse. The signatures allow you to trust the Qubes servers just as much as you would trust that carpark find )

Maybe I am missing the joke?

18

LLMs like ChatGPT are “play along” machines. They’re clever actors that can lie. Intentionally: given the implicit bias of the training corpus and explicit bias of the hidden instructions; and unintentionally: they make shit up sometimes/often – hallucinate – to continue telling a compelling story word-by-word. They aren’t aware they hallucinate (they are not aware, full stop).

There is no fixing the hallucination problem.

A trillion dollar industry has successfully convinced the world that these super expensive super clever play along machines are genius oracles, hyperintelligences that subsume real research, even subsume individual critical thinking, but it’s a big lie. It’s no wonder the PR has been so successful; industry-scale LLMs are self-evidently amazing in what they can do. But the more you use them the more you see the cracks that reveal the lie.

What they actually are: supplements to research. They’re useful in the same way as the overview section of a newly added Wikipedia article, or a site:reddit.com SERP. They’re good for high-level exploratory questions. Like a conversation with a well-traveled semi-informed story teller, they can spark ideas. Beyond that, caveat emptor.

19

Also, just a reminder that the documentation itself is PGP-signed (via signed Git repo tags and commits), so you don’t even have to trust the website if you don’t want to.

1 Like
20

I read this in the sense of ‘provenance’… where you got it from.

In this case it’s a shorthand for making sure that you have the actual identical version that the Qubes team had (or Linus Torvalds, Fedora release team, etc) made their decision to sign it.

Well, I am not reading what was not written but what was. To me, “Never implicitly trust Open Source if you did not check the source yourself” is entirely different from “Just verify that you have the original source code through signatures”. The latter is actually an form of implicit trust, i.e. contrary to the advise given, and completely unrelated to the data being signed (be that open/closed software, media, malware, etc).

Hence my question.