"AMD appears to lack sufficient testing...and may harbor latent bugs" - Trail of Bits SecureDrop Workstation Review

User Support Hardware Issues
1

Oleg_Artemiev’s qubes-users mailing list thread AMD with latest Qubes - is it now “all relatively okay”? reminded me of something I’d read in the Freedom of the Press Foundation SecureDrop Workstation Assessment.

In the Trail of Bits SecureDrop Workstation Audit 2020 on page 20 in the section “Only support Intel hardware, as AMD appears to lack sufficient testing” Trail of Bits states:

“AMD systems have likely received far less testing on Qubes, and may harbor latent bugs due to differences in low-level support for hardware virtualization.”

“It would not be prudent to recommend AMD systems due to the possibility of latent bugs in Qubes OS on AMD platforms.”

My threat landscape is certainly very different from SecureDrop Workstations, but…

Anyone care to comment?

Full disclosure: I have been happily using Qubes on a AMD IOMMU system for two years without issue.

2 Likes
2

Interesting. It seems to advocate using the worst-known brand in terms of security by employing ‘the devil we know’ reasoning while ignoring the larger body of security research.

The Qubes organization should do something to address this lack of understanding, lest it develop into a (poorly justified) rule of thumb in the community. Qubes should be supportive of users having hardware brand choices, most particularly with CPUs as that component is the most trusted here. And Qubes claims to portability look pale if it can’t even vouch for more than one brand of x86.

2 Likes
3

There’s talk that side-channel attacks like Spectre and Meltdown are here to stay, since it seems to be an unfixable side effect of speculative execution. Intel has dropped the ball in its recent history, and I doubt AMD is better by a wide margin, even though I have a better impression of the red team right now. I’m just daydreaming here, but I’d love for there to eventually be an ARM-compatible Qubes.

4

There is something even better on the horizon: Port Qubes to ppc64 [3 bitcoin bounty] · Issue #4318 · QubesOS/qubes-issues · GitHub

1 Like
5

Any suggestions?

(One thing to bear in mind is that we want to be somewhat selective about which misunderstandings we address. If we had a policy of always trying to correct everyone’s misunderstandings, trolls could feign misunderstanding to DoS the project. Come to think of it, even the sheer volume of sincere misunderstands might have the same effect.)

I think it’s fair to say that we currently already do support users having hardware brand choices and already do support more than one brand of x86. I take it that you agree, and the sense of “should” here is that of “ought to continue to be the case” rather than “is not the case but ought to be”?

6

The SecureDrop auditors appear to be reacting to Qubes developers not running/testing AMD hardware themselves. From the Hardware Testing page:

Tested Models

These hardware models have been (and continue to be) tested and work well with Qubes OS:

Lenovo ThinkPad X1 Carbon Gen 5
Lenovo ThinkPad x230

Note: The Lenovo X and T series are similar enough to assume similar compatibility of the matching model from the other series.
Desired Models

The Qubes devs would like to test these models, but the hardware is not available to us. If anyone is willing to lend or donate these models to us, we would be happy to test them:

Lenovo Thinkpad T or X series with Intel 8th Gen CPU and integrated graphics
Dell Latitude with Intel 8th Gen CPU and integrated graphics

There is no mention of AMD on that page or the Certified Hardware page, no postings or other discussion showing developer enthusiasm for AMD anti-sidechannel features like SEV and wider availability of ECC RAM support, despite Qubes developers having a long history of bemoaning side-channel vulnerabilities.

But what SecureDrop has pointed out is a distinct lack of interest on the part of Qubes for even the basics, giving the appearance that AMD support is all user-driven.

1 Like
7

I’m having AMD Ryzen 1800XT, Ryzen 3900XT and a ThreadRipper 3960X. I would say they work as expected under Qubes OS, the first for years and the two last for months.

2 Likes
8

I’ll also take a moment to stress that this is a BAD state of affairs!

Qubes has made portability a big deal, and now that all looks like a joke when an auditor comes along and says a nearly identical ISA isn’t sufficiently supported. What do we then make of the money and effort thrown at alternative ISAs like OpenPOWER? Also user-driven… and also to be ignored as insufficiently supported?

One of the main reasons Qubes supposedly placed emphasis on portability was due to trust issues with major component manufacturers. You had an easy path forward with this one; to robustly support an x86 alternative implementation. But what we got is an entrenched pattern of Intel leg-hugging and a crappy schism over what is considered respectable hardware that IMO leads concerned users to the wrong vendor.

1 Like
9

From my perspective, a big problem is that the source of this misunderstanding comes from Trail of Bits.
https://www.trailofbits.com/about
Not to mention the Freedom of the Press Foundation and The New York Times involvement.

Wish I had useful suggestions.

10

Egads… “We help our clientele — ranging from Facebook to DARPA — lead their industries.”

Is that supposed to be reassuring?

Edit:

If this group defines themselves in terms of shoring up market dominance–if that is the first impression they’re comfortable making–then nothing whatsoever this group says about Intel vs AMD can be taken as unbiased. My kinder side suggests these people are akin to stewards on the S.S. Titanic.

2 Likes
11

Glad to hear you say that @tasket

Perhaps, but now a forum identified Qubes Team member @fepitre and yourself have begun to change that appearance.

My main concern is the audit report’s use of qualified pseudo technical statements like “may harbor latent bugs due to differences in low-level support for hardware virtualization” and "the possibility of latent bugs in Qubes OS on AMD platforms.” If this was just some guy on qubes-users I’d probably just ignore them but it wasn’t.

1 Like
12
13

Perhaps, but now a forum identified Qubes Team member @fepitre and yourself have begun to change that appearance.

Talking in the forums is one thing, but there is still no official profile for AMD systems that would give it any weight. For that matter, see my prior message & edit; we may be barking up the wrong tree.

To put it plainly, I don’t like industry groups that are about dominant players and keeping them on top. I don’t like assemblages of the “Top Talent” who ostensibly got the IT industry in such an awful situation in the first place. It smacks of corruption.

And I also now have to question the direction and leadership of the Qubes project for doing, lately, I don’t know what in the face of this Titanic chair-shuffling. From my perspective, the 4.1 release has languished under a lost cause style objective of relegating GPUs as somehow untrusted and separate from users’ core experience. It has failed to express a vision of hardware that would better fit Qubes and better serve a typical Qubes user, thereby rending its potential for influence in this vital area; we are still a community that merely surfs atop what the consumer market throws at us.

We have configuration management snafus that are compounded each passing month, because devs don’t see anything wrong with picking various settings as needed to be managed under saltstack. We don’t know what new settings we’re going to have to play whack-a-mole with bc there’s no plan to how salt will expand over normal configs. And there’s no regard for a regular user’s interests amongst these changes, such as providing a way to opt out.

1 Like
14

@tasket @Justin Kind reminder that the community guidelines encourage to “[be] respectful of the topics and the people discussing them, even if you disagree with some of what is being said” and to avoid ad hominen attacks.

The way I see it: if we’re here to learn and contribute to making people safer, we need to understand rather that assume the reasons for the behaviors we disagee with. We can only do that if people feel welcome and respected enough to join the conversation, and we all make an effort to provide reasoned arguments that improve it.

We may disagree on what that means to some extent, but I think we can agree to criticizing ideas (or behaviors), not people.

4 Likes
15

To be clear, my point in describing these Qubes developments is to highlight how much effort goes into navel-gazing, even counterproductive misfeatures while something as basic as in-house testing of a valuable x86 implementation has been ignored.

There was not even a mention of AMD in Qubes’ hardware donation requests.

And now Qubes is tacitly accepting vague aspersions that AMD virtualization should be avoided because its a “different implementation” than Intel. Its as if Xen and KVM users are strangers to IOMMU passthrough; What Qubes doesn’t test on AMD means no one else tests it. Very puzzling indeed. Imagine what Trail of Bits think of data centers that are drawn to AMD SEV to handle sensitive information; assuming they do have an opinion of AMD in data centers (bc “deep expertise”, I think they must) the mental gymnastics must be impressive.

Extending the devil we know logic to software, Qubes in its entirety becomes unacceptably insecure. If the same firm is hired not for the benefit of helping a group of Qubes users, but for some other purpose… Their assessment–should they cast their gaze back in this direction–could very well be just that. Qubes is not well tested in relation to more established operating systems as it is a different implementation of isolation mechanisms on the desktop––AVOID. By accepting work from a group invested in Qubes technology, and taking Qubes as a given, a glass house has been constructed.

In spring of 2018–on the heels of Meltdown and Spectre–Trail of Bits was hired by CTS Labs to back up their claims of a supposedly severe set of AMD vulns headlined as Ryzenfall, having given AMD only 1 day to respond. They turned out to be essentially a dud (required admin and/or firmware flashing privs) but not before CTS whipped up a tech media storm against AMD. The Anandtech interview is interesting, especially when CTS cut it short:

DK: I think the biggest question that I still have is that ultimately who originated this request for analysis – who was the customer that kicked this all off?

ILO: I definitely am not going to comment on our customers.

DK: What about the flavor of customer: is it a semiconductor company, is it someone in the industry, or is it someone outside the industry? I don’t expect you to disclose the name but the genre seems quite reasonable.

ILO: Guys I’m sorry we’re really going to need to jump off this call but feel free to follow up with any more questions.

CTS Labs demanded AMD recall all Ryzen processors “for public safety” and also simultaneously displayed on their ryzenfall site a disclaimer that the contents were merely opinions and not statements of fact… That is not only unheard-of for security research, it is how propaganda orgs operate to manipulate the public.

Many people, including Linus Torvalds, said CTS’ claims (which appear to have been channeled in advance through an investment firm) looked like stock manipulation, and I have to agree.

Trail of Bits’ role as CTS hired hands appears to be quashing skepticism about the severity of the bugs. The “Twitter hot takes” ToB complained about were emphasizing the need to have local admin privs first. ToB bemoaned stirring a “hornet’s nest”, an uncanny act of projection.

There has to be remediation of Qubes’ lack of judgement and courage here, and I’m not talking about damage-control PR tactics. Merely repeating “AMD is supported” won’t dispel the shadow that has been cast on Qubes’ integrity; it will darken the OpenPOWER port (if/when its finished) because that hardware will have even less testing on Qubes, orders of magnitude less than AMD. Any aspect of Qubes development could be (and arguably has been) undermined by this kind of bias.

1 Like
16

@tasket Thank you for the research and your passion. I learned a lot reading about the ryzenfall episode. Makes me wonder about a 3rd party with an agenda.

I agree and believe this is one of those issues that must be addressed.

Below are just some thoughts

@adw One suggestion is rewording and/or updating the Can I use AMD-v instead of VT-x? FAQ entry to try to head off this “misunderstanding”. Damage-control, for sure, but perhaps worth considering.

Has an automated testing expert been found as a result of the Get paid to support Qubes development through automated testing! announcements?
Curious if you think ensuring AMD systems are included in future testing plans might help combat misinformation/misperceptions about the project’s support for AMD x86 implementations? I realize this could merely be a PR tactic, and maybe I’m being naive, but if done openly couldn’t that be demonstrative evidence?

Also, would I be mistaken to assume that someone on the team would accept an AMD system donation?
My ignorance about the process involved in editing the Qubes’ hardware donation page may be showing here, but isn’t it easily altered?

@fepitre Any AMD systems you would like to try? Another ThreadRipper 3960X? Something else?

1 Like
17

What do you mean exactly? The only remaining CPU family which is not really tested is EPYC (by tested is only my usage). Currently, we are trying to setup automated tests on bare-metal hardware like certified laptops but we need more time notably to find an easy way to interact automatically with the hardware (like any LOM). Also, previous listed CPU are bought by myself. I don’t have spare/fund for buying such CPUs on my own.

18

@Justin Those are good suggestions.

Specifically, AMD systems should have representation in a preferred hardware list (which already exists for Intel) and the hardware donation section and also be visible in the testing regimen. But this is not enough.

Qubes should push back against the Trail of Bits assessment in a public statement. IMO, it should follow the rationale that the features Qubes uses at the virtualization level are no longer new or unusual.

1 Like
19

The Trail of Bits recommendation to “only use Intel” was targeted at Secure Drop - not Invisible Things Labs or “random person interested in using Qubes OS on their laptop”. Threat models may differ significantly. Qubes OS, Xen, and CPU architecture was not the primary focus. According to SD:

Too much is being read into Trail of Bits’ recommendation, though they could have worded the recommendation better. Trail of Bits may have searched for evidence of AMD support or testing and found very little. While that may not be the reality, to their point as well as @tasket’s - more outward statements on AMD support would benefit the community. ToB linked to the official Qubes OS FAQ that asks “can I use AMD” - while the answer is yes, it links to a 10 year old mailing list post. The more worthy action from the recommendation is to improve that answer by stating the level of support AMD does receive, major issues or concerns if one is using an AMD-powered device, and what the team would like from the community to help improve support [does not always have to be funding or donated hardware - it could start with more HCL reports].

The focus of the audit was not Qubes OS, but a solution that uses Qubes OS to fulfill a goal. If there is not clear evidence of testing, maintaining, or supporting something, it is not unreasonable to suggest moving to a platform that is, despite reservations. This is partly security, but moreso avoiding usability and availability impacts. The recommendation does not say AMD or other architectures are inherently bad. There would have been more than 4 usages of “x86” if they were to go to that conclusion.

I also doubt level of testing by the Qubes team is testing for architectural exploits. Evaluating and publishing of QSBs that are mostly Xen or CPU related is sufficient from a security standpoint, and Qubes already does that. I count 7 QSBs that at least mention AMD. This is why I believe the “latent bugs” comment is targeted at usability than security, and perhaps is being evaluated too closely.

2 Likes
20

No, we’re not. We have neither the time nor the inclination to reply to everything that is publicly said about Qubes OS. Not addressing something that someone else said doesn’t mean we tacitly endorse it. I already explained this to you above.

This is libel, and it will get you nowhere.

PRs welcome: Redirecting…

I believe so, but I’m not sure if the folks involved are ready to make a public statement about it yet.

I don’t make hardware testing decisions, but my personal opinion is that hardware testing should be about whatever makes the most sense and is the most useful for Qubes users and for the project, not concerns about whether we’re publicly displaying enough affection for one corporate brand over another.

@marmarek, could you use any?

Of course, just as easy as the rest of the docs: Redirecting…

2 Likes