Need Help with QubesOS GPU Passthrough for AI-Powered Data Management System (PIVOT & TOOL)

[otter2]

There is no TL;DR and my attention span is far too short to keep reading until I get the full concept

As far as I can see the idea is weird. Users querying your server? How is that better than just asking OpenAI?

Sorry but I don’t think that I can help that much with any of your technical questions because I’m not that knowledgeable, but here’s my take:

@sly0bvio

Making sure the GPU is isolated correctly for use with specific VMs handling AI workloads.

Check IOMMU groups, passed dGPU alone in the group is good.

@sly0bvio

What’s the best way to manage inputs and outputs securely through QubesOS? For instance, ensuring that user queries are safely processed and that results go through the proper layers or filters before being returned.

idk what are they coming from and to what? Probably whatever is the best generic network practice or something like the clipboard mechanism

@sly0bvio

I’m also concerned about how to safely allow access to certain VMs while ensuring that each process is properly filtered through the hardware and software.

Why don’t just run these in disposables and flush them for new queries?

@sly0bvio

How do I set up QubesOS to handle incoming queries and outputs securely while ensuring that the VMs handling data transfers don’t leak sensitive information?

I don’t think this is qubes-related, more like generic network security. Disposable data transfer handling vms?

@sly0bvio

What are the best practices for sandboxing AI models in VMs without exposing data or models to unnecessary risks?

Don’t give them access to network, I guess

@sly0bvio

Which VM setups or configurations work best for this kind of workload?

For now, it is one (or more if you need it) dGPU per vm. In development. There are alternatives, like this:

Seamless GPU passthrough on Qubes OS with VirtualGL

We are presenting to you a guide regarding GPU passthrough created in collaboration with peers. Qubes OS already has extensive documentation available about GPU passthrough for 3D accelerated tasks but they all require in depth configuration, extra displays, and extra input devices. With VirtualGL (https://virtualgl.org) one can take advantage of the existing Qubes OS framework for audio/display by offloading OpenGL onto the secondary graphics card directly with VirtualGL. This document will mo…

And also hybrid graphics and OPTIMUS if hardware allows:

Using Hybrid Graphics to play games with seamless GPU acceleration

Recently got a new motherboard (X670E Tomahawk) that supports Hybrid Graphics, and it works with Qubes OS. [hybrid-graphics-small] Here is a picture of my games qube running Magic Arena in window mode, with the rendering being done by my 4060 GPU. The GPU needs to be passed to the qube running the game, it doesn’t allow you to share the GPU with multiple qubes, it only allows you to use the GPU without having a dedicated display connected. It doesn’t really require any extra configuration, …

Apart from that, CPUs and RAM are very flexible. Main restriction is that initial memory of a qube (with memory balancing enabled) cannot be less than 10% of the max dedicated memory of a qube.

Sorry if what I’m stating here is obvious to you. This all is surface-level stuff and I’m pretty sure you can find lesser-known solutions and projects related to your questions in the community.