Need Help with QubesOS GPU Passthrough for AI-Powered Data Management System (PIVOT & TOOL)

[sly0bvio]

Hello QubesOS community

The following is a post I have created with project details processed through a GPT-4o model, creating an output that was proofread then modified (information was added for clarity or removed for ambiguity or inaccuracy) and then further revised in this manner until it was time to post.

I’m currently working on Ethical AI Governance of Ecosystemic Resources, I’d be glad to explain more if you ask… There is a sub-project called PIVOT (Personalized Information Vault & Organization TOOL). The goal of this project is to empower individuals to take control of their personal data, enabling them to store, organize, and process it in a way that maximizes autonomy, privacy, and impact. Through TOOL (Transfer & Organization Operations Layer), users will be able to offload AI tasks like data processing to systems running QubesOS, in a secure way for AI inference and training, running QubesOS or another OS-Virtualization system, using GPU passthrough to allow AI tasks to operate with GPU power.

• PIVOT will give users control over their personal data by leveraging layered AI to answer key questions, helping them organize data and make informed decisions.
• TOOL is the “Layered AI Information & Discussions” system, designed to transfer and process user data through AI while maintaining high levels of privacy and security.
• The system will support querying from external nodes, allowing offloaded AI tasks via GPU passthrough to handle complex, resource-heavy processes.

Why QubesOS?

QubesOS is the perfect foundation for this project because of its security-by-isolation model, and the large community of passionate experts who browse the forums is a perfect space to get ideas flowing for something new and different, for ALL of US The idea is to set up virtual machines (VMs) that will handle different layers of data processing and transfer, securely communicating between various nodes. The GPU passthrough will enable AI processing tasks to be offloaded, while QubesOS ensures that each process runs in its own isolated environment.

Challenges I Need Help With:

1. Setting Up GPU & AI in QubesOS:
   I’ll be using a Xeon-based server with a dedicated GPU to handle AI inference and training. While I’ve looked into PCI passthrough for QubesOS, I’m encountering some challenges in:

   • Making sure the GPU is isolated correctly for use with specific VMs handling AI workloads.
   • Ensuring security when passing the GPU to VMs, especially when handling sensitive data.

2. Input/Output Security:
   The system will need to handle inputs from multiple sources (other computers/nodes on the network) and return outputs after processing.

   • What’s the best way to manage inputs and outputs securely through QubesOS? For instance, ensuring that user queries are safely processed and that results go through the proper layers or filters before being returned.
   • I’m also concerned about how to safely allow access to certain VMs while ensuring that each process is properly filtered through the hardware and software.

3. Operational Security:
   Maintaining operational security (OpSec) is a major priority for this project, as the system will be dealing with sensitive user data.

   • How do I set up QubesOS to handle incoming queries and outputs securely while ensuring that the VMs handling data transfers don’t leak sensitive information?
   • What are the best practices for sandboxing AI models in VMs without exposing data or models to unnecessary risks?

4. Optimizing Hardware and Software Layers:
   With QubesOS, I want to make sure that the hardware (e.g., CPU, GPU, RAM) and software are optimized to handle multiple VMs running AI models while maintaining high performance. Any advice on:

   • Which VM setups or configurations work best for this kind of workload?
   • How to balance the hardware resources between different VMs handling various stages of data processing?

Why This Matters:

By successfully implementing this system, we will enable individuals to process their personal data more securely and effectively, without needing to rely on external services that often come at the expense of privacy. With QubesOS and the power of AI, we can reduce the need for currency trade by enabling resource-sharing and smart data management.

I’m hoping the community can provide guidance on how best to set up this system. Any advice, resources, or experience with GPU passthrough and secure VM setups in QubesOS would be greatly appreciated.

Additionally, if YOU made a system like this… what would YOU do with that system? I want to understand YOU…

Thanks in advance for any help or support!

Regards,
Sly 0bvio
0BV.IO/U/SLY
I will update this later…

[sly0bvio]

Nobody has a response for this?

[otter2]

There is no TL;DR and my attention span is far too short to keep reading until I get the full concept

As far as I can see the idea is weird. Users querying your server? How is that better than just asking OpenAI?

Sorry but I don’t think that I can help that much with any of your technical questions because I’m not that knowledgeable, but here’s my take:

@sly0bvio

Making sure the GPU is isolated correctly for use with specific VMs handling AI workloads.

Check IOMMU groups, passed dGPU alone in the group is good.

@sly0bvio

What’s the best way to manage inputs and outputs securely through QubesOS? For instance, ensuring that user queries are safely processed and that results go through the proper layers or filters before being returned.

idk what are they coming from and to what? Probably whatever is the best generic network practice or something like the clipboard mechanism

@sly0bvio

I’m also concerned about how to safely allow access to certain VMs while ensuring that each process is properly filtered through the hardware and software.

Why don’t just run these in disposables and flush them for new queries?

@sly0bvio

How do I set up QubesOS to handle incoming queries and outputs securely while ensuring that the VMs handling data transfers don’t leak sensitive information?

I don’t think this is qubes-related, more like generic network security. Disposable data transfer handling vms?

@sly0bvio

What are the best practices for sandboxing AI models in VMs without exposing data or models to unnecessary risks?

Don’t give them access to network, I guess

@sly0bvio

Which VM setups or configurations work best for this kind of workload?

For now, it is one (or more if you need it) dGPU per vm. In development. There are alternatives, like this:

Seamless GPU passthrough on Qubes OS with VirtualGL

We are presenting to you a guide regarding GPU passthrough created in collaboration with peers. Qubes OS already has extensive documentation available about GPU passthrough for 3D accelerated tasks but they all require in depth configuration, extra displays, and extra input devices. With VirtualGL (https://virtualgl.org) one can take advantage of the existing Qubes OS framework for audio/display by offloading OpenGL onto the secondary graphics card directly with VirtualGL. This document will mo…

And also hybrid graphics and OPTIMUS if hardware allows:

Using Hybrid Graphics to play games with seamless GPU acceleration

Recently got a new motherboard (X670E Tomahawk) that supports Hybrid Graphics, and it works with Qubes OS. [hybrid-graphics-small] Here is a picture of my games qube running Magic Arena in window mode, with the rendering being done by my 4060 GPU. The GPU needs to be passed to the qube running the game, it doesn’t allow you to share the GPU with multiple qubes, it only allows you to use the GPU without having a dedicated display connected. It doesn’t really require any extra configuration, …

Apart from that, CPUs and RAM are very flexible. Main restriction is that initial memory of a qube (with memory balancing enabled) cannot be less than 10% of the max dedicated memory of a qube.

Sorry if what I’m stating here is obvious to you. This all is surface-level stuff and I’m pretty sure you can find lesser-known solutions and projects related to your questions in the community.