Another question that pops up a lot, not just on mine, but other channels aswell, is how to configure a GPU passthrough in proxmox, and how to install nvidia drivers. In order to run an AI model locally (e.g. in a VM natively or with docker) using an nvidia GPU fully you need to install 3 essential packages:
- CUDA Drivers
- Nvidia Drivers
- Docker Containers Nvidia Toolkit (if you are running the models from a docker container in Linux)
However, these drivers alone are not enough. You also need to install a bunch of pre-requisites such as linux-headers and other things to get the drivers and GPU up and running.
So, I decided to make a detailed video about how to run AI models (Censored and Uncensored) on Windows, Mac, Linux, Docker and how you can get all that virtualized via proxmox. It also includes how to conduct a GPU passthrough.
The video can be seen here and it covers the following:
00:00 Overview of what's to come
01:02 Deepseek Local Windows and Mac
2:54 Uncensored Models on Windows and MAc
5:02 Creating Proxmox VM with Debian (Linux) & GPU Passthrough in your homelab
6:50 Debian Linux pre-requirements (headers, sudo, etc)
8:51 Cuda, Drivers and Docker-Toolkit for Nvidia GPU
12:35 Running Ollama & OpenWebUI on Docker (Linux)
18:34 Running uncensored models with docker linux setup
19:00 Running Ollama & OpenWebUI Natively on Linux
22:48 Alternatives - AI on your NAS
Along with the video, I also created a medium article with all the commands and step by step how to get all of this working available here .