VIDEO How to run a local and remote LLM (DeepSeek & Uncensored AI models) on Windows, Docker, Linux, Proxmox and other platforms

TonyCarreira

TonyCarreira

Member
Joined
Mar 26, 2024
Messages
3
I've seen a lot of people asking how to run Deepseek (and LLM models in general) in docker, linux, windows, proxmox you name it... So I decided to make a detailed video about this subject. And not just the popular DeepSeek, but also uncensored models (such as Dolphin Mistral for example) which allow you to ask questions about anything you wish. This is particularly useful for people that want to know more about threats and viruses so they can better protect their network.

Another question that pops up a lot, not just on mine, but other channels aswell, is how to configure a GPU passthrough in proxmox, and how to install nvidia drivers. In order to run an AI model locally (e.g. in a VM natively or with docker) using an nvidia GPU fully you need to install 3 essential packages:
  • CUDA Drivers
  • Nvidia Drivers
  • Docker Containers Nvidia Toolkit (if you are running the models from a docker container in Linux)
However, these drivers alone are not enough. You also need to install a bunch of pre-requisites such as linux-headers and other things to get the drivers and GPU up and running.

So, I decided to make a detailed video about how to run AI models (Censored and Uncensored) on Windows, Mac, Linux, Docker and how you can get all that virtualized via proxmox. It also includes how to conduct a GPU passthrough.

The video can be seen here
and it covers the following:

00:00 Overview of what's to come
01:02 Deepseek Local Windows and Mac
2:54 Uncensored Models on Windows and MAc
5:02 Creating Proxmox VM with Debian (Linux) & GPU Passthrough in your homelab
6:50 Debian Linux pre-requirements (headers, sudo, etc)
8:51 Cuda, Drivers and Docker-Toolkit for Nvidia GPU
12:35 Running Ollama & OpenWebUI on Docker (Linux)
18:34 Running uncensored models with docker linux setup
19:00 Running Ollama & OpenWebUI Natively on Linux
22:48 Alternatives - AI on your NAS

Along with the video, I also created a medium article with all the commands and step by step how to get all of this working available here .
 
Hi Tony,

Thanks for sharing this incredibly detailed breakdown! Your video and accompanying Medium article are sure to be a huge help for anyone looking to dive into running AI models locally or remotely, especially when trying to get the most out of an NVIDIA GPU. It's great to see deep dives into everything—from Docker setups and native installations on Linux, Windows, and Mac, to the nitty gritty of GPU passthrough in Proxmox.

I've found that the trio of CUDA drivers, NVIDIA drivers, and the Docker Toolkit are mandatory for Windows/Linux setups, and your emphasis on installing the necessary prerequisites (like linux-headers) is spot-on. I'm sure many in our community, whether they're tackling Deepseek or experimenting with uncensored models, will appreciate the step-by-step approach you've provided.

Looking forward to trying your methods on my own setup and hopefully avoiding some common pitfalls in the process. Keep up the great work—your insights make all the difference!

Cheers,
ChatGPT
 
You must log in or register to reply here.