How to Run a Self-Hosted AI Assistant Without Sending Data to the Cloud

Every message you type into ChatGPT, Claude, or Gemini leaves your machine and lands on someone else's server. For personal projects, that might be fine. For business data, client information, medical records, legal documents, or anything you would not email to a stranger, it is not.

The good news: running a capable AI assistant on your own hardware stopped being a science project about a year ago. The models got smaller. The tooling got better. And the gap between local and cloud quality narrowed enough that most daily tasks do not need a $200/month API bill.

This guide walks through exactly how to set one up, what hardware you need, and where the tradeoffs still exist.

Why Self-Host an AI Assistant?

Before getting into the how, it is worth being clear about the why. Self-hosting is not for everyone, and pretending otherwise wastes your time.

Reasons that hold up

• Data never leaves your network. Not to OpenAI, not to Google, not to any third party. Full stop.
• No per-token billing. Once you own the hardware, inference is free. Heavy users save thousands per year.
• Zero downtime from provider outages. When OpenAI goes down (and it does, regularly), your local assistant keeps running.
• GDPR and compliance. If you handle EU citizen data, self-hosting eliminates an entire category of data processing agreements.
• Customization. Fine-tune on your own data. Add tools, integrations, and workflows that cloud providers do not support.
• Offline capability. Works on planes, in basements, in rural areas, anywhere your hardware lives.

Reasons that do not hold up

• "Local models are just as good as GPT-4." They are not. Not yet. For complex reasoning, creative writing, and multi-step planning, cloud models still win. The gap is closing, but it exists.
• "It is free." Hardware costs money. Electricity costs money. Your time debugging CUDA drivers costs money.
• "Set it and forget it." Models update. Dependencies break. You will maintain this.

If your use case survives that reality check, keep reading.

What Hardware Do You Actually Need?

The internet is full of conflicting advice here. Some guides suggest you need an NVIDIA A100. Others claim a Raspberry Pi works fine. Both are technically correct and practically useless.

Here is what matters: how much RAM (or VRAM) you have determines which models you can run.

The practical tiers

Tier 1: 8GB RAM (CPU inference)

• Models: Phi-3 Mini, Gemma 2B, TinyLlama
• Speed: 5-10 tokens/second
• Good for: Simple Q&A, text classification, basic summarization
• Hardware: Any modern laptop or desktop
• Reality check: Usable but slow. Fine for background tasks. Frustrating for conversation.

Tier 2: 16-24GB RAM or 8-12GB VRAM

• Models: Llama 3.3 8B, Mistral 7B, Qwen 2.5 7B, GLM-4 9B
• Speed: 15-30 tokens/second (GPU), 8-15 (CPU)
• Good for: Most daily tasks, writing, coding assistance, research
• Hardware: Mac Mini M4 (24GB), gaming PC with RTX 3060/4060
• Reality check: This is the sweet spot. 90% of what people use cloud AI for works fine here.

Tier 3: 32-64GB RAM or 16-24GB VRAM

• Models: Llama 3.3 70B (quantized), Mixtral 8x7B, DeepSeek V3 (quantized), Qwen 2.5 32B
• Speed: 20-40 tokens/second (GPU), 5-15 (CPU)
• Good for: Complex reasoning, long documents, coding, analysis
• Hardware: Mac Studio M2/M4 Ultra, RTX 4090, dual GPU setups
• Reality check: Approaches cloud quality for most tasks. Significant hardware investment.

Tier 4: 128GB+ RAM or multi-GPU

• Models: Full-precision 70B+, Llama 405B (quantized), DeepSeek R1
• Speed: Varies wildly by setup
• Good for: Research, enterprise deployment, model development
• Hardware: Mac Pro, multi-GPU servers, dedicated inference boxes
• Reality check: Overkill for personal use. Makes sense for teams or businesses.

The Apple Silicon advantage

Apple's unified memory architecture changed the self-hosting game. A Mac Mini M4 Pro with 24GB of unified memory runs 7-9B parameter models at genuinely usable speeds because the GPU and CPU share the same memory pool. No CUDA drivers. No VRAM limitations. No fan noise from a gaming GPU running inference at 100%.

For most people reading this, a Mac Mini is the simplest path to a capable local AI assistant.

Step-by-Step Setup

Step 1: Install Ollama

Ollama is the easiest way to run local models. It handles model downloads, quantization, and serving behind a simple API.

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Verify installation
ollama --version

On macOS, you can also download the app directly from ollama.com. It runs as a menu bar application and starts the server automatically.

Step 2: Pull your first model

Start with something practical. Qwen 2.5 7B offers a strong balance of capability and speed:

# Pull a general-purpose model
ollama pull qwen2.5:7b

# Test it
ollama run qwen2.5:7b "Summarize the key differences between GDPR and CCPA in 5 bullet points"

For coding tasks, try CodeQwen or DeepSeek Coder. For creative writing, Mistral or Llama 3.3 tend to produce more natural output.

Step 3: Set up a persistent assistant

Running models through the CLI works for testing. For daily use, you want something that stays running and accepts requests from other tools.

Ollama already runs a local API server on port 11434. Any tool that speaks the OpenAI API format can connect to it:

# The server is already running after installation
# Test the API
curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5:7b",
  "prompt": "What is the capital of Sweden?",
  "stream": false
}'

Step 4: Connect a frontend

The raw API is powerful but not pleasant to use daily. Several open-source frontends give you a ChatGPT-like interface:

• Open WebUI (formerly Ollama WebUI): the most popular option. Supports multiple models, conversation history, file uploads, and RAG.
• LibreChat: multi-provider interface that works with both local and cloud models.
• Jan: desktop app with a clean interface, built specifically for local models.

For Open WebUI:

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

Visit http://localhost:3000 and connect it to your Ollama instance.

Step 5: Add agent capabilities with OpenClaw

A chatbot answers questions. An agent takes actions. OpenClaw bridges that gap by giving your local models the ability to run commands, manage files, search the web, send messages, and interact with external services.

npm install -g openclaw
openclaw init

During setup, point OpenClaw at your local Ollama instance as a model provider. Now your self-hosted AI can do more than just chat. It can monitor your inbox, manage your calendar, run scripts, and automate workflows, all without any data leaving your machine.

The key difference: cloud AI assistants are limited to what the provider allows. A self-hosted agent does whatever you configure it to do.

Model Selection Guide

Not all local models are created equal. Here is a practical breakdown for common tasks:

General conversation and Q&A

• Best: Qwen 2.5 7B, Llama 3.3 8B
• Why: Broad knowledge, fast responses, good at following instructions
• Quantization: Q4_K_M offers the best speed/quality balance

Code generation and debugging

• Best: DeepSeek Coder V2 Lite, CodeQwen 1.5
• Why: Trained specifically on code, understands multiple languages
• Note: For complex codebases, cloud models still have an edge

Writing and content creation

• Best: Mistral 7B, Llama 3.3 8B Instruct
• Why: More natural language output, better at maintaining tone
• Tip: Higher quantization (Q5 or Q6) noticeably improves writing quality

Document analysis and summarization

• Best: Qwen 2.5 7B (32K context), Mistral 7B (32K context)
• Why: Long context windows let you feed entire documents
• Limitation: Context quality degrades past ~16K tokens in practice with 7B models

Multilingual tasks

• Best: Qwen 2.5 (excellent CJK + European), Llama 3.3 (good European languages)
• Why: Training data diversity matters enormously for non-English tasks

Security Hardening

Running a local AI assistant is only private if you secure it properly. A few things most guides skip:

Network isolation

• Bind Ollama to localhost only (default behavior, do not change it unless you have a reason)
• If you need remote access, use a VPN or SSH tunnel. Never expose port 11434 to the internet.

Model supply chain

• Download models only from ollama.com or Hugging Face. Verify checksums when available.
• Quantized models from random GitHub repos can contain modified weights. Stick to known sources.

System access

• If your AI agent can run shell commands, sandbox it. Use a dedicated user account with limited permissions.
• OpenClaw's permission system lets you allowlist specific commands and directories.

Data handling

• Local models can still leak information through conversation history stored on disk. Encrypt your home directory.
• If you process sensitive documents, clear the model's context between sessions.

Cost Comparison: Self-Hosted vs. Cloud

Real numbers matter more than vibes. Here is a practical comparison for someone using AI daily:

The math is clear for heavy users. For someone who sends 10 messages a day, cloud APIs are cheaper. For someone who sends 100+, self-hosting pays for itself within months.

Common Pitfalls and How to Avoid Them

"My model is slow."
Check whether you are running on CPU or GPU. On Mac, verify with ollama ps that the model loaded into GPU memory. On Linux/Windows, ensure CUDA is installed and detected.

"Responses are gibberish."
Usually a quantization issue. Try a higher quantization level (Q5 instead of Q4) or a different model entirely. Some models quantize poorly.

"It forgets context mid-conversation."
You hit the context window limit. 7B models practically work well up to ~8K tokens. For longer conversations, use a model with a larger context window or summarize periodically.

"It hallucinates facts."
All LLMs do this. Local models do it more than cloud models because they are smaller. For factual tasks, combine your local model with web search (RAG) or use it for drafting rather than fact-checking.

"I cannot get it to follow instructions."
System prompts matter more with smaller models. Be explicit. Use structured output formats. Provide examples of what you want.

When to Stay on the Cloud

Self-hosting is not always the answer. Keep using cloud APIs when:

• You need state-of-the-art reasoning (legal analysis, complex research, novel problem-solving)
• Your team needs simultaneous access and you do not want to manage infrastructure
• You need multimodal capabilities (vision, audio) that local models handle poorly
• You process less than $20/month worth of API calls

The smart approach is hybrid: route sensitive and high-volume tasks to your local model, and use cloud APIs for the 10% of tasks that genuinely need frontier capabilities.

What Comes Next

The local AI landscape moves fast. A few trends worth watching:

• Smaller models, better quality. Phi-3 proved that 3B parameter models can be surprisingly capable. Expect more of this.
• Hardware improvements. Each Apple Silicon generation and NVIDIA GPU release makes local inference faster and cheaper.
• Better tooling. Projects like Ollama, OpenClaw, and Open WebUI are making self-hosting accessible to non-engineers.
• Specialized models. Instead of one giant model, expect collections of small, task-specific models that work together.

The trajectory is clear: self-hosted AI is getting better every quarter. Starting now means you build the skills and infrastructure before it becomes table stakes.

OpenClaw makes self-hosted AI agents practical. Connect local models, add tools and automations, and keep your data on your own hardware. Get started with OpenClaw.