Run a Capable Local Model with Ollama

Tier: Core · Hands-on time: ~45 minutes

This playbook gets you from nothing to a working, honest setup — including the hardware reality check most tutorials skip.

When to use

• •
  You handle data that must not leave your machine (client deliverables, HR, legal, personal finances).
• •
  You want free unlimited usage for high-volume tasks (extraction, summarization, drafts).
• •
  You're building the restricted-environment variants of the other playbooks in this track.

Prerequisites

Hardware reality check — do this first. The model must fit in memory; what you can run well:

• •
  8 GB RAM: 3–4B models only. Usable for extraction and short summaries; will disappoint on reasoning.
• •
  16 GB RAM (or 8 GB GPU VRAM): 7–9B models — the sweet spot for this track's workflows.
• •
  32 GB RAM / 16+ GB VRAM: 12–14B comfortably, 27–32B with patience.
• •
  Apple Silicon Macs punch above their weight (unified memory); pure-CPU laptops run 7–9B models at a slow-but-usable few words per second.

Also: ~10 GB free disk, admin rights to install, terminal basics.

The workflow

1. 1.

   Install Ollama. Download the installer from the official Ollama site (macOS/Windows), or on Linux:

   Copy

   curl -fsSL https://ollama.com/install.sh | sh
   

   Verify: ollama --version.

2. 2.

   Pull a current 7–9B instruct model. Browse the Ollama model library for a current-generation small instruct model (the library front page lists them; names change quarterly — pick the recent one sized to your RAM class):

   Copy

   ollama pull <model-name:8b>
   ollama list
   

3. 3.

   Smoke-test in chat.

   Copy

   ollama run <model-name:8b>
   

   Try "Summarize in 3 bullets:" followed by two pasted paragraphs from any document. Exit with /bye. Check resource behavior with ollama ps (shows loaded model and memory) while it runs.

4. 4.

   Calibrate honestly — 10 minutes that set correct expectations. Run these and grade the answers:

   • •
     An extraction task from pb-03 (expect: good).
   • •
     A summary of a 2-page text (expect: good).
   • •
     A niche factual question from your domain (expect: confident errors — local small models hallucinate more than frontier hosted ones).
   • •
     A multi-step reasoning task (expect: mediocre). Write down what it's good for; that's your routing rule for pb-10.
5. 5.

   Use it from other tools. Ollama serves an API on localhost:11434:

   Copy

   curl http://localhost:11434/api/generate -d '{
     "model": "<model-name:8b>",
     "prompt": "Extract all dates from: Invoice due 15 Aug 2026, issued July 1.",
     "stream": false
   }'
   

   For JSON extraction add "format": "json". Most local-AI GUIs and editor extensions can point at this endpoint.

6. 6.

   Set your defaults. Keep one small fast model and (RAM permitting) one larger model. Update occasionally: ollama pull re-fetches; ollama rm <model> reclaims disk.

Tool variants

• Ollama (this playbook): terminal-first, scriptable API, the de-facto standard local runtime.
• LM Studio: GUI-first alternative — model browser, chat UI, and an OpenAI-compatible local server; friendlier for non-terminal users, same models underneath.
• llama.cpp: the engine underlying much of the ecosystem; maximum control and performance tuning, meaningful setup cost — choose it only when you need what it uniquely offers.

Verification

• •
  ollama run answers a prompt at a tolerable speed with no swapping (ollama ps memory fits your RAM; system stays responsive).
• •
  The curl API call returns valid output — proves other tools can integrate.
• •
  Your calibration notes exist: at least two task types graded "good," at least one honestly graded "poor."
• •
  Disk and memory accounted for: you know what ollama list shows and what each model costs in GB.

Pitfalls

• Expecting frontier quality. An 8B local model is roughly a capable intern for mechanical text tasks, not a senior analyst. Use it where pb-01/02/03 workflows apply; route hard reasoning to hosted tiers (pb-10).
• Oversized model syndrome. Pulling the biggest model that technically loads yields 1 token/sec and abandonment. Size to the reality check.
• Skipping calibration. Users who skip step 4 either overtrust (dangerous) or dismiss local models entirely (wasteful).
• Exposing the API.localhost:11434 has no authentication. Never bind it to 0.0.0.0 on a shared network without putting a proxy with auth in front.
• Quantization confusion. Same model name, different quant tags = different quality/size. Default tags are sensible; note which one you tested.

Restricted-environment variant

This playbook is the restricted-environment foundation — data never leaves the machine. Additional hardening for strict client environments:

• •
  Confirm policy allows installing local software and downloading model weights (several GB from the internet); air-gapped sites may require offline transfer of model files.
• •
  Keep client work in a separate model context: start fresh sessions per client; Ollama doesn't persist chat history across run sessions by default, but your terminal scrollback does — clear it.
• •
  Document for compliance: model name, version tag, and that inference is fully local — auditors ask.

Success criteria

• •
  A model appropriate to your hardware answers prompts locally at usable speed.
• •
  API endpoint verified with curl — you can wire it into other playbooks' restricted variants.
• •
  Written calibration notes: three task types you'll trust it with, one you won't.
• •
  You have completed one real work task (an extraction or summary) fully offline.