June 15, 2026 · build log
The smallest model wins.
I built a free, all-local web-research harness for the smallest useful model I had — llama3.2:3b, 2 GB, running on a laptop. Then I froze a benchmark of real research questions, answered by Claude doing live browsing, and made the little model clear the same bar. It hits 36/41 (88%) — and beats the 8B at roughly 3× the speed. The lesson is the whole point: small models fail on harness, not intelligence.
Repo: github.com/alejandroclvi/tamagotchi · MIT · free, local, offline. Bring your own 2 GB model.
The question
I run a lot of agents. Most of the bill is paid API calls — a frontier model doing work a much cheaper model could do, if only it were steered correctly. So I asked a concrete question: how much of that can a free local model replace once the task is broken into simple steps and the model is kept honest?
Not as a vibe. As a number. I picked the hardest, most expensive thing a small model is “supposed” to be bad at — open-ended web research — and built a harness to measure exactly where it breaks.
The benchmark forces real work
A small model can recite “Canberra” from its weights. That tests memory, not research. So every task in the set is built to be unanswerable from the model's own head — version numbers, a line off a specific GitHub repo, a fact that lives on no single page. The answer key is frozen: each question was solved by Claude doing live browsing, and that answer is the bar.
The set is ranked easiest → hardest, and the ranking is the build order:
- Tiers 1–3 — land on the canonical page, read one fact, then chain across hops (a film → its director → his birth year). The test: can a 3B carry data between steps?
- Tier 4 — facts on no wiki page: a latest version number, a product site, a tagline on a GitHub repo. Needs a real search backend.
- Tier 6 — “scattered web”: the right answer is on the authoritative doc, and a plausible-but-wrong distractor outranks it.
- Tier 7 — gated properties: recognize a login / paywall, stay out, and report honestly instead of fabricating from behind the wall.
The scoreboard
Every line below is the same frozen set, same machine, the harness getting one capability smarter each pass:
| model | score | what changed |
|---|---|---|
| 3b | 7/16 | baseline — Wikipedia-only fetcher |
| 3b | 14/16 | + auto-finish + real search (SearXNG) → ~5× faster |
| 8b | 13/16 | same harness, 2.3× bigger model — slower AND lower |
| 3b | 22/23 | + multi-hop decompose-chain + a critic faculty |
| 3b | 36/41 | + deep-read, anti-fabrication, gate-awareness, grounding |
Read the highlighted row again: the 8B is bigger, slower, and scores lower than the 2 GB model on the exact same harness. The extra parameters bought nothing here — they just cost latency. Once “sense” is baked into deterministic primitives, the intelligence lives in the harness, not the weights.
What the harness actually does
The model is demoted to one job: a text-locator over content it actually retrieved. Everything that makes the research correct is deterministic code wrapped around it.
- Don't trust the model. Every extracted answer must literally appear in the retrieved snippets or page. If it doesn't, the model spoke from memory — so it's re-verified on a real page, or it's a miss, not a guess. This single rule killed the confabulations.
- Deep-read the authoritative source. The top snippet is often a plausible distractor that ranks high. The harness scores results by authority (entity-in-host, official-doc hosts, content-farms penalized), opens the real source, and reads it whole — the answer often sits past where a snippet stops.
- URL-liveness gate. Every link in the final answer is HTTP-checked. A fabricated or dead URL gets replaced with a real retrieved one or stripped. The model doesn't get to invent citations.
- Multi-hop decompose-chain. A planner breaks a question into a chain of one-fact lookups; the fast model forages each; answers carry forward. A 2 GB model resolving “2016 Olympics host → country → capital → Brasília” is orchestration of small models, not one big one.
- Gate-awareness. It recognizes a login / signup / paywall, does not try to log in, reads whatever is public, and if everything funnels to a wall it stops and says so — honest by construction.
Why I'm building it this way
The whole industry is racing to put more intelligence in the model. That intelligence is rented, per-call, and gone the moment the request ends. The cheaper bet is to put it in the harness — where it compounds, persists, and runs on hardware you already own. A flaky small model behind a smart harness beats a smart model behind no harness, and it does it for free.
The build is eval-driven end to end: a frozen answer key, tiers as the build order, and a rule that every regression has to show up as a dropped number before I'll believe a change was an improvement. That's what keeps it honest — about the model, and about me.
Next, it stops being a benchmark and starts being a tool: wrap the crawler as enrich(lead) and point it at a real outreach list — name in, a real LinkedIn URL, phone, site, and one grounded fact out. Same harness, doing paid work for free.