Measuring autocomplete without fooling ourselves
Our original 28-scenario autocomplete check had five flaws: it treated an intentional confidence-gate silence as a safety failure, accepted only one exact word tail for partial-word completions, mixed 10–40 s cold startup into steady-state latency, stole keyboard focus by launching the full app, and never measured next-word accuracy.
We replaced it with a headless engine runner (--cotyping-bench) that exercises the exact production LlamaCotypingRuntime with hybrid decoding, certified prefix search, and prefix healing across 28 safety/behaviour checks (5 deterministic passes) plus 96 phrases replayed at all 788 word boundaries and 8 phrases typed one character at a time.
Gemma 4 E2B Base vs. LFM2.5 1.2B Instruct (Apple M4 Max)
| Metric | Gemma 4 E2B Base (Balanced) | LFM2.5 1.2B Instruct (Lightweight) |
|---|---|---|
| Unsafe results (28 checks × 5 passes) | 0 | 0 |
| Word completions | 12 / 13 | 12 / 13 |
| Next-word accuracy (788 positions) | 23.1% | 20.7% |
| English accuracy (624 positions) | 25.3% | 24.2% |
| Non-English accuracy (164 positions) | 14.6% | 7.3% |
| Right when shown (precision) | 50.1% | 43.0% |
| Per keystroke while typing (median / p95) | 56 / 102 ms | 26 / 76 ms |
| Model load time | 0.65 s | 0.16 s |
What the numbers actually say
- In English, the two models are effectively tied (+1.1 points, 95% bootstrap interval −1.4 to +3.5), while LFM2.5 is 2–3× faster per keystroke (26 ms median) and weighs only 0.73 GB.
- Gemma 4 E2B earns the Balanced default on precision and non-English text. When it chooses to show a suggestion, it is right 50.1% of the time vs. 43.0% for LFM2.5, and its non-English lead (+7.3 points) comes largely from languages like Serbian where LFM2.5 stayed silent.
- Silence is a feature. Both models withheld suggestions on the exact same 3 open-ended prompts (
notes-list,comma-clause,sentence-start) and stayed quiet mid-word (rec|eive), producing zero unsafe suggestions across all 140 passes.
First suggestion after a pause (November 18, 2026)
Steady-state speed only tells half the story: after three minutes without a decode, llama.cpp releases Metal weight residency (180 s default keep-alive), and the GPU clocks down within seconds. On an M4 Max (Gemma 4 E2B Q6_K), warming the runtime with a single throwaway token on wake, unlock, click, or the first keystroke (then removing that token so the cached prompt stays intact) brings the first suggestion back to steady-state latency:
| Idle pause | Before | Warm-up at key | Warm-up just before | Warm reference |
|---|---|---|---|---|
| 20 s | 61 ms | 51 ms | 47 ms | 46–50 ms |
| 60 s | 59 ms | 48 ms | 50 ms | 48–50 ms |
| 300 s (5 min) | 95–130 ms | 50–53 ms | 49 ms | 47–49 ms |
Four warm-up touches over a five-minute pause added 18 mJ of CPU energy (468 vs. 450 mJ) and never reload a model freed under memory pressure. A 30-minute pause under memory pressure—where OS file paging dominates—has not been re-measured yet. Full methodology and raw logs live in Benchmarks/autocomplete/Cotyping/results/2026-11-18-first-after-pause/REPORT.md.
Choosing between them in LokalBot
Both presets ship built into LokalBot: Balanced local uses Gemma 4 E2B Base, and Lightweight local switches to LFM2.5 1.2B Instruct for faster typing and a smaller disk footprint. For a feature-by-feature comparison with Cotypist and Cotabby, see Autocomplete that knows your meetings. Full category breakdowns and commands to rerun the engine harness are in Benchmarks/autocomplete/Cotyping/results/2026-10-15-engine-benchmark/REPORT.md.