Benchmark

56 ms local system-wide autocomplete: Gemma 4 E2B vs. LFM2.5 1.2B

System-wide inline autocomplete only feels invisible when it stays under ~100 ms per keystroke and knows when to stay silent. Here are the measured numbers behind LokalBot's Balanced and Lightweight presets.

Measuring autocomplete without fooling ourselves

Our original 28-scenario autocomplete check had five flaws: it treated an intentional confidence-gate silence as a safety failure, accepted only one exact word tail for partial-word completions, mixed 10–40 s cold startup into steady-state latency, stole keyboard focus by launching the full app, and never measured next-word accuracy.

We replaced it with a headless engine runner (--cotyping-bench) that exercises the exact production LlamaCotypingRuntime with hybrid decoding, certified prefix search, and prefix healing across 28 safety/behaviour checks (5 deterministic passes) plus 96 phrases replayed at all 788 word boundaries and 8 phrases typed one character at a time.

Gemma 4 E2B Base vs. LFM2.5 1.2B Instruct (Apple M4 Max)

Autocomplete engine benchmark comparing Gemma 4 E2B Base Q6_K and LFM2.5 1.2B Instruct Q4_K_M
MetricGemma 4 E2B Base (Balanced)LFM2.5 1.2B Instruct (Lightweight)
Unsafe results (28 checks × 5 passes)00
Word completions12 / 1312 / 13
Next-word accuracy (788 positions)23.1%20.7%
English accuracy (624 positions)25.3%24.2%
Non-English accuracy (164 positions)14.6%7.3%
Right when shown (precision)50.1%43.0%
Per keystroke while typing (median / p95)56 / 102 ms26 / 76 ms
Model load time0.65 s0.16 s

What the numbers actually say

First suggestion after a pause (November 18, 2026)

Steady-state speed only tells half the story: after three minutes without a decode, llama.cpp releases Metal weight residency (180 s default keep-alive), and the GPU clocks down within seconds. On an M4 Max (Gemma 4 E2B Q6_K), warming the runtime with a single throwaway token on wake, unlock, click, or the first keystroke (then removing that token so the cached prompt stays intact) brings the first suggestion back to steady-state latency:

First suggestion latency after idle pauses on Apple M4 Max (Gemma 4 E2B Q6_K)
Idle pauseBeforeWarm-up at keyWarm-up just beforeWarm reference
20 s61 ms51 ms47 ms46–50 ms
60 s59 ms48 ms50 ms48–50 ms
300 s (5 min)95–130 ms50–53 ms49 ms47–49 ms

Four warm-up touches over a five-minute pause added 18 mJ of CPU energy (468 vs. 450 mJ) and never reload a model freed under memory pressure. A 30-minute pause under memory pressure—where OS file paging dominates—has not been re-measured yet. Full methodology and raw logs live in Benchmarks/autocomplete/Cotyping/results/2026-11-18-first-after-pause/REPORT.md.

Choosing between them in LokalBot

Both presets ship built into LokalBot: Balanced local uses Gemma 4 E2B Base, and Lightweight local switches to LFM2.5 1.2B Instruct for faster typing and a smaller disk footprint. For a feature-by-feature comparison with Cotypist and Cotabby, see Autocomplete that knows your meetings. Full category breakdowns and commands to rerun the engine harness are in Benchmarks/autocomplete/Cotyping/results/2026-10-15-engine-benchmark/REPORT.md.

Try it locally

Meeting notes that stay on your computer by default.

Free, open source, and no account. For Mac, with Windows and Linux in alpha.

Download for macOS

Apple Silicon · macOS 15+ · or with Homebrew: brew install --cask stevyhacker/tap/lokalbot

macOS · Windows · Linux (alpha) · requirements