Local Desktop AI Leaderboard

Measured local models for everyday desktop work

Independent, reproducible benchmarks across seven real desktop AI roles—speech-to-text, speaker diarization, personal memory restraint, inline autocomplete, hybrid meeting search, structured extraction, and OCR—scored head-to-head against cloud baselines with every harness and raw report checked into Git.

Current local reference stack (~9.6 GB total on disk)

Click any role to jump to its interactive leaderboard.

Headline benchmark tables

AMI Speech-to-Text, Memory Restraint, Hybrid Retrieval & Autocomplete

Enable JavaScript to switch across all 17 suites, sort columns, filter by model footprint, view the Memory Restraint Pareto chart, or copy Markdown tables. All raw data is also available at /benchmarks.json.

SuiteLeading local configurationHeadline measurementComparison baseline
AMI Speech-to-Text (27.4 min)Qwen3-ASR 1.7B (MLX 8-bit, $0/hr)17.6% WER · 44× RT · 4.2% meeting disagreementGemini 3.8 Flash 16.7% (115 invented words) · GPT-4o Transcribe 23.0%
Speaker Diarization (8 AMI runs)Nemotron 3 offline (Core ML)14.56% DER (0 ms collar) · 8/8 speakers · 500× RTPyannote Community-1 43.44% · Muse Voice Transcribe cloud 60.4%
Memory Recall vs. Restraint (96 EN)MemT-1.0-Nano / Qwen3.5 4B defaultMemT 51/52 recall, 6 unw. · Qwen 4B 30/52, 3 unw., 95.5% emptyHy-Memory 30B 52/52 recall, 76 unwanted, 0/44 empty
Inline Autocomplete (788 pos.)Gemma 4 E2B Base Q6_K / LFM2.5 1.2B50.1% precision · 56 ms p50 (49–53 ms post-5m pause)LFM2.5 1.2B 43.0% precision · 26 ms p50 · 0.73 GB
Hybrid Retrieval (QMSum, 1,576 Qs)LokalBot Hybrid (FTS5 + EmbeddingGemma 2)0.280 R@1 · 0.475 R@4 · 0.595 R@10 · 39 ms · 50 MBGBrain Hybrid 0.239 R@1 · 0.367 R@4 · 255 ms · ~1 GB
Extraction & Tool Calls (4B)Qwen3.5 4B Q4_K_M93.0% fields · 6/6 native tools · 33.4 s warm recapMiniCPM5 2B 79.1% · Spark-X2.6-4B 12/12 short, 5/6 tools
Screenshot OCR & Local TTSApple Vision / Kokoro 82M0.971 F1 at 119.6 ms (243 ms real) · Kokoro 0.81 GiBPP-OCRv6 0.974 F1 at 6.78 s · Breeze TTS 2 6.42 GiB

Open Leaderboard

Submit a model or hardware run

Model creators, quantizers, and community members are welcome to test new models or fill missing hardware profiles (especially M1 / M2 / M3 16 GB, Linux Vulkan/CUDA, and Windows).

  1. Pick a harness under Benchmarks/ (speech, diarization, memory restraint, autocomplete, QMSum retrieval, extraction, or OCR) and keep the frozen dataset and scorer unchanged.
  2. Record exact provenance: pinned Hugging Face revision + SHA-256, runtime version, quantization, OS, chip/GPU, and raw outputs.
  3. Open a PR adding your REPORT.md and row to Website/public/benchmarks.json—or open an issue to suggest a model for the next run.

Radical Honesty

What has not been measured yet

Every score here reflects a specific harness and machine, not a universal ranking. Known gaps we still want to close:

  • M1, M2, and M3 Macs (especially 16 GB): Nearly all Apple Silicon timings are from one M4 Max (48 GB).
  • Windows & Linux Vulkan/CUDA/CPU stack: Only memory restraint and Spark-X2.6 ran on Linux/Windows so far; full sherpa-onnx ASR and GPUI Vulkan/CUDA timings are pending.
  • 30-minute idle pauses under RAM pressure: The 5-minute autocomplete warm-up fix is measured (49–53 ms), not 30-minute OS file paging.
  • Blind human summary grading: Extraction suites check structured fields and tools, not blinded human ratings of full meeting recaps.
  • 5–8 speaker calls & noisy multilingual rooms: Diarization covers 4-speaker English AMI rooms; non-English ASR slices are small FLEURS sets.

FAQ

Methodology & common questions

What hardware were these benchmarks measured on?

Primary desktop runs used an Apple M4 Max MacBook Pro with 48 GB of unified memory. The memory-restraint and Spark-X2.6 suites ran on an Ubuntu Linux AMD Ryzen 7 9700X host (64 GB RAM), and the community-suggested MemT-1.0-Nano extension ran with experimental NF4 expert offload on Windows with a Radeon RX 6600 XT. M1, M2, and M3 Macs have not been measured yet.

How did local Qwen3-ASR 1.7B compare with closed speech-to-text APIs?

On 27.4 minutes of overlapped AMI Mix-Headset audio, local Qwen3-ASR 1.7B scored 17.6% WER at 44× realtime and $0/hr, ahead of GPT-4o Transcribe (23.0%), GPT Transcribe (22.2%), GPT-4o mini Transcribe (21.8%), and Alibaba's cloud Qwen APIs, and within confidence intervals of Gemini 3.8 Flash (16.7%). On three real two-track meetings, Qwen3-ASR 1.7B had the lowest consensus disagreement (4.2%), while Gemini 3.8 Flash invented 115 words on quiet clips and GPT-4o Transcribe dropped 278 words.

Why do memory benchmarks score both recall and restraint?

A personal memory writer that saves every sentence can reach 100% recall while polluting a user's profile with temporary tasks, third-party details, and excluded sensitive facts. Our 120-case suite scores eligible recall alongside unwanted proposals and empty cases correctly left blank so tradeoffs stay visible.

Can I reproduce these results or submit a new model or chip run?

Yes. Every suite links directly to its harness, frozen cases, pinned model hashes, and raw results in the Benchmarks folder on GitHub, and all aggregated rows are served as machine-readable JSON at lokalbot.com/benchmarks.json. Open a pull request with a new model or hardware run (such as M1–M3, Linux Vulkan/CUDA, or Windows) or file an issue to suggest a model.

Built for LokalBot

Run this entire stack locally.

LokalBot is the free, GPLv3 desktop app behind these harnesses—no account, no meeting bot, no cloud upload.

Download for macOS

Apple Silicon · macOS 15+ · or with Homebrew: brew install --cask stevyhacker/tap/lokalbot

macOS · Windows · Linux (alpha) · requirements