voculinux/docs/PERF_BASELINE_PROTOCOL.md

Installed-build performance baseline protocol (ticket #19)

Locked measurement protocol for the Voculinux port's deferred performance numbers. The numbers themselves are recorded as evidence on the ticket, never here: until measured they are protocol, not results.

Machine class

Void Linux (glibc, x86_64), niri Wayland session, NVIDIA RTX 2050 4 GB, no CUDA toolkit installed (Vulkan then CPU backend order).

Prerequisites

• Installed build from install.sh (venv or system install), default whisper.cpp engine, debug logging available (--debug).
• Ladder models on disk with pinned checksums verified against src/vocalinux/utils/model_checksums.txt: large-v3-turbo-q8_0 (default rung), large-v3-turbo-q5_0 (first fallback), small and small.en (floor, by language).
• Isolated XDG directories so the pass never touches a live install: point XDG_CONFIG_HOME, XDG_DATA_HOME, XDG_STATE_HOME at scratch roots and link the verified models under <data>/voculinux/models/.
• Seed config.json from the app's own DEFAULT_CONFIG: engine whisper_cpp, the rung under test, keep-alive enabled with the minimum 60 s timeout, first_run false, update checks off.

The three metrics

All commands are scripts/measure_perf_baseline.py subcommands run with the installed build's interpreter.

1. Idle tray memory, both states. Start with --debug --start-minimized, sample RSS with sample-rss --pid, and read VRAM with nvidia-smi. Loaded state is right after startup; the unloaded state follows the keep-alive timeout (watch the log for reason=idle_keepalive). RSS does not reflect the unload (freed native pages stay mapped); VRAM-used is the state signal.
2. Wake latency, toggle to capture start. Drive starts over D-Bus (busctl --user call com.voculinux.Voculinux /com/voculinux/Voculinux com.voculinux.Voculinux Toggle; the --toggle client also works but spawns the full app stack). Parse repeated runs with parse-log --log: each Toggle requested in state IDLE pairs with the next Opened capture stream. Report steady-state spread separately from the first-capture cost (model reload after an idle unload lands on the first toggle).
3. Transcription real-time factor, fixed sample. Regenerate the sample with make-sample --out scripts/perf_baseline_sample.wav --seed 19 (30 s speech-like babble, 16 kHz mono 16-bit, committed next to the script so reruns are bit-identical), then transcribe --model-file … --backend vulkan|cpu --wav … --runs 3. Report the first (cold: shader/pipeline build, one-time) run separately from warm runs. CUDA is measured only where a complete toolkit exists; otherwise its absence is recorded, not worked around.

Same-pass extras

• Per-rung load footprint: probe-load --model-file … --backend … reports load wall time, child RSS, and VRAM used while the model is held. Free versus total memory is distinguished via nvidia-smi before/during/after. CPU loads touch no VRAM; their RSS is the footprint.
• Injection latency: inject-probe records the RemoteDesktop version-probe result and timing, then a session-establishment attempt (fast failure versus hang matters more than the version answer), plus the resolved first rung and a no-emission wtype "" spawn cost as a lower bound. On niri the portal probe is skipped, as the injector skips it, and the report says so with portal_skip_reason: "niri" and portal_available: false. End-to-end key-to-screen latency is deliberately not taken here: emitting keystrokes into a live desktop from a background run would disturb the session. It belongs to the hardware checklist with a dedicated target window.
• Thresholds for the model ladder: derive free-memory cutoffs from the maximum observed per-rung footprint (transcribe peak, not just load) plus explicit headroom for the compositor and other GPU clients, and record the derivation next to the numbers.

Report

render_markdown_report (and the JSON the subcommands print) carries the protocol version, full host facts, and observed numbers only. tests/test_perf_baseline_protocol.py covers the parsing, statistics, threshold-derivation, and sample-determinism helpers offline.