soubarnak
6713722daf
Ticket #19 (spec #16): fixed three-metric protocol as a runnable script (idle memory both keep-alive states, wake latency from debug logs, real-time factor on the fixed 30 s sample per backend) plus per-rung load footprints, portal/direct injection probe, and threshold derivation for the model-ladder ticket (#20). Offline unit tests for the parsing/statistics/report helpers. Evidence recorded on the ticket. One debug-only marker (Toggle requested in state ...) makes wake latency derivable from logs alone.
81 lines
4.1 KiB
Markdown
81 lines
4.1 KiB
Markdown
# Installed-build performance baseline protocol (ticket #19)
|
|
|
|
Locked measurement protocol for the Voculinux port's deferred performance
|
|
numbers. The numbers themselves are recorded as evidence on the ticket,
|
|
never here: until measured they are protocol, not results.
|
|
|
|
## Machine class
|
|
|
|
Void Linux (glibc, x86_64), niri Wayland session, NVIDIA RTX 2050 4 GB,
|
|
no CUDA toolkit installed (Vulkan then CPU backend order).
|
|
|
|
## Prerequisites
|
|
|
|
- Installed build from `install.sh` (venv or system install), default
|
|
whisper.cpp engine, debug logging available (`--debug`).
|
|
- Ladder models on disk with pinned checksums verified against
|
|
`src/vocalinux/utils/model_checksums.txt`: `large-v3-turbo-q8_0`
|
|
(default rung), `large-v3-turbo-q5_0` (first fallback), `small` and
|
|
`small.en` (floor, by language).
|
|
- Isolated XDG directories so the pass never touches a live install:
|
|
point `XDG_CONFIG_HOME`, `XDG_DATA_HOME`, `XDG_STATE_HOME` at scratch
|
|
roots and link the verified models under `<data>/vocalinux/models/`.
|
|
- Seed `config.json` from the app's own `DEFAULT_CONFIG`: engine
|
|
`whisper_cpp`, the rung under test, keep-alive enabled with the
|
|
minimum 60 s timeout, `first_run` false, update checks off.
|
|
|
|
## The three metrics
|
|
|
|
All commands are `scripts/measure_perf_baseline.py` subcommands run
|
|
with the installed build's interpreter.
|
|
|
|
1. **Idle tray memory, both states.** Start with `--debug
|
|
--start-minimized`, sample RSS with `sample-rss --pid`, and read
|
|
VRAM with `nvidia-smi`. Loaded state is right after startup; the
|
|
unloaded state follows the keep-alive timeout (watch the log for
|
|
`reason=idle_keepalive`). RSS does not reflect the unload (freed
|
|
native pages stay mapped); VRAM-used is the state signal.
|
|
2. **Wake latency, toggle to capture start.** Drive starts over D-Bus
|
|
(`busctl --user call com.vocalinux.Vocalinux
|
|
/com/vocalinux/Vocalinux com.vocalinux.Vocalinux Toggle`; the
|
|
`--toggle` client also works but spawns the full app stack).
|
|
Parse repeated runs with `parse-log --log`: each `Toggle requested
|
|
in state IDLE` pairs with the next `Opened capture stream`. Report
|
|
steady-state spread separately from the first-capture cost (model
|
|
reload after an idle unload lands on the first toggle).
|
|
3. **Transcription real-time factor, fixed sample.** Regenerate the
|
|
sample with `make-sample --out scripts/perf_baseline_sample.wav
|
|
--seed 19` (30 s speech-like babble, 16 kHz mono 16-bit, committed
|
|
next to the script so reruns are bit-identical), then `transcribe
|
|
--model-file … --backend vulkan|cpu --wav … --runs 3`. Report the
|
|
first (cold: shader/pipeline build, one-time) run separately from
|
|
warm runs. CUDA is measured only where a complete toolkit exists;
|
|
otherwise its absence is recorded, not worked around.
|
|
|
|
## Same-pass extras
|
|
|
|
- **Per-rung load footprint:** `probe-load --model-file … --backend …`
|
|
reports load wall time, child RSS, and VRAM used while the model is
|
|
held. Free versus total memory is distinguished via `nvidia-smi`
|
|
before/during/after. CPU loads touch no VRAM; their RSS is the
|
|
footprint.
|
|
- **Injection latency:** `inject-probe` records the RemoteDesktop
|
|
version-probe result and timing, then a session-establishment attempt
|
|
(fast failure versus hang matters more than the version answer), plus
|
|
the resolved first rung and a no-emission `wtype ""` spawn cost as a
|
|
lower bound. End-to-end key-to-screen latency is deliberately not
|
|
taken here: emitting keystrokes into a live desktop from a background
|
|
run would disturb the session. It belongs to the hardware checklist
|
|
with a dedicated target window.
|
|
- **Thresholds for the model ladder:** derive free-memory cutoffs from
|
|
the maximum observed per-rung footprint (transcribe peak, not just
|
|
load) plus explicit headroom for the compositor and other GPU
|
|
clients, and record the derivation next to the numbers.
|
|
|
|
## Report
|
|
|
|
`render_markdown_report` (and the JSON the subcommands print) carries
|
|
the protocol version, full host facts, and observed numbers only.
|
|
`tests/test_perf_baseline_protocol.py` covers the parsing, statistics,
|
|
threshold-derivation, and sample-determinism helpers offline.
|