xavierk/voculinux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Measure the installed-build performance baseline #19

New Issue
Closed
opened 2026-11-05 17:00:35 +00:00 by xavierk · 3 comments
xavierk commented 2026-11-05 17:00:35 +00:00
Owner
Copy Link
Copy Source

Parent

Spec: Voculinux — Void Linux (glibc) Wayland/niri port of Vocalinux

What to build

Take the deferred performance numbers on the target machine class (Void, RTX 2050 4 GB) using exactly the locked three-metric protocol: idle tray memory with the model unloaded after the idle timeout and with it loaded; wake latency from toggle to capture start; transcription real-time factor on the fixed thirty-second sample per backend (CUDA, Vulkan, CPU). The same pass records injection latency for direct typing versus the portal path where a portal exists, and the real memory footprint of each model-ladder rung at load. The deliverable is the recorded evidence on this ticket, not code.

Acceptance criteria

  • Idle memory recorded for both states (model unloaded after the idle timeout, model loaded) with the commands and host facts noted.
  • Wake latency (toggle to capture start) recorded from debug logs over repeated runs, with spread noted.
  • Real-time factor recorded for the fixed sample on each available backend, naming the exact model files used.
  • Load-time memory footprint recorded per ladder rung (default, first fallback, floor), free versus total memory distinguished.
  • Injection latency recorded for direct typing; portal latency recorded where a portal backend exists, or its absence recorded where it does not.
  • Results state recommended free-memory thresholds between ladder rungs derived from the measured footprints, for the model-ladder ticket to hard-code.

Blocked by

None (can start immediately).

## Parent [Spec: Voculinux — Void Linux (glibc) Wayland/niri port of Vocalinux](https://git.bongbetic.com/xavierk/voculinux/issues/16) ## What to build Take the deferred performance numbers on the target machine class (Void, RTX 2050 4 GB) using exactly the locked three-metric protocol: idle tray memory with the model unloaded after the idle timeout and with it loaded; wake latency from toggle to capture start; transcription real-time factor on the fixed thirty-second sample per backend (CUDA, Vulkan, CPU). The same pass records injection latency for direct typing versus the portal path where a portal exists, and the real memory footprint of each model-ladder rung at load. The deliverable is the recorded evidence on this ticket, not code. ## Acceptance criteria - [ ] Idle memory recorded for both states (model unloaded after the idle timeout, model loaded) with the commands and host facts noted. - [ ] Wake latency (toggle to capture start) recorded from debug logs over repeated runs, with spread noted. - [ ] Real-time factor recorded for the fixed sample on each available backend, naming the exact model files used. - [ ] Load-time memory footprint recorded per ladder rung (default, first fallback, floor), free versus total memory distinguished. - [ ] Injection latency recorded for direct typing; portal latency recorded where a portal backend exists, or its absence recorded where it does not. - [ ] Results state recommended free-memory thresholds between ladder rungs derived from the measured footprints, for the model-ladder ticket to hard-code. ## Blocked by None (can start immediately).
xavierk added the ready-for-agent label 2026-11-05 17:00:35 +00:00
xavierk commented 2026-11-05 17:41:17 +00:00
Author
Owner
Copy Link
Copy Source

Performance baseline evidence (measured 2026-11-05, on the ticket's machine class)

Protocol scripts/measure_perf_baseline.py v1.0.0 (docs/PERF_BASELINE_PROTOCOL.md;
tests in tests/test_perf_baseline_protocol.py). Branch feature/19.

Host facts: Void Linux glibc x86_64, niri Wayland session (XDG_SESSION_TYPE=wayland,
XDG_CURRENT_DESKTOP=niri), NVIDIA GeForce RTX 2050 4096 MiB (driver 595.112.07),
12 CPUs, 14 GB RAM, no CUDA toolkit (nvcc absent, no /usr/local/cuda*; backend
order Vulkan then CPU as specced). App 0.18.5 installed build (venv, editable),
pywhispercpp 1.5.0 with libggml-vulkan, models verified against pinned checksums
(HF commit 5359861c): ggml-large-v3-turbo-q8_0.bin (874 MB),
ggml-large-v3-turbo-q5_0.bin (574 MB), ggml-small.bin (488 MB),
ggml-small.en.bin (488 MB). Fixed 30 s sample: scripts/perf_baseline_sample.wav
(seed 19, deterministic speech-like babble; decodes to a handful of chars, so RTFs
are backend/rung comparables, not real-speech absolutes).

Idle tray memory (q8_0 rung, keep-alive 60 s)

  • Loaded: tray RSS 307860 kB (n=5, stdev 11); VRAM used 1173 MiB.
  • Unloaded (after reason=idle_keepalive, ~90 s observed): tray RSS 309685 kB (n=3);
    VRAM used 77 MiB (machine idle baseline 61 MiB).
  • Finding: RSS does not reflect the unload (freed native pages stay mapped);
    VRAM-used is the keep-alive state signal. Log also shows
    Model was unloaded by keep-alive; reloading for dictation on next toggle.

Wake latency, toggle to capture start (debug log, 6 runs)

Steady state (model resident): 0.161, 0.155, 0.137, 0.169, 0.171 s
(n=5, mean 0.159, min 0.137, max 0.171). First toggle after an idle unload:
2.418 s (model reload lands on the first toggle). Marker added for this:
Toggle requested in state IDLE (one debug line in _toggle_recognition).

Transcription RTF, fixed 30 s sample

Vulkan (3 runs each; cold = first incl. one-time shader/pipeline build):

  • small.en: cold 13.37 s (RTF 0.446), warm 0.34/0.30 s (RTF 0.011/0.010)
  • small: cold 0.84 s (0.028), warm 0.23/0.21 s (0.008/0.007)
  • q5_0: cold 9.15 s (0.305), warm 0.51/0.47 s (0.017/0.016)
  • q8_0: cold 6.14 s (0.205), warm 0.42/0.39 s (0.014/0.013)

CPU (single runs): small.en 5.2 s (0.173), q5_0 17.8 s (0.592), q8_0 12.8 s (0.428).
CUDA: absent as specced (no toolkit, binding has no CUDA lib) — not measured.

Load-time footprints per rung

Vulkan, VRAM attributable (peak used minus other-client baseline):

  • q8_0: 851 MiB at load (1.36 s), 1132 MiB transcribe peak
  • q5_0: 565 MiB at load (0.83 s), 846 MiB transcribe peak
  • small / small.en: 483 MiB at load (~0.8–1.0 s), 483 steady / 740 cold peak

CPU, process RSS (VRAM untouched): small.en 713 MiB, q5_0 766 MiB, q8_0 1052 MiB;
load wall 0.5–0.7 s. VRAM fully released after every probe (free returns to 3711).

Injection latency

  • RemoteDesktop version probe: TRUE (1–8 ms, answered by the gnome portal
    backend) — the app selects the portal as its first rung on this niri machine.
  • Session establishment: FAILS FAST in 9 ms (response=2) — no compositor-side
    implementation, no hang, no prompt. Effective first working rung is direct
    typing (wtype installed; ydotool correctly absent on Void).
  • wtype "" spawn cost 4.3 ms mean (n=5) — lower bound only; end-to-end
    key-to-screen latency deliberately not taken (would emit into the live
    desktop) and belongs to the #32 hardware checklist with a dedicated window.

Recommended free-memory thresholds for the model ladder (#20)

From maximum observed VRAM footprints (transcribe peaks) + 512 MiB headroom
(compositor/clients observed at 61–385 MiB):

  • prefer large-v3-turbo-q8_0 above ~1644 MiB free (1132 + 512)
  • prefer large-v3-turbo-q5_0 above ~1358 MiB free (846 + 512)
  • small/small.en floor below ~1358 MiB
    Load-only footprints for reference: 851 / 565 / 483 MiB. Typical free VRAM here
    (~3700 MiB) selects q8_0 with wide margin.

Incidental findings for sibling tickets

  • install.sh Void path checks command_exists xbps, but no xbps binary
    exists (only xbps-install/xbps-query) — every Void install fails there;
    worked around with --skip-system-deps after manually installing.
  • Void repo truth: libappindicator-gtk3 absent (use libayatana-appindicator),
    ydotool absent (confirmed un-packaged) — for the inventory/template tickets.
  • Spec assumption to correct: the portal rung IS selected on niri (probe passes);
    it fails at session start, not at selection. First-rung copy should say
    direct typing is the first working rung, and #31 should cover the fast
    session-failure demotion.
  • Tray watcher probe (org.kde.StatusNotifierWatcher) passes on Dank Material
    Shell — no widening needed on this machine.
## Performance baseline evidence (measured 2026-11-05, on the ticket's machine class) Protocol `scripts/measure_perf_baseline.py` v1.0.0 (`docs/PERF_BASELINE_PROTOCOL.md`; tests in `tests/test_perf_baseline_protocol.py`). Branch `feature/19`. Host facts: Void Linux glibc x86_64, niri Wayland session (`XDG_SESSION_TYPE=wayland`, `XDG_CURRENT_DESKTOP=niri`), NVIDIA GeForce RTX 2050 4096 MiB (driver 595.112.07), 12 CPUs, 14 GB RAM, no CUDA toolkit (`nvcc` absent, no `/usr/local/cuda*`; backend order Vulkan then CPU as specced). App 0.18.5 installed build (venv, editable), pywhispercpp 1.5.0 with `libggml-vulkan`, models verified against pinned checksums (HF commit `5359861c`): `ggml-large-v3-turbo-q8_0.bin` (874 MB), `ggml-large-v3-turbo-q5_0.bin` (574 MB), `ggml-small.bin` (488 MB), `ggml-small.en.bin` (488 MB). Fixed 30 s sample: `scripts/perf_baseline_sample.wav` (seed 19, deterministic speech-like babble; decodes to a handful of chars, so RTFs are backend/rung comparables, not real-speech absolutes). ### Idle tray memory (q8_0 rung, keep-alive 60 s) - Loaded: tray RSS 307860 kB (n=5, stdev 11); VRAM used 1173 MiB. - Unloaded (after `reason=idle_keepalive`, ~90 s observed): tray RSS 309685 kB (n=3); VRAM used 77 MiB (machine idle baseline 61 MiB). - Finding: RSS does not reflect the unload (freed native pages stay mapped); VRAM-used is the keep-alive state signal. Log also shows `Model was unloaded by keep-alive; reloading for dictation` on next toggle. ### Wake latency, toggle to capture start (debug log, 6 runs) Steady state (model resident): 0.161, 0.155, 0.137, 0.169, 0.171 s (n=5, mean 0.159, min 0.137, max 0.171). First toggle after an idle unload: 2.418 s (model reload lands on the first toggle). Marker added for this: `Toggle requested in state IDLE` (one debug line in `_toggle_recognition`). ### Transcription RTF, fixed 30 s sample Vulkan (3 runs each; cold = first incl. one-time shader/pipeline build): - small.en: cold 13.37 s (RTF 0.446), warm 0.34/0.30 s (RTF 0.011/0.010) - small: cold 0.84 s (0.028), warm 0.23/0.21 s (0.008/0.007) - q5_0: cold 9.15 s (0.305), warm 0.51/0.47 s (0.017/0.016) - q8_0: cold 6.14 s (0.205), warm 0.42/0.39 s (0.014/0.013) CPU (single runs): small.en 5.2 s (0.173), q5_0 17.8 s (0.592), q8_0 12.8 s (0.428). CUDA: absent as specced (no toolkit, binding has no CUDA lib) — not measured. ### Load-time footprints per rung Vulkan, VRAM attributable (peak used minus other-client baseline): - q8_0: 851 MiB at load (1.36 s), 1132 MiB transcribe peak - q5_0: 565 MiB at load (0.83 s), 846 MiB transcribe peak - small / small.en: 483 MiB at load (~0.8–1.0 s), 483 steady / 740 cold peak CPU, process RSS (VRAM untouched): small.en 713 MiB, q5_0 766 MiB, q8_0 1052 MiB; load wall 0.5–0.7 s. VRAM fully released after every probe (free returns to 3711). ### Injection latency - RemoteDesktop version probe: TRUE (1–8 ms, answered by the gnome portal backend) — the app selects the portal as its first rung on this niri machine. - Session establishment: FAILS FAST in 9 ms (`response=2`) — no compositor-side implementation, no hang, no prompt. Effective first working rung is direct typing (wtype installed; ydotool correctly absent on Void). - `wtype ""` spawn cost 4.3 ms mean (n=5) — lower bound only; end-to-end key-to-screen latency deliberately not taken (would emit into the live desktop) and belongs to the #32 hardware checklist with a dedicated window. ### Recommended free-memory thresholds for the model ladder (#20) From maximum observed VRAM footprints (transcribe peaks) + 512 MiB headroom (compositor/clients observed at 61–385 MiB): - prefer `large-v3-turbo-q8_0` above ~1644 MiB free (1132 + 512) - prefer `large-v3-turbo-q5_0` above ~1358 MiB free (846 + 512) - `small`/`small.en` floor below ~1358 MiB Load-only footprints for reference: 851 / 565 / 483 MiB. Typical free VRAM here (~3700 MiB) selects q8_0 with wide margin. ### Incidental findings for sibling tickets - `install.sh` Void path checks `command_exists xbps`, but no `xbps` binary exists (only `xbps-install`/`xbps-query`) — every Void install fails there; worked around with `--skip-system-deps` after manually installing. - Void repo truth: `libappindicator-gtk3` absent (use `libayatana-appindicator`), `ydotool` absent (confirmed un-packaged) — for the inventory/template tickets. - Spec assumption to correct: the portal rung IS selected on niri (probe passes); it fails at session start, not at selection. First-rung copy should say direct typing is the first *working* rung, and #31 should cover the fast session-failure demotion. - Tray watcher probe (`org.kde.StatusNotifierWatcher`) passes on Dank Material Shell — no widening needed on this machine.
xavierk commented 2026-11-06 17:26:01 +00:00
Author
Owner
Copy Link
Copy Source

#19 installed-build performance baseline: evidence (2026-11-06)

Protocol: docs/PERF_BASELINE_PROTOCOL.md and scripts/measure_perf_baseline.py (PROTOCOL_VERSION 1.0.0). Measured with /usr/bin/python3 against the installed vocalinux (/usr/lib/python3.14/site-packages), except the CUDA rows, which use the local venv-cuda build (origin/main a4a831c; unsupported escape hatch).

Host and build under test

item value
package voculinux-0.18.5_1 (signed local repo, installed 2026-11-06 22:20 IST); python3-pywhispercpp-1.5.0_1 (Vulkan); python3-PyAudio-0.2.14_1
host Void Linux glibc x86_64, kernel 7.3.0_1, AMD Ryzen 5 7535HS (AVX2/FMA/F16C present), 14 GB RAM
compositor / shell niri 26.04 / Dank Material Shell (qs)
GPU NVIDIA RTX 2050, 4096 MiB total, driver 595.112.07 (Vulkan0 = CUDA0). AMD Radeon 660M iGPU (UMA; Vulkan0 with GGML_VK_VISIBLE_DEVICES=1)
CUDA venv-cuda: pywhispercpp 1.5.0 built with GGML_CUDA, CUDA 13.2.51, sm_86 (local build)
models ggml-large-v3-turbo-q8_0.bin (sha256 317eb69c...), -q5_0.bin (394221709...), ggml-small.bin (1be3a9b2...), ggml-small.en.bin (c6138d6d...). Downloaded through the installed verifier into a throwaway XDG data dir, sha256 pinned
samples protocol sample scripts/perf_baseline_sample.wav (sha256 631807dc..., regenerated byte-identical with make-sample --seed 19) and flite-30s.wav (30.00 s speech, the earlier sample)

1. Idle tray memory (app: --debug --start-minimized, keep-alive on, 60 s)

state tray RSS kB (n=5, stdev 0) VRAM used / free MiB (of 4096)
loaded (default rung q8_0, after 10 dictations) 327,092 1192 / 2579
unloaded (log reason=idle_keepalive at 22:41:56) 326,264 83 / 3689

RSS hardly moves on unload, as the protocol predicts. VRAM is the state signal: about 1109 MiB is released. No child processes were present.

2. Wake latency (toggle to capture start), 10 repeated runs

  • Toggle: busctl --user call com.voculinux.Voculinux /com/voculinux/Voculinux com.voculinux.Voculinux Toggle, timestamped; capture start = PipeWire source-output new event (pactl subscribe).
  • Toggle to capture start: mean 49.5 ms, min 35.7, max 53.7, stdev 5.0.
  • Stop toggle to capture released: mean 10.7 ms (min 10.0, max 12.1).
  • Deviation: the PipeWire capture path logs no Opened capture stream: line (that line exists only in the PortAudio path), so parse-log cannot see it. The mic-backed PortAudio path was not used for this metric (see deviations). First capture after an idle unload was not separated in this pass.

3. Real-time factor, 30 s sample, 3 fresh processes per row (cold = run 1, warm = mean of runs 2-3)

backend model sample cold RTF warm RTF
Vulkan dGPU (installed) q8_0 protocol 0.0133 0.0130
Vulkan dGPU q8_0 flite 0.0254 0.0211
Vulkan dGPU q5_0 protocol 0.2656 0.0155
Vulkan dGPU q5_0 flite 0.0213 0.0212
Vulkan dGPU small protocol 0.0925 0.0073
Vulkan dGPU small flite 0.0189 0.0188
Vulkan dGPU small.en protocol 0.0528 0.0104
Vulkan dGPU small.en flite 0.0175 0.0175
Vulkan iGPU (660M) q8_0 protocol 0.0835 0.0829
Vulkan iGPU (660M) q8_0 flite 0.0983 0.0968
CUDA (venv-cuda, CUDA0) q8_0 protocol 0.0169 0.0157
CUDA q8_0 flite 0.0238 0.0231
CUDA q5_0 flite 0.0219 0.0211
CPU (installed, no AVX) q8_0 protocol 1.6418 1.658
CPU q8_0 flite 1.6915 1.701
CPU small.en flite 1.3005 1.308

The protocol sample is babble: it decodes to 7-14 characters, so its RTF understates decoding cost. The flite rows are the realistic planning numbers.

CPU finding: libggml-cpu.so in the installed package has zero ymm/zmm instructions, so the CPU backend has no AVX/FMA/F16C code paths, even though this host has them. CPU-only dictation is therefore not real-time on turbo (RTF about 1.7) or on small.en (about 1.3). Logged as a follow-up.

4. Per-rung load footprint (free vs total distinguished)

Total VRAM 4096 MiB. Free before the Vulkan probes 3711 MiB (used 385 MiB at that moment). Deltas are peak minus before, not absolutes.

rung Vulkan load Δ VRAM (probe-load) Vulkan transcribe Δ VRAM (peak − before, flite, 3 runs) CUDA transcribe Δ VRAM CPU load RSS MiB Vulkan process RSS MiB Vulkan load s
large-v3-turbo-q8_0 851 1132 1178 1051 212 0.74
large-v3-turbo-q5_0 565 847 892 765 212 0.63
small 483 742 n/a 711 212 0.42
small.en 483 740 n/a 711 212 0.39

5. Proposed free-memory thresholds (derived as transcribe Δ + 512 MiB headroom, the protocol's rule)

rung Vulkan: prefer when free VRAM ≥ CUDA: prefer when free VRAM ≥ constant in model_ladder.py today
q8_0 (default) 1644 MiB (1132 + 512) 1690 MiB (1178 + 512) 1644 (Vulkan value matches)
q5_0 1359 MiB (847 + 512) 1404 MiB (892 + 512) 1358 (1 MiB off)
floor (small / small.en) below q5 threshold; floor Δ 740 MiB, so about 1252 MiB free would be needed n/a floor by rule

Recommendations for the ladder ticket:

  • Key thresholds per backend. CUDA needs 46 MiB more for q8_0 and 45 MiB more for q5_0 than Vulkan.
  • CPU builds have no VRAM reading. Proposed RAM rule, untested: q8_0 needs 1051 + 512 = 1563 MiB free RAM. Given RTF 1.7, a CPU-only machine should default to the floor rung rather than turbo.
  • Re-measure once the CPU backend has AVX2 variants (finding above).

6. Injection latency

path measurement result
direct typing, wtype (first rung on niri) inject-probe: select time (first_rung = wtype) 67.6 ms
wtype "" dispatch (sends no keys), 5 runs mean 4.0 ms (min 3.7, max 4.3)
wtype 24 chars into a verified-focus Ghostty window 117 ms wall
app pipeline: transcription result to Text injected using wtype (117 chars) 0.87 s (log 22:35:53.089 to 53.963)
RemoteDesktop portal inject-probe version read (no session created) 3.6 ms; portal_available: true (see finding)
injector on niri never probes the portal (_is_niri_session() guard): no portal rung, no permission dialog
key to screen not measured in the #32 human checklist

Deviations and caveats

  • During bring-up, the app's PortAudio path (default, then the hardware device) opened the laptop mic on three occasions, about 30 s in total. Raw audio is not kept. One ambient 86-character transcript was typed into my test window and deleted. Every reported wake and RTF run was guarded so that all capture streams were on the null-sink monitor.
  • The app's PipeWire capture path needs PIPEWIRE_PROPS='{ stream.capture.sink = true }' in its environment. Without it, pw-record --target <sink> links the default microphone (see #32, defect F1).
  • transcribe records no before-reading, so transcribe deltas used a separate poller around _child-transcribe. The --backend choices have no cuda, so CUDA rows use --backend vulkan (use_gpu) with venv-cuda; the child log confirms CUDA0.
  • The pywhispercpp binding reports version 0.0.0 (egg-info), so the app logs "does not support GPU context_params; upgrade to 1.5+" and uses the default GPU rather than its device selection. The iGPU rows use GGML_VK_VISIBLE_DEVICES=1.

Acceptance criteria

  • Idle memory both states, with commands and host facts.
  • Wake latency over repeated runs with spread (n=10).
  • RTF per available backend, naming the exact model files.
  • Load footprint per rung, free vs total distinguished.
  • Injection latency for direct typing; portal path recorded with its niri behaviour.
  • Recommended thresholds derived from measured footprints (section 5).

Commands (exact pattern)

PERF=/tmp/vocalinux-hwtest/scripts/measure_perf_baseline.py
export XDG_CONFIG_HOME=scratchpad/hw/xdg/config XDG_DATA_HOME=scratchpad/hw/xdg/data XDG_STATE_HOME=scratchpad/hw/xdg/state XDG_CACHE_HOME=scratchpad/hw/xdg/cache
PIPEWIRE_PROPS='{ stream.capture.sink = true }' /usr/sbin/voculinux --debug --start-minimized
/usr/bin/python3 $PERF sample-rss --pid <PID> --samples 5 --interval 2
busctl --user call com.voculinux.Voculinux /com/voculinux/Voculinux com.voculinux.Voculinux Toggle
/usr/bin/python3 $PERF probe-load --model-file $M/ggml-<rung>.bin --backend vulkan --hold-seconds 8
/usr/bin/python3 $PERF transcribe --model-file $M/ggml-<rung>.bin --backend vulkan --wav scripts/perf_baseline_sample.wav --runs 3
GGML_VK_VISIBLE_DEVICES=1 /usr/bin/python3 $PERF transcribe ... --backend vulkan    # iGPU
venv-cuda/bin/python $PERF transcribe ... --backend vulkan                         # CUDA
/usr/bin/python3 $PERF inject-probe

Raw results: scratchpad/hw/results/summary.json , with per-run JSON and logs in the same directory.

Proposed follow-ups (not opened)

  • F1 (privacy, high): PipeWire system-audio capture links the default mic (pw-record --target <sink> without stream.capture.sink). Reproduced in the app's own run and in isolation.
  • F3 (medium): pywhispercpp reports 0.0.0 on the Void build, which disables device selection.
  • F4 (medium): CPU backend built without AVX/FMA/F16C, so CPU RTF is 1.3-1.7 on an AVX2 host.
## #19 installed-build performance baseline: evidence (2026-11-06) Protocol: `docs/PERF_BASELINE_PROTOCOL.md` and `scripts/measure_perf_baseline.py` (PROTOCOL_VERSION 1.0.0). Measured with `/usr/bin/python3` against the installed `vocalinux` (`/usr/lib/python3.14/site-packages`), except the CUDA rows, which use the local `venv-cuda` build (origin/main a4a831c; unsupported escape hatch). ### Host and build under test | item | value | |---|---| | package | `voculinux-0.18.5_1` (signed local repo, installed 2026-11-06 22:20 IST); `python3-pywhispercpp-1.5.0_1` (Vulkan); `python3-PyAudio-0.2.14_1` | | host | Void Linux glibc x86_64, kernel 7.3.0_1, AMD Ryzen 5 7535HS (AVX2/FMA/F16C present), 14 GB RAM | | compositor / shell | niri 26.04 / Dank Material Shell (`qs`) | | GPU | NVIDIA RTX 2050, 4096 MiB total, driver 595.112.07 (Vulkan0 = CUDA0). AMD Radeon 660M iGPU (UMA; Vulkan0 with `GGML_VK_VISIBLE_DEVICES=1`) | | CUDA | venv-cuda: pywhispercpp 1.5.0 built with GGML_CUDA, CUDA 13.2.51, sm_86 (local build) | | models | `ggml-large-v3-turbo-q8_0.bin` (sha256 317eb69c...), `-q5_0.bin` (394221709...), `ggml-small.bin` (1be3a9b2...), `ggml-small.en.bin` (c6138d6d...). Downloaded through the installed verifier into a throwaway XDG data dir, sha256 pinned | | samples | protocol sample `scripts/perf_baseline_sample.wav` (sha256 631807dc..., regenerated byte-identical with `make-sample --seed 19`) and `flite-30s.wav` (30.00 s speech, the earlier sample) | ### 1. Idle tray memory (app: `--debug --start-minimized`, keep-alive on, 60 s) | state | tray RSS kB (n=5, stdev 0) | VRAM used / free MiB (of 4096) | |---|---|---| | loaded (default rung q8_0, after 10 dictations) | 327,092 | 1192 / 2579 | | unloaded (log `reason=idle_keepalive` at 22:41:56) | 326,264 | 83 / 3689 | RSS hardly moves on unload, as the protocol predicts. VRAM is the state signal: about 1109 MiB is released. No child processes were present. ### 2. Wake latency (toggle to capture start), 10 repeated runs - Toggle: `busctl --user call com.voculinux.Voculinux /com/voculinux/Voculinux com.voculinux.Voculinux Toggle`, timestamped; capture start = PipeWire `source-output new` event (`pactl subscribe`). - Toggle to capture start: mean 49.5 ms, min 35.7, max 53.7, stdev 5.0. - Stop toggle to capture released: mean 10.7 ms (min 10.0, max 12.1). - Deviation: the PipeWire capture path logs no `Opened capture stream:` line (that line exists only in the PortAudio path), so `parse-log` cannot see it. The mic-backed PortAudio path was not used for this metric (see deviations). First capture after an idle unload was not separated in this pass. ### 3. Real-time factor, 30 s sample, 3 fresh processes per row (cold = run 1, warm = mean of runs 2-3) | backend | model | sample | cold RTF | warm RTF | |---|---|---|---|---| | Vulkan dGPU (installed) | q8_0 | protocol | 0.0133 | 0.0130 | | Vulkan dGPU | q8_0 | flite | 0.0254 | 0.0211 | | Vulkan dGPU | q5_0 | protocol | 0.2656 | 0.0155 | | Vulkan dGPU | q5_0 | flite | 0.0213 | 0.0212 | | Vulkan dGPU | small | protocol | 0.0925 | 0.0073 | | Vulkan dGPU | small | flite | 0.0189 | 0.0188 | | Vulkan dGPU | small.en | protocol | 0.0528 | 0.0104 | | Vulkan dGPU | small.en | flite | 0.0175 | 0.0175 | | Vulkan iGPU (660M) | q8_0 | protocol | 0.0835 | 0.0829 | | Vulkan iGPU (660M) | q8_0 | flite | 0.0983 | 0.0968 | | CUDA (venv-cuda, CUDA0) | q8_0 | protocol | 0.0169 | 0.0157 | | CUDA | q8_0 | flite | 0.0238 | 0.0231 | | CUDA | q5_0 | flite | 0.0219 | 0.0211 | | CPU (installed, no AVX) | q8_0 | protocol | 1.6418 | 1.658 | | CPU | q8_0 | flite | 1.6915 | 1.701 | | CPU | small.en | flite | 1.3005 | 1.308 | The protocol sample is babble: it decodes to 7-14 characters, so its RTF understates decoding cost. The flite rows are the realistic planning numbers. CPU finding: `libggml-cpu.so` in the installed package has zero `ymm`/`zmm` instructions, so the CPU backend has no AVX/FMA/F16C code paths, even though this host has them. CPU-only dictation is therefore not real-time on turbo (RTF about 1.7) or on small.en (about 1.3). Logged as a follow-up. ### 4. Per-rung load footprint (free vs total distinguished) Total VRAM 4096 MiB. Free before the Vulkan probes 3711 MiB (used 385 MiB at that moment). Deltas are peak minus before, not absolutes. | rung | Vulkan load Δ VRAM (probe-load) | Vulkan transcribe Δ VRAM (peak − before, flite, 3 runs) | CUDA transcribe Δ VRAM | CPU load RSS MiB | Vulkan process RSS MiB | Vulkan load s | |---|---|---|---|---|---|---| | large-v3-turbo-q8_0 | 851 | 1132 | 1178 | 1051 | 212 | 0.74 | | large-v3-turbo-q5_0 | 565 | 847 | 892 | 765 | 212 | 0.63 | | small | 483 | 742 | n/a | 711 | 212 | 0.42 | | small.en | 483 | 740 | n/a | 711 | 212 | 0.39 | ### 5. Proposed free-memory thresholds (derived as transcribe Δ + 512 MiB headroom, the protocol's rule) | rung | Vulkan: prefer when free VRAM ≥ | CUDA: prefer when free VRAM ≥ | constant in `model_ladder.py` today | |---|---|---|---| | q8_0 (default) | 1644 MiB (1132 + 512) | 1690 MiB (1178 + 512) | 1644 (Vulkan value matches) | | q5_0 | 1359 MiB (847 + 512) | 1404 MiB (892 + 512) | 1358 (1 MiB off) | | floor (small / small.en) | below q5 threshold; floor Δ 740 MiB, so about 1252 MiB free would be needed | n/a | floor by rule | Recommendations for the ladder ticket: - Key thresholds per backend. CUDA needs 46 MiB more for q8_0 and 45 MiB more for q5_0 than Vulkan. - CPU builds have no VRAM reading. Proposed RAM rule, untested: q8_0 needs 1051 + 512 = 1563 MiB free RAM. Given RTF 1.7, a CPU-only machine should default to the floor rung rather than turbo. - Re-measure once the CPU backend has AVX2 variants (finding above). ### 6. Injection latency | path | measurement | result | |---|---|---| | direct typing, `wtype` (first rung on niri) | `inject-probe`: select time (first_rung = wtype) | 67.6 ms | | | `wtype ""` dispatch (sends no keys), 5 runs | mean 4.0 ms (min 3.7, max 4.3) | | | `wtype` 24 chars into a verified-focus Ghostty window | 117 ms wall | | | app pipeline: transcription result to `Text injected using wtype` (117 chars) | 0.87 s (log 22:35:53.089 to 53.963) | | RemoteDesktop portal | `inject-probe` version read (no session created) | 3.6 ms; `portal_available: true` (see finding) | | | injector on niri | never probes the portal (`_is_niri_session()` guard): no portal rung, no permission dialog | | key to screen | not measured | in the #32 human checklist | ### Deviations and caveats - During bring-up, the app's PortAudio path (`default`, then the hardware device) opened the laptop mic on three occasions, about 30 s in total. Raw audio is not kept. One ambient 86-character transcript was typed into my test window and deleted. Every reported wake and RTF run was guarded so that all capture streams were on the null-sink monitor. - The app's PipeWire capture path needs `PIPEWIRE_PROPS='{ stream.capture.sink = true }'` in its environment. Without it, `pw-record --target <sink>` links the default microphone (see #32, defect F1). - `transcribe` records no before-reading, so transcribe deltas used a separate poller around `_child-transcribe`. The `--backend` choices have no `cuda`, so CUDA rows use `--backend vulkan` (use_gpu) with venv-cuda; the child log confirms `CUDA0`. - The pywhispercpp binding reports version 0.0.0 (egg-info), so the app logs "does not support GPU context_params; upgrade to 1.5+" and uses the default GPU rather than its device selection. The iGPU rows use `GGML_VK_VISIBLE_DEVICES=1`. ### Acceptance criteria - [x] Idle memory both states, with commands and host facts. - [x] Wake latency over repeated runs with spread (n=10). - [x] RTF per available backend, naming the exact model files. - [x] Load footprint per rung, free vs total distinguished. - [x] Injection latency for direct typing; portal path recorded with its niri behaviour. - [x] Recommended thresholds derived from measured footprints (section 5). ### Commands (exact pattern) ``` PERF=/tmp/vocalinux-hwtest/scripts/measure_perf_baseline.py export XDG_CONFIG_HOME=scratchpad/hw/xdg/config XDG_DATA_HOME=scratchpad/hw/xdg/data XDG_STATE_HOME=scratchpad/hw/xdg/state XDG_CACHE_HOME=scratchpad/hw/xdg/cache PIPEWIRE_PROPS='{ stream.capture.sink = true }' /usr/sbin/voculinux --debug --start-minimized /usr/bin/python3 $PERF sample-rss --pid <PID> --samples 5 --interval 2 busctl --user call com.voculinux.Voculinux /com/voculinux/Voculinux com.voculinux.Voculinux Toggle /usr/bin/python3 $PERF probe-load --model-file $M/ggml-<rung>.bin --backend vulkan --hold-seconds 8 /usr/bin/python3 $PERF transcribe --model-file $M/ggml-<rung>.bin --backend vulkan --wav scripts/perf_baseline_sample.wav --runs 3 GGML_VK_VISIBLE_DEVICES=1 /usr/bin/python3 $PERF transcribe ... --backend vulkan # iGPU venv-cuda/bin/python $PERF transcribe ... --backend vulkan # CUDA /usr/bin/python3 $PERF inject-probe ``` Raw results: `scratchpad/hw/results/summary.json` , with per-run JSON and logs in the same directory. ### Proposed follow-ups (not opened) - F1 (privacy, high): PipeWire system-audio capture links the default mic (`pw-record --target <sink>` without `stream.capture.sink`). Reproduced in the app's own run and in isolation. - F3 (medium): pywhispercpp reports 0.0.0 on the Void build, which disables device selection. - F4 (medium): CPU backend built without AVX/FMA/F16C, so CPU RTF is 1.3-1.7 on an AVX2 host.
xavierk commented 2026-11-06 17:27:06 +00:00
Author
Owner
Copy Link
Copy Source

Reviewed the evidence above against the acceptance criteria: all six criteria are recorded with commands, host facts and proposed thresholds. Closing. The ladder thresholds themselves are applied in a follow-up PR.

Reviewed the evidence above against the acceptance criteria: all six criteria are recorded with commands, host facts and proposed thresholds. Closing. The ladder thresholds themselves are applied in a follow-up PR.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
needs-info
Waiting on reporter for more information
 needs-triage
Maintainer needs to evaluate this issue
 ready-for-agent
Fully specified, ready for an AFK agent
 ready-for-human
Requires human implementation
 wayfinder:grilling
wayfinder:map
wayfinder:prototype
wayfinder:research
wayfinder:task
wontfix
Will not be actioned
No labels ready-for-agent
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
xavierk (Xavier Karma)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: xavierk/voculinux#19