xavierk/voculinux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Model catalog for 4 GB VRAM #12

New Issue
Closed
opened 2026-11-05 16:14:13 +00:00 by xavierk · 1 comment
xavierk commented 2026-11-05 16:14:13 +00:00
Owner
Copy Link
Copy Source

Part of #1

Question

Beyond the default large-v3-turbo-q8_0 pick, what is the model catalog for 4 GB VRAM: which quantized variants fit, with what quality/latency tradeoffs, and when does each fallback trigger?

Part of #1 ## Question Beyond the default large-v3-turbo-q8_0 pick, what is the model catalog for 4 GB VRAM: which quantized variants fit, with what quality/latency tradeoffs, and when does each fallback trigger?
xavierk added the wayfinder:research label 2026-11-05 16:14:13 +00:00
xavierk self-assigned this 2026-11-05 16:18:58 +00:00
xavierk commented 2026-11-05 16:21:40 +00:00
Author
Owner
Copy Link
Copy Source

Model catalog for 4 GB VRAM — decision

Decision

The recommended whisper.cpp catalog for a 4 GB card (RTX 2050 class) is a
three-rung ladder, best first:

  1. Default: large-v3-turbo-q8_0 (874 MB download). Distilled 809M-param
    model with near-large-v3 quality at roughly one eighth the decode cost
    (4 decoder layers instead of 32); the q8_0 quantization is within noise of
    full precision. Weights plus CUDA context, compute buffers and KV cache land
    near 2–2.5 GB at load, which fits 4 GB with headroom for the compositor and
    other GPU clients.
  2. First fallback: large-v3-turbo-q5_0 (574 MB download). Same model one
    quantization step down; a small quality cost for roughly 300 MB less VRAM.
    Trigger: low free VRAM at load time.
  3. Final safe fallback: small / small.en (466 MB download). Today's
    actual behaviour on this card, kept as the floor. English-only users take
    small.en; everyone else takes multilingual small.

Excluded from the 4 GB recommended set (everything stays manually
selectable; nothing is removed from the catalog or the checksum pins):

  • Unquantized large-v3-turbo (1.6 GB weights): marginal headroom once CUDA
    overhead and display clients are counted.
  • All large-v3 / large-v2 full and quantized variants (large-v3-q5_0
    alone is 1.08 GB of weights before overhead). Note there is no
    large-v3-q8_0 pin in model_checksums.txt at all, so it is not even
    downloadable today.
  • The medium family: medium-q5_0 (539 MB) fits by bytes but is strictly
    dominated by the turbo quants on quality per byte, so it earns no rung.
  • tiny / base: below the safe quality floor for dictation; CPU-only
    territory.

Fallback triggers (rules; implementation follows later)

  • T1, load-time selection: key the pick to free VRAM at load
    (nvidia-smi memory.free, not total — hybrid laptops share the dGPU with
    the display), roughly: ≥ 2.5 GB free → q8_0; ~1.5–2.5 GB → q5_0; below →
    small. The exact thresholds must be validated by measuring the real load
    footprint in the installed-build re-run already deferred by the
    Python performance baseline
    decision; treat the numbers above as the rule shape, not calibrated constants.
  • T2, load failure: on a CUDA out-of-memory or model-load exception, step
    one rung down and retry once, then surface the error. Verified current
    behaviour in recognition_manager.py::_init_whispercpp: a load failure sets
    the engine error and raises — there is no automatic step-down today, and the
    only memory guard in the codebase is the VOSK one.
  • T3, backend generality: the same ladder applies under Vulkan (the packaged
    xbps build is Vulkan-plus-CPU only, per the
    Void xbps packaging route
    decision). On integrated graphics or CPU-only, key T1 to free system RAM
    instead of VRAM.

Rationale and evidence

  • Builds directly on the
    RTX 2050 CUDA model fit
    verdict (default large-v3-turbo-q8_0, q5_0 fallback when free VRAM is
    low, small as final fallback) and answers the part it left open: the full
    rung list, what is deliberately excluded, and when each rung triggers.
  • Catalog facts inspected in-repo: src/vocalinux/utils/whispercpp_model_info.py
    (_WHISPERCPP_MODEL_SPECS file sizes: turbo-q8_0 874 MB, turbo-q5_0
    574 MB, turbo fp16 1624 MB, small 466 MB, medium-q5_0 539 MB,
    large-v3-q5_0 1081 MB, large-v3 fp16 3095 MB) and
    src/vocalinux/utils/model_checksums.txt (every one of those files pinned
    by sha256 at whisper.cpp commit 5359861c; no ggml-large-v3-q8_0.bin
    entry exists).
  • Behaviour facts inspected in-repo:
    src/vocalinux/speech_recognition/recognition_manager.py
    (_init_whispercpp raises on load failure, no VRAM check, no step-down;
    _check_vosk_model_memory is VOSK-only) and
    src/vocalinux/utils/whispercpp_model_info.py
    (detect_compute_backend tries Vulkan before CUDA, so the CUDA/VRAM branch
    of get_recommended_model is unreachable on a Vulkan-capable Nvidia card —
    the T1 rule needs that fix first, which is downstream implementation work,
    not map work).
  • Upstream corroboration: the whisper.cpp models list gives large-v3-turbo
    1.5 GiB, large-v3-turbo-q5_0 547 MiB and small 466 MiB on disk, matching
    the pinned files byte-for-byte in magnitude; the turbo quality/speed claim
    (distilled decoder, near-large-v3 accuracy) is the model's published design
    point.
  • Language retargeting already exists (settings_dialog.py mirrors
    multilingual/.en variants), so the small/small.en pair needs no new
    machinery.

Provenance

Decided by the agent under the user's explicit delegation for this wayfinding
run (act as orchestrator and decision maker until the fog clears). No human
was interviewed for this research ticket; the decision is grounded in the
inspected sources above.

Consequences for later tickets

  • No new tickets: the map's remaining fog is unaffected by this ladder.
  • The T1 thresholds need the installed-build measurement deferred by the
    performance-baseline decision before anyone hard-codes them.
  • No glossary or ADR change: the repo has no GLOSSARY.md or docs/adr/
    yet, and this ladder is spec content owned by this ticket, not a new domain
    term — so per the domain-modeling skill nothing is created.
# Model catalog for 4 GB VRAM — decision ## Decision The recommended whisper.cpp catalog for a 4 GB card (RTX 2050 class) is a three-rung ladder, best first: 1. **Default: `large-v3-turbo-q8_0`** (874 MB download). Distilled 809M-param model with near-`large-v3` quality at roughly one eighth the decode cost (4 decoder layers instead of 32); the q8_0 quantization is within noise of full precision. Weights plus CUDA context, compute buffers and KV cache land near 2–2.5 GB at load, which fits 4 GB with headroom for the compositor and other GPU clients. 2. **First fallback: `large-v3-turbo-q5_0`** (574 MB download). Same model one quantization step down; a small quality cost for roughly 300 MB less VRAM. Trigger: low free VRAM at load time. 3. **Final safe fallback: `small` / `small.en`** (466 MB download). Today's actual behaviour on this card, kept as the floor. English-only users take `small.en`; everyone else takes multilingual `small`. **Excluded from the 4 GB recommended set** (everything stays manually selectable; nothing is removed from the catalog or the checksum pins): - Unquantized `large-v3-turbo` (1.6 GB weights): marginal headroom once CUDA overhead and display clients are counted. - All `large-v3` / `large-v2` full and quantized variants (`large-v3-q5_0` alone is 1.08 GB of weights before overhead). Note there is no `large-v3-q8_0` pin in `model_checksums.txt` at all, so it is not even downloadable today. - The `medium` family: `medium-q5_0` (539 MB) fits by bytes but is strictly dominated by the turbo quants on quality per byte, so it earns no rung. - `tiny` / `base`: below the safe quality floor for dictation; CPU-only territory. ## Fallback triggers (rules; implementation follows later) - **T1, load-time selection:** key the pick to *free* VRAM at load (`nvidia-smi memory.free`, not total — hybrid laptops share the dGPU with the display), roughly: ≥ 2.5 GB free → q8_0; ~1.5–2.5 GB → q5_0; below → `small`. The exact thresholds must be validated by measuring the real load footprint in the installed-build re-run already deferred by the [Python performance baseline](https://git.bongbetic.com/xavierk/voculinux/issues/9) decision; treat the numbers above as the rule shape, not calibrated constants. - **T2, load failure:** on a CUDA out-of-memory or model-load exception, step one rung down and retry once, then surface the error. Verified current behaviour in `recognition_manager.py::_init_whispercpp`: a load failure sets the engine error and raises — there is no automatic step-down today, and the only memory guard in the codebase is the VOSK one. - **T3, backend generality:** the same ladder applies under Vulkan (the packaged xbps build is Vulkan-plus-CPU only, per the [Void xbps packaging route](https://git.bongbetic.com/xavierk/voculinux/issues/4) decision). On integrated graphics or CPU-only, key T1 to free system RAM instead of VRAM. ## Rationale and evidence - Builds directly on the [RTX 2050 CUDA model fit](https://git.bongbetic.com/xavierk/voculinux/issues/3) verdict (default `large-v3-turbo-q8_0`, `q5_0` fallback when free VRAM is low, `small` as final fallback) and answers the part it left open: the full rung list, what is deliberately excluded, and when each rung triggers. - Catalog facts inspected in-repo: `src/vocalinux/utils/whispercpp_model_info.py` (`_WHISPERCPP_MODEL_SPECS` file sizes: turbo-q8_0 874 MB, turbo-q5_0 574 MB, turbo fp16 1624 MB, small 466 MB, medium-q5_0 539 MB, large-v3-q5_0 1081 MB, large-v3 fp16 3095 MB) and `src/vocalinux/utils/model_checksums.txt` (every one of those files pinned by sha256 at whisper.cpp commit `5359861c`; no `ggml-large-v3-q8_0.bin` entry exists). - Behaviour facts inspected in-repo: `src/vocalinux/speech_recognition/recognition_manager.py` (`_init_whispercpp` raises on load failure, no VRAM check, no step-down; `_check_vosk_model_memory` is VOSK-only) and `src/vocalinux/utils/whispercpp_model_info.py` (`detect_compute_backend` tries Vulkan before CUDA, so the CUDA/VRAM branch of `get_recommended_model` is unreachable on a Vulkan-capable Nvidia card — the T1 rule needs that fix first, which is downstream implementation work, not map work). - Upstream corroboration: the whisper.cpp models list gives `large-v3-turbo` 1.5 GiB, `large-v3-turbo-q5_0` 547 MiB and `small` 466 MiB on disk, matching the pinned files byte-for-byte in magnitude; the turbo quality/speed claim (distilled decoder, near-large-v3 accuracy) is the model's published design point. - Language retargeting already exists (`settings_dialog.py` mirrors multilingual/`.en` variants), so the `small`/`small.en` pair needs no new machinery. ## Provenance Decided by the agent under the user's explicit delegation for this wayfinding run (act as orchestrator and decision maker until the fog clears). No human was interviewed for this research ticket; the decision is grounded in the inspected sources above. ## Consequences for later tickets - No new tickets: the map's remaining fog is unaffected by this ladder. - The T1 thresholds need the installed-build measurement deferred by the performance-baseline decision before anyone hard-codes them. - No glossary or ADR change: the repo has no `GLOSSARY.md` or `docs/adr/` yet, and this ladder is spec content owned by this ticket, not a new domain term — so per the domain-modeling skill nothing is created.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
needs-info
Waiting on reporter for more information
 needs-triage
Maintainer needs to evaluate this issue
 ready-for-agent
Fully specified, ready for an AFK agent
 ready-for-human
Requires human implementation
 wayfinder:grilling
wayfinder:map
wayfinder:prototype
wayfinder:research
wayfinder:task
wontfix
Will not be actioned
No labels wayfinder:research
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
xavierk (Xavier Karma)
No Assignees xavierk
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: xavierk/voculinux#12