xavierk/voculinux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

docs(cuda): integration plan for an optional CUDA backend on Void (#57)
CI / security (push) Successful in 1m10s
Details
CI / lint (push) Successful in 21s
Details
CI / ai-review (push) Skipped
Details

Browse Source
This commit was merged in pull request #57.
This commit is contained in:
xavierk
2026-11-07 03:17:22 +00:00
parent 808bd17704
commit 594686d0f7
2 changed files with 345 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/README.md
+1
View File
@@ -9,6 +9,7 @@ User and maintainer documentation for Voculinux.
| [TROUBLESHOOTING.md](TROUBLESHOOTING.md) | Users | Common install and runtime failures |
| [USER_GUIDE.md](USER_GUIDE.md) | Users | Dictation, engines, models, shortcuts, tips |
| [NIRI.md](NIRI.md) | Users | niri keybind pack and Dank Material Shell notes |
| [cuda-vocu.md](cuda-vocu.md) | Maintainers | Plan: optional CUDA backend in the Void repository |
| [UPDATE.md](UPDATE.md) | Users | How to upgrade; release notes by version |
| [DISTRO_COMPATIBILITY.md](DISTRO_COMPATIBILITY.md) | Users | Distro matrix, session caveats |
| [HTTP_REMOTE.md](HTTP_REMOTE.md) | Power users | Remote HTTP transcription |
docs/cuda-vocu.md
+344
View File
@@ -0,0 +1,344 @@
# CUDA for Voculinux on Void Linux: integration plan
Status: plan, not started. Target release: 0.20.0. Owner decisions needed: section 9.
This plan adds an optional CUDA backend for the whisper.cpp engine to the signed Void
xbps repository, without changing what users who do not opt in receive. Each phase
lists the files it touches, the tests that gate it, and its acceptance check, so it can
be split into tickets as written (section 8).
## 1. Goal and non-goals
Goal: `sudo xbps-install -S voculinux-cuda` on Void Linux (glibc, x86_64) with an NVIDIA
GPU of compute capability 7.5 or newer gives whisper.cpp a CUDA backend. The app
selects it automatically and falls back to Vulkan, then CPU, when it cannot run.
Non-goals:
- No change to the default `voculinux` package. It stays Vulkan plus CPU and pulls in
nothing from NVIDIA.
- No CUDA for the optional Python engines (OpenAI Whisper, Faster Whisper). They stay in
the user's own Python environment.
- No musl, no aarch64. Voculinux publishes glibc x86_64 only.
- No user-installed toolkit. Users never install `nvcc` or a CUDA toolkit.
## 2. Facts this plan is built on
Measured or checked on 2026-11-07, on the maintainer host (Void, RTX 2050 4 GB, compute
capability 8.6, driver 595.112.07, local CUDA 13.2 toolkit):
| Fact | Value | Consequence |
|------|-------|-------------|
| Void CUDA toolkit package | none, in main or nonfree (`xbps-query -Rs cuda` is empty) | the toolkit must come from NVIDIA's redistributable archives, at build time only |
| NVIDIA driver library `libcuda.so.1` | in Void nonfree package `nvidia-libs` | the CUDA package depends on `nvidia-libs`; users need the nonfree repo |
| Performance on RTX 2050, `large-v3-turbo-q8_0`, warm real-time factor | Vulkan 0.013, CUDA 0.016 (local build) | CUDA is not faster on this card. Value is for larger NVIDIA GPUs, still unmeasured |
| VRAM to prefer q8_0 / q5_0 | Vulkan 1644 / 1359 MiB, CUDA 1690 / 1404 MiB | already in `model_ladder.py` (`_PREFER_ABOVE_MIB`) |
| `libggml-cuda.so` from the local build (sm_86 only) | 57 MiB, 36 MiB zstd | one architecture; a multi-architecture build is larger (phase 1 measures it) |
| `libcublasLt.so.13` / `libcublas.so.13` / `libcudart.so.13` | 482 / 52 / 0.7 MiB, about 365 MiB zstd together | the runtime package is large; check the release upload limit (phase 0) |
| Gitea `[attachment] MAX_SIZE` reported by `/api/v1/settings/attachment` | 100 MB | that is the issue-attachment limit; the release-asset limit must be checked separately (phase 0) |
| Host GCC | 14.2.1 | nvcc 13.2 must accept the container's GCC without `-allow-unsupported-compiler` (phase 1 checks) |
| CUDA EULA, Attachment A | `libcudart.so`, `libcublas.so`, `libcublasLt.so` are redistributable with an application, unmodified; `nvcc` is not | ship those three libraries, never the compiler |
The backend is a loadable library. The `python3-pywhispercpp` template already builds
whisper.cpp with `GGML_BACKEND_DL=ON`, and the carried patch
`packaging/void/srcpkgs/python3-pywhispercpp/patches/load-ggml-backends-beside-module.patch`
loads every `libggml-*.so` that sits beside the extension. A `libggml-cuda.so` dropped into
that directory is picked up without changing the extension. This is what makes a separate,
optional CUDA package possible.
## 3. Target package layout
| Package | New | Contents | Depends on |
|---------|-----|----------|------------|
| `python3-pywhispercpp` | no | extension, `libggml-base`, CPU variants, `libggml-vulkan.so` (unchanged) | as today |
| `voculinux-cuda-libs` | yes | the unmodified `libcudart.so.13`, `libcublas.so.13`, `libcublasLt.so.13` in the private directory `/usr/lib/voculinux/cuda/`, plus the NVIDIA EULA as its license | nothing |
| `python3-pywhispercpp-cuda` | yes | only `libggml-cuda.so`, installed beside the extension in `${py3_sitelib}` | `python3-pywhispercpp-${version}_${revision}` (exact), `voculinux-cuda-libs`, `nvidia-libs` |
| `voculinux-cuda` | yes | meta package, no files | `voculinux`, `python3-pywhispercpp-cuda` |
Rules that come out of the EULA and xbps:
- The NVIDIA libraries live in a private directory that is not on the loader path, so only
Voculinux's backend loads them ("accessed only by your application"). `libggml-cuda.so`
finds them through its rpath: `$ORIGIN` for `libggml-base`, then `/usr/lib/voculinux/cuda`.
- `voculinux-cuda-libs` sets `noshlibprovides=yes`, so no other Void package can link
against its copies, and keeps the files byte-identical to NVIDIA's archive (no strip:
`nostrip=yes`).
- Both new build templates set `restricted=yes`: Void's builders must never build or
publish them. The release container sets `XBPS_ALLOW_RESTRICTED=yes` for these two only.
- `python3-pywhispercpp-cuda` must be built from the identical pywhispercpp sdist,
version and revision as `python3-pywhispercpp`. A ggml backend built from a different
ggml source is refused at load time or, worse, crashes. The exact `depends` pin and a
test enforce this.
- `libcuda.so.1` and the CUDA libraries are not in Void's `common/shlibs`, so the CUDA
template lists them in `skiprdeps`, or sets `noverifyrdeps=yes`, and declares the real
dependencies in `depends`. Phase 1 decides which xbps-src accepts.
## 4. Phases
### Phase 0: preflight (maintainer, no code)
1. Check the release upload limit on git.bongbetic.com: upload a 400 MiB file to a draft
release in a scratch repository with `tea`. If refused, raise
`[repository.release] FILE_MAX_SIZE` in the Gitea server config (infra change, needs
the owner's OK). Delete the draft release afterwards.
2. The owner accepts redistributing the NVIDIA libraries under the CUDA EULA (section 9).
Acceptance: a 400 MiB asset uploads and downloads anonymously from a public release.
### Phase 1: spike, measured before any template is committed
Work in a throwaway worktree, never on `main`.
1. In the pinned build image (`packaging/void/tool_checksums.txt`, `base-image`), unpack
the redistributable archives from section 5, build pywhispercpp 1.5.0 with:
`GGML_CUDA=1`, `-DGGML_BACKEND_DL=ON -DBUILD_SHARED_LIBS=ON -DGGML_NATIVE=OFF`,
`-DCMAKE_CUDA_ARCHITECTURES="75-real;80-real;86-real;89-real;90-real;100-real;120-real;120-virtual"`
and fatbin compression, if ggml's CMake offers it for this nvcc
(`GGML_CUDA_COMPRESSION_MODE=size`; confirm the option exists in the vendored ggml).
2. Record: build time, `libggml-cuda.so` size (raw and zstd), and whether nvcc accepts
the image's GCC.
3. On the RTX 2050, with the packaged `python3-pywhispercpp` installed, copy the built
`libggml-cuda.so` beside the extension, put the three runtime libraries in a private
directory reached through rpath, and check:
- the extension registers the CUDA device (`voculinux --debug` shows `CUDA0`);
- with both `libggml-cuda.so` and `libggml-vulkan.so` present, which device whisper.cpp
takes as GPU device 0 (registration order in `ggml_backend_load_all_from_path`);
- `CUDA_VISIBLE_DEVICES=` (empty) set before the first import makes it use Vulkan;
- with `nvidia-libs` absent (any non-NVIDIA box or a container), loading the CUDA
backend fails quietly and Vulkan or CPU still works.
4. Measure the real-time factor with `scripts/measure_perf_baseline.py`, CUDA against
Vulkan, for `large-v3-turbo-q8_0`, `q5_0` and `small`.
Acceptance: a short report in the ticket with the numbers from steps 2 to 4. If the
backend does not load beside the packaged extension, stop and revise this plan before
phase 2.
### Phase 2: `voculinux-cuda-libs` template
Files:
- `packaging/void/srcpkgs/voculinux-cuda-libs/template` (new): `distfiles` = the
`cuda_cudart` and `libcublas` archives, `checksum` from section 5, `do_install` copies
only `lib/libcudart.so.13*`, `lib/libcublas.so.13*`, `lib/libcublasLt.so.13*` to
`/usr/lib/voculinux/cuda/` (symlinks kept), `vlicense` the archive's `LICENSE`;
`restricted=yes`, `nostrip=yes`, `noshlibprovides=yes`, `archs="x86_64"`,
`license="custom:NVIDIA-CUDA-EULA"`.
- `tests/test_void_cuda_libs_template.py` (new): checksums equal the pinned manifest
values, only the three libraries are installed, the install path is private, the flags
above are set.
Acceptance: `xbps-src pkg voculinux-cuda-libs` in the release container; a test that the
installed files hash equal to the files inside the NVIDIA archive (unmodified).
### Phase 3: `python3-pywhispercpp-cuda` template
Files:
- `packaging/void/srcpkgs/python3-pywhispercpp-cuda/template` (new). Same `version`,
`distfiles` (the pywhispercpp sdist) and `checksum` as `python3-pywhispercpp`, plus the
build-only CUDA archives (`cuda_nvcc`, `cuda_crt`, `libnvvm`, `cuda_cudart`,
`cuda_cccl`, `libcublas`). `pre_build` assembles a toolkit root under `$wrksrc/cuda`
from those archives and sets `CUDACXX`, `CUDAToolkit_ROOT`, the `CMAKE_ARGS` from phase
1 and `-DCMAKE_INSTALL_RPATH='$ORIGIN:/usr/lib/voculinux/cuda'`. Same
`patches/load-ggml-backends-beside-module.patch` (symlink or copy; the test checks they
are identical). `do_install` installs only `libggml-cuda.so` into `${py3_sitelib}`.
`depends="python3-pywhispercpp-${version}_${revision} voculinux-cuda-libs nvidia-libs"`.
`restricted=yes`, `archs="x86_64"`.
- `tests/test_void_pywhispercpp_template.py`: extend to assert that the CUDA template's
version, revision, sdist checksum and patch match `python3-pywhispercpp` exactly.
- `tests/test_void_pywhispercpp_cuda_template.py` (new): architecture list, rpath, only
one installed file, pinned CUDA checksums, exact-version dependency.
Acceptance: the package builds in the release container, contains exactly one file, and
`readelf -d` shows the rpath and `NEEDED` entries for `libcudart.so.13`,
`libcublas.so.13`, `libcuda.so.1` and `libggml-base.so.0`.
### Phase 4: `voculinux-cuda` meta package and install message
Files:
- `packaging/void/srcpkgs/voculinux-cuda/template` (new): `build_style=meta`,
`depends="voculinux>=${version}_1 python3-pywhispercpp-cuda"`, its `version` kept equal
to `voculinux`'s.
- `packaging/void/srcpkgs/voculinux-cuda/INSTALL.msg` (new): needs the nonfree repository
(`xbps-install -S void-repo-nonfree`) and an NVIDIA driver that supports CUDA 13.2;
GPUs below compute capability 7.5 keep using Vulkan; how to check the backend
(`voculinux --debug`).
- `tests/test_void_app_template.py`: version of `voculinux-cuda` equals `voculinux`'s.
- `packaging/void/build-release.sh` `check-tag`: refuse a tag where the two differ.
Acceptance: `xbps-install -S voculinux-cuda` from a local staged repository pulls all
four packages.
### Phase 5: application backend selection
Today `detect_pywhispercpp_gpu_backend()`
(`src/vocalinux/speech_recognition/recognition_manager.py`) returns the first match of
`libggml-vulkan*` then `libggml-cuda*`, so with both installed it says `vulkan`. And
`cuda_toolkit_ready()` (`src/vocalinux/speech_recognition/model_ladder.py`) requires
`nvcc` or a complete toolkit root, which a packaged install never has.
Changes:
1. `detect_pywhispercpp_gpu_backends() -> frozenset[str]`: every GPU backend library
present beside the extension (`{"cuda", "vulkan"}`, `{"vulkan"}`, or empty). Keep
`detect_pywhispercpp_gpu_backend()` as a thin wrapper that returns the selected one,
for current callers.
2. `cuda_backend_ready(installed: frozenset[str]) -> bool` in `model_ladder.py`, pure
over its probe inputs: `"cuda"` is installed, `nvidia-smi` answers, compute capability
at least `MIN_CUDA_COMPUTE_CAPABILITY`, and the CUDA runtime is reachable, meaning
`/usr/lib/voculinux/cuda/libcudart.so.13` exists or a complete toolkit root exists
(the current probe, kept for source builds).
3. `MIN_CUDA_COMPUTE_CAPABILITY` becomes `(7, 5)`: CUDA 13 does not build for older
GPUs, so a 7.0 card must not be offered CUDA.
4. Selection, one function, `select_gpu_backend(installed, cuda_ready, preference)`:
preference `auto` gives CUDA when ready, else Vulkan when installed, else CPU;
`vulkan` and `cpu` force those; `cuda` falls back with a logged warning when not ready.
5. Enforcement: when the selection is not CUDA, set `CUDA_VISIBLE_DEVICES=""` in
`os.environ` before the first `pywhispercpp` import, so ggml registers no CUDA device
and GPU device 0 is the Vulkan one. When it is CUDA and both are present, rely on the
registration order measured in phase 1; if Vulkan registers first, also set
`GGML_VK_VISIBLE_DEVICES` to hide the NVIDIA Vulkan device. The import happens in
`src/vocalinux/utils/pywhispercpp_loader.py`; set the variables there, once.
6. Config key `whispercpp_gpu_backend` (`auto` default; `cuda`, `vulkan`, `cpu`) in
`src/vocalinux/ui/config_manager.py`. A Settings control is a follow-up, not part of
this plan.
7. The model ladder already takes the backend name; pass the selected one, so CUDA uses
its own thresholds (1690 / 1404 MiB).
8. Log once at load: installed backends, selected backend, reason, device name.
Tests:
- `tests/test_model_ladder.py`: `cuda_backend_ready` truth table (no driver, low
capability, packaged runtime only, toolkit only, neither); 7.0 now refused, 7.5 allowed.
- `tests/test_recognition_manager.py` (or a new `tests/test_gpu_backend_selection.py`):
detection with `cuda` only, `vulkan` only, both, none; selection for each preference;
the environment variables are set before import and only when needed.
- `tests/test_host_process.py` keeps passing: `nvidia-smi` probes run with
`env=host_env()`.
Acceptance: unit tests pass; on the RTX 2050 with the CUDA package installed, `auto`
picks CUDA, `vulkan` picks Vulkan, and removing `python3-pywhispercpp-cuda` returns to
Vulkan with no error.
### Phase 6: release pipeline
Files:
- `packaging/void/smoke.sh`: add the three new packages to `PKGS`, in dependency order
(`voculinux-cuda-libs python3-pywhispercpp-cuda voculinux-cuda` after
`python3-pywhispercpp` and `voculinux`).
- `packaging/void/container-release.sh`: `XBPS_ALLOW_RESTRICTED=yes` in the build
container's `etc/conf`; payload checks for the new packages (one file in
`python3-pywhispercpp-cuda`, three libraries plus license in `voculinux-cuda-libs`);
install smoke: install `voculinux-cuda` from the staged signed repository and check
that, with no GPU and no driver in the container, `voculinux` still imports and selects
CPU without error. The container has no GPU, so CUDA itself is verified by hand
(phase 7).
- `packaging/void/build-release.sh`: nothing beyond `check-tag` (phase 4), unless phase 1
shows the build needs more time or memory limits.
- `packaging/void/publish-release.sh`: no change in calls; add a size preflight that
fails before any upload when an asset exceeds the limit found in phase 0.
- `packaging/void/tool_checksums.txt`: one `version` row naming the CUDA redist manifest
(`redistrib_13.2.0.json`) the archive checksums come from.
- Tests: `tests/test_void_smoke_gate.py`, `tests/test_void_release_build.py`,
`tests/test_void_publish_release.py` for the above.
Acceptance: `just void-release vX.Y.Z` builds and signs all eight packages; the dry run of
`bash packaging/void/publish-release.sh` lists every asset and passes the size preflight.
### Phase 7: documentation and hardware acceptance
Docs:
- `README.md`, section "GPU and CUDA": add the `voculinux-cuda` row and the install line.
- `packaging/void/README.md`: the CUDA packages, the private library directory, the EULA
rules, how to bump the CUDA version (new manifest, new checksums, rebuild both).
- `docs/RELEASE_PROCESS.md`: build time and size expectations; restricted packages.
- `docs/TROUBLESHOOTING.md`: "CUDA not used" checklist (nonfree repo, driver, capability,
`whispercpp_gpu_backend`, debug log line).
- `docs/UPDATE.md`, `CHANGELOG.md`: 0.20.0 entry.
Hardware acceptance on the RTX 2050, from the published repository:
1. `sudo xbps-install -Su voculinux-cuda` installs from git.bongbetic.com with the
pinned key and no prompt.
2. `voculinux --debug` logs the selected backend `cuda` and device `CUDA0`; dictation
types text.
3. Real-time factor and VRAM recorded with `scripts/measure_perf_baseline.py` and
compared with the Vulkan numbers in section 2.
4. `sudo xbps-remove voculinux-cuda python3-pywhispercpp-cuda voculinux-cuda-libs`, then
relaunch: Vulkan, no error.
5. On the AMD iGPU only (`CUDA_VISIBLE_DEVICES=` and the NVIDIA Vulkan device hidden),
Vulkan still works with the CUDA package installed.
## 5. Pinned CUDA inputs
From NVIDIA's `redistrib_13.2.0.json` (release date 2026-03-09), the version of the toolkit
already proven on the maintainer host. Base URL:
`https://developer.download.nvidia.com/compute/cuda/redist/`.
| Component | Version | sha256 | Used for |
|-----------|---------|--------|----------|
| `cuda_nvcc` | 13.2.51 | `706b996fefc59dc8d64d317fdf48d0aa84c4ae004eff43009dd918f40c5cc66a` | build only |
| `cuda_crt` | 13.2.51 | `fbc31fed55b7255591f3a19f575ca078827f5e6757d317d009f7ec1e69fcde4b` | build only |
| `libnvvm` | 13.2.51 | `e013fce38130d2337ea695aadc5ddd5dcfb78f9107903d72492b9819539749bb` | build only |
| `cuda_cccl` | 13.2.27 | `56e1bafb29faa87375b0484814870046530b88c0a421909096892f027ec1927b` | build only (headers) |
| `cuda_cudart` | 13.2.51 | `539edc1056e44d319f2112e9971c6415d78d4dde04b3f6ffbd20ec808e718526` | build and shipped (`libcudart.so.13`) |
| `libcublas` | 13.3.0.5 | `9f3c23d55b098d9159c435db59f228df79493152a778d4d367e7d60470d8c64b` | build and shipped (`libcublas.so.13`, `libcublasLt.so.13`) |
Each archive path is `<component>/linux-x86_64/<component>-linux-x86_64-<version>-archive.tar.xz`.
The `libcublas` archive is about 765 MiB to download, most of it static libraries that are
never shipped. xbps-src caches distfiles, so the two templates fetch it once per host.
Bumping CUDA: take the new `redistrib_<version>.json`, replace the six rows here and in
both templates in one pull request, rebuild both packages together, and re-run phase 7.
## 6. Size and time budget
| Item | Estimate | Source |
|------|----------|--------|
| `voculinux-cuda-libs` package | about 365 MiB compressed, 535 MiB installed | zstd of the 13.2 libraries |
| `python3-pywhispercpp-cuda` package | 36 MiB per architecture before fatbin compression; total measured in phase 1 | local sm_86 build |
| Build time of the CUDA backend | measured in phase 1 | eight architectures with nvcc |
| User download for `voculinux-cuda` | the two packages above | |
If the multi-architecture backend is too large, drop `90-real` and `100-real`, which are
data-center GPUs, keeping `120-virtual` so newer cards still JIT-compile from PTX.
## 7. Risks
| Risk | Mitigation |
|------|------------|
| Release asset upload limit below the package size | phase 0 measures and raises it before any template work |
| ggml backend ABI mismatch between the two pywhispercpp builds | exact `version_revision` dependency plus a template-equality test |
| nvcc 13.2 rejects the image's GCC | phase 1 finds out; if so, use a GCC the image provides and nvcc supports, never `-allow-unsupported-compiler` |
| Driver too old for CUDA 13.2 | `INSTALL.msg` states the minimum driver from NVIDIA's CUDA 13.2 release notes; selection falls back when `cudaGetDeviceCount` fails |
| Vulkan and CUDA both register the same NVIDIA GPU | phase 1 measures order; phase 5 hides the unused one with environment variables |
| No speed gain on small GPUs | `auto` keeps CUDA first only where measured useful; the `vulkan` preference is one config key away; phase 7 records numbers |
| Void's `nvidia-libs` upgrade changes the driver ABI | `libcuda.so.1` is the stable driver ABI; phase 7 re-run after major driver updates |
## 8. Ticket breakdown
Dependencies in brackets.
| # | Ticket | Type |
|---|--------|------|
| 1 | Phase 0: release upload limit and EULA decision | human / infra |
| 2 | Phase 1: CUDA backend spike and measurements [1] | spike |
| 3 | Phase 2: `voculinux-cuda-libs` template [2] | AFK |
| 4 | Phase 3: `python3-pywhispercpp-cuda` template [2, 3] | AFK |
| 5 | Phase 4: `voculinux-cuda` meta package [4] | AFK |
| 6 | Phase 5: backend detection, selection and config key [2] | AFK, parallel with 3 to 5 |
| 7 | Phase 6: release pipeline [3, 4, 5] | AFK |
| 8 | Phase 7: docs, 0.20.0 release and hardware acceptance [6, 7] | human-in-the-loop |
## 9. Decisions for the owner
1. Accept redistributing `libcudart`, `libcublas` and `libcublasLt` under the NVIDIA CUDA
EULA from this repository (required).
2. Raise the Gitea release upload limit if phase 0 needs it (infra change on the Gitea
host).
3. Default preference: `auto` (CUDA first when ready, as the plan says) or `vulkan`
first, given that Vulkan matched CUDA on the RTX 2050.
4. Architecture list: the eight in phase 1, or a smaller consumer-only set.