diff --git a/docs/README.md b/docs/README.md index 0740e76..facbd1d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -9,6 +9,7 @@ User and maintainer documentation for Voculinux. | [TROUBLESHOOTING.md](TROUBLESHOOTING.md) | Users | Common install and runtime failures | | [USER_GUIDE.md](USER_GUIDE.md) | Users | Dictation, engines, models, shortcuts, tips | | [NIRI.md](NIRI.md) | Users | niri keybind pack and Dank Material Shell notes | +| [cuda-vocu.md](cuda-vocu.md) | Maintainers | Plan: optional CUDA backend in the Void repository | | [UPDATE.md](UPDATE.md) | Users | How to upgrade; release notes by version | | [DISTRO_COMPATIBILITY.md](DISTRO_COMPATIBILITY.md) | Users | Distro matrix, session caveats | | [HTTP_REMOTE.md](HTTP_REMOTE.md) | Power users | Remote HTTP transcription | diff --git a/docs/cuda-vocu.md b/docs/cuda-vocu.md new file mode 100644 index 0000000..2fd2b81 --- /dev/null +++ b/docs/cuda-vocu.md @@ -0,0 +1,344 @@ +# CUDA for Voculinux on Void Linux: integration plan + +Status: plan, not started. Target release: 0.20.0. Owner decisions needed: section 9. + +This plan adds an optional CUDA backend for the whisper.cpp engine to the signed Void +xbps repository, without changing what users who do not opt in receive. Each phase +lists the files it touches, the tests that gate it, and its acceptance check, so it can +be split into tickets as written (section 8). + +## 1. Goal and non-goals + +Goal: `sudo xbps-install -S voculinux-cuda` on Void Linux (glibc, x86_64) with an NVIDIA +GPU of compute capability 7.5 or newer gives whisper.cpp a CUDA backend. The app +selects it automatically and falls back to Vulkan, then CPU, when it cannot run. + +Non-goals: + +- No change to the default `voculinux` package. It stays Vulkan plus CPU and pulls in + nothing from NVIDIA. +- No CUDA for the optional Python engines (OpenAI Whisper, Faster Whisper). They stay in + the user's own Python environment. +- No musl, no aarch64. Voculinux publishes glibc x86_64 only. +- No user-installed toolkit. Users never install `nvcc` or a CUDA toolkit. + +## 2. Facts this plan is built on + +Measured or checked on 2026-11-07, on the maintainer host (Void, RTX 2050 4 GB, compute +capability 8.6, driver 595.112.07, local CUDA 13.2 toolkit): + +| Fact | Value | Consequence | +|------|-------|-------------| +| Void CUDA toolkit package | none, in main or nonfree (`xbps-query -Rs cuda` is empty) | the toolkit must come from NVIDIA's redistributable archives, at build time only | +| NVIDIA driver library `libcuda.so.1` | in Void nonfree package `nvidia-libs` | the CUDA package depends on `nvidia-libs`; users need the nonfree repo | +| Performance on RTX 2050, `large-v3-turbo-q8_0`, warm real-time factor | Vulkan 0.013, CUDA 0.016 (local build) | CUDA is not faster on this card. Value is for larger NVIDIA GPUs, still unmeasured | +| VRAM to prefer q8_0 / q5_0 | Vulkan 1644 / 1359 MiB, CUDA 1690 / 1404 MiB | already in `model_ladder.py` (`_PREFER_ABOVE_MIB`) | +| `libggml-cuda.so` from the local build (sm_86 only) | 57 MiB, 36 MiB zstd | one architecture; a multi-architecture build is larger (phase 1 measures it) | +| `libcublasLt.so.13` / `libcublas.so.13` / `libcudart.so.13` | 482 / 52 / 0.7 MiB, about 365 MiB zstd together | the runtime package is large; check the release upload limit (phase 0) | +| Gitea `[attachment] MAX_SIZE` reported by `/api/v1/settings/attachment` | 100 MB | that is the issue-attachment limit; the release-asset limit must be checked separately (phase 0) | +| Host GCC | 14.2.1 | nvcc 13.2 must accept the container's GCC without `-allow-unsupported-compiler` (phase 1 checks) | +| CUDA EULA, Attachment A | `libcudart.so`, `libcublas.so`, `libcublasLt.so` are redistributable with an application, unmodified; `nvcc` is not | ship those three libraries, never the compiler | + +The backend is a loadable library. The `python3-pywhispercpp` template already builds +whisper.cpp with `GGML_BACKEND_DL=ON`, and the carried patch +`packaging/void/srcpkgs/python3-pywhispercpp/patches/load-ggml-backends-beside-module.patch` +loads every `libggml-*.so` that sits beside the extension. A `libggml-cuda.so` dropped into +that directory is picked up without changing the extension. This is what makes a separate, +optional CUDA package possible. + +## 3. Target package layout + +| Package | New | Contents | Depends on | +|---------|-----|----------|------------| +| `python3-pywhispercpp` | no | extension, `libggml-base`, CPU variants, `libggml-vulkan.so` (unchanged) | as today | +| `voculinux-cuda-libs` | yes | the unmodified `libcudart.so.13`, `libcublas.so.13`, `libcublasLt.so.13` in the private directory `/usr/lib/voculinux/cuda/`, plus the NVIDIA EULA as its license | nothing | +| `python3-pywhispercpp-cuda` | yes | only `libggml-cuda.so`, installed beside the extension in `${py3_sitelib}` | `python3-pywhispercpp-${version}_${revision}` (exact), `voculinux-cuda-libs`, `nvidia-libs` | +| `voculinux-cuda` | yes | meta package, no files | `voculinux`, `python3-pywhispercpp-cuda` | + +Rules that come out of the EULA and xbps: + +- The NVIDIA libraries live in a private directory that is not on the loader path, so only + Voculinux's backend loads them ("accessed only by your application"). `libggml-cuda.so` + finds them through its rpath: `$ORIGIN` for `libggml-base`, then `/usr/lib/voculinux/cuda`. +- `voculinux-cuda-libs` sets `noshlibprovides=yes`, so no other Void package can link + against its copies, and keeps the files byte-identical to NVIDIA's archive (no strip: + `nostrip=yes`). +- Both new build templates set `restricted=yes`: Void's builders must never build or + publish them. The release container sets `XBPS_ALLOW_RESTRICTED=yes` for these two only. +- `python3-pywhispercpp-cuda` must be built from the identical pywhispercpp sdist, + version and revision as `python3-pywhispercpp`. A ggml backend built from a different + ggml source is refused at load time or, worse, crashes. The exact `depends` pin and a + test enforce this. +- `libcuda.so.1` and the CUDA libraries are not in Void's `common/shlibs`, so the CUDA + template lists them in `skiprdeps`, or sets `noverifyrdeps=yes`, and declares the real + dependencies in `depends`. Phase 1 decides which xbps-src accepts. + +## 4. Phases + +### Phase 0: preflight (maintainer, no code) + +1. Check the release upload limit on git.bongbetic.com: upload a 400 MiB file to a draft + release in a scratch repository with `tea`. If refused, raise + `[repository.release] FILE_MAX_SIZE` in the Gitea server config (infra change, needs + the owner's OK). Delete the draft release afterwards. +2. The owner accepts redistributing the NVIDIA libraries under the CUDA EULA (section 9). + +Acceptance: a 400 MiB asset uploads and downloads anonymously from a public release. + +### Phase 1: spike, measured before any template is committed + +Work in a throwaway worktree, never on `main`. + +1. In the pinned build image (`packaging/void/tool_checksums.txt`, `base-image`), unpack + the redistributable archives from section 5, build pywhispercpp 1.5.0 with: + `GGML_CUDA=1`, `-DGGML_BACKEND_DL=ON -DBUILD_SHARED_LIBS=ON -DGGML_NATIVE=OFF`, + `-DCMAKE_CUDA_ARCHITECTURES="75-real;80-real;86-real;89-real;90-real;100-real;120-real;120-virtual"` + and fatbin compression, if ggml's CMake offers it for this nvcc + (`GGML_CUDA_COMPRESSION_MODE=size`; confirm the option exists in the vendored ggml). +2. Record: build time, `libggml-cuda.so` size (raw and zstd), and whether nvcc accepts + the image's GCC. +3. On the RTX 2050, with the packaged `python3-pywhispercpp` installed, copy the built + `libggml-cuda.so` beside the extension, put the three runtime libraries in a private + directory reached through rpath, and check: + - the extension registers the CUDA device (`voculinux --debug` shows `CUDA0`); + - with both `libggml-cuda.so` and `libggml-vulkan.so` present, which device whisper.cpp + takes as GPU device 0 (registration order in `ggml_backend_load_all_from_path`); + - `CUDA_VISIBLE_DEVICES=` (empty) set before the first import makes it use Vulkan; + - with `nvidia-libs` absent (any non-NVIDIA box or a container), loading the CUDA + backend fails quietly and Vulkan or CPU still works. +4. Measure the real-time factor with `scripts/measure_perf_baseline.py`, CUDA against + Vulkan, for `large-v3-turbo-q8_0`, `q5_0` and `small`. + +Acceptance: a short report in the ticket with the numbers from steps 2 to 4. If the +backend does not load beside the packaged extension, stop and revise this plan before +phase 2. + +### Phase 2: `voculinux-cuda-libs` template + +Files: + +- `packaging/void/srcpkgs/voculinux-cuda-libs/template` (new): `distfiles` = the + `cuda_cudart` and `libcublas` archives, `checksum` from section 5, `do_install` copies + only `lib/libcudart.so.13*`, `lib/libcublas.so.13*`, `lib/libcublasLt.so.13*` to + `/usr/lib/voculinux/cuda/` (symlinks kept), `vlicense` the archive's `LICENSE`; + `restricted=yes`, `nostrip=yes`, `noshlibprovides=yes`, `archs="x86_64"`, + `license="custom:NVIDIA-CUDA-EULA"`. +- `tests/test_void_cuda_libs_template.py` (new): checksums equal the pinned manifest + values, only the three libraries are installed, the install path is private, the flags + above are set. + +Acceptance: `xbps-src pkg voculinux-cuda-libs` in the release container; a test that the +installed files hash equal to the files inside the NVIDIA archive (unmodified). + +### Phase 3: `python3-pywhispercpp-cuda` template + +Files: + +- `packaging/void/srcpkgs/python3-pywhispercpp-cuda/template` (new). Same `version`, + `distfiles` (the pywhispercpp sdist) and `checksum` as `python3-pywhispercpp`, plus the + build-only CUDA archives (`cuda_nvcc`, `cuda_crt`, `libnvvm`, `cuda_cudart`, + `cuda_cccl`, `libcublas`). `pre_build` assembles a toolkit root under `$wrksrc/cuda` + from those archives and sets `CUDACXX`, `CUDAToolkit_ROOT`, the `CMAKE_ARGS` from phase + 1 and `-DCMAKE_INSTALL_RPATH='$ORIGIN:/usr/lib/voculinux/cuda'`. Same + `patches/load-ggml-backends-beside-module.patch` (symlink or copy; the test checks they + are identical). `do_install` installs only `libggml-cuda.so` into `${py3_sitelib}`. + `depends="python3-pywhispercpp-${version}_${revision} voculinux-cuda-libs nvidia-libs"`. + `restricted=yes`, `archs="x86_64"`. +- `tests/test_void_pywhispercpp_template.py`: extend to assert that the CUDA template's + version, revision, sdist checksum and patch match `python3-pywhispercpp` exactly. +- `tests/test_void_pywhispercpp_cuda_template.py` (new): architecture list, rpath, only + one installed file, pinned CUDA checksums, exact-version dependency. + +Acceptance: the package builds in the release container, contains exactly one file, and +`readelf -d` shows the rpath and `NEEDED` entries for `libcudart.so.13`, +`libcublas.so.13`, `libcuda.so.1` and `libggml-base.so.0`. + +### Phase 4: `voculinux-cuda` meta package and install message + +Files: + +- `packaging/void/srcpkgs/voculinux-cuda/template` (new): `build_style=meta`, + `depends="voculinux>=${version}_1 python3-pywhispercpp-cuda"`, its `version` kept equal + to `voculinux`'s. +- `packaging/void/srcpkgs/voculinux-cuda/INSTALL.msg` (new): needs the nonfree repository + (`xbps-install -S void-repo-nonfree`) and an NVIDIA driver that supports CUDA 13.2; + GPUs below compute capability 7.5 keep using Vulkan; how to check the backend + (`voculinux --debug`). +- `tests/test_void_app_template.py`: version of `voculinux-cuda` equals `voculinux`'s. +- `packaging/void/build-release.sh` `check-tag`: refuse a tag where the two differ. + +Acceptance: `xbps-install -S voculinux-cuda` from a local staged repository pulls all +four packages. + +### Phase 5: application backend selection + +Today `detect_pywhispercpp_gpu_backend()` +(`src/vocalinux/speech_recognition/recognition_manager.py`) returns the first match of +`libggml-vulkan*` then `libggml-cuda*`, so with both installed it says `vulkan`. And +`cuda_toolkit_ready()` (`src/vocalinux/speech_recognition/model_ladder.py`) requires +`nvcc` or a complete toolkit root, which a packaged install never has. + +Changes: + +1. `detect_pywhispercpp_gpu_backends() -> frozenset[str]`: every GPU backend library + present beside the extension (`{"cuda", "vulkan"}`, `{"vulkan"}`, or empty). Keep + `detect_pywhispercpp_gpu_backend()` as a thin wrapper that returns the selected one, + for current callers. +2. `cuda_backend_ready(installed: frozenset[str]) -> bool` in `model_ladder.py`, pure + over its probe inputs: `"cuda"` is installed, `nvidia-smi` answers, compute capability + at least `MIN_CUDA_COMPUTE_CAPABILITY`, and the CUDA runtime is reachable, meaning + `/usr/lib/voculinux/cuda/libcudart.so.13` exists or a complete toolkit root exists + (the current probe, kept for source builds). +3. `MIN_CUDA_COMPUTE_CAPABILITY` becomes `(7, 5)`: CUDA 13 does not build for older + GPUs, so a 7.0 card must not be offered CUDA. +4. Selection, one function, `select_gpu_backend(installed, cuda_ready, preference)`: + preference `auto` gives CUDA when ready, else Vulkan when installed, else CPU; + `vulkan` and `cpu` force those; `cuda` falls back with a logged warning when not ready. +5. Enforcement: when the selection is not CUDA, set `CUDA_VISIBLE_DEVICES=""` in + `os.environ` before the first `pywhispercpp` import, so ggml registers no CUDA device + and GPU device 0 is the Vulkan one. When it is CUDA and both are present, rely on the + registration order measured in phase 1; if Vulkan registers first, also set + `GGML_VK_VISIBLE_DEVICES` to hide the NVIDIA Vulkan device. The import happens in + `src/vocalinux/utils/pywhispercpp_loader.py`; set the variables there, once. +6. Config key `whispercpp_gpu_backend` (`auto` default; `cuda`, `vulkan`, `cpu`) in + `src/vocalinux/ui/config_manager.py`. A Settings control is a follow-up, not part of + this plan. +7. The model ladder already takes the backend name; pass the selected one, so CUDA uses + its own thresholds (1690 / 1404 MiB). +8. Log once at load: installed backends, selected backend, reason, device name. + +Tests: + +- `tests/test_model_ladder.py`: `cuda_backend_ready` truth table (no driver, low + capability, packaged runtime only, toolkit only, neither); 7.0 now refused, 7.5 allowed. +- `tests/test_recognition_manager.py` (or a new `tests/test_gpu_backend_selection.py`): + detection with `cuda` only, `vulkan` only, both, none; selection for each preference; + the environment variables are set before import and only when needed. +- `tests/test_host_process.py` keeps passing: `nvidia-smi` probes run with + `env=host_env()`. + +Acceptance: unit tests pass; on the RTX 2050 with the CUDA package installed, `auto` +picks CUDA, `vulkan` picks Vulkan, and removing `python3-pywhispercpp-cuda` returns to +Vulkan with no error. + +### Phase 6: release pipeline + +Files: + +- `packaging/void/smoke.sh`: add the three new packages to `PKGS`, in dependency order + (`voculinux-cuda-libs python3-pywhispercpp-cuda voculinux-cuda` after + `python3-pywhispercpp` and `voculinux`). +- `packaging/void/container-release.sh`: `XBPS_ALLOW_RESTRICTED=yes` in the build + container's `etc/conf`; payload checks for the new packages (one file in + `python3-pywhispercpp-cuda`, three libraries plus license in `voculinux-cuda-libs`); + install smoke: install `voculinux-cuda` from the staged signed repository and check + that, with no GPU and no driver in the container, `voculinux` still imports and selects + CPU without error. The container has no GPU, so CUDA itself is verified by hand + (phase 7). +- `packaging/void/build-release.sh`: nothing beyond `check-tag` (phase 4), unless phase 1 + shows the build needs more time or memory limits. +- `packaging/void/publish-release.sh`: no change in calls; add a size preflight that + fails before any upload when an asset exceeds the limit found in phase 0. +- `packaging/void/tool_checksums.txt`: one `version` row naming the CUDA redist manifest + (`redistrib_13.2.0.json`) the archive checksums come from. +- Tests: `tests/test_void_smoke_gate.py`, `tests/test_void_release_build.py`, + `tests/test_void_publish_release.py` for the above. + +Acceptance: `just void-release vX.Y.Z` builds and signs all eight packages; the dry run of +`bash packaging/void/publish-release.sh` lists every asset and passes the size preflight. + +### Phase 7: documentation and hardware acceptance + +Docs: + +- `README.md`, section "GPU and CUDA": add the `voculinux-cuda` row and the install line. +- `packaging/void/README.md`: the CUDA packages, the private library directory, the EULA + rules, how to bump the CUDA version (new manifest, new checksums, rebuild both). +- `docs/RELEASE_PROCESS.md`: build time and size expectations; restricted packages. +- `docs/TROUBLESHOOTING.md`: "CUDA not used" checklist (nonfree repo, driver, capability, + `whispercpp_gpu_backend`, debug log line). +- `docs/UPDATE.md`, `CHANGELOG.md`: 0.20.0 entry. + +Hardware acceptance on the RTX 2050, from the published repository: + +1. `sudo xbps-install -Su voculinux-cuda` installs from git.bongbetic.com with the + pinned key and no prompt. +2. `voculinux --debug` logs the selected backend `cuda` and device `CUDA0`; dictation + types text. +3. Real-time factor and VRAM recorded with `scripts/measure_perf_baseline.py` and + compared with the Vulkan numbers in section 2. +4. `sudo xbps-remove voculinux-cuda python3-pywhispercpp-cuda voculinux-cuda-libs`, then + relaunch: Vulkan, no error. +5. On the AMD iGPU only (`CUDA_VISIBLE_DEVICES=` and the NVIDIA Vulkan device hidden), + Vulkan still works with the CUDA package installed. + +## 5. Pinned CUDA inputs + +From NVIDIA's `redistrib_13.2.0.json` (release date 2026-03-09), the version of the toolkit +already proven on the maintainer host. Base URL: +`https://developer.download.nvidia.com/compute/cuda/redist/`. + +| Component | Version | sha256 | Used for | +|-----------|---------|--------|----------| +| `cuda_nvcc` | 13.2.51 | `706b996fefc59dc8d64d317fdf48d0aa84c4ae004eff43009dd918f40c5cc66a` | build only | +| `cuda_crt` | 13.2.51 | `fbc31fed55b7255591f3a19f575ca078827f5e6757d317d009f7ec1e69fcde4b` | build only | +| `libnvvm` | 13.2.51 | `e013fce38130d2337ea695aadc5ddd5dcfb78f9107903d72492b9819539749bb` | build only | +| `cuda_cccl` | 13.2.27 | `56e1bafb29faa87375b0484814870046530b88c0a421909096892f027ec1927b` | build only (headers) | +| `cuda_cudart` | 13.2.51 | `539edc1056e44d319f2112e9971c6415d78d4dde04b3f6ffbd20ec808e718526` | build and shipped (`libcudart.so.13`) | +| `libcublas` | 13.3.0.5 | `9f3c23d55b098d9159c435db59f228df79493152a778d4d367e7d60470d8c64b` | build and shipped (`libcublas.so.13`, `libcublasLt.so.13`) | + +Each archive path is `/linux-x86_64/-linux-x86_64--archive.tar.xz`. +The `libcublas` archive is about 765 MiB to download, most of it static libraries that are +never shipped. xbps-src caches distfiles, so the two templates fetch it once per host. + +Bumping CUDA: take the new `redistrib_.json`, replace the six rows here and in +both templates in one pull request, rebuild both packages together, and re-run phase 7. + +## 6. Size and time budget + +| Item | Estimate | Source | +|------|----------|--------| +| `voculinux-cuda-libs` package | about 365 MiB compressed, 535 MiB installed | zstd of the 13.2 libraries | +| `python3-pywhispercpp-cuda` package | 36 MiB per architecture before fatbin compression; total measured in phase 1 | local sm_86 build | +| Build time of the CUDA backend | measured in phase 1 | eight architectures with nvcc | +| User download for `voculinux-cuda` | the two packages above | | + +If the multi-architecture backend is too large, drop `90-real` and `100-real`, which are +data-center GPUs, keeping `120-virtual` so newer cards still JIT-compile from PTX. + +## 7. Risks + +| Risk | Mitigation | +|------|------------| +| Release asset upload limit below the package size | phase 0 measures and raises it before any template work | +| ggml backend ABI mismatch between the two pywhispercpp builds | exact `version_revision` dependency plus a template-equality test | +| nvcc 13.2 rejects the image's GCC | phase 1 finds out; if so, use a GCC the image provides and nvcc supports, never `-allow-unsupported-compiler` | +| Driver too old for CUDA 13.2 | `INSTALL.msg` states the minimum driver from NVIDIA's CUDA 13.2 release notes; selection falls back when `cudaGetDeviceCount` fails | +| Vulkan and CUDA both register the same NVIDIA GPU | phase 1 measures order; phase 5 hides the unused one with environment variables | +| No speed gain on small GPUs | `auto` keeps CUDA first only where measured useful; the `vulkan` preference is one config key away; phase 7 records numbers | +| Void's `nvidia-libs` upgrade changes the driver ABI | `libcuda.so.1` is the stable driver ABI; phase 7 re-run after major driver updates | + +## 8. Ticket breakdown + +Dependencies in brackets. + +| # | Ticket | Type | +|---|--------|------| +| 1 | Phase 0: release upload limit and EULA decision | human / infra | +| 2 | Phase 1: CUDA backend spike and measurements [1] | spike | +| 3 | Phase 2: `voculinux-cuda-libs` template [2] | AFK | +| 4 | Phase 3: `python3-pywhispercpp-cuda` template [2, 3] | AFK | +| 5 | Phase 4: `voculinux-cuda` meta package [4] | AFK | +| 6 | Phase 5: backend detection, selection and config key [2] | AFK, parallel with 3 to 5 | +| 7 | Phase 6: release pipeline [3, 4, 5] | AFK | +| 8 | Phase 7: docs, 0.20.0 release and hardware acceptance [6, 7] | human-in-the-loop | + +## 9. Decisions for the owner + +1. Accept redistributing `libcudart`, `libcublas` and `libcublasLt` under the NVIDIA CUDA + EULA from this repository (required). +2. Raise the Gitea release upload limit if phase 0 needs it (infra change on the Gitea + host). +3. Default preference: `auto` (CUDA first when ready, as the plan says) or `vulkan` + first, given that Vulkan matched CUDA on the RTX 2050. +4. Architecture list: the eight in phase 1, or a smaller consumer-only set.