voculinux/docs/cuda-vocu.md

CUDA for Voculinux on Void Linux: integration plan

Status: plan, not started. Target release: 0.20.0. Owner decisions taken 2026-11-07 (section 9); phase 0 is complete.

This plan adds an optional CUDA backend for the whisper.cpp engine to the signed Void xbps repository, without changing what users who do not opt in receive. Each phase lists the files it touches, the tests that gate it, and its acceptance check, so it can be split into tickets as written (section 8).

1. Goal and non-goals

Goal: sudo xbps-install -S voculinux-cuda on Void Linux (glibc, x86_64) with an NVIDIA GPU of compute capability 7.5 or newer gives whisper.cpp a CUDA backend. The app selects it automatically and falls back to Vulkan, then CPU, when it cannot run.

Non-goals:

• No change to the default voculinux package. It stays Vulkan plus CPU and pulls in nothing from NVIDIA.
• No CUDA for the optional Python engines (OpenAI Whisper, Faster Whisper). They stay in the user's own Python environment.
• No musl, no aarch64. Voculinux publishes glibc x86_64 only.
• No user-installed toolkit. Users never install nvcc or a CUDA toolkit.

2. Facts this plan is built on

Measured or checked on 2026-11-07, on the maintainer host (Void, RTX 2050 4 GB, compute capability 8.6, driver 595.112.07, local CUDA 13.2 toolkit):

Fact	Value	Consequence
Void CUDA toolkit package	none, in main or nonfree (xbps-query -Rs cuda is empty)	the toolkit must come from NVIDIA's redistributable archives, at build time only
NVIDIA driver library libcuda.so.1	in Void nonfree package nvidia-libs	the CUDA package depends on nvidia-libs; users need the nonfree repo
Performance on RTX 2050, large-v3-turbo-q8_0, warm real-time factor	Vulkan 0.013, CUDA 0.016 (local build)	CUDA is not faster on this card. Value is for larger NVIDIA GPUs, still unmeasured
VRAM to prefer q8_0 / q5_0	Vulkan 1644 / 1359 MiB, CUDA 1690 / 1404 MiB	already in model_ladder.py (_PREFER_ABOVE_MIB)
libggml-cuda.so from the local build (sm_86 only)	57 MiB, 36 MiB zstd	one architecture; a multi-architecture build is larger (phase 1 measures it)
libcublasLt.so.13 / libcublas.so.13 / libcudart.so.13	482 / 52 / 0.7 MiB, about 365 MiB zstd together	the runtime package is large; check the release upload limit (phase 0)
Gitea [attachment] MAX_SIZE reported by /api/v1/settings/attachment	100 MB	that is the issue-attachment limit; the release-asset limit must be checked separately (phase 0)
Host GCC	14.2.1	nvcc 13.2 must accept the container's GCC without -allow-unsupported-compiler (phase 1 checks)
CUDA EULA, Attachment A	libcudart.so, libcublas.so, libcublasLt.so are redistributable with an application, unmodified; nvcc is not	ship those three libraries, never the compiler

The backend is a loadable library. The python3-pywhispercpp template already builds whisper.cpp with GGML_BACKEND_DL=ON, and the carried patch packaging/void/srcpkgs/python3-pywhispercpp/patches/load-ggml-backends-beside-module.patch loads every libggml-*.so that sits beside the extension. A libggml-cuda.so dropped into that directory is picked up without changing the extension. This is what makes a separate, optional CUDA package possible.

3. Target package layout

Package	New	Contents	Depends on
python3-pywhispercpp	no	extension, libggml-base, CPU variants, libggml-vulkan.so (unchanged)	as today
voculinux-cuda-libs	yes	the unmodified libcudart.so.13, libcublas.so.13, libcublasLt.so.13 in the private directory /usr/lib/voculinux/cuda/, plus the NVIDIA EULA as its license	nothing
python3-pywhispercpp-cuda	yes	only libggml-cuda.so, installed beside the extension in ${py3_sitelib}	python3-pywhispercpp-${version}_${revision} (exact), voculinux-cuda-libs, nvidia-libs
voculinux-cuda	yes	meta package, no files	voculinux, python3-pywhispercpp-cuda

Rules that come out of the EULA and xbps:

• The NVIDIA libraries live in a private directory that is not on the loader path, so only Voculinux's backend loads them ("accessed only by your application"). libggml-cuda.so finds them through its rpath: $ORIGIN for libggml-base, then /usr/lib/voculinux/cuda.
• voculinux-cuda-libs sets noshlibprovides=yes, so no other Void package can link against its copies, and keeps the files byte-identical to NVIDIA's archive (no strip: nostrip=yes).
• Both new build templates set restricted=yes: Void's builders must never build or publish them. The release container sets XBPS_ALLOW_RESTRICTED=yes for these two only.
• python3-pywhispercpp-cuda must be built from the identical pywhispercpp sdist, version and revision as python3-pywhispercpp. A ggml backend built from a different ggml source is refused at load time or, worse, crashes. The exact depends pin and a test enforce this.
• libcuda.so.1 and the CUDA libraries are not in Void's common/shlibs, so the CUDA template lists them in skiprdeps, or sets noverifyrdeps=yes, and declares the real dependencies in depends. Phase 1 decides which xbps-src accepts.

4. Phases

Phase 0: preflight (maintainer, no code) — done 2026-11-07

Result: Gitea 1.27.1 on xServer accepted a 419,430,400-byte release asset (HTTP 201) in a private scratch repository, and the authenticated download matched its sha256. The scratch repository was deleted. No server limit change is needed, and the Coolify service gitea carries no upload-limit overrides. The owner accepted the EULA terms (section 9).

Original steps, kept for re-checking after a Gitea upgrade:

1. Check the release upload limit on git.bongbetic.com: upload a 400 MiB file to a draft release in a scratch repository with tea. If refused, raise [repository.release] FILE_MAX_SIZE in the Gitea server config (infra change, needs the owner's OK). Delete the draft release afterwards.
2. The owner accepts redistributing the NVIDIA libraries under the CUDA EULA (section 9).

Acceptance: a 400 MiB asset uploads and downloads anonymously from a public release.

Phase 1: spike, measured before any template is committed

Work in a throwaway worktree, never on main.

1. In the pinned build image (packaging/void/tool_checksums.txt, base-image), unpack the redistributable archives from section 5, build pywhispercpp 1.5.0 with: GGML_CUDA=1, -DGGML_BACKEND_DL=ON -DBUILD_SHARED_LIBS=ON -DGGML_NATIVE=OFF, -DCMAKE_CUDA_ARCHITECTURES="75-real;86-real;89-real;120-real;120-virtual" (consumer GPUs only, section 9) and fatbin compression, if ggml's CMake offers it for this nvcc (GGML_CUDA_COMPRESSION_MODE=size; confirm the option exists in the vendored ggml).
2. Record: build time, libggml-cuda.so size (raw and zstd), and whether nvcc accepts the image's GCC.
3. On the RTX 2050, with the packaged python3-pywhispercpp installed, copy the built libggml-cuda.so beside the extension, put the three runtime libraries in a private directory reached through rpath, and check:
   • the extension registers the CUDA device (voculinux --debug shows CUDA0);
   • with both libggml-cuda.so and libggml-vulkan.so present, which device whisper.cpp takes as GPU device 0 (registration order in ggml_backend_load_all_from_path);
   • CUDA_VISIBLE_DEVICES= (empty) set before the first import makes it use Vulkan;
   • with nvidia-libs absent (any non-NVIDIA box or a container), loading the CUDA backend fails quietly and Vulkan or CPU still works.
4. Measure the real-time factor with scripts/measure_perf_baseline.py, CUDA against Vulkan, for large-v3-turbo-q8_0, q5_0 and small.

Acceptance: a short report in the ticket with the numbers from steps 2 to 4. If the backend does not load beside the packaged extension, stop and revise this plan before phase 2.

Phase 2: voculinux-cuda-libs template

Files:

• packaging/void/srcpkgs/voculinux-cuda-libs/template (new): distfiles = the cuda_cudart and libcublas archives, checksum from section 5, do_install copies only lib/libcudart.so.13*, lib/libcublas.so.13*, lib/libcublasLt.so.13* to /usr/lib/voculinux/cuda/ (symlinks kept), vlicense the archive's LICENSE; restricted=yes, nostrip=yes, noshlibprovides=yes, archs="x86_64", license="custom:NVIDIA-CUDA-EULA".
• tests/test_void_cuda_libs_template.py (new): checksums equal the pinned manifest values, only the three libraries are installed, the install path is private, the flags above are set.

Acceptance: xbps-src pkg voculinux-cuda-libs in the release container; a test that the installed files hash equal to the files inside the NVIDIA archive (unmodified).

Phase 3: python3-pywhispercpp-cuda template

Files:

• packaging/void/srcpkgs/python3-pywhispercpp-cuda/template (new). Same version, distfiles (the pywhispercpp sdist) and checksum as python3-pywhispercpp, plus the build-only CUDA archives (cuda_nvcc, cuda_crt, libnvvm, cuda_cudart, cuda_cccl, libcublas). pre_build assembles a toolkit root under $wrksrc/cuda from those archives and sets CUDACXX, CUDAToolkit_ROOT, the CMAKE_ARGS from phase 1 and -DCMAKE_INSTALL_RPATH='$ORIGIN:/usr/lib/voculinux/cuda'. Same patches/load-ggml-backends-beside-module.patch (symlink or copy; the test checks they are identical). do_install installs only libggml-cuda.so into ${py3_sitelib}. depends="python3-pywhispercpp-${version}_${revision} voculinux-cuda-libs nvidia-libs". restricted=yes, archs="x86_64".
• tests/test_void_pywhispercpp_template.py: extend to assert that the CUDA template's version, revision, sdist checksum and patch match python3-pywhispercpp exactly.
• tests/test_void_pywhispercpp_cuda_template.py (new): architecture list, rpath, only one installed file, pinned CUDA checksums, exact-version dependency.

Acceptance: the package builds in the release container, contains exactly one file, and readelf -d shows the rpath and NEEDED entries for libcudart.so.13, libcublas.so.13, libcuda.so.1 and libggml-base.so.0.

Phase 4: voculinux-cuda meta package and install message

Files:

• packaging/void/srcpkgs/voculinux-cuda/template (new): build_style=meta, depends="voculinux>=${version}_1 python3-pywhispercpp-cuda", its version kept equal to voculinux's.
• packaging/void/srcpkgs/voculinux-cuda/INSTALL.msg (new): needs the nonfree repository (xbps-install -S void-repo-nonfree) and an NVIDIA driver that supports CUDA 13.2; GPUs below compute capability 7.5 keep using Vulkan; how to check the backend (voculinux --debug).
• tests/test_void_app_template.py: version of voculinux-cuda equals voculinux's.
• packaging/void/build-release.sh check-tag: refuse a tag where the two differ.

Acceptance: xbps-install -S voculinux-cuda from a local staged repository pulls all four packages.

Phase 5: application backend selection

Today detect_pywhispercpp_gpu_backend() (src/vocalinux/speech_recognition/recognition_manager.py) returns the first match of libggml-vulkan* then libggml-cuda*, so with both installed it says vulkan. And cuda_toolkit_ready() (src/vocalinux/speech_recognition/model_ladder.py) requires nvcc or a complete toolkit root, which a packaged install never has.

Changes:

1. detect_pywhispercpp_gpu_backends() -> frozenset[str]: every GPU backend library present beside the extension ({"cuda", "vulkan"}, {"vulkan"}, or empty). Keep detect_pywhispercpp_gpu_backend() as a thin wrapper that returns the selected one, for current callers.
2. cuda_backend_ready(installed: frozenset[str]) -> bool in model_ladder.py, pure over its probe inputs: "cuda" is installed, nvidia-smi answers, compute capability at least MIN_CUDA_COMPUTE_CAPABILITY, and the CUDA runtime is reachable, meaning /usr/lib/voculinux/cuda/libcudart.so.13 exists or a complete toolkit root exists (the current probe, kept for source builds).
3. MIN_CUDA_COMPUTE_CAPABILITY becomes (7, 5): CUDA 13 does not build for older GPUs, so a 7.0 card must not be offered CUDA.
4. Selection, one function, select_gpu_backend(installed, cuda_ready, preference): preference auto gives CUDA when ready, else Vulkan when installed, else CPU; vulkan and cpu force those; cuda falls back with a logged warning when not ready.
5. Enforcement: when the selection is not CUDA, set CUDA_VISIBLE_DEVICES="" in os.environ before the first pywhispercpp import, so ggml registers no CUDA device and GPU device 0 is the Vulkan one. When it is CUDA and both are present, rely on the registration order measured in phase 1; if Vulkan registers first, also set GGML_VK_VISIBLE_DEVICES to hide the NVIDIA Vulkan device. The import happens in src/vocalinux/utils/pywhispercpp_loader.py; set the variables there, once.
6. Config key whispercpp_gpu_backend (auto default; cuda, vulkan, cpu) in src/vocalinux/ui/config_manager.py. A Settings control is a follow-up, not part of this plan.
7. The model ladder already takes the backend name; pass the selected one, so CUDA uses its own thresholds (1690 / 1404 MiB).
8. Log once at load: installed backends, selected backend, reason, device name.

Tests:

• tests/test_model_ladder.py: cuda_backend_ready truth table (no driver, low capability, packaged runtime only, toolkit only, neither); 7.0 now refused, 7.5 allowed.
• tests/test_recognition_manager.py (or a new tests/test_gpu_backend_selection.py): detection with cuda only, vulkan only, both, none; selection for each preference; the environment variables are set before import and only when needed.
• tests/test_host_process.py keeps passing: nvidia-smi probes run with env=host_env().

Acceptance: unit tests pass; on the RTX 2050 with the CUDA package installed, auto picks CUDA, vulkan picks Vulkan, and removing python3-pywhispercpp-cuda returns to Vulkan with no error.

Phase 6: release pipeline

Files:

• packaging/void/smoke.sh: add the three new packages to PKGS, in dependency order (voculinux-cuda-libs python3-pywhispercpp-cuda voculinux-cuda after python3-pywhispercpp and voculinux).
• packaging/void/container-release.sh: XBPS_ALLOW_RESTRICTED=yes in the build container's etc/conf; payload checks for the new packages (one file in python3-pywhispercpp-cuda, three libraries plus license in voculinux-cuda-libs); install smoke: install voculinux-cuda from the staged signed repository and check that, with no GPU and no driver in the container, voculinux still imports and selects CPU without error. The container has no GPU, so CUDA itself is verified by hand (phase 7).
• packaging/void/build-release.sh: nothing beyond check-tag (phase 4), unless phase 1 shows the build needs more time or memory limits.
• packaging/void/publish-release.sh: no change in calls; add a size preflight that fails before any upload when an asset exceeds the limit found in phase 0.
• packaging/void/tool_checksums.txt: one version row naming the CUDA redist manifest (redistrib_13.2.0.json) the archive checksums come from.
• Tests: tests/test_void_smoke_gate.py, tests/test_void_release_build.py, tests/test_void_publish_release.py for the above.

Acceptance: just void-release vX.Y.Z builds and signs all eight packages; the dry run of bash packaging/void/publish-release.sh lists every asset and passes the size preflight.

Phase 7: documentation and hardware acceptance

Docs:

• README.md, section "GPU and CUDA": add the voculinux-cuda row and the install line.
• packaging/void/README.md: the CUDA packages, the private library directory, the EULA rules, how to bump the CUDA version (new manifest, new checksums, rebuild both).
• docs/RELEASE_PROCESS.md: build time and size expectations; restricted packages.
• docs/TROUBLESHOOTING.md: "CUDA not used" checklist (nonfree repo, driver, capability, whispercpp_gpu_backend, debug log line).
• docs/UPDATE.md, CHANGELOG.md: 0.20.0 entry.

Hardware acceptance on the RTX 2050, from the published repository:

1. sudo xbps-install -Su voculinux-cuda installs from git.bongbetic.com with the pinned key and no prompt.
2. voculinux --debug logs the selected backend cuda and device CUDA0; dictation types text.
3. Real-time factor and VRAM recorded with scripts/measure_perf_baseline.py and compared with the Vulkan numbers in section 2.
4. sudo xbps-remove voculinux-cuda python3-pywhispercpp-cuda voculinux-cuda-libs, then relaunch: Vulkan, no error.
5. On the AMD iGPU only (CUDA_VISIBLE_DEVICES= and the NVIDIA Vulkan device hidden), Vulkan still works with the CUDA package installed.

5. Pinned CUDA inputs

From NVIDIA's redistrib_13.2.0.json (release date 2026-03-09), the version of the toolkit already proven on the maintainer host. Base URL: https://developer.download.nvidia.com/compute/cuda/redist/.

Component	Version	sha256	Used for
cuda_nvcc	13.2.51	706b996fefc59dc8d64d317fdf48d0aa84c4ae004eff43009dd918f40c5cc66a	build only
cuda_crt	13.2.51	fbc31fed55b7255591f3a19f575ca078827f5e6757d317d009f7ec1e69fcde4b	build only
libnvvm	13.2.51	e013fce38130d2337ea695aadc5ddd5dcfb78f9107903d72492b9819539749bb	build only
cuda_cccl	13.2.27	56e1bafb29faa87375b0484814870046530b88c0a421909096892f027ec1927b	build only (headers)
cuda_cudart	13.2.51	539edc1056e44d319f2112e9971c6415d78d4dde04b3f6ffbd20ec808e718526	build and shipped (libcudart.so.13)
libcublas	13.3.0.5	9f3c23d55b098d9159c435db59f228df79493152a778d4d367e7d60470d8c64b	build and shipped (libcublas.so.13, libcublasLt.so.13)

Each archive path is <component>/linux-x86_64/<component>-linux-x86_64-<version>-archive.tar.xz. The libcublas archive is about 765 MiB to download, most of it static libraries that are never shipped. xbps-src caches distfiles, so the two templates fetch it once per host.

Bumping CUDA: take the new redistrib_<version>.json, replace the six rows here and in both templates in one pull request, rebuild both packages together, and re-run phase 7.

6. Size and time budget

Item	Estimate	Source
voculinux-cuda-libs package	about 365 MiB compressed, 535 MiB installed	zstd of the 13.2 libraries
python3-pywhispercpp-cuda package	36 MiB per architecture before fatbin compression; total measured in phase 1	local sm_86 build
Build time of the CUDA backend	measured in phase 1	four architectures plus PTX with nvcc
User download for voculinux-cuda	the two packages above

Architecture set (consumer GPUs only, owner decision):

Target	GPUs
75-real	Turing: GeForce GTX 16 series, RTX 20 series
86-real	Ampere consumer: RTX 30 series and RTX 2050
89-real	Ada: RTX 40 series
120-real	Blackwell consumer: RTX 50 series
120-virtual	PTX, so a future consumer architecture JIT-compiles on first load

Left out on purpose: 80 (A100), 87 (Jetson Orin), 90 (H100), 100 and 103 (data-center Blackwell). Those GPUs are not Void desktop targets. An Ampere data-center card would JIT from no matching PTX and fail to load the backend, so the app falls back to Vulkan there.

7. Risks

Risk	Mitigation
Release asset upload limit below the package size	closed: phase 0 uploaded 400 MiB; publish-release.sh still gets a size preflight (phase 6)
ggml backend ABI mismatch between the two pywhispercpp builds	exact version_revision dependency plus a template-equality test
nvcc 13.2 rejects the image's GCC	phase 1 finds out; if so, use a GCC the image provides and nvcc supports, never -allow-unsupported-compiler
Driver too old for CUDA 13.2	INSTALL.msg states the minimum driver from NVIDIA's CUDA 13.2 release notes; selection falls back when cudaGetDeviceCount fails
Vulkan and CUDA both register the same NVIDIA GPU	phase 1 measures order; phase 5 hides the unused one with environment variables
No speed gain on small GPUs	owner chose CUDA first regardless; the vulkan preference is one config key away; phase 7 records numbers
Void's nvidia-libs upgrade changes the driver ABI	libcuda.so.1 is the stable driver ABI; phase 7 re-run after major driver updates

8. Ticket breakdown

Dependencies in brackets.

#	Ticket	Type
1	Phase 0: release upload limit and EULA decision (done 2026-11-07)	human / infra
2	Phase 1: CUDA backend spike and measurements [1]	spike
3	Phase 2: voculinux-cuda-libs template [2]	AFK
4	Phase 3: python3-pywhispercpp-cuda template [2, 3]	AFK
5	Phase 4: voculinux-cuda meta package [4]	AFK
6	Phase 5: backend detection, selection and config key [2]	AFK, parallel with 3 to 5
7	Phase 6: release pipeline [3, 4, 5]	AFK
8	Phase 7: docs, 0.20.0 release and hardware acceptance [6, 7]	human-in-the-loop

9. Owner decisions (taken 2026-11-07)

1. EULA: accepted. libcudart, libcublas and libcublasLt are redistributed from this repository under the NVIDIA CUDA EULA, unmodified, with the license shipped.
2. Gitea upload limit: may be raised. Phase 0 showed no change is needed.
3. Default preference: CUDA first. auto selects CUDA whenever it is ready, then Vulkan, then CPU, even though Vulkan matched CUDA on the RTX 2050.
4. Architectures: consumer GPUs only. 75-real;86-real;89-real;120-real;120-virtual (section 6).