diff --git a/.github/workflows/unified-pipeline.yml b/.github/workflows/unified-pipeline.yml index 6dc9b53..d5834df 100644 --- a/.github/workflows/unified-pipeline.yml +++ b/.github/workflows/unified-pipeline.yml @@ -135,6 +135,8 @@ jobs: - name: Set up uv uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0 with: + # Keep export formatting aligned with the documented generation version. + version: "0.13.3" enable-cache: true # --only-group lint installs the linters without the project itself, so @@ -185,6 +187,7 @@ jobs: - name: Set up uv uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0 with: + version: "0.13.3" enable-cache: true python-version: ${{ matrix.python-version }} @@ -251,6 +254,7 @@ jobs: - name: Set up uv uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0 with: + version: "0.13.3" # Caches uv's download cache, not .venv/, so the checkout below stays # the fresh clone this job exists to exercise. enable-cache: true diff --git a/AGENTS.md b/AGENTS.md index 7531ac7..6c0a887 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,15 +34,15 @@ Rules: | Piece | Current | |---|---| | Python | `>=3.11` (`requires-python` in `pyproject.toml`; CI: 3.11–3.14). Floor, classifiers, CI matrix and `install.sh` must agree — `tests/test_python_version_policy.py` | -| GTK | GTK 3 via distro `python3-gi` (PyGObject). Never pip-install it | -| uv | `>=0.12,<0.13` (`[tool.uv]` in `pyproject.toml`). `uv.lock` is the source of truth; CI runs `uv sync --locked` / `uv run --locked`, which fails on drift | +| GTK | Runtime `venv/` uses distro `python3-gi` (PyGObject), never pip-install it; uv builds the locked PyGObject in the separate dev `.venv/` on supported distros | +| uv | `>=0.12,<0.14` (`[tool.uv]` in `pyproject.toml`). `uv.lock` is the source of truth; CI runs `uv sync --locked` / `uv run --locked`, which fails on drift | | just | https://just.systems or distro package `just` | | Format / lint / types | Black + isort (line length 100), flake8 (`E9,F63,F7,F82` only), mypy `src/` (targets 3.11 and runs as a CI lint gate). The tools live in the `lint` dependency group, not the `dev` extra, so CI can install them without building the project | | Website | Next.js / TypeScript in `web/` — see `web/AGENTS.md` | Two virtualenvs, on purpose: -- **`.venv/`** — uv's, for dev tooling. `just deps` syncs it; every other Python `just` recipe uses `uv run --no-sync` so it does not prune extras that `just deps-all` installed. No activation needed. +- **`.venv/`** — uv's, for dev tooling. `just deps` checks native headers before syncing it (on Fedora 40+, install `glib2-devel`, `cairo-gobject-devel`, `gobject-introspection-devel`, `libffi-devel` and the PortAudio headers); every other Python `just` recipe uses `uv run --no-sync` so it does not prune extras that `just deps-all` installed. No activation needed. - **`venv/`** — what `install.sh` builds for the app itself, from the *system* Python so distro `gi` is importable. Never run `install.sh` from an activated `.venv`: it drops an inherited `VIRTUAL_ENV` from `PATH` and picks `$SYSTEM_PYTHON` (default `/usr/bin/python3`) precisely because a venv built from uv's interpreter cannot see distro PyGObject. @@ -115,11 +115,13 @@ Website: `web/AGENTS.md`, `web/PRODUCT.md`, `web/DESIGN.md`. Do not duplicate si ## Dependencies (uv) +Use **uv 0.13.3** for `just lock` and `just export-check`, matching CI. Other versions in the supported contributor range can order equivalent platform markers differently; export verification compares files byte-for-byte. Update the CI pins and regenerate exports together when changing the generation version. + `uv.lock` is authoritative. `just lock` regenerates it, the hash-pinned `requirements/*.txt` exports, and `packaging/flatpak/python3-dependencies.yaml`. The AppImage build installs from `runtime`/`vad` plus `appimage.txt`; `install.sh` consumes the runtime and selected extras with `--require-hashes --no-deps --no-build-isolation`, then installs the project with `--no-deps --no-build-isolation`. **Do not edit `requirements/*.txt` by hand**, nor the url/sha256 pairs in `packaging/flatpak/python3-dependencies.yaml`. Change `pyproject.toml` (or `requirements/whisper.in` for the Whisper engine, `requirements/appimage*.in` for the AppImage), run `just lock`, and commit the lock plus the exports with the manifest change. | Constraint | Rule | |---|---| -| PyGObject | Distro `python3-gi` only, via `--system-site-packages`. Pip install fails on Ubuntu 24.04 (`girepository-2.0`). uv-managed interpreters do not see distro `gi` unless the venv is created that way | +| PyGObject | The runtime installer uses distro `python3-gi` via `--system-site-packages`; do not pip-install it into `venv/`. The dev `.venv/` instead builds the locked version with uv when glib 2.80+ and native headers are available. uv-managed interpreters do not see distro `gi` unless the venv is created with `--system-site-packages` | | `[vosk]` extra | Wheel-only on PyPI (no sdist). Never part of a source-buildable lock. `install.sh --engine=vosk` installs it | | Whisper CPU torch | `requirements/whisper.txt` is compiled from `requirements/whisper.in`. Pin `torch` to a `+cpu` local version — PyPI CUDA wheels win resolution regardless of index order. OpenAI Whisper does not use `torchaudio`, and the CPU index can lag torch | | pywhispercpp | Pinned in `install.sh` as `PYWHISPERCPP_VERSION` (keep in sync with `uv.lock`) | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 24965f4..cf71026 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -23,9 +23,8 @@ cd vocalinux ./install.sh --dev ``` -This installs system dependencies, creates a venv from the system Python, installs the package in editable mode with dev tools, and runs the test suite. - -`install.sh` always builds `venv/` from the system Python (`/usr/bin/python3`, or `$SYSTEM_PYTHON`), because distro PyGObject is only importable from that interpreter. It ignores an activated virtualenv, so you can run it from a shell that still has uv's `.venv` active. See [The two environments](#the-two-environments). +This installs system dependencies, creates `venv/` with an editable install and dev +tools, and runs the test suite. See [The two environments](#the-two-environments). ### Manual setup @@ -33,20 +32,20 @@ This installs system dependencies, creates a venv from the system Python, instal 2. Install system dependencies (examples): - **Ubuntu 24.04+** (`just deps` pip-builds PyGObject 3.56 from the lock): + **Ubuntu 24.04+:** ```bash sudo apt update sudo apt install -y python3-pip python3-gi python3-gi-cairo \ gir1.2-gtk-3.0 gir1.2-ibus-1.0 libgirepository-2.0-dev libgirepository1.0-dev \ - build-essential pkg-config libcairo2-dev \ + build-essential pkg-config libcairo2-dev libffi-dev \ python3-dev portaudio19-dev python3-venv xdotool ``` - **Debian 12** cannot pip-build PyGObject 3.56 (glib 2.74). Use `./install.sh --dev` for tests and running from source, then `venv/bin/pytest` / `venv/bin/python -m vocalinux.main --debug`. + **Debian 12 (installer environment only):** ```bash sudo apt install -y python3-pip python3-gi python3-gi-cairo \ gir1.2-gtk-3.0 gir1.2-ibus-1.0 libgirepository1.0-dev libcairo2-dev \ - build-essential pkg-config \ + build-essential pkg-config libffi-dev \ python3-dev portaudio19-dev python3-venv xdotool ``` @@ -54,10 +53,18 @@ This installs system dependencies, creates a venv from the system Python, instal ```bash sudo apt install -y python3-pip python3-gi python3-gi-cairo \ gir1.2-gtk-3.0 gir1.2-ibus-1.0 libgirepository-2.0-dev libcairo2-dev \ - build-essential pkg-config \ + build-essential pkg-config libffi-dev \ python3-dev portaudio19-dev python3-venv xdotool ``` + **Fedora 40+:** + ```bash + sudo dnf install -y python3-pip python3-gobject gtk3 ibus-devel \ + gobject-introspection-devel glib2-devel python3-devel portaudio-devel \ + cairo-devel cairo-gobject-devel libffi-devel python3-virtualenv pkg-config cmake \ + gcc-c++ make xdotool + ``` + AppIndicator (system tray): ```bash # Older Ubuntu: @@ -66,6 +73,10 @@ This installs system dependencies, creates a venv from the system Python, instal sudo apt install -y gir1.2-ayatanaappindicator3-0.1 ``` + **macOS (tooling only):** `brew install portaudio pkg-config`; use + `xcode-select --install` if a C compiler is missing. PyGObject/pycairo and their + header checks apply only on Linux, which is required to run the desktop app. + 3. Create the environment and install: ```bash @@ -73,7 +84,8 @@ This installs system dependencies, creates a venv from the system Python, instal just deps ``` - `just deps` compiles PyGObject from the lock into `.venv`. That needs `libgirepository-2.0-dev` (Ubuntu 24.04+). Debian 12 cannot build it; use Option 1 and the installer `venv/` instead. + Missing native build headers are reported with install commands before syncing. + For older Linux releases, use the installer environment described below. 4. Run: @@ -98,7 +110,16 @@ The repository uses two virtual environments on purpose. Do not merge them: | `.venv/` | `just deps` (uv) | pinned in `.python-version` | dev tooling: pytest, black, mypy, `just` recipes | | `venv/` | `./install.sh` | the system Python | running the installed app, which needs distro PyGObject | -`gi` (PyGObject) is the reason. `just deps` / `uv sync` build it from source into `.venv/`, which works wherever glib 2.80+ and `libgirepository-2.0-dev` are present (Arch, Fedora, Ubuntu 24.04 with that package, CI). Debian 12 cannot build PyGObject 3.56; use `./install.sh --dev` and `venv/bin/pytest`. `install.sh` never reuses `.venv/`: it builds `venv/` from the system Python with `--system-site-packages`, and rebuilds it if another interpreter created it. Set `SYSTEM_PYTHON=/usr/bin/python3.12 ./install.sh` on systems that ship several system interpreters. +- On Linux, `.venv/` builds the locked PyGObject, requiring glib 2.80+ and the + headers above. On Debian 12 or Fedora 39 and older, use `./install.sh --dev`, + then `venv/bin/pytest` or `venv/bin/python -m vocalinux.main --debug` instead. +- `venv/` uses distro PyGObject via `--system-site-packages`. The installer ignores + an activated venv and rebuilds one created by the wrong interpreter. It uses + `/usr/bin/python3` unless overridden, e.g. `SYSTEM_PYTHON=/usr/bin/python3.12 ./install.sh`. + +`just appimage` bundles its own Python/PyGObject in the pinned Ubuntu Docker image. +For GI failures, check `Checking bundled PyGObject after linuxdeploy` and the smoke +test's path diagnostics; installing `python3-gobject` on the host won't fix the bundle. ## Making changes diff --git a/justfile b/justfile index fcea2d2..574a409 100644 --- a/justfile +++ b/justfile @@ -5,8 +5,9 @@ # # Two environments, deliberately separate: # .venv/ dev tooling, built by uv from .python-version (3.13). `just deps` -# pip-builds PyGObject from the lock and needs libgirepository-2.0-dev -# (Ubuntu 24.04+). Debian 12 cannot build 3.56; use `just install-dev`. +# builds PyGObject from the lock and needs libgirepository-2.0-dev +# (Ubuntu 24.04+) or glib2-devel (Fedora 40+), plus cairo-gobject +# headers on Fedora. Debian 12 cannot build 3.56; use `just install-dev`. # venv/ what `just install` creates, always from the system Python so the # distro PyGObject package is importable. # install.sh ignores an activated .venv and rebuilds @@ -27,13 +28,23 @@ DEV_EXTRAS := "--extra dev --extra vad --group lint" default: @just --list +# Fail before `uv sync` when the headers pyaudio, pycairo and PyGObject need are +# absent. Without this, the first error is a C compiler traceback from inside a +# dependency build, which reads as a uv problem rather than a distro package. +# PyGObject/pycairo checks apply only on Linux, matching the dependency marker. +# An existing .venv/ with the platform's native packages skips the header check. +[no-exit-message] +[private] +_check_build_deps: + @python3 scripts/check_build_deps.py + # Create .venv/ and keep it matching the lock. --inexact is what makes this safe # to run ahead of every recipe: a plain `uv sync` removes extraneous packages, so # it would uninstall whatever `just deps-all` installed — the very thing the # --no-sync flags below exist to prevent. `version` skips this; it reads one # string out of version.py and needs a bare interpreter, not a sync. [private] -_tooling: +_tooling: _check_build_deps uv sync --inexact {{DEV_EXTRAS}} # Install Vocalinux @@ -45,11 +56,11 @@ install-dev: ./install.sh --dev # Install development dependencies into .venv/ (dev + vad extras) -deps: +deps: _check_build_deps uv sync {{DEV_EXTRAS}} # Install every optional extra — whisper/vosk engines and docs (CUDA torch, multi-GB) -deps-all: +deps-all: _check_build_deps uv sync --all-extras --group lint # Run test suite diff --git a/packaging/appimage/build.sh b/packaging/appimage/build.sh index ce7ed09..bc2b9c1 100755 --- a/packaging/appimage/build.sh +++ b/packaging/appimage/build.sh @@ -596,6 +596,32 @@ export DEPLOY_GTK_VERSION=3 -d "$APPDIR/usr/share/applications/vocalinux.desktop" \ -i "$APPDIR/usr/share/icons/hicolor/scalable/apps/vocalinux.svg" +# The plugin can change the bundle after PyGObject was installed. Check the +# package and the version-agnostic Python path that AppRun and the GI smoke use +# before entering a new mount/user namespace; a missing module there is not a +# missing python3-gi package on the builder's host. +echo "== Checking bundled PyGObject after linuxdeploy ==" +GI_SITE="$APPDIR/usr/lib/python${PY_VER}/site-packages" +if [ ! -f "$GI_SITE/gi/__init__.py" ]; then + echo "PyGObject is missing from $GI_SITE after linuxdeploy." >&2 + echo "Check the earlier uv install of requirements/appimage.txt." >&2 + exit 1 +fi +if [ -d "$APPDIR/usr/lib/python3" ] && [ ! -L "$APPDIR/usr/lib/python3" ]; then + echo "Expected usr/lib/python3 to be a symlink to python${PY_VER}, not a directory." >&2 + exit 1 +fi +# Restore this link if linuxdeploy replaced it; a broken alias hides gi even +# though its versioned site-packages directory is present in the bundle. +ln -sfn "python${PY_VER}" "$APPDIR/usr/lib/python3" +if ! PYTHONHOME="$APPDIR/usr" \ + PYTHONPATH="$APPDIR/usr/lib/python3:$APPDIR/usr/lib/python3/site-packages" \ + LD_LIBRARY_PATH="$APPDIR/usr/lib" \ + "$APPDIR/usr/bin/python3" -c 'import gi; print(" gi:", gi.__file__)'; then + echo "The bundled Python cannot import gi before the isolated smoke test." >&2 + exit 1 +fi + # linuxdeploy copies the host's full typelib set; keep those extras (transitive # GIR deps vary by distro) and re-assert our required seed on top. echo "== Ensuring required typelibs are present (keeping linuxdeploy extras) ==" @@ -715,7 +741,32 @@ smoke_gi_imports() { export GI_TYPELIB_PATH='$appdir/usr/lib/girepository-1.0' export LD_LIBRARY_PATH='$appdir/usr/lib' '$appdir/usr/bin/python3' - <<'PY' -import gi +try: + import gi +except ModuleNotFoundError: + import os + import pathlib + import stat + import sys + + print('Bundled gi is not importable inside the isolated smoke test.', file=sys.stderr) + print(f' python: {sys.executable}, uid: {os.geteuid()}', file=sys.stderr) + print(' PYTHONHOME:', os.environ.get('PYTHONHOME'), file=sys.stderr) + print(' PYTHONPATH:', os.environ.get('PYTHONPATH'), file=sys.stderr) + print(f' sys.path: {sys.path}', file=sys.stderr) + lib = pathlib.Path(sys.prefix) / 'lib' + versioned = lib / f'python{sys.version_info.major}.{sys.version_info.minor}' + for candidate in (lib / 'python3', versioned / 'site-packages' / 'gi', + lib / 'python3' / 'site-packages' / 'gi'): + try: + details = candidate.stat() + except OSError as exc: + print(f' {candidate}: {exc}', file=sys.stderr) + else: + print(f' {candidate}: mode={stat.S_IMODE(details.st_mode):o}, ' + f'uid={details.st_uid}, accessible={os.access(candidate, os.R_OK | os.X_OK)}', + file=sys.stderr) + raise gi.require_version('Gtk', '3.0') from gi.repository import Gtk # noqa: F401 try: diff --git a/pyproject.toml b/pyproject.toml index f2824a0..663110a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -130,9 +130,9 @@ Issues = "https://github.com/VocaHQ/vocalinux/issues" Changelog = "https://github.com/VocaHQ/vocalinux/releases" [tool.uv] -# Lock/exports are generated with uv; keep the minor pinned so lockfile -# upgrades are deliberate (see `just lock`). -required-version = ">=0.12,<0.13" +# Contributors may use uv 0.12 or 0.13. CI and lock/export generation use +# uv 0.13.3 for reproducible export formatting (see `just lock`). +required-version = ">=0.12,<0.14" [tool.setuptools] # Resource dirs carry no .py files, but setuptools still treats them as packages. diff --git a/requirements/dev.txt b/requirements/dev.txt index 11804e6..473f681 100644 --- a/requirements/dev.txt +++ b/requirements/dev.txt @@ -356,7 +356,7 @@ distlib==0.4.3 \ --hash=sha256:4b0ce306c966eb73bc3a7b6abad017c556dadd92c44701562cd528ac7fde4d5b \ --hash=sha256:f152097224a0ae24be5a0f6bae1b9359af82133bce63f98a95f86cae1aede9ed # via virtualenv -evdev==1.9.3 ; 'linux' in sys_platform or sys_platform == 'linux' \ +evdev==1.9.3 ; sys_platform == 'linux' or 'linux' in sys_platform \ --hash=sha256:2c140e01ac8437758fa23fe5c871397412461f42d421aa20241dc8fe8cfccbc9 # via # pynput diff --git a/requirements/faster-whisper.txt b/requirements/faster-whisper.txt index 7d24da1..611dbcb 100644 --- a/requirements/faster-whisper.txt +++ b/requirements/faster-whisper.txt @@ -211,7 +211,7 @@ ctranslate2==4.8.2 \ --hash=sha256:f6f0b576c247984d3fc299a372ccc9319b668d6d25b0539f3c861beba15d0504 \ --hash=sha256:fedda669421a57f8164568ee86ea265727016988d310fcc2e24ca30e9ec457cd # via faster-whisper -evdev==1.9.3 ; 'linux' in sys_platform or sys_platform == 'linux' \ +evdev==1.9.3 ; sys_platform == 'linux' or 'linux' in sys_platform \ --hash=sha256:2c140e01ac8437758fa23fe5c871397412461f42d421aa20241dc8fe8cfccbc9 # via # pynput diff --git a/requirements/parakeet.txt b/requirements/parakeet.txt index 39d7084..26cd13f 100644 --- a/requirements/parakeet.txt +++ b/requirements/parakeet.txt @@ -150,7 +150,7 @@ colorama==0.4.6 ; sys_platform == 'win32' \ --hash=sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44 \ --hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6 # via tqdm -evdev==1.9.3 ; 'linux' in sys_platform or sys_platform == 'linux' \ +evdev==1.9.3 ; sys_platform == 'linux' or 'linux' in sys_platform \ --hash=sha256:2c140e01ac8437758fa23fe5c871397412461f42d421aa20241dc8fe8cfccbc9 # via # pynput diff --git a/requirements/runtime.txt b/requirements/runtime.txt index da19847..e6c7a45 100644 --- a/requirements/runtime.txt +++ b/requirements/runtime.txt @@ -150,7 +150,7 @@ colorama==0.4.6 ; sys_platform == 'win32' \ --hash=sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44 \ --hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6 # via tqdm -evdev==1.9.3 ; 'linux' in sys_platform or sys_platform == 'linux' \ +evdev==1.9.3 ; sys_platform == 'linux' or 'linux' in sys_platform \ --hash=sha256:2c140e01ac8437758fa23fe5c871397412461f42d421aa20241dc8fe8cfccbc9 # via # pynput diff --git a/requirements/vad.txt b/requirements/vad.txt index 5f03bf9..3be2e38 100644 --- a/requirements/vad.txt +++ b/requirements/vad.txt @@ -150,7 +150,7 @@ colorama==0.4.6 ; sys_platform == 'win32' \ --hash=sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44 \ --hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6 # via tqdm -evdev==1.9.3 ; 'linux' in sys_platform or sys_platform == 'linux' \ +evdev==1.9.3 ; sys_platform == 'linux' or 'linux' in sys_platform \ --hash=sha256:2c140e01ac8437758fa23fe5c871397412461f42d421aa20241dc8fe8cfccbc9 # via # pynput diff --git a/requirements/vosk.txt b/requirements/vosk.txt index acb9dcb..d1471a2 100644 --- a/requirements/vosk.txt +++ b/requirements/vosk.txt @@ -240,7 +240,7 @@ colorama==0.4.6 ; sys_platform == 'win32' \ --hash=sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44 \ --hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6 # via tqdm -evdev==1.9.3 ; 'linux' in sys_platform or sys_platform == 'linux' \ +evdev==1.9.3 ; sys_platform == 'linux' or 'linux' in sys_platform \ --hash=sha256:2c140e01ac8437758fa23fe5c871397412461f42d421aa20241dc8fe8cfccbc9 # via # pynput diff --git a/scripts/check_build_deps.py b/scripts/check_build_deps.py new file mode 100755 index 0000000..0e5250f --- /dev/null +++ b/scripts/check_build_deps.py @@ -0,0 +1,212 @@ +#!/usr/bin/env python3 +"""Fail fast when the native headers `just deps` needs are missing. + +`uv sync` builds pyaudio, pycairo and PyGObject from source on Linux. Without +their development headers it fails deep inside a C compiler invocation, which +reads as a uv or dependency problem even though the fix is a distro package. +This check runs before the sync and names what is missing. +""" + +from __future__ import annotations + +import shutil +import subprocess +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +VENV_PYTHON = ROOT / ".venv" / "bin" / "python" + +# pkg-config modules the source builds query. Labels explain the module to a +# contributor who does not know which package owns it. +BUILD_MODULES = (("portaudio-2.0", "PortAudio headers (pyaudio)"),) +# PyGObject and its pycairo dependency are selected only on Linux by the +# sys_platform marker in pyproject.toml. They must not gate macOS tooling. +PYGOBJECT_BUILD_MODULES = ( + ("cairo", "cairo headers (pycairo)"), + ("cairo-gobject", "cairo-gobject headers (PyGObject)"), + ("girepository-2.0", "GObject Introspection headers (glib 2.80+)"), + ("libffi", "libffi headers (PyGObject)"), +) +MIN_GIREPOSITORY_VERSION = "2.80" + +APT_HINT = [ + "Ubuntu 24.04+ / Debian 13+:", + " sudo apt install -y build-essential pkg-config libcairo2-dev portaudio19-dev \\", + " libgirepository-2.0-dev libffi-dev", +] + +DEBIAN_12_NOTE = [ + "", + "Debian 12 cannot build PyGObject 3.56 (glib is too old). Use ./install.sh --dev", + "there and run the suite with venv/bin/pytest.", +] + +DNF_HINT = [ + "Fedora 40+ / RHEL-based systems with glib 2.80+:", + " sudo dnf install -y gcc gcc-c++ make pkg-config cairo-devel \\", + " cairo-gobject-devel portaudio-devel glib2-devel \\", + " gobject-introspection-devel libffi-devel", + "If your system's glib is older than 2.80, use ./install.sh --dev instead.", +] + +FEDORA_OLD_NOTE = [ + "Fedora 39 and older cannot build PyGObject 3.56 (glib is too old).", + "Use ./install.sh --dev there and run the suite with venv/bin/pytest.", +] + +PACMAN_HINT = [ + "Arch / Manjaro:", + " sudo pacman -S --needed base-devel pkgconf cairo portaudio gobject-introspection", +] + +ZYPPER_HINT = [ + "openSUSE:", + " sudo zypper install -y gcc gcc-c++ make pkg-config cairo-devel \\", + " portaudio-devel gobject-introspection-devel libffi-devel", +] + +GENERIC_HINT = [ + "Install the development packages for PortAudio, cairo, GObject Introspection", + "(glib 2.80+) and libffi, plus a C compiler and pkg-config.", +] + +MACOS_HINT = [ + "macOS (Homebrew):", + " brew install portaudio pkg-config", + "Install the Xcode Command Line Tools if a C compiler is missing:", + " xcode-select --install", +] + + +def read_os_release(path: Path = Path("/etc/os-release")) -> dict[str, str]: + """Parse os-release without executing it as shell.""" + values: dict[str, str] = {} + try: + lines = path.read_text(encoding="utf-8").splitlines() + except OSError: + return values + + for line in lines: + key, separator, value = line.partition("=") + if not separator or not key or key.startswith("#"): + continue + values[key] = value.strip().strip('"').strip("'") + return values + + +def install_hint(release: dict[str, str]) -> list[str]: + """Return distro-specific install commands for the missing build headers.""" + distro = release.get("ID", "").lower() + like = release.get("ID_LIKE", "").lower() + + if distro == "ubuntu" or "ubuntu" in like: + return APT_HINT + + if distro == "debian" or "debian" in like: + return [*APT_HINT, *DEBIAN_12_NOTE] + + if distro in {"fedora", "rhel", "centos", "rocky", "almalinux"} or any( + family in like for family in ("fedora", "rhel") + ): + if distro == "fedora" and release.get("VERSION_ID", "").isdigit(): + if int(release["VERSION_ID"]) < 40: + return FEDORA_OLD_NOTE + return DNF_HINT + + if distro in {"arch", "manjaro", "endeavouros"} or "arch" in like: + return PACMAN_HINT + + if distro == "opensuse" or distro.startswith("opensuse-") or "suse" in like: + return ZYPPER_HINT + + return GENERIC_HINT + + +def venv_has_compiled_packages() -> bool: + """True when `.venv` has the native packages selected for this platform.""" + if not VENV_PYTHON.is_file(): + return False + packages = ("pyaudio",) + if sys.platform == "linux": + packages += ("pycairo", "pygobject") + check = "import importlib.metadata as metadata; " + "; ".join( + f"metadata.version({package!r})" for package in packages + ) + result = subprocess.run( + [str(VENV_PYTHON), "-c", check], + capture_output=True, + check=False, + ) + return result.returncode == 0 + + +def missing_build_dependencies() -> list[str]: + """Return the compiler or pkg-config modules uv cannot find, if any.""" + missing: list[str] = [] + + if shutil.which("pkg-config") is None: + missing.append("pkg-config") + if not any(shutil.which(compiler) for compiler in ("cc", "gcc", "clang")): + missing.append("a C compiler") + + # Module probes need pkg-config; report the missing tool instead of a list + # of modules the user cannot check yet. + if missing: + return missing + + modules = BUILD_MODULES + if sys.platform == "linux": + modules += PYGOBJECT_BUILD_MODULES + for module, description in modules: + # PyGObject 3.56 requires girepository-2.0 >= 2.80; finding an older + # .pc file is not enough to build it successfully. + args = ["pkg-config", "--exists", module] + if module == "girepository-2.0": + args = ["pkg-config", f"--atleast-version={MIN_GIREPOSITORY_VERSION}", module] + result = subprocess.run( + args, + capture_output=True, + check=False, + ) + if result.returncode != 0: + missing.append(description) + + return missing + + +def main() -> int: + """Print an actionable dependency report when the build cannot succeed.""" + if venv_has_compiled_packages(): + return 0 + + missing = missing_build_dependencies() + if not missing: + return 0 + + print("error: missing native build dependencies for .venv/", file=sys.stderr) + print(file=sys.stderr) + print( + "`just deps` builds native Python packages from source, so uv needs their", + file=sys.stderr, + ) + print("development headers before it can create the environment.", file=sys.stderr) + print(file=sys.stderr) + print("Missing:", file=sys.stderr) + for item in missing: + print(f" - {item}", file=sys.stderr) + print(file=sys.stderr) + print("Install them for your system:", file=sys.stderr) + hints = MACOS_HINT if sys.platform == "darwin" else install_hint(read_os_release()) + for line in hints: + print(line, file=sys.stderr) + print(file=sys.stderr) + print( + "See the Manual setup section in CONTRIBUTING.md for the full package list.", + file=sys.stderr, + ) + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_appimage_packaging.py b/tests/test_appimage_packaging.py index 2d653b2..8197e50 100644 --- a/tests/test_appimage_packaging.py +++ b/tests/test_appimage_packaging.py @@ -142,6 +142,21 @@ def test_appimage_build_smokes_gi_without_host_typelibs(): assert "unshare --user --mount" in text +def test_appimage_build_checks_gi_after_linuxdeploy() -> None: + """The GTK plugin must not silently remove gi or the path AppRun imports it from.""" + text = BUILD_SH.read_text(encoding="utf-8") + deploy = text.index('"$TOOLDIR/linuxdeploy" --appdir') + check = text.index("== Checking bundled PyGObject after linuxdeploy ==", deploy) + smoke = text.index('smoke_gi_imports "$APPDIR"', check) + between = text[check:smoke] + + assert 'GI_SITE="$APPDIR/usr/lib/python${PY_VER}/site-packages"' in between + assert '"$GI_SITE/gi/__init__.py"' in between + assert 'ln -sfn "python${PY_VER}" "$APPDIR/usr/lib/python3"' in between + assert "'import gi; print(" in between + assert "sys.path:" in text, "isolated GI failures should report where Python looked" + + def test_appimage_build_rebuilds_pywhispercpp_with_vulkan(): text = BUILD_SH.read_text() assert "GGML_VULKAN=1" in text diff --git a/tests/test_check_build_deps.py b/tests/test_check_build_deps.py new file mode 100644 index 0000000..e6c0f41 --- /dev/null +++ b/tests/test_check_build_deps.py @@ -0,0 +1,250 @@ +"""Guards for the `just deps` native build-dependency preflight.""" + +import importlib.metadata +import importlib.util +import subprocess +from pathlib import Path +from typing import Any + +import pytest + +SCRIPT = Path(__file__).resolve().parents[1] / "scripts" / "check_build_deps.py" +SPEC = importlib.util.spec_from_file_location("check_build_deps", SCRIPT) +assert SPEC is not None and SPEC.loader is not None +PREFLIGHT: Any = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(PREFLIGHT) + + +@pytest.fixture(autouse=True) +def linux_platform(monkeypatch: pytest.MonkeyPatch) -> None: + """Keep Linux checks covered even when contributors run the suite on macOS.""" + monkeypatch.setattr(PREFLIGHT.sys, "platform", "linux") + + +def test_os_release_parser_strips_quotes_and_ignores_comments(tmp_path: Path) -> None: + """os-release is parsed, not sourced; quoted values lose their quotes.""" + release = tmp_path / "os-release" + release.write_text( + 'ID=fedora\nID_LIKE="rhel centos"\n# comment\nVERSION_ID="45"\n', + encoding="utf-8", + ) + + assert PREFLIGHT.read_os_release(release) == { + "ID": "fedora", + "ID_LIKE": "rhel centos", + "VERSION_ID": "45", + } + + +@pytest.mark.parametrize( + "distro,like,expected", + [ + ("ubuntu", "debian", "sudo apt install"), + ("fedora", "rhel", "sudo dnf install"), + ("rocky", "rhel fedora", "sudo dnf install"), + ("arch", "", "sudo pacman -S"), + ("opensuse-tumbleweed", "suse", "sudo zypper install"), + ], +) +def test_install_hint_matches_the_distro_family(distro: str, like: str, expected: str) -> None: + """The failure names the package manager the contributor actually has.""" + hint = "\n".join(PREFLIGHT.install_hint({"ID": distro, "ID_LIKE": like})) + assert expected in hint + + +def test_debian_hint_points_older_releases_at_the_installer() -> None: + """Debian 12 cannot build PyGObject 3.56, so the hint says what works.""" + hint = "\n".join(PREFLIGHT.install_hint({"ID": "debian", "ID_LIKE": ""})) + assert "sudo apt install" in hint + assert "./install.sh --dev" in hint + + +def test_fedora_hint_keeps_the_split_cairo_gobject_package() -> None: + """Fedora splits cairo-gobject.pc and girepository-2.0.pc across packages.""" + hint = "\n".join(PREFLIGHT.install_hint({"ID": "fedora", "ID_LIKE": ""})) + assert "cairo-gobject-devel" in hint + assert "glib2-devel" in hint + + +def test_old_fedora_hint_points_at_the_installer() -> None: + """Installing more headers cannot make Fedora 39's glib new enough.""" + hint = "\n".join(PREFLIGHT.install_hint({"ID": "fedora", "VERSION_ID": "39"})) + assert "./install.sh --dev" in hint + assert "sudo dnf install" not in hint + + +def test_unknown_distro_gets_a_generic_hint() -> None: + """No false package-manager command when the family is unknown.""" + hint = "\n".join(PREFLIGHT.install_hint({"ID": "plan9", "ID_LIKE": ""})) + assert "development packages" in hint + assert "sudo " not in hint + + +def test_missing_pkg_config_is_reported_without_probing_modules( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The tool itself missing is the first actionable failure.""" + monkeypatch.setattr( + PREFLIGHT.shutil, + "which", + lambda name: "/usr/bin/cc" if name == "cc" else None, + ) + + def fail_run(*args: Any, **kwargs: Any) -> Any: + raise AssertionError("pkg-config cannot be queried when it is absent") + + monkeypatch.setattr(PREFLIGHT.subprocess, "run", fail_run) + + assert PREFLIGHT.missing_build_dependencies() == ["pkg-config"] + + +def test_missing_modules_keep_their_explanations(monkeypatch: pytest.MonkeyPatch) -> None: + """Every absent pkg-config module is reported with the package it comes from.""" + monkeypatch.setattr(PREFLIGHT.shutil, "which", lambda name: f"/usr/bin/{name}") + + class Result: + def __init__(self, returncode: int) -> None: + self.returncode = returncode + + def fake_run(args: list[str], **kwargs: Any) -> Result: + return Result(0 if args[-1] == "portaudio-2.0" else 1) + + monkeypatch.setattr(PREFLIGHT.subprocess, "run", fake_run) + + missing = PREFLIGHT.missing_build_dependencies() + assert any("cairo headers" in item for item in missing) + assert any("libffi" in item for item in missing) + assert not any("portaudio" in item.lower() for item in missing) + + +def test_girepository_requires_glib_280(monkeypatch: pytest.MonkeyPatch) -> None: + """A too-old girepository-2.0.pc must not pass the preflight.""" + monkeypatch.setattr(PREFLIGHT.shutil, "which", lambda name: f"/usr/bin/{name}") + checked: list[list[str]] = [] + + class Result: + def __init__(self, returncode: int) -> None: + self.returncode = returncode + + def fake_run(args: list[str], **kwargs: Any) -> Result: + checked.append(args) + return Result(1 if args[-1] == "girepository-2.0" else 0) + + monkeypatch.setattr(PREFLIGHT.subprocess, "run", fake_run) + + assert PREFLIGHT.missing_build_dependencies() == ["GObject Introspection headers (glib 2.80+)"] + assert ["pkg-config", "--atleast-version=2.80", "girepository-2.0"] in checked + + +def test_fresh_macos_environment_does_not_need_gi_headers( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + """PyAudio's build prerequisites suffice when the lock excludes PyGObject.""" + monkeypatch.setattr(PREFLIGHT.sys, "platform", "darwin") + monkeypatch.setattr(PREFLIGHT, "VENV_PYTHON", tmp_path / "absent-python") + monkeypatch.setattr(PREFLIGHT.shutil, "which", lambda name: f"/usr/bin/{name}") + checked: list[str] = [] + + def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: + checked.append(args[-1]) + return subprocess.CompletedProcess(args, 0 if args[-1] == "portaudio-2.0" else 1) + + monkeypatch.setattr(PREFLIGHT.subprocess, "run", fake_run) + + assert PREFLIGHT.main() == 0 + assert checked == ["portaudio-2.0"] + + +@pytest.mark.parametrize( + "platform,installed,expected", + [ + ("darwin", {"pyaudio"}, True), + ("darwin", set(), False), + ("linux", {"pyaudio"}, False), + ("linux", {"pyaudio", "pygobject"}, False), + ("linux", {"pyaudio", "pycairo"}, False), + ("linux", {"pyaudio", "pycairo", "pygobject"}, True), + ], +) +def test_venv_probe_matches_platform_dependencies( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + platform: str, + installed: set[str], + expected: bool, +) -> None: + """Run the metadata probe against complete and partial per-platform environments.""" + monkeypatch.setattr(PREFLIGHT.sys, "platform", platform) + python = tmp_path / "python" + python.touch() + monkeypatch.setattr(PREFLIGHT, "VENV_PYTHON", python) + + def version(package: str) -> str: + if package not in installed: + raise importlib.metadata.PackageNotFoundError(package) + return "1.0" + + def fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: + assert args[:2] == [str(python), "-c"] + try: + exec(args[2], {}) + except importlib.metadata.PackageNotFoundError: + return subprocess.CompletedProcess(args, 1) + return subprocess.CompletedProcess(args, 0) + + monkeypatch.setattr(importlib.metadata, "version", version) + monkeypatch.setattr(PREFLIGHT.subprocess, "run", fake_run) + + assert PREFLIGHT.venv_has_compiled_packages() is expected + + +def test_macos_missing_portaudio_has_no_gi_install_hint( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + """Failure guidance should request only packages macOS tooling actually needs.""" + monkeypatch.setattr(PREFLIGHT.sys, "platform", "darwin") + monkeypatch.setattr(PREFLIGHT, "VENV_PYTHON", tmp_path / "absent-python") + monkeypatch.setattr(PREFLIGHT.shutil, "which", lambda name: f"/usr/bin/{name}") + monkeypatch.setattr( + PREFLIGHT.subprocess, "run", lambda args, **kwargs: subprocess.CompletedProcess(args, 1) + ) + + assert PREFLIGHT.main() == 1 + error = capsys.readouterr().err + assert "PortAudio headers" in error + assert "brew install portaudio pkg-config" in error + assert "xcode-select --install" in error + assert "PyGObject" not in error + assert "cairo" not in error + assert "libffi" not in error + + +def test_existing_environment_short_circuits_the_check(monkeypatch: pytest.MonkeyPatch) -> None: + """An environment that already holds its native packages needs no header check.""" + monkeypatch.setattr(PREFLIGHT, "venv_has_compiled_packages", lambda: True) + monkeypatch.setattr( + PREFLIGHT, + "missing_build_dependencies", + lambda: pytest.fail("the preflight should not inspect headers"), + ) + + assert PREFLIGHT.main() == 0 + + +def test_missing_dependencies_fail_with_install_commands( + monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """The default failure is the actionable report, not a compiler traceback.""" + monkeypatch.setattr(PREFLIGHT, "venv_has_compiled_packages", lambda: False) + monkeypatch.setattr( + PREFLIGHT, "missing_build_dependencies", lambda: ["PortAudio headers (pyaudio)"] + ) + monkeypatch.setattr(PREFLIGHT, "read_os_release", lambda: {"ID": "fedora"}) + + assert PREFLIGHT.main() == 1 + + error = capsys.readouterr().err + assert "missing native build dependencies" in error + assert "PortAudio headers (pyaudio)" in error + assert "sudo dnf install" in error + assert "CONTRIBUTING.md" in error diff --git a/tests/test_dependency_exports.py b/tests/test_dependency_exports.py index 5e8c1fa..42b16cf 100644 --- a/tests/test_dependency_exports.py +++ b/tests/test_dependency_exports.py @@ -39,6 +39,27 @@ def _pyproject() -> dict[str, Any]: return tomllib.load(handle) +def test_ci_uv_pin_is_supported_and_consistent() -> None: + """All CI jobs must use one supported uv to reproduce the committed exports.""" + required = _pyproject()["tool"]["uv"]["required-version"] + bounds = re.fullmatch(r">=(\d+\.\d+),<(\d+\.\d+)", required) + assert bounds, f"unexpected uv required-version: {required!r}" + + workflow = WORKFLOW.read_text(encoding="utf-8") + steps = re.findall(r"- name: Set up uv\n(.*?)(?=\n\s+- name:|\Z)", workflow, re.DOTALL) + assert steps and len(steps) == workflow.count("uses: astral-sh/setup-uv@") + pins = [] + for step in steps: + pin = re.search(r'^\s+version: "(\d+\.\d+\.\d+)"$', step, re.MULTILINE) + assert pin, "setup-uv in CI must explicitly pin the export generation version" + pins.append(pin.group(1)) + assert len(set(pins)) == 1, "uv versions in CI disagree; exports are byte-for-byte checked" + + minimum, maximum = (tuple(map(int, bound.split("."))) for bound in bounds.groups()) + minor = tuple(map(int, pins[0].split(".")[:2])) + assert minimum <= minor < maximum + + def _requirement_names(specs: list[str]) -> list[str]: """Distribution names out of PEP 508 specifiers, lowercased.""" return [re.split(r"[<>=!~\[; ]", spec, maxsplit=1)[0].strip().lower() for spec in specs]