fix(packaging): derive the Flatpak and AUR dependency sets from the lock (#819)
* fix(packaging): derive the Flatpak and AUR dependency sets from the lock * test(packaging): add return annotations on new packaging tests Greptile asked for -> None on the new tests, matching AGENTS.md. * test(packaging): type parametrized Flatpak source arguments Greptile asked for type hints on the source parameter, matching AGENTS.md. --------- Co-authored-by: jatinkrmalik <jatinkrmalik@gmail.com>
This commit is contained in:
Sergey
@@ -7,6 +7,10 @@ on:
|
||||
- 'packaging/flatpak/**'
|
||||
- 'src/**'
|
||||
- 'pyproject.toml'
|
||||
# The dependency manifest is generated from the hash-pinned export, so a
|
||||
# lock refresh moves what this build ships. Without this, `just lock`
|
||||
# could change every version in it and start no Flatpak job.
|
||||
- 'requirements/**'
|
||||
- '.github/workflows/flatpak.yml'
|
||||
pull_request:
|
||||
branches: [ main, master ]
|
||||
@@ -14,6 +18,10 @@ on:
|
||||
- 'packaging/flatpak/**'
|
||||
- 'src/**'
|
||||
- 'pyproject.toml'
|
||||
# The dependency manifest is generated from the hash-pinned export, so a
|
||||
# lock refresh moves what this build ships. Without this, `just lock`
|
||||
# could change every version in it and start no Flatpak job.
|
||||
- 'requirements/**'
|
||||
- '.github/workflows/flatpak.yml'
|
||||
workflow_dispatch:
|
||||
|
||||
|
||||
@@ -80,8 +80,10 @@ just test # pytest -v
|
||||
just test-cov # pytest --cov=src --cov-report=html
|
||||
just deps # sync .venv with dev+vad extras and the lint group
|
||||
just deps-all # also whisper/vosk/docs; later recipes use --no-sync so they keep it
|
||||
just lock # regenerate uv.lock + requirements/*.txt
|
||||
just lock # regenerate uv.lock, requirements/*.txt and the Flatpak dep manifest
|
||||
just lock-check # fail if uv.lock is stale vs pyproject.toml
|
||||
just flatpak-deps # regenerate the Flatpak dep manifest from requirements/runtime.txt (needs PyPI)
|
||||
just flatpak-deps-check # fail if that manifest is behind the export
|
||||
just model-checksums # refresh pinned model digests after adding a model
|
||||
just appimage # build the AppImage in its pinned base image (needs docker)
|
||||
just appimage-boot fedora:42 # boot that AppImage in a distro container
|
||||
@@ -106,7 +108,7 @@ Website: `web/AGENTS.md`, `web/PRODUCT.md`, `web/DESIGN.md`. Do not duplicate si
|
||||
|
||||
## Dependencies (uv)
|
||||
|
||||
`uv.lock` is authoritative. `just lock` regenerates it and the hash-pinned `requirements/*.txt` exports. The AppImage build installs from `runtime`/`vad` plus `appimage.txt`; `install.sh` still resolves at install time (phase 2 of #701). **Do not edit `requirements/*.txt` by hand.** Change `pyproject.toml` (or `requirements/whisper.in` for the Whisper engine, `requirements/appimage*.in` for the AppImage), run `just lock`, and commit the lock plus the exports with the manifest change.
|
||||
`uv.lock` is authoritative. `just lock` regenerates it, the hash-pinned `requirements/*.txt` exports, and `packaging/flatpak/python3-dependencies.yaml`. The AppImage build installs from `runtime`/`vad` plus `appimage.txt`; `install.sh` still resolves at install time (phase 2 of #701). **Do not edit `requirements/*.txt` by hand**, nor the url/sha256 pairs in `packaging/flatpak/python3-dependencies.yaml`. Change `pyproject.toml` (or `requirements/whisper.in` for the Whisper engine, `requirements/appimage*.in` for the AppImage), run `just lock`, and commit the lock plus the exports with the manifest change.
|
||||
|
||||
| Constraint | Rule |
|
||||
|---|---|
|
||||
@@ -118,6 +120,7 @@ Website: `web/AGENTS.md`, `web/PRODUCT.md`, `web/DESIGN.md`. Do not duplicate si
|
||||
| AppImage PyGObject | Pinned separately in `requirements/appimage.in`, and below 3.52: the AppImage bundles its own interpreter and builds PyGObject against the base image's girepository-1.0, while uv.lock's 3.56 needs girepository-2.0 (glib 2.80+) |
|
||||
| AppImage build inputs | Base image, tooling, interpreter, shaderc and the Vulkan headers are pinned in `packaging/appimage/tool_checksums.txt`. Build with `just appimage` (docker) — building on the host ships the host's glibc, which is what kept the AppImage off Debian 12 |
|
||||
| AppImage boot matrix | `packaging/appimage/boot-test.sh` runs the finished AppImage in distro containers, and the matrix in `unified-pipeline.yml` must keep distros both older and newer than the build image: the old ones prove the glibc floor, the new ones catch a bundle that breaks the host binaries it spawns. A distro added there needs its package recipe in the script — `tests/test_appimage_packaging.py` checks both |
|
||||
| Flatpak deps | `packaging/flatpak/python3-dependencies.yaml` is generated from `requirements/runtime.txt` by `scripts/sync_flatpak_deps.py` (`just flatpak-deps`, which `just lock` runs itself). It resolves each artifact by the digest uv already recorded, so the Flatpak downloads the bytes in `uv.lock`. The project installs there with `pip3 install --no-deps`, so nothing inside the build evaluates `pyproject.toml` — `tests/test_flatpak_packaging.py` checks the invariant offline |
|
||||
| Speech models | Verified against digests pinned in `src/vocalinux/utils/model_checksums.txt` before install, by both `install.sh` and the runtime downloaders. **Fails closed** — an unpinned model is refused. Regenerate with `just model-checksums` (never by hand); `tests/test_model_checksums.py` fails if it falls behind. whisper.cpp URLs use a pinned Hugging Face commit, never `main` |
|
||||
|
||||
Optional extras: `vosk`, `whisper`, `vad`, `dev`.
|
||||
|
||||
@@ -168,11 +168,26 @@ lock:
|
||||
-o requirements/appimage.txt
|
||||
uv pip compile requirements/appimage-tools.in --universal --no-deps --generate-hashes \
|
||||
-o requirements/appimage-tools.txt
|
||||
# The Flatpak is a fourth copy of the dependency set, and it drifted ten
|
||||
# packages behind before anything compared it to source. Regenerate it here
|
||||
# so a lock refresh cannot leave it behind again.
|
||||
just flatpak-deps
|
||||
|
||||
# Fail if uv.lock is stale relative to pyproject.toml
|
||||
lock-check:
|
||||
uv lock --check
|
||||
|
||||
# The export pins names, versions and digests; this looks up the URL that serves
|
||||
# those bytes, so it needs PyPI. tests/test_flatpak_packaging.py checks the same
|
||||
# invariant offline, which is what CI gates on.
|
||||
# Point the Flatpak's dependency manifest at what uv.lock resolved.
|
||||
flatpak-deps: _tooling
|
||||
uv run --no-sync python scripts/sync_flatpak_deps.py
|
||||
|
||||
# Fail if the Flatpak manifest is behind requirements/runtime.txt.
|
||||
flatpak-deps-check: _tooling
|
||||
uv run --no-sync python scripts/sync_flatpak_deps.py --check
|
||||
|
||||
# Refresh the pinned model digests. whisper.cpp digests come from Hugging Face
|
||||
# `lfs` metadata and cost no bandwidth; VOSK is pinned by the bytes we fetch, so
|
||||
# every zip not already in the manifest is downloaded. A first run, or any run
|
||||
|
||||
@@ -174,7 +174,7 @@ pacman -U --noconfirm "$BUILD"/*.pkg.tar.* >/dev/null
|
||||
|
||||
# The console script proves the wheel's entry points landed; the imports walk
|
||||
# every depends entry the package declares (gi/Gtk/AppIndicator, IBus, cairo,
|
||||
# pyaudio, numpy, requests, tqdm, psutil, lxml, pydub, evdev, xlib, pynput).
|
||||
# pyaudio, numpy, requests, tqdm, psutil, evdev, xlib, pynput).
|
||||
vocalinux --version
|
||||
python - <<'PY' || fail "the installed package does not import"
|
||||
import vocalinux.main
|
||||
|
||||
@@ -26,8 +26,6 @@ depends=(
|
||||
'python-requests'
|
||||
'python-tqdm'
|
||||
'python-psutil'
|
||||
'python-lxml'
|
||||
'python-pydub'
|
||||
'python-evdev'
|
||||
'python-xlib'
|
||||
'hicolor-icon-theme'
|
||||
@@ -45,6 +43,12 @@ optdepends=(
|
||||
'python-pywhispercpp-cuda: NVIDIA GPU whisper.cpp backend'
|
||||
'python-pywhispercpp-rocm: AMD GPU whisper.cpp backend'
|
||||
'python-onnxruntime: Silero VAD'
|
||||
# One line per selectable engine. Without it the engine is offered in
|
||||
# Settings and fails at import, which is how vosk and whisper shipped too.
|
||||
'python-vosk: Vosk engine'
|
||||
'python-openai-whisper: OpenAI Whisper engine'
|
||||
'python-sherpa-onnx: Parakeet TDT 0.6B engine'
|
||||
'python-faster-whisper: faster-whisper engine'
|
||||
'xdotool: X11 injection fallback'
|
||||
'xclip: X11 clipboard tools (copy/paste injection fallbacks)'
|
||||
'wl-clipboard: Wayland clipboard (wl-copy/wl-paste) injection fallbacks'
|
||||
|
||||
@@ -74,19 +74,38 @@ flatpak-builder --run build-dir packaging/flatpak/com.vocalinux.Vocalinux.yml \
|
||||
|
||||
## Python Dependencies
|
||||
|
||||
`python3-dependencies.yaml` is generated with `flatpak-pip-generator` (offline
|
||||
builds). Regenerate when `pyproject.toml` deps change:
|
||||
`python3-dependencies.yaml` is generated from `requirements/runtime.txt`, the
|
||||
hash-pinned export `just lock` writes out of `uv.lock`. Refresh it whenever the
|
||||
dependency set changes:
|
||||
|
||||
```bash
|
||||
pipx run flatpak-pip-generator \
|
||||
--runtime org.gnome.Sdk//50 \
|
||||
--yaml \
|
||||
--ignore-pkg 'vosk>=0.3.45' "PyGObject; sys_platform == 'linux'" \
|
||||
--output packaging/flatpak/python3-dependencies \
|
||||
pywhispercpp pydub pynput evdev requests tqdm numpy pyaudio python-xlib psutil lxml \
|
||||
meson-python pyproject-metadata
|
||||
just flatpak-deps # rewrite the manifest (needs PyPI)
|
||||
just flatpak-deps-check # report drift, change nothing
|
||||
```
|
||||
|
||||
`just lock` runs the first of those itself, so a normal dependency change needs
|
||||
no extra step.
|
||||
|
||||
The script resolves each package's URL by looking up the digest uv already
|
||||
recorded, so the bytes this build downloads are the bytes in `uv.lock`, not
|
||||
merely the same version number. It takes a universal wheel where one exists and
|
||||
the sdist otherwise; an ABI-tagged wheel would pin the build to one SDK Python.
|
||||
|
||||
This replaced a `flatpak-pip-generator` invocation whose package list was typed
|
||||
out on the command line. Nothing regenerated it and no test compared it to
|
||||
anything, so it drifted: by 2026-09-10 ten of its fifteen shared packages were
|
||||
behind the lock, `pydub` and `lxml` were still built four months after #705
|
||||
deleted them for having no imports, and pywhispercpp sat at 1.4.1 under the
|
||||
project's own `>=1.5.0`. The project is installed here with
|
||||
`pip3 install --no-deps`, so nothing inside the build could notice.
|
||||
`tests/test_flatpak_packaging.py` is what notices now, and it runs offline.
|
||||
|
||||
Two things in that file are hand-written and the script preserves them, because
|
||||
no generator produces them: pywhispercpp's Vulkan build, which injects a version
|
||||
into `setup.py` (the sdist ships no `version.txt`, `setup()` takes no `version=`
|
||||
argument, and setuptools-scm has no git tree to read in the sandbox), and the
|
||||
symlinking of whisper.cpp's shared libraries onto the loader path.
|
||||
|
||||
`python3-build-dependencies.yaml` is a small hand-maintained helper so
|
||||
`--no-build-isolation` builds can import `mesonpy` before NumPy is built.
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ command: vocalinux
|
||||
build-options:
|
||||
env:
|
||||
NO_REPAIR: "1"
|
||||
PYWHISPERCPP_VERSION: "1.4.1"
|
||||
PYWHISPERCPP_VERSION: "1.5.0"
|
||||
|
||||
finish-args:
|
||||
# Download speech-recognition models on first run.
|
||||
|
||||
@@ -1,4 +1,12 @@
|
||||
# Generated with flatpak-pip-generator --runtime org.gnome.Sdk//50 --yaml --ignore-pkg vosk>=0.3.45 PyGObject; sys_platform == 'linux' --output /tmp/vocalinux-flatpak-gen3/python3-dependencies pywhispercpp pydub pynput evdev requests tqdm numpy pyaudio python-xlib psutil lxml meson-python pyproject-metadata
|
||||
# Generated by scripts/sync_flatpak_deps.py from requirements/runtime.txt, the
|
||||
# hash-pinned export `just lock` writes out of uv.lock. Run `just flatpak-deps`
|
||||
# after any dependency change; `just flatpak-deps-check` and
|
||||
# tests/test_flatpak_packaging.py fail when this file falls behind.
|
||||
#
|
||||
# Do not hand-edit the url/sha256 pairs. Every other line is hand-written and
|
||||
# the script preserves it: pywhispercpp needs a Vulkan build, a version
|
||||
# injected into setup.py, and whisper.cpp's shared libraries symlinked onto
|
||||
# the loader path, none of which a generator produces.
|
||||
name: python3-dependencies
|
||||
buildsystem: simple
|
||||
build-commands: []
|
||||
@@ -10,10 +18,15 @@ modules:
|
||||
# the offline Flatpak build, so inject the version explicitly. --no-same-owner
|
||||
# stops GNU tar from trying to restore the archive's uid inside the build
|
||||
# sandbox (where chown to an arbitrary uid is not permitted).
|
||||
- tar --no-same-owner -xzf pywhispercpp-1.4.1.tar.gz
|
||||
- python3 -c "from pathlib import Path; p = Path('pywhispercpp-1.4.1/setup.py'); s = p.read_text(); p.write_text(s.replace('setup(\\n name=\"pywhispercpp\",', 'setup(\\n name=\"pywhispercpp\",\\n version=\"1.4.1\",'))"
|
||||
- tar --owner=0 --group=0 -czf pywhispercpp-1.4.1.tar.gz pywhispercpp-1.4.1
|
||||
- PYWHISPERCPP_VERSION=1.4.1 NO_REPAIR=1 GGML_VULKAN=1 pip3 install --verbose --exists-action=i --no-index --find-links="file://${PWD}"
|
||||
- tar --no-same-owner -xzf pywhispercpp-1.5.0.tar.gz
|
||||
# The build runs under --no-build-isolation, where setuptools-scm is absent,
|
||||
# so a replace that matched nothing would not fail -- it would produce a wheel
|
||||
# carrying version 0.0.0. recognition_manager.py reads that metadata and
|
||||
# degrades GPU context_params below the floor in pyproject.toml, silently.
|
||||
# Assert the shape instead.
|
||||
- python3 -c "from pathlib import Path; p = Path('pywhispercpp-1.5.0/setup.py'); s = p.read_text(); n = 'setup(\\n name=\"pywhispercpp\",'; assert n in s, 'pywhispercpp setup.py no longer matches the injection point'; p.write_text(s.replace(n, n + '\\n version=\"1.5.0\",'))"
|
||||
- tar --owner=0 --group=0 -czf pywhispercpp-1.5.0.tar.gz pywhispercpp-1.5.0
|
||||
- PYWHISPERCPP_VERSION=1.5.0 NO_REPAIR=1 GGML_VULKAN=1 pip3 install --verbose --exists-action=i --no-index --find-links="file://${PWD}"
|
||||
--prefix=${FLATPAK_DEST} "pywhispercpp" --no-build-isolation
|
||||
# whisper.cpp installs its shared libraries next to the extension module.
|
||||
# Symlink them onto the default loader path so the extension resolves them
|
||||
@@ -29,47 +42,38 @@ modules:
|
||||
sources:
|
||||
- &id002
|
||||
type: file
|
||||
url: https://files.pythonhosted.org/packages/22/30/7cd8fdcdfbc5b869528b079bfb76dcdf6056b1a2097a662e5e8c04f42965/certifi-2026.4.22-py3-none-any.whl
|
||||
sha256: 3cb2210c8f88ba2318d29b0388d1023c8492ff72ecdde4ebdaddbb13a31b1c4a
|
||||
url: https://files.pythonhosted.org/packages/0b/a7/71ac2cff56fec219ed242bb11b8efb69fcc4bec75db06fb7bfe35de520e6/certifi-2026.7.22-py3-none-any.whl
|
||||
sha256: 62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775
|
||||
- &id003
|
||||
type: file
|
||||
url: https://files.pythonhosted.org/packages/db/8f/61959034484a4a7c527811f4721e75d02d653a35afb0b6054474d8185d4c/charset_normalizer-3.4.7-py3-none-any.whl
|
||||
sha256: 3dce51d0f5e7951f8bb4900c257dad282f49190fdbebecd4ba99bcc41fef404d
|
||||
url: https://files.pythonhosted.org/packages/cc/61/d01fc49b8dea277640b55a9e15960dbca9fdc8c9fde18e572d39c59f4019/charset_normalizer-3.5.1-py3-none-any.whl
|
||||
sha256: 6df0ec430f9a831772c23ca5a224cba36517a58a84bb32c32bb59a9fa67c47f6
|
||||
- &id004
|
||||
type: file
|
||||
url: https://files.pythonhosted.org/packages/5d/13/ad7d7ca3808a898b4612b6fe93cde56b53f3034dcde235acb1f0e1df24c6/idna-3.13-py3-none-any.whl
|
||||
sha256: 892ea0cde124a99ce773decba204c5552b69c3c67ffd5f232eb7696135bc8bb3
|
||||
url: https://files.pythonhosted.org/packages/57/b0/0e52c878c53f245edd3a11020f20979b3f490f245af532c7cae3027754b5/idna-3.19-py3-none-any.whl
|
||||
sha256: 815e7be7a7806d54abb586dc943addc79e8b2ee16915059658cbeff4b1b43bf4
|
||||
- &id008
|
||||
type: file
|
||||
url: https://files.pythonhosted.org/packages/d7/9f/b8cef5bffa569759033adda9481211426f12f53299629b410340795c2514/numpy-2.4.4.tar.gz
|
||||
sha256: 2d390634c5182175533585cc89f3608a4682ccb173cc9bb940b2881c8d6f8fa0
|
||||
url: https://files.pythonhosted.org/packages/9a/80/db0b4559e57ec36362bedbb05530a87fafbcb6067708c946967a41d449e7/numpy-2.5.2.tar.gz
|
||||
sha256: d482d171c406ae88c5b19cad3b6a1c4c5209f886ab74bc44c2c865c23f52d860
|
||||
- type: file
|
||||
url: https://files.pythonhosted.org/packages/75/a6/a0a304dc33b49145b21f4808d763822111e67d1c3a32b524a1baf947b6e1/platformdirs-4.9.6-py3-none-any.whl
|
||||
sha256: e61adb1d5e5cb3441b4b7710bea7e4c12250ca49439228cc1021c00dcfac0917
|
||||
url: https://files.pythonhosted.org/packages/19/a9/c34aebedd3a4c9afe5101b1b8713710b3fec18087c8a36c35d2f909861bd/platformdirs-4.11.3-py3-none-any.whl
|
||||
sha256: 5ed065d443751de711da036041a7a214122efc4a4de393b3f4137ba5576540e7
|
||||
- type: file
|
||||
url: https://files.pythonhosted.org/packages/f6/7d/fe4b2b190d6cdcf0d5e4ca2b946d73bc449a586606a24d25ff50ffec5a5c/pywhispercpp-1.4.1.tar.gz
|
||||
sha256: 520ce9275bb6fc81e0daa2e08144a80ee006b117a18058a77860c5b1c9c122f4
|
||||
url: https://files.pythonhosted.org/packages/1c/43/8778f0685a4d32d74efdfa78fc5f745ca9998d96f8121dbf7793db2da5d3/pywhispercpp-1.5.0.tar.gz
|
||||
sha256: 5aa54c73ec96617d97aa519dbbf817e44255eddd9bd52ee1243be61028dbd018
|
||||
- &id005
|
||||
type: file
|
||||
url: https://files.pythonhosted.org/packages/d7/8e/7540e8a2036f79a125c1d2ebadf69ed7901608859186c856fa0388ef4197/requests-2.33.1-py3-none-any.whl
|
||||
sha256: 4e6d1ef462f3626a1f0a0a9c42dd93c63bad33f9f1c1937509b8c5c8718ab56a
|
||||
url: https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl
|
||||
sha256: 2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0
|
||||
- &id007
|
||||
type: file
|
||||
url: https://files.pythonhosted.org/packages/16/e1/3079a9ff9b8e11b846c6ac5c8b5bfb7ff225eee721825310c91b3b50304f/tqdm-4.67.3-py3-none-any.whl
|
||||
sha256: ee1e4c0e59148062281c49d80b25b67771a127c85fc9676d3be5f243206826bf
|
||||
url: https://files.pythonhosted.org/packages/f9/1c/01bfd571a64e7f270e6bab5e33777debe0edc56759233ce84f27dec92d14/tqdm-4.70.0-py3-none-any.whl
|
||||
sha256: 7f585706bfddbdebf89daac705b2dfcc16890130727d3197ca62c732b4310953
|
||||
- &id006
|
||||
type: file
|
||||
url: https://files.pythonhosted.org/packages/39/08/aaaad47bc4e9dc8c725e68f9d04865dbcb2052843ff09c97b08904852d84/urllib3-2.6.3-py3-none-any.whl
|
||||
sha256: bf272323e553dfb2e87d9bfd225ca7b0f467b919d7bbd355436d3fd37cb0acd4
|
||||
- name: python3-pydub
|
||||
buildsystem: simple
|
||||
build-commands:
|
||||
- pip3 install --verbose --exists-action=i --no-index --find-links="file://${PWD}"
|
||||
--prefix=${FLATPAK_DEST} "pydub" --no-build-isolation
|
||||
sources:
|
||||
- type: file
|
||||
url: https://files.pythonhosted.org/packages/a6/53/d78dc063216e62fc55f6b2eebb447f6a4b0a59f55c8406376f76bf959b08/pydub-0.25.1-py2.py3-none-any.whl
|
||||
sha256: 65617e33033874b59d87db603aa1ed450633288aefead953b30bded59cb599a6
|
||||
url: https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl
|
||||
sha256: 9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897
|
||||
- name: python3-pynput
|
||||
buildsystem: simple
|
||||
build-commands:
|
||||
@@ -81,8 +85,8 @@ modules:
|
||||
url: https://files.pythonhosted.org/packages/a5/f5/397b61091120a9ca5001041dd7bf76c385b3bfd67a0e5bcb74b852bd22a4/evdev-1.9.3.tar.gz
|
||||
sha256: 2c140e01ac8437758fa23fe5c871397412461f42d421aa20241dc8fe8cfccbc9
|
||||
- type: file
|
||||
url: https://files.pythonhosted.org/packages/59/4f/ac3fa906ae8a375a536b12794128c5efacade9eaa917a35dfd27ce0c7400/pynput-1.8.1-py2.py3-none-any.whl
|
||||
sha256: 42dfcf27404459ca16ca889c8fb8ffe42a9fe54f722fd1a3e130728e59e768d2
|
||||
url: https://files.pythonhosted.org/packages/6d/98/bbeb760852adb27f166ce1617f0e51aabb15f21b1e60ea703f2aed3c78ac/pynput-1.8.2-py2.py3-none-any.whl
|
||||
sha256: 8cc38cf13a6ab2749cb375678be8a0fd705d7ce49c8001ff5db4007a723bbef1
|
||||
- &id009
|
||||
type: file
|
||||
url: https://files.pythonhosted.org/packages/fc/b8/ff33610932e0ee81ae7f1269c890f697d56ff74b9f5b2ee5d9b7fa2c5355/python_xlib-0.33-py2.py3-none-any.whl
|
||||
@@ -149,15 +153,6 @@ modules:
|
||||
- type: file
|
||||
url: https://files.pythonhosted.org/packages/aa/c6/d1ddf4abb55e93cebc4f2ed8b5d6dbad109ecb8d63748dd2b20ab5e57ebe/psutil-7.2.2.tar.gz
|
||||
sha256: 0746f5f8d406af344fd547f1c8daa5f5c33dbc293bb8d6a16d80b4bb88f59372
|
||||
- name: python3-lxml
|
||||
buildsystem: simple
|
||||
build-commands:
|
||||
- pip3 install --verbose --exists-action=i --no-index --find-links="file://${PWD}"
|
||||
--prefix=${FLATPAK_DEST} "lxml" --no-build-isolation
|
||||
sources:
|
||||
- type: file
|
||||
url: https://files.pythonhosted.org/packages/28/30/9abc9e34c657c33834eaf6cd02124c61bdf5944d802aa48e69be8da3585d/lxml-6.1.0.tar.gz
|
||||
sha256: bfd57d8008c4965709a919c3e9a98f76c2c7cb319086b3d26858250620023b13
|
||||
- name: python3-meson-python
|
||||
buildsystem: simple
|
||||
build-commands:
|
||||
|
||||
Executable
+294
@@ -0,0 +1,294 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Point the Flatpak's dependency manifest at the versions uv.lock resolved.
|
||||
|
||||
`packaging/flatpak/python3-dependencies.yaml` was produced once by
|
||||
flatpak-pip-generator, from a package list typed out by hand on the command
|
||||
line recorded in its first line, and then hand-edited. Nothing regenerated it
|
||||
afterwards and no test compared it to anything, so it drifted: by 2026-09-10 ten
|
||||
of its fifteen shared packages were behind `requirements/runtime.txt`, including
|
||||
pywhispercpp, which sat at 1.4.1 under the project's own `>=1.5.0` floor. The
|
||||
Flatpak installs the project with `pip3 install --no-deps`, so that constraint
|
||||
is never evaluated and nothing there could notice.
|
||||
|
||||
This script closes the loop the other way round: it reads the hash-pinned export
|
||||
that `just lock` produces and rewrites every source in the manifest to the
|
||||
artifact whose digest uv already recorded. The bytes the Flatpak downloads are
|
||||
then the bytes in `uv.lock`, verified by hash rather than by convention.
|
||||
|
||||
It edits in place rather than regenerating, because two things in that file are
|
||||
hand-written and no generator produces them: pywhispercpp's Vulkan build with
|
||||
its version injection, and the symlinking of whisper.cpp's shared libraries onto
|
||||
the loader path. Keeping the structure keeps the diff to versions and digests.
|
||||
|
||||
Usage:
|
||||
python scripts/sync_flatpak_deps.py # rewrite the manifest
|
||||
python scripts/sync_flatpak_deps.py --check # report drift, change nothing
|
||||
|
||||
`tests/test_flatpak_packaging.py` asserts the same invariant offline, so CI
|
||||
catches drift without reaching PyPI.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
from packaging.markers import Marker
|
||||
from packaging.utils import canonicalize_name
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
EXPORT = REPO_ROOT / "requirements" / "runtime.txt"
|
||||
DEPS_YAML = REPO_ROOT / "packaging" / "flatpak" / "python3-dependencies.yaml"
|
||||
MANIFEST = REPO_ROOT / "packaging" / "flatpak" / "com.vocalinux.Vocalinux.yml"
|
||||
|
||||
#: The environment the Flatpak build resolves for. `flatpak remote-info flathub
|
||||
#: org.gnome.Sdk//50 --show-metadata` reports a freedesktop 25.08 base, which
|
||||
#: ships Python 3.13. Only numpy reads this today: the lock splits it at
|
||||
#: `python_full_version >= '3.12'`. Everything else is marked on sys_platform
|
||||
#: alone. Bump it when the manifest's runtime-version moves.
|
||||
TARGET_ENV = {
|
||||
"sys_platform": "linux",
|
||||
"platform_system": "Linux",
|
||||
"platform_machine": "x86_64",
|
||||
"os_name": "posix",
|
||||
"python_version": "3.13",
|
||||
"python_full_version": "3.13.0",
|
||||
"implementation_name": "cpython",
|
||||
"platform_python_implementation": "CPython",
|
||||
"extra": "",
|
||||
}
|
||||
|
||||
#: org.gnome.Platform//50 ships these; a second copy under /app would shadow the
|
||||
#: runtime's own GI stack. PyGObject is already absent from the export, which
|
||||
#: `just lock` builds with --no-emit-package pygobject for the same reason.
|
||||
RUNTIME_PROVIDED = {"pycairo"}
|
||||
|
||||
#: Build backends for the sdists the manifest builds offline under
|
||||
#: --no-build-isolation. They are not runtime dependencies, so they never appear
|
||||
#: in requirements/runtime.txt and stay pinned here by hand.
|
||||
BUILD_BACKENDS = {"meson-python", "pyproject-metadata"}
|
||||
|
||||
PYPI_JSON = "https://pypi.org/pypi/{name}/{version}/json"
|
||||
|
||||
|
||||
def parse_export(path: Path) -> dict[str, tuple[str, set[str]]]:
|
||||
"""Map every package the Flatpak target selects to its version and digests.
|
||||
|
||||
uv writes one `name==version [; marker]` line followed by indented
|
||||
`--hash=sha256:` lines, continued with backslashes. A package can appear
|
||||
more than once under mutually exclusive markers, which is how numpy is
|
||||
split across Python versions, so the marker decides rather than the order.
|
||||
"""
|
||||
selected: dict[str, tuple[str, set[str]]] = {}
|
||||
current: str | None = None
|
||||
for raw in path.read_text(encoding="utf-8").splitlines():
|
||||
line = raw.strip().rstrip("\\").strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
head = re.match(r"^([A-Za-z0-9._-]+)==([^ ;]+)\s*(?:;\s*(.*))?$", line)
|
||||
if head:
|
||||
name, version, marker = head.group(1), head.group(2), head.group(3)
|
||||
current = None
|
||||
if marker and not Marker(marker).evaluate(TARGET_ENV):
|
||||
continue
|
||||
key = canonicalize_name(name)
|
||||
if key in selected:
|
||||
raise SystemExit(
|
||||
f"{path.name}: {key} resolves twice for the Flatpak target "
|
||||
f"({selected[key][0]} and {version}); the markers overlap"
|
||||
)
|
||||
selected[key] = (version, set())
|
||||
current = key
|
||||
continue
|
||||
digest = re.match(r"^--hash=sha256:([0-9a-f]{64})$", line)
|
||||
if digest and current:
|
||||
selected[current][1].add(digest.group(1))
|
||||
return selected
|
||||
|
||||
|
||||
def pypi_artifacts(name: str, version: str) -> list[dict]:
|
||||
url = PYPI_JSON.format(name=name, version=version)
|
||||
with urllib.request.urlopen(url, timeout=30) as response: # noqa: S310
|
||||
return json.load(response)["urls"]
|
||||
|
||||
|
||||
def choose_artifact(name: str, version: str, digests: set[str]) -> tuple[str, str]:
|
||||
"""The one file the Flatpak should download, as (url, sha256).
|
||||
|
||||
Prefer a universal wheel, fall back to the sdist. That is the rule the
|
||||
original generator followed and it holds for every package here: the pure
|
||||
Python ones ship `py3-none-any`, and the compiled ones (numpy, evdev,
|
||||
PyAudio, psutil, pywhispercpp) ship no universal wheel, so the sandbox
|
||||
builds them from source against the SDK's own interpreter. Never an
|
||||
ABI-tagged wheel, whose cp tag would pin us to one SDK Python.
|
||||
"""
|
||||
artifacts = pypi_artifacts(name, version)
|
||||
wheels = [
|
||||
a
|
||||
for a in artifacts
|
||||
if a["packagetype"] == "bdist_wheel" and a["filename"].endswith("-none-any.whl")
|
||||
]
|
||||
sdists = [a for a in artifacts if a["packagetype"] == "sdist"]
|
||||
chosen = wheels or sdists
|
||||
if not chosen:
|
||||
raise SystemExit(
|
||||
f"{name} {version} publishes neither a universal wheel nor an sdist; "
|
||||
"the Flatpak cannot build it without an ABI-tagged wheel"
|
||||
)
|
||||
picked = chosen[0]
|
||||
sha = picked["digests"]["sha256"]
|
||||
if sha not in digests:
|
||||
raise SystemExit(
|
||||
f"{name} {version}: PyPI serves {picked['filename']} with digest "
|
||||
f"{sha}, which is not among the hashes uv locked. Re-run `just lock`."
|
||||
)
|
||||
return picked["url"], sha
|
||||
|
||||
|
||||
def artifact_name(filename: str) -> str:
|
||||
if filename.endswith(".whl"):
|
||||
return canonicalize_name(filename.split("-")[0])
|
||||
base = re.sub(r"\.(tar\.gz|zip|tar\.bz2)$", "", filename)
|
||||
return canonicalize_name(base.rpartition("-")[0])
|
||||
|
||||
|
||||
def manifest_sources(text: str) -> list[tuple[int, str, str]]:
|
||||
"""Every PyPI source in the manifest, as (line index, filename, sha256)."""
|
||||
lines = text.splitlines()
|
||||
found = []
|
||||
for index, line in enumerate(lines):
|
||||
url = re.match(r"^\s*url: (https://files\.pythonhosted\.org/\S+/([^/\s]+))$", line)
|
||||
if not url:
|
||||
continue
|
||||
digest = re.match(r"^\s*sha256: ([0-9a-f]{64})\s*$", lines[index + 1])
|
||||
if not digest:
|
||||
raise SystemExit(
|
||||
f"{DEPS_YAML.name}:{index + 2}: a PyPI source with no sha256 beneath it"
|
||||
)
|
||||
found.append((index, url.group(2), digest.group(1)))
|
||||
return found
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
"--check",
|
||||
action="store_true",
|
||||
help="report what would change and exit non-zero, without writing",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
locked = parse_export(EXPORT)
|
||||
text = DEPS_YAML.read_text(encoding="utf-8")
|
||||
lines = text.splitlines(keepends=True)
|
||||
|
||||
in_manifest = {artifact_name(name) for _, name, _ in manifest_sources(text)}
|
||||
stale = sorted(in_manifest - set(locked) - BUILD_BACKENDS)
|
||||
missing = sorted(set(locked) - in_manifest - RUNTIME_PROVIDED)
|
||||
if stale:
|
||||
print(
|
||||
f"{DEPS_YAML.name} builds packages the lock does not resolve: "
|
||||
f"{', '.join(stale)}\n"
|
||||
"Delete their modules and sources by hand: whether a package is a "
|
||||
"build backend the sandbox needs or a leftover this script cannot "
|
||||
"tell.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
if missing:
|
||||
print(
|
||||
f"{DEPS_YAML.name} is missing packages the lock resolves: "
|
||||
f"{', '.join(missing)}\n"
|
||||
"The Flatpak installs with --no-deps, so a missing one fails at "
|
||||
"runtime rather than at build time. Add a module for each.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
if stale or missing:
|
||||
return 1
|
||||
|
||||
changes: list[str] = []
|
||||
for index, filename, old_sha in manifest_sources(text):
|
||||
name = artifact_name(filename)
|
||||
if name in BUILD_BACKENDS:
|
||||
# Absent from the lock by definition: nothing imports them at
|
||||
# runtime, they only let the sandbox build the sdists offline.
|
||||
continue
|
||||
version, digests = locked[name]
|
||||
url, sha = choose_artifact(name, version, digests)
|
||||
if sha == old_sha:
|
||||
continue
|
||||
indent = re.match(r"^(\s*)", lines[index]).group(1)
|
||||
lines[index] = f"{indent}url: {url}\n"
|
||||
lines[index + 1] = f"{indent}sha256: {sha}\n"
|
||||
changes.append(f" {name}: {filename} -> {url.rsplit('/', 1)[-1]}")
|
||||
|
||||
updated = "".join(lines)
|
||||
|
||||
# pywhispercpp's version is spelled out in its build commands: the sdist
|
||||
# carries no version.txt and setup() takes no version= argument, so the
|
||||
# module rewrites setup.py before building. setuptools-scm would otherwise
|
||||
# read a git tree that does not exist in the sandbox.
|
||||
pywhispercpp_version = locked["pywhispercpp"][0]
|
||||
updated, injected = re.subn(
|
||||
# The version appears as a tarball name, a directory name and a
|
||||
# literal inside the setup.py rewrite. Refuse to match rather than
|
||||
# match short: a partial match would leave the tail of the old version
|
||||
# glued to the new one and rename the directory tar just wrote.
|
||||
r"pywhispercpp-\d+(?:\.\d+)*(?![\d.]\d)",
|
||||
f"pywhispercpp-{pywhispercpp_version}",
|
||||
updated,
|
||||
)
|
||||
updated = re.sub(
|
||||
r"PYWHISPERCPP_VERSION=\d+(?:\.\d+)*",
|
||||
f"PYWHISPERCPP_VERSION={pywhispercpp_version}",
|
||||
updated,
|
||||
)
|
||||
updated = re.sub(
|
||||
r'(version=\\")\d+(?:\.\d+)*(\\")',
|
||||
rf"\g<1>{pywhispercpp_version}\g<2>",
|
||||
updated,
|
||||
)
|
||||
if injected and updated != text and not changes:
|
||||
changes.append(f" pywhispercpp: build commands -> {pywhispercpp_version}")
|
||||
|
||||
manifest_text = MANIFEST.read_text(encoding="utf-8")
|
||||
manifest_updated = re.sub(
|
||||
r'(PYWHISPERCPP_VERSION: ")\d+(?:\.\d+)*(")',
|
||||
rf"\g<1>{pywhispercpp_version}\g<2>",
|
||||
manifest_text,
|
||||
)
|
||||
|
||||
if updated == text and manifest_updated == manifest_text:
|
||||
print("Flatpak dependencies already match requirements/runtime.txt")
|
||||
return 0
|
||||
|
||||
if args.check:
|
||||
print("Flatpak dependencies are behind requirements/runtime.txt:", file=sys.stderr)
|
||||
for change in changes:
|
||||
print(change, file=sys.stderr)
|
||||
if manifest_updated != manifest_text:
|
||||
print(
|
||||
f" {MANIFEST.name}: PYWHISPERCPP_VERSION -> {pywhispercpp_version}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
print("Run `just flatpak-deps` to update.", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
DEPS_YAML.write_text(updated, encoding="utf-8")
|
||||
MANIFEST.write_text(manifest_updated, encoding="utf-8")
|
||||
print(f"Updated {DEPS_YAML.relative_to(REPO_ROOT)}:")
|
||||
for change in changes:
|
||||
print(change)
|
||||
if manifest_updated != manifest_text:
|
||||
print(
|
||||
f"Updated {MANIFEST.relative_to(REPO_ROOT)}: "
|
||||
f"PYWHISPERCPP_VERSION={pywhispercpp_version}"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -6,6 +6,7 @@ first place, because no paths filter matched packaging/aur/**.
|
||||
"""
|
||||
|
||||
import re
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
@@ -172,3 +173,149 @@ def test_the_pkgbuild_builds_without_isolation_on_distro_setuptools():
|
||||
changes what the gate proves, not a slip to make in passing."""
|
||||
pkgbuild = PKGBUILD.read_text(encoding="utf-8")
|
||||
assert "--no-isolation" in pkgbuild
|
||||
|
||||
|
||||
#: Arch names in depends=() that are not Python distributions from our lock:
|
||||
#: the GI stack and the C libraries behind it. `python-gobject` and
|
||||
#: `python-cairo` are Python packages, but they come from Arch rather than from
|
||||
#: the export -- `just lock` builds it with --no-emit-package pygobject, and
|
||||
#: PyGObject cannot be pip-installed on a user machine anyway.
|
||||
NOT_FROM_THE_LOCK = {
|
||||
"python",
|
||||
"python-gobject",
|
||||
"python-cairo",
|
||||
"gtk3",
|
||||
"libayatana-appindicator",
|
||||
"ibus",
|
||||
"gobject-introspection",
|
||||
"portaudio",
|
||||
"hicolor-icon-theme",
|
||||
}
|
||||
|
||||
#: Arch name -> PyPI name, where stripping `python-` does not get there.
|
||||
ARCH_TO_PYPI = {
|
||||
"python-pywhispercpp": "pywhispercpp",
|
||||
"python-pyaudio": "pyaudio",
|
||||
# Arch keeps the distribution name, which already starts with `python-`.
|
||||
"python-xlib": "python-xlib",
|
||||
}
|
||||
|
||||
|
||||
def _locked_distributions() -> set:
|
||||
"""PyPI names in the hash-pinned runtime export, canonicalized.
|
||||
|
||||
Read the resolved set, not `pyproject.toml`'s direct one. #705 deleted
|
||||
tqdm and python-xlib from the direct list, but pywhispercpp still needs the
|
||||
first and pynput the second, so both are still installed on every machine
|
||||
and the PKGBUILD is right to name them. Comparing against the direct list
|
||||
would call for deleting two dependencies Arch users need.
|
||||
"""
|
||||
names = set()
|
||||
for line in RUNTIME_EXPORT.read_text(encoding="utf-8").splitlines():
|
||||
match = re.match(r"^([A-Za-z0-9._-]+)==", line)
|
||||
if match:
|
||||
names.add(match.group(1).lower().replace("_", "-"))
|
||||
assert names, "requirements/runtime.txt parsed to nothing"
|
||||
return names
|
||||
|
||||
|
||||
def _direct_dependencies() -> set:
|
||||
"""`[project].dependencies` from pyproject.toml, canonicalized.
|
||||
|
||||
PyGObject is dropped: Arch ships it as python-gobject, which is already in
|
||||
NOT_FROM_THE_LOCK, and it is absent from the export for the same reason.
|
||||
"""
|
||||
with (REPO_ROOT / "pyproject.toml").open("rb") as handle:
|
||||
specs = tomllib.load(handle)["project"]["dependencies"]
|
||||
names = {
|
||||
re.split(r"[<>=!~\[; ]", spec, maxsplit=1)[0].strip().lower().replace("_", "-")
|
||||
for spec in specs
|
||||
}
|
||||
return names - {"pygobject"}
|
||||
|
||||
|
||||
def _pypi_name(arch_name: str) -> str:
|
||||
"""PyPI distribution behind an Arch package name, minus any version bound.
|
||||
|
||||
depends=() entries may carry a constraint (`python>=3.11`); pacman treats
|
||||
the name and the bound separately and so must this.
|
||||
"""
|
||||
bare = re.split(r"[<>=]", arch_name, maxsplit=1)[0]
|
||||
if bare in ARCH_TO_PYPI:
|
||||
return ARCH_TO_PYPI[bare]
|
||||
return bare.removeprefix("python-").lower().replace("_", "-")
|
||||
|
||||
|
||||
def test_depends_declares_no_python_package_nothing_depends_on() -> None:
|
||||
"""#705 deleted pydub, lxml, tqdm and python-xlib from `pyproject.toml`
|
||||
and left all four here, so AUR users installed four packages for nothing.
|
||||
|
||||
Two of them came back through the dependency chain and belong here; pydub
|
||||
and lxml had no path back and stayed for four months. #772's build gate
|
||||
could not see it: `makepkg` verifies that every depends entry *resolves*,
|
||||
never that anything needs it, and the depends test above compares the
|
||||
gate's stub to this list rather than either to source.
|
||||
"""
|
||||
orphans = sorted(
|
||||
name
|
||||
for name in _aur_depends()
|
||||
if re.split(r"[<>=]", name, maxsplit=1)[0] not in NOT_FROM_THE_LOCK
|
||||
and _pypi_name(name) not in _locked_distributions()
|
||||
)
|
||||
assert not orphans, (
|
||||
f"depends=() names packages the lock does not resolve: {', '.join(orphans)}."
|
||||
" Drop them, or add the dependency to pyproject.toml and re-run `just lock`."
|
||||
)
|
||||
|
||||
|
||||
def test_depends_covers_every_direct_dependency() -> None:
|
||||
"""The other direction, and against the direct list rather than the
|
||||
resolved one.
|
||||
|
||||
pacman resolves transitive dependencies itself, so certifi, idna, six and
|
||||
urllib3 have no business in depends=(); naming them would pin us to
|
||||
requests' internals. What must be here is what `src/` imports directly. A
|
||||
dependency added to `pyproject.toml` and not here builds fine, because
|
||||
`python -m build` never reads depends=(), and then fails at import on a
|
||||
user's machine.
|
||||
"""
|
||||
declared = {_pypi_name(name) for name in _aur_depends()}
|
||||
missing = sorted(
|
||||
dist
|
||||
for dist in _direct_dependencies()
|
||||
if dist not in declared and f"python-{dist}" not in NOT_FROM_THE_LOCK
|
||||
)
|
||||
assert (
|
||||
not missing
|
||||
), f"pyproject.toml requires these and depends=() omits them: {', '.join(missing)}"
|
||||
|
||||
|
||||
def test_every_selectable_engine_reaches_optdepends() -> None:
|
||||
"""Each optional extra is an engine a user can pick in Settings. One with
|
||||
no optdepends line is an engine the AUR install cannot run and does not
|
||||
say so. #802 and #543 both landed without one."""
|
||||
with (REPO_ROOT / "pyproject.toml").open("rb") as handle:
|
||||
extras = tomllib.load(handle)["project"]["optional-dependencies"]
|
||||
#: Extras that are tooling, not engines. Everything else a user can pick
|
||||
#: in Settings, and a new engine extra must land here without this test
|
||||
#: being edited: an allowlist would let it slip past silently, which is
|
||||
#: exactly how #802 and #543 shipped.
|
||||
not_engines = {"dev", "docs"}
|
||||
lines = PKGBUILD.read_text(encoding="utf-8").splitlines()
|
||||
start = next(i for i, line in enumerate(lines) if line.startswith("optdepends=("))
|
||||
end = next(i for i in range(start + 1, len(lines)) if lines[i].strip() == ")")
|
||||
optdepends = "\n".join(lines[start + 1 : end])
|
||||
|
||||
missing = []
|
||||
for extra in sorted(set(extras) - not_engines):
|
||||
for spec in extras[extra]:
|
||||
dist = re.split(r"[<>=!~\[; ]", spec, maxsplit=1)[0].strip().lower()
|
||||
if dist in {"torch", "torchaudio"}:
|
||||
continue # pulled in by python-openai-whisper, not named alone
|
||||
arch_name = f"python-{dist.replace('_', '-')}"
|
||||
if arch_name not in optdepends:
|
||||
missing.append(f"{extra} -> {arch_name}")
|
||||
assert not missing, (
|
||||
"these optional engines have no optdepends line, so an AUR user who"
|
||||
f" selects them gets an ImportError: {', '.join(missing)}"
|
||||
)
|
||||
|
||||
@@ -0,0 +1,253 @@
|
||||
"""Tie the Flatpak's dependency set back to the lock that everything else uses.
|
||||
|
||||
`packaging/flatpak/python3-dependencies.yaml` was generated once from a package
|
||||
list typed on a command line, then hand-edited, then never compared to anything.
|
||||
By 2026-09-10 ten of its fifteen shared packages were behind
|
||||
`requirements/runtime.txt` and two more (pydub, lxml) were still built after
|
||||
#705 deleted them for having no imports in `src/`.
|
||||
|
||||
The one that mattered was pywhispercpp 1.4.1, under the project's own
|
||||
`>=1.5.0`. Nothing could catch it: the Flatpak installs the project with
|
||||
`pip3 install --no-deps`, so the constraint in `pyproject.toml` is never
|
||||
evaluated, and `recognition_manager.py` degrades GPU `context_params` below
|
||||
1.5.0, so Flatpak users lost that quietly rather than loudly.
|
||||
|
||||
`packaging/appimage/build.sh` has done this check for the AppImage since #743.
|
||||
These are the Flatpak's, and they run offline: `scripts/sync_flatpak_deps.py`
|
||||
needs PyPI to look a URL up, but everything asserted here is already in the
|
||||
repo.
|
||||
"""
|
||||
|
||||
import re
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from packaging.markers import Marker
|
||||
from packaging.specifiers import SpecifierSet
|
||||
from packaging.utils import canonicalize_name
|
||||
from packaging.version import Version
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
PYPROJECT = REPO_ROOT / "pyproject.toml"
|
||||
EXPORT = REPO_ROOT / "requirements" / "runtime.txt"
|
||||
FLATPAK_DIR = REPO_ROOT / "packaging" / "flatpak"
|
||||
DEPS_YAML = FLATPAK_DIR / "python3-dependencies.yaml"
|
||||
MANIFEST = FLATPAK_DIR / "com.vocalinux.Vocalinux.yml"
|
||||
WORKFLOW = REPO_ROOT / ".github" / "workflows" / "flatpak.yml"
|
||||
|
||||
#: Kept in step with scripts/sync_flatpak_deps.py, which resolves the same
|
||||
#: markers when it picks a version. GNOME 50 is a freedesktop 25.08 base.
|
||||
TARGET_ENV = {
|
||||
"sys_platform": "linux",
|
||||
"platform_system": "Linux",
|
||||
"platform_machine": "x86_64",
|
||||
"os_name": "posix",
|
||||
"python_version": "3.13",
|
||||
"python_full_version": "3.13.0",
|
||||
"implementation_name": "cpython",
|
||||
"platform_python_implementation": "CPython",
|
||||
"extra": "",
|
||||
}
|
||||
|
||||
#: org.gnome.Platform//50 ships these. PyGObject is already absent from the
|
||||
#: export, which `just lock` builds with --no-emit-package pygobject.
|
||||
RUNTIME_PROVIDED = {"pycairo"}
|
||||
|
||||
#: Build backends for the sdists the sandbox compiles under
|
||||
#: --no-build-isolation. Never imported at runtime, so never in the export.
|
||||
BUILD_BACKENDS = {"meson-python", "pyproject-metadata"}
|
||||
|
||||
|
||||
def _locked() -> dict:
|
||||
"""Every package the export selects for the Flatpak target: version, digests."""
|
||||
selected = {}
|
||||
current = None
|
||||
for raw in EXPORT.read_text(encoding="utf-8").splitlines():
|
||||
line = raw.strip().rstrip("\\").strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
head = re.match(r"^([A-Za-z0-9._-]+)==([^ ;]+)\s*(?:;\s*(.*))?$", line)
|
||||
if head:
|
||||
name, version, marker = head.groups()
|
||||
current = None
|
||||
if marker and not Marker(marker).evaluate(TARGET_ENV):
|
||||
continue
|
||||
current = canonicalize_name(name)
|
||||
selected[current] = {"version": version, "hashes": set()}
|
||||
continue
|
||||
digest = re.match(r"^--hash=sha256:([0-9a-f]{64})$", line)
|
||||
if digest and current:
|
||||
selected[current]["hashes"].add(digest.group(1))
|
||||
return selected
|
||||
|
||||
|
||||
def _flatpak_sources() -> list:
|
||||
"""Every PyPI artifact the Flatpak downloads: name, version, digest, filename."""
|
||||
lines = DEPS_YAML.read_text(encoding="utf-8").splitlines()
|
||||
sources = []
|
||||
for index, line in enumerate(lines):
|
||||
url = re.match(r"^\s*url: https://files\.pythonhosted\.org/\S+/([^/\s]+)$", line)
|
||||
if not url:
|
||||
continue
|
||||
filename = url.group(1)
|
||||
digest = re.match(r"^\s*sha256: ([0-9a-f]{64})\s*$", lines[index + 1])
|
||||
assert digest, f"{DEPS_YAML.name}:{index + 2}: PyPI source with no sha256 under it"
|
||||
if filename.endswith(".whl"):
|
||||
name, version = filename.split("-")[0], filename.split("-")[1]
|
||||
else:
|
||||
base = re.sub(r"\.(tar\.gz|zip|tar\.bz2)$", "", filename)
|
||||
name, _, version = base.rpartition("-")
|
||||
sources.append(
|
||||
{
|
||||
"name": canonicalize_name(name),
|
||||
"version": version,
|
||||
"sha256": digest.group(1),
|
||||
"filename": filename,
|
||||
}
|
||||
)
|
||||
assert sources, "no PyPI sources found; the parser and the file disagree"
|
||||
return sources
|
||||
|
||||
|
||||
def _flatpak_versions() -> dict:
|
||||
return {source["name"]: source["version"] for source in _flatpak_sources()}
|
||||
|
||||
|
||||
def test_the_project_is_installed_without_deps() -> None:
|
||||
"""The reason every other test here has to exist.
|
||||
|
||||
`--no-deps` is correct (the runtime provides PyGObject and the modules
|
||||
below provide the rest), but it means pip never reads `pyproject.toml`'s
|
||||
constraints. Nothing inside a Flatpak build can notice a dependency that
|
||||
violates them, so the check has to live out here.
|
||||
"""
|
||||
assert "pip3 install --no-deps" in MANIFEST.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("source", _flatpak_sources(), ids=lambda s: s["name"])
|
||||
def test_every_source_is_the_version_the_lock_resolved(source: dict[str, str]) -> None:
|
||||
"""A version here that uv did not resolve is a package nothing tested."""
|
||||
if source["name"] in BUILD_BACKENDS:
|
||||
pytest.skip("build backend; absent from the runtime export by design")
|
||||
locked = _locked().get(source["name"])
|
||||
assert locked, (
|
||||
f"{source['name']} is built by the Flatpak but the lock does not"
|
||||
" resolve it for Linux. Drop its module, or add it to pyproject.toml."
|
||||
)
|
||||
assert source["version"] == locked["version"], (
|
||||
f"{source['name']}: Flatpak builds {source['version']},"
|
||||
f" requirements/runtime.txt pins {locked['version']}."
|
||||
" Run `just flatpak-deps`."
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("source", _flatpak_sources(), ids=lambda s: s["name"])
|
||||
def test_every_digest_is_one_uv_recorded(source: dict[str, str]) -> None:
|
||||
"""The bytes the Flatpak fetches are the bytes in uv.lock, not merely the
|
||||
same version number. A digest uv never saw is an artifact nothing pinned."""
|
||||
if source["name"] in BUILD_BACKENDS:
|
||||
pytest.skip("build backend; absent from the runtime export by design")
|
||||
hashes = _locked()[source["name"]]["hashes"]
|
||||
assert source["sha256"] in hashes, (
|
||||
f"{source['filename']} carries a digest absent from"
|
||||
" requirements/runtime.txt. Run `just flatpak-deps`."
|
||||
)
|
||||
|
||||
|
||||
def test_the_flatpak_carries_every_runtime_dependency() -> None:
|
||||
"""A missing one fails at first launch, not at build time, because the
|
||||
project is installed with --no-deps and nothing resolves the gap."""
|
||||
missing = sorted(set(_locked()) - set(_flatpak_versions()) - RUNTIME_PROVIDED)
|
||||
assert not missing, (
|
||||
"the lock resolves these for Linux and the Flatpak does not build them:"
|
||||
f" {', '.join(missing)}"
|
||||
)
|
||||
|
||||
|
||||
def test_the_flatpak_builds_nothing_the_lock_does_not_resolve() -> None:
|
||||
"""#705 deleted pydub, lxml, tqdm and python-xlib from `pyproject.toml`.
|
||||
|
||||
Two of those four came back through the dependency chain: pywhispercpp
|
||||
needs tqdm and pynput needs python-xlib, so the lock still resolves them
|
||||
and the Flatpak must still build them. pydub and lxml are the ones with no
|
||||
path back, and they stayed here for four months. Compare against the
|
||||
resolved set rather than the direct one, or this test deletes two packages
|
||||
the Flatpak needs.
|
||||
"""
|
||||
extra = sorted(set(_flatpak_versions()) - set(_locked()) - BUILD_BACKENDS)
|
||||
assert not extra, (
|
||||
f"the Flatpak builds packages nothing depends on: {', '.join(extra)}."
|
||||
" Delete their modules from python3-dependencies.yaml."
|
||||
)
|
||||
|
||||
|
||||
def test_pywhispercpp_satisfies_the_projects_own_floor() -> None:
|
||||
"""The finding this file was written for. Stated as the constraint rather
|
||||
than a literal, so bumping the floor cannot leave the check behind it."""
|
||||
with PYPROJECT.open("rb") as handle:
|
||||
dependencies = tomllib.load(handle)["project"]["dependencies"]
|
||||
spec = next(d for d in dependencies if canonicalize_name(d.split(">")[0]) == "pywhispercpp")
|
||||
constraint = SpecifierSet(spec[len("pywhispercpp") :])
|
||||
shipped = _flatpak_versions()["pywhispercpp"]
|
||||
assert Version(shipped) in constraint, (
|
||||
f"the Flatpak ships pywhispercpp {shipped}, outside the project's own"
|
||||
f" {constraint}. recognition_manager.py degrades GPU context_params"
|
||||
" below 1.5.0, so this is silent for the user."
|
||||
)
|
||||
|
||||
|
||||
def test_the_version_injection_names_the_version_being_built() -> None:
|
||||
"""pywhispercpp's sdist carries no version.txt and its setup() takes no
|
||||
version= argument, so the module rewrites setup.py before building and
|
||||
spells the version out four times. All four have to agree with the tarball
|
||||
actually downloaded, or the build renames a directory tar just wrote.
|
||||
"""
|
||||
module = re.search(
|
||||
r"^ - name: python3-pywhispercpp$(.*?)(?=^ - name: |\Z)",
|
||||
DEPS_YAML.read_text(encoding="utf-8"),
|
||||
re.M | re.S,
|
||||
)
|
||||
assert module, "the pywhispercpp module is gone; this test needs rewriting"
|
||||
spelled = set(re.findall(r"pywhispercpp-(\d+(?:\.\d+)+)(?![\d.]\d)", module.group(1)))
|
||||
spelled |= set(re.findall(r"PYWHISPERCPP_VERSION=(\d+(?:\.\d+)+)", module.group(1)))
|
||||
spelled |= set(re.findall(r'version=\\"(\d+(?:\.\d+)+)\\"', module.group(1)))
|
||||
assert spelled == {_flatpak_versions()["pywhispercpp"]}, (
|
||||
f"the pywhispercpp module spells out {sorted(spelled)}; the tarball it"
|
||||
f" downloads is {_flatpak_versions()['pywhispercpp']}"
|
||||
)
|
||||
|
||||
|
||||
def test_the_manifest_env_agrees_with_the_module() -> None:
|
||||
"""build-options.env sets PYWHISPERCPP_VERSION for the whole build. Left
|
||||
behind, it tells the build one version while the module builds another."""
|
||||
declared = re.search(r'PYWHISPERCPP_VERSION: "([^"]+)"', MANIFEST.read_text(encoding="utf-8"))
|
||||
assert declared, "the manifest no longer sets PYWHISPERCPP_VERSION"
|
||||
assert declared.group(1) == _flatpak_versions()["pywhispercpp"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("source", _flatpak_sources(), ids=lambda s: s["name"])
|
||||
def test_no_source_is_pinned_to_one_python_abi(source: dict[str, str]) -> None:
|
||||
"""Universal wheel or sdist, never a cp-tagged wheel.
|
||||
|
||||
The SDK's Python minor version is not ours to choose and moves with the
|
||||
GNOME runtime. An abi3 or cp313 wheel builds today and stops resolving the
|
||||
day the runtime bumps, with nothing in this repo changed.
|
||||
"""
|
||||
if not source["filename"].endswith(".whl"):
|
||||
return
|
||||
assert source["filename"].endswith("-none-any.whl"), (
|
||||
f"{source['filename']} is pinned to one interpreter ABI; take the sdist"
|
||||
" and let the sandbox build it against the SDK's own Python"
|
||||
)
|
||||
|
||||
|
||||
def test_a_lock_refresh_reaches_the_flatpak_build() -> None:
|
||||
"""The manifest is generated from requirements/, so a change there changes
|
||||
what the Flatpak ships. Without the path filter, `just lock` could move
|
||||
every version in it and start no Flatpak job."""
|
||||
text = WORKFLOW.read_text(encoding="utf-8")
|
||||
assert text.count("'requirements/**'") >= 2, (
|
||||
"flatpak.yml does not watch requirements/**, so a lock refresh changes"
|
||||
" what the Flatpak builds and runs no build"
|
||||
)
|
||||
Reference in New Issue
Block a user