ed0ed8ba49
* fix(packaging): build pywhispercpp at the portable x86-64 baseline Every packaged channel shipped a libggml-cpu compiled for AVX2+FMA: the manylinux PyPI wheel itself carries -mavx2 -mfma -mbmi2 code, so any package that bundles it dies with SIGILL when whisper.cpp loads a model on a pre-Haswell CPU (the reporter's i5-3337U). GGML_NATIVE=OFF alone is not the portable baseline either; while the native default is on, ggml turns every GGML_* SIMD option on, so each is pinned off. - Snap: force the sdist (PIP_NO_BINARY) with the flag floor via CMAKE_ARGS in the part's build-environment. - Flatpak: extend the module's CMAKE_ARGS with the same floor. - Native .deb/.rpm: the build still vendors the wheels, then rebuilds libggml-cpu alone from the same hash-pinned sdist and swaps it into the vendored copies, asserting no AVX/FMA survives; the container image gains cmake, g++, make and binutils. - AppImage: extract the pinned-sdist reinstall into a shared helper and route every CPU-only fallback of the Vulkan path through a new baseline rebuild at the same floor; the Vulkan compile keeps its own flags via the same helper. Keeping the wheel on those paths was the bug's exact binary. Fixes #957 * style(tests): apply black/isort to the new packaging guards * fix(native): keep the wheel's libggml providers, alias its base/gomp sonames Swapping the whole libggml set for the baseline build broke the import smoke: the wheel's auditwheel-mangled core lib doubles as the extension's __cxa_call_terminate provider, and its NEEDED entries are the mangled names. Swap only libggml-cpu, and satisfy the baseline library's own unmangled NEEDEDs (libggml-base.so.0, libgomp.so.1) with aliases of the vendored copies, which carry no generated SIMD and are the same upstream version. Fail closed when the vendored set has restructured instead of silently shipping the AVX2 wheel. * fix(packaging): fail builds that lose the speech engine, unblock the ISA assert Greptile findings on the baseline work: - A failed baseline rebuild in the AppImage returned success after remove_appdir_pywhispercpp, so the bundle would ship with no speech engine at all while logging that the wheel was kept. Propagate the failure through every CPU-only fallback so the build stops instead. - The native libggml-cpu assertion used grep -q under pipefail, where grep's early exit SIGPIPEs objdump and the pipeline reports a miss, letting an AVX2 library through. Read the full disassembly instead. * fix(installer): build pywhispercpp from sdist on the CPU path install_cpu_pywhispercpp let pip pick the manylinux wheel, whose libggml-cpu is compiled for AVX2+FMA and SIGILLs on pre-Haswell CPUs (#957). Pass --no-binary pywhispercpp like the Vulkan/CUDA paths so the sdist compiles against this machine's own instruction set. * fix(installer): install C++ toolchain on distros that ship without one Forcing the pywhispercpp sdist on the CPU path needs a compiler, and the fedora, mageia, alpine, void and solus package maps did not request one. Add gcc-c++/make (rpm), build-base (alpine), base-devel (void) and gcc/g++/make (solus) so source installs compile on minimal systems. --------- Co-authored-by: Jatin K Malik <jatinkrmalik@gmail.com>
221 lines
10 KiB
Bash
Executable File
221 lines
10 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Build vocalinux_<version>_<arch>.deb and vocalinux-<version>.<arch>.rpm.
|
|
#
|
|
# Usage: build.sh <path-to-vocalinux-wheel> <version> [output-dir]
|
|
# version release version, e.g. 0.18.0 or 0.19.0-beta
|
|
#
|
|
# CI and releases run this inside docker-build.sh's pinned Debian 12 image;
|
|
# it works on any host with bash, python3 + pip, curl and tar. The packages
|
|
# stay thin on purpose (#600): the interpreter, GTK, PyGObject and the gi
|
|
# typelibs come from the distribution, and so does every Python dependency
|
|
# the distros package. Only pywhispercpp and pynput are vendored — the first
|
|
# is unpackaged on both distros and the second is missing from Fedora.
|
|
set -euo pipefail
|
|
|
|
REPO_ROOT="$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/../.." && pwd)"
|
|
PINS="$REPO_ROOT/packaging/native/tool_checksums.txt"
|
|
|
|
fail() {
|
|
echo "FAIL: $*" >&2
|
|
exit 1
|
|
}
|
|
|
|
WHEEL="$(readlink -f "${1:?usage: build.sh <wheel> <version> [outdir]}")"
|
|
RAW_VERSION="${2:?usage: build.sh <wheel> <version> [outdir]}"
|
|
OUTDIR="$(mkdir -p "${3:-dist}" && cd "${3:-dist}" && pwd)"
|
|
[ -f "$WHEEL" ] || fail "wheel not found: $WHEEL"
|
|
|
|
# Split "0.19.0-beta" into version + prerelease; nfpm renders the prerelease
|
|
# as a "~beta" suffix, which orders before the stable release in both dpkg
|
|
# and rpm. An empty prerelease expands to nothing.
|
|
VERSION="${RAW_VERSION#v}"
|
|
PRERELEASE=""
|
|
case "$VERSION" in
|
|
*-*)
|
|
PRERELEASE="${VERSION#*-}"
|
|
VERSION="${VERSION%%-*}"
|
|
;;
|
|
esac
|
|
|
|
case "$(uname -m)" in
|
|
x86_64) NFPM_PIN="nfpm-x86_64"; NFPM_ARCH="amd64" ;;
|
|
aarch64|arm64) NFPM_PIN="nfpm-aarch64"; NFPM_ARCH="arm64" ;;
|
|
*) fail "unsupported architecture $(uname -m)" ;;
|
|
esac
|
|
|
|
TOOLS="$(mktemp -d)"
|
|
trap 'rm -rf "$TOOLS"' EXIT
|
|
|
|
fetch_pinned() {
|
|
local name="$1" dest="$2" url sum
|
|
sum="$(awk -v n="$name" '$1==n && $2=="sha256" {print $3}' "$PINS")"
|
|
url="$(awk -v n="$name" '$1==n && $2=="sha256" {print $4}' "$PINS")"
|
|
[ -n "$sum" ] && [ -n "$url" ] || fail "no sha256 pin for $name in $PINS"
|
|
curl -fSL --retry 3 -o "$dest" "$url"
|
|
echo "$sum $dest" | sha256sum -c - >/dev/null
|
|
}
|
|
|
|
# nfpm is a single static binary inside the release tarball; unpack just it.
|
|
fetch_pinned "$NFPM_PIN" "$TOOLS/nfpm.tar.gz"
|
|
tar -xzf "$TOOLS/nfpm.tar.gz" -C "$TOOLS" nfpm
|
|
NFPM="$TOOLS/nfpm"
|
|
|
|
PAYLOAD="$TOOLS/payload"
|
|
DOC="$TOOLS/pkg-doc"
|
|
install -d \
|
|
"$PAYLOAD/usr/lib/vocalinux/app" \
|
|
"$PAYLOAD/usr/lib/vocalinux/vendor" \
|
|
"$PAYLOAD/usr/bin" \
|
|
"$PAYLOAD/usr/share/applications" \
|
|
"$PAYLOAD/usr/share/icons/hicolor/scalable/apps" \
|
|
"$DOC"
|
|
|
|
# --- application payload ------------------------------------------------
|
|
# The wheel is a zip; -m zipfile lays out vocalinux/ and its dist-info
|
|
# exactly where pip --target would, without pip's EXTERNALLY-MANAGED refusal
|
|
# on Debian 12.
|
|
python3 -m zipfile -e "$WHEEL" "$PAYLOAD/usr/lib/vocalinux/app/"
|
|
|
|
# --- vendored wheels -----------------------------------------------------
|
|
# Download each supported CPython's pywhispercpp wheel — it has no abi3 tag,
|
|
# so there is one wheel per interpreter — plus the pure-Python pynput, per
|
|
# the hashes pinned in requirements/runtime.txt, and merge them flat into
|
|
# the vendor dir. The extension filenames carry their tag
|
|
# (_pywhispercpp.cpython-311-*.so next to .cpython-313-*.so) so all four
|
|
# coexist, and the flat layout reproduces what pip would place in
|
|
# site-packages, which is what utils/pywhispercpp_loader.py scans for.
|
|
#
|
|
# VENDOR_PYS tracks requires-python in pyproject.toml and the CI matrix.
|
|
VENDOR_PYS="3.11 3.12 3.13 3.14"
|
|
PINS_VENDOR="$TOOLS/vendor-requirements.txt"
|
|
awk '/^(pywhispercpp|pynput)==/{p=1} p{print; if ($0 !~ /\\$/) p=0}' \
|
|
"$REPO_ROOT/requirements/runtime.txt" > "$PINS_VENDOR"
|
|
grep -q "^pywhispercpp==" "$PINS_VENDOR" || fail "pywhispercpp pin not found in requirements/runtime.txt"
|
|
grep -q "^pynput==" "$PINS_VENDOR" || fail "pynput pin not found in requirements/runtime.txt"
|
|
|
|
for PYV in $VENDOR_PYS; do
|
|
mkdir -p "$TOOLS/wheels/$PYV"
|
|
pip3 download --require-hashes --no-deps --only-binary=:all: \
|
|
--python-version "$PYV" -r "$PINS_VENDOR" -d "$TOOLS/wheels/$PYV" >/dev/null
|
|
done
|
|
for VENDOR_WHEEL in "$TOOLS"/wheels/*/*.whl; do
|
|
python3 -m zipfile -e "$VENDOR_WHEEL" "$PAYLOAD/usr/lib/vocalinux/vendor/"
|
|
done
|
|
|
|
# --- portable libggml-cpu -------------------------------------------------
|
|
# The pinned pywhispercpp wheels bundle a libggml-cpu compiled for AVX2+FMA,
|
|
# which dies with SIGILL at model load on pre-Haswell CPUs (#957). Rebuild
|
|
# the ggml libraries from the same pinned sdist at the plain x86-64 baseline —
|
|
# GGML_NATIVE=OFF alone still compiles -mavx2 -mfma, because ggml turns every
|
|
# GGML_* SIMD option on while its native default is on — and swap them into
|
|
# the vendored copies. libwhisper and the Python extension carry no generated
|
|
# SIMD, so only the libggml set needs replacing.
|
|
SDIST_DIR="$TOOLS/pywhispercpp-sdist"
|
|
mkdir -p "$SDIST_DIR"
|
|
pip3 download --require-hashes --no-deps --no-binary pywhispercpp \
|
|
-r "$PINS_VENDOR" -d "$SDIST_DIR" >/dev/null
|
|
tar -xzf "$SDIST_DIR"/pywhispercpp-*.tar.gz -C "$SDIST_DIR"
|
|
WHISPER_GGML="$(echo "$SDIST_DIR"/pywhispercpp-*/whisper.cpp/ggml)"
|
|
[ -d "$WHISPER_GGML" ] || fail "whisper.cpp/ggml missing from pywhispercpp sdist"
|
|
# ggml.pc.in is generated upstream and absent from the sdist; it only feeds
|
|
# the standalone install rules this build never runs.
|
|
touch "$WHISPER_GGML/ggml.pc.in"
|
|
cmake -S "$WHISPER_GGML" -B "$TOOLS/ggml-build" \
|
|
-DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON \
|
|
-DGGML_NATIVE=OFF -DGGML_SSE42=OFF -DGGML_AVX=OFF -DGGML_AVX2=OFF \
|
|
-DGGML_BMI2=OFF -DGGML_FMA=OFF -DGGML_F16C=OFF \
|
|
-DGGML_BUILD_TESTS=OFF -DGGML_BUILD_EXAMPLES=OFF
|
|
cmake --build "$TOOLS/ggml-build" --target ggml-cpu -j"$(nproc)"
|
|
BASELINE_CPU="$(find "$TOOLS/ggml-build" -name 'libggml-cpu.so.*' -type f | sort -V | tail -1)"
|
|
[ -f "$BASELINE_CPU" ] || fail "baseline libggml-cpu build produced no library"
|
|
# objdump, not a feature check: the SIGILL is ISA-level, so assert no 256-bit
|
|
# or FMA instructions survive in the library that ships. No grep -q: under
|
|
# pipefail an early exit SIGPIPEs objdump and the pipeline reports a miss.
|
|
if objdump -d "$BASELINE_CPU" | grep -E 'ymm|vfmadd|vfnmadd|vfmsub|vzeroupper' >/dev/null; then
|
|
fail "baseline libggml-cpu still carries AVX/FMA instructions"
|
|
fi
|
|
# The vendored wheel files carry auditwheel-mangled basenames
|
|
# (libggml-cpu-<hash>.so.0.9.8) that the extension's NEEDED entries point at,
|
|
# so overwrite the cpu library in place and keep its filename. Only cpu is
|
|
# swapped: the wheel's other libggml files carry no generated SIMD, and the
|
|
# mangled core lib doubles as the extension's __cxa_call_terminate provider —
|
|
# replacing it with a locally built copy drops that export.
|
|
swapped=0
|
|
while IFS= read -r vendored; do
|
|
cp "$BASELINE_CPU" "$vendored"
|
|
swapped=$((swapped + 1))
|
|
done < <(find "$PAYLOAD/usr/lib/vocalinux/vendor" -name 'libggml-cpu*.so*')
|
|
[ "$swapped" -gt 0 ] || fail "found no vendored libggml-cpu to replace"
|
|
# The baseline library's own NEEDED entries use the unmangled sonames
|
|
# (libggml-base.so.0, libgomp.so.1), so alias the vendored copies — the same
|
|
# upstream version, and base/gomp carry no generated SIMD anyway — under
|
|
# those names inside pywhispercpp.libs.
|
|
while IFS= read -r libs_dir; do
|
|
vendored_base="$(find "$libs_dir" -name 'libggml-base*.so*' | head -1)"
|
|
[ -n "$vendored_base" ] || fail "no vendored libggml-base in $libs_dir"
|
|
cp "$vendored_base" "$libs_dir/libggml-base.so.0"
|
|
vendored_gomp="$(find "$libs_dir" -name 'libgomp*.so*' | head -1)"
|
|
[ -n "$vendored_gomp" ] || fail "no vendored libgomp in $libs_dir"
|
|
cp "$vendored_gomp" "$libs_dir/libgomp.so.1"
|
|
done < <(find "$PAYLOAD/usr/lib/vocalinux/vendor" -type d -name 'pywhispercpp.libs')
|
|
|
|
# --- launchers -----------------------------------------------------------
|
|
# Same environment install.sh's wrapper establishes: the vendored
|
|
# pywhispercpp.libs on LD_LIBRARY_PATH, user site-packages off so they cannot
|
|
# shadow the distro modules, and sg input so evdev hotkeys work on Wayland
|
|
# when the user is already in the input group but the session has not picked
|
|
# up the membership yet. vocalinux-gui exists because the wheel declares a
|
|
# gui-script of that name; on Linux it runs the same entry.
|
|
write_launcher() {
|
|
local path="$1"
|
|
cat > "$path" <<'LAUNCHER_EOF'
|
|
#!/bin/sh
|
|
# Launcher for the distro-packaged Vocalinux. The vendored wheels (the
|
|
# pywhispercpp/pynput copies Debian and Fedora do not ship) live under
|
|
# /usr/lib/vocalinux/vendor with the app itself under .../app; the loader in
|
|
# utils/pywhispercpp_loader.py finds the bundled .so files from sys.path.
|
|
PYTHONNOUSERSITE=1
|
|
VOCALINUX_VENDOR="/usr/lib/vocalinux/vendor"
|
|
if [ -d "$VOCALINUX_VENDOR/pywhispercpp.libs" ]; then
|
|
LD_LIBRARY_PATH="$VOCALINUX_VENDOR/pywhispercpp.libs${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
|
|
export LD_LIBRARY_PATH
|
|
fi
|
|
export PYTHONNOUSERSITE PYTHONPATH="/usr/lib/vocalinux/app:$VOCALINUX_VENDOR${PYTHONPATH:+:$PYTHONPATH}"
|
|
if grep -q "^input:.*\b$(whoami)\b" /etc/group 2>/dev/null \
|
|
&& ! groups | grep -q "\binput\b" && command -v sg >/dev/null 2>&1; then
|
|
# sg(1) takes a single command string, so each argument is re-quoted for
|
|
# the inner shell; joining $* would re-split paths containing spaces.
|
|
QUOTED_ARGS=""
|
|
for arg in "$@"; do
|
|
arg="$(printf %s "$arg" | sed "s/'/'\\\\''/g")"
|
|
QUOTED_ARGS="$QUOTED_ARGS '$arg'"
|
|
done
|
|
exec sg input -c "/usr/bin/python3 -m vocalinux.main$QUOTED_ARGS"
|
|
fi
|
|
exec /usr/bin/python3 -m vocalinux.main "$@"
|
|
LAUNCHER_EOF
|
|
chmod 755 "$path"
|
|
}
|
|
write_launcher "$PAYLOAD/usr/bin/vocalinux"
|
|
write_launcher "$PAYLOAD/usr/bin/vocalinux-gui"
|
|
|
|
# --- desktop entry, icon, license ----------------------------------------
|
|
install -Dm644 "$REPO_ROOT/vocalinux.desktop" \
|
|
"$PAYLOAD/usr/share/applications/vocalinux.desktop"
|
|
install -Dm644 "$REPO_ROOT/resources/icons/scalable/vocalinux.svg" \
|
|
"$PAYLOAD/usr/share/icons/hicolor/scalable/apps/vocalinux.svg"
|
|
install -Dm644 "$REPO_ROOT/LICENSE" "$DOC/LICENSE"
|
|
|
|
# --- package -------------------------------------------------------------
|
|
# nfpm resolves contents src paths relative to its working directory, so run
|
|
# it from $TOOLS where ./payload and ./pkg-doc live.
|
|
cd "$TOOLS"
|
|
VOCA_ARCH="$NFPM_ARCH" VOCA_VERSION="$VERSION" VOCA_PRERELEASE="$PRERELEASE" \
|
|
"$NFPM" package --config "$REPO_ROOT/packaging/native/nfpm.yaml" \
|
|
--packager deb --target "$OUTDIR/"
|
|
VOCA_ARCH="$NFPM_ARCH" VOCA_VERSION="$VERSION" VOCA_PRERELEASE="$PRERELEASE" \
|
|
"$NFPM" package --config "$REPO_ROOT/packaging/native/nfpm.yaml" \
|
|
--packager rpm --target "$OUTDIR/"
|
|
|
|
ls -la "$OUTDIR"/*.deb "$OUTDIR"/*.rpm
|