xavierk/voculinux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

chore(build): uv tooling, a Python 3.11 floor, and verified model downloads (epic #701, phases 2.5 & 5) (#713)

Browse Source 
* chore(build): migrate justfile tooling to uv

All recipes now run through 'uv run' against .venv, with a shared
DEV_EXTRAS variable (dev + vad) because uv syncs the environment
exactly and would otherwise uninstall the dev tools between
recipes. 'deps' installs dev+vad; new 'deps-all' covers every
extra. 'build' switches to 'uv build'; 'version' uses --no-sync.

Bump [tool.uv] required-version to >=0.12,<0.13 for uv 0.12.x, and
raise the mypy python_version target to 3.12: mypy 1.19 in the lock
cannot target 3.9 and numpy 2.x stubs need a 3.12 target to parse
(mypy is not run in CI or pre-commit; this only affects 'just
typecheck').

* fix(tests): isolate test runs from real XDG state; fix Arch model-path crash

Local test runs failed en masse (185 failures) while CI stayed
green: tests picked up the developer's real ~/.local/share/vocalinux
models via the default XDG paths and switched IBus/keyboard code
paths through the desktop session env (XDG_CURRENT_DESKTOP/
XDG_SESSION_TYPE on GNOME/Wayland, issue #478 branches). conftest
now points XDG base dirs at a throwaway session-wide directory and
clears session variables at import time, mirroring a fresh CI
runner (paths like MODELS_DIR freeze at first import during
collection, so fixture-time isolation is too late).

Also guard the Arch branch of _get_system_model_paths(): it removed
/usr/local/share/vocalinux/models unconditionally and crashed with
ValueError on Arch hosts when a custom XDG_DATA_DIRS never added
that path. Add regression tests, and make the lock-file test assert
the actual $XDG_DATA_HOME/vocalinux contract instead of the
hardcoded ~/.local/share default.

* fix(tests): keep the IBus engine double a plain class

Python 3.14's unittest.mock re-instantiates a user subclass when it builds
child mocks (klass(name=...)). VocalinuxEngine inherits from whatever
IBus.Engine resolves to, and the test module bound that to the MagicMock
class itself, so every engine instantiation raised

    TypeError: VocalinuxEngine.__init__() got an unexpected keyword argument 'name'

on 3.14 while passing on 3.13. Replace it with a plain stand-in class that
still hands out per-instance MagicMocks for the IBus API surface the engine
touches.

The same pattern remains in test_ibus_engine.py, test_ibus_engine_utils.py,
test_ibus_active_engine_guard.py and test_ibus_xkb_wayland.py; those never
instantiate the engine, so they are unaffected for now.

* fix(install): build the venv from the system Python

install.sh looked up a bare `python3` from PATH. Started from an activated
virtualenv — a shell left in uv's .venv after `just deps` — it built venv/
from that interpreter, where distro PyGObject is invisible even with
--system-site-packages, and the run aborted with

    Distro PyGObject (python3-gi / python3-gobject) is not importable in the venv.

A mismatched venv/ then survived every retry, because the installer reuses
whatever venv it finds.

- drop an inherited VIRTUAL_ENV (and its bin from PATH) before any
  interpreter lookup
- select the interpreter explicitly: $SYSTEM_PYTHON (default
  /usr/bin/python3) or PATH python3, preferring one that can import gi,
  and create the venv with it instead of `python3`
- recreate venv/ when it was built by a different interpreter
- name the interpreter in the PyGObject guard's error

Documents the two environments (uv's .venv for tooling, venv/ for the
installed app) in CONTRIBUTING, the justfile header and AGENTS.md, and adds
the mismatch symptom to the INSTALL troubleshooting section.

* chore(python): raise the supported floor to 3.11

requires-python promised 3.9 while the CI matrix ran 3.9/3.10/3.11/3.13,
the classifiers advertised up to 3.14, mypy targeted 3.12 and install.sh
enforced 3.9 — five different answers to "which Python does this support".

Settle on 3.11 everywhere: pyproject (requires-python, classifiers, black
targets), install.sh's MIN_VERSION, check-system-deps.sh, the distro package
map, the CI matrix (now 3.11-3.14, matching the classifiers), the release and
nightly workflows, and the docs.

mypy now targets the floor instead of 3.12, so type checking covers what we
promise; numpy 2.x ships stubs in 3.12 syntax that a 3.11 target cannot
parse, so its imports are skipped. (mypy has 71 pre-existing errors either
way; no CI job gates on it.)

uv.lock and the requirements/* exports are regenerated — dropping 3.9/3.10
removes most of the version-conditional resolution branches.

tests/test_python_version_policy.py pins the invariant: install.sh's floor,
requires-python, the classifiers and the CI matrix must agree.

* ci: run lint and tests from uv.lock

The lock file and the hash-pinned exports were not enforced anywhere: CI
still built a venv by hand and installed unpinned `PyGObject` plus
`-e ".[dev,vad]"`, so the tool versions it checked against had nothing to do
with the ones `just lint` uses locally.

Both Python jobs now use astral-sh/setup-uv and run through `uv sync
--locked` / `uv run --locked`, which also fails the build when uv.lock drifts
from pyproject.toml.

Partially covers phase 5 of #701 (tests install from the lock, lock freshness
gate); trusted publishing, artifact checksums and action SHA pinning are
still open there.

* ci: lint from a dedicated dependency group

The lint job failed on this branch: `uv run --extra dev` syncs the whole
project, so it tried to build pyaudio from source on a runner that installs
no system dependencies.

    src/pyaudio/device_api.c:9:10: fatal error: portaudio.h: No such file or directory

Move black/isort/flake8 out of the `dev` extra into a `lint` dependency
group. CI runs `uv run --locked --only-group lint`, which installs the three
pure-Python tools and skips the project entirely — no portaudio or
girepository headers needed. `just deps` and DEV_EXTRAS request the group
alongside the dev extras, so the local environment still carries test and
lint tooling at once and `just lint` matches CI version for version.

pre-commit is unaffected: its hooks pin their own tool versions upstream.

* fix(install): never configure an engine the venv cannot import

vosk became an optional extra in #705, but the installer still writes
`"engine": "vosk"` into config.json on the whisper.cpp fallback path even
when installing the extra only produced a warning. config.json also survives
reinstalls — every writer guards on "file does not exist" — so an engine
picked by an earlier attempt outlives the venv that supported it. Either way
Vocalinux dies at startup:

    ModuleNotFoundError: No module named 'vosk'

- the fallback now writes the vosk config only when `import vosk` actually
  succeeds, and otherwise leaves the engine configuration alone
- a new verify_configured_engine() runs near the end of the install: if
  config.json selects vosk and the venv cannot import it, it says so and
  prints both fixes instead of leaving a broken install behind

* feat(models): verify every model download against a pinned digest

Pins a sha256/md5 per downloadable speech model in
src/vocalinux/utils/model_checksums.txt and verifies against it in both
install.sh and the runtime downloaders, before the file is moved into place.
Verification fails closed. Covers phase 2.5 of #701; see the PR for details.

* docs(agents): rebase AGENTS.md onto the #727 rewrite

* fix(install): hash models that are already on disk instead of trusting them

* refactor(models): one download path per engine, and fix Whisper's

* docs(install): drop distros that cannot meet the 3.11 floor

* docs: retire the distros and Python versions the 3.11 floor rules out

* fix(install): abort instead of warning when Python is below the floor

* fix(install): fall back to the system Python, not the one on PATH

* fix(install): match an existing venv by installation, not by X.Y

* fix(install): repair a config naming an engine the venv cannot import

* fix(models): pin VOSK by the sha256 of bytes we fetched, not upstream's md5

* fix(justfile): request the dev set in every recipe that syncs

* test(version): scan the docs and website trees for a stale Python floor

* docs: correct the stale model-checksum and distro claims

* fix(models): stamp the VOSK tree the app extracts, as the installer does

* fix(install): replace an unverified VOSK model only once its successor is ready

* docs(install): align the Fedora floor with the rest of the tree

* test(install): drop an f-string with no placeholders

* test(models): stub the downloader probe the VOSK installer test relies on

* docs: drop the long dashes from two model-checksum comments

* fix(install): gate on the interpreter, not on the distro release label

* fix(justfile): export the lint group into requirements/dev.txt

* fix(models): unpack the VOSK tree into staging and stamp it before the swap

* fix(whisper): adopt a checkpoint an earlier release stored under the catalog name

* docs(settings): state what a stray Whisper copy really means for deletion

* fix(models): hash a whisper.cpp model found on disk before it reaches ctypes

* docs(contributing): stop telling readers to uninstall their linters

* fix(install): stop warning off distros the project documents as supported

* docs(distros): file the supported derivatives under supported
This commit is contained in:
Sergey Silaev
2026-08-24 00:25:06 -07:00
committed by GitHub
parent ba9cca844b
commit 582d494633
49 changed files with 3691 additions and 4122 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/vocalinux/speech_recognition/recognition_manager.py
+158 -209
View File
@@ -21,8 +21,18 @@ from typing import Callable, Optional
from ..common_types import RecognitionState
from ..ui.audio_feedback import play_error_sound, play_start_sound, play_stop_sound
from ..utils.model_checksums import (
ChecksumError,
verify_model_file,
write_verification_stamp,
)
from ..utils.paths import models_dir
from ..utils.vosk_model_info import SUPPORTED_LANGUAGES, VOSK_MODEL_INFO
from ..utils.whisper_model_info import (
migrate_legacy_checkpoint_names,
whisper_model_file,
whisper_model_url,
)
from ..utils.whispercpp_model_info import WHISPERCPP_MODEL_INFO, get_model_path, is_model_downloaded
from ..version import __version__
from .command_processor import CommandProcessor
@@ -875,7 +885,9 @@ def _get_system_model_paths() -> list:
# Arch Linux doesn't use /usr/local
if "arch" in os_release:
paths.remove("/usr/local/share/vocalinux/models")
local_share_path = "/usr/local/share/vocalinux/models"
if local_share_path in paths:
paths.remove(local_share_path)
except (IOError, OSError, FileNotFoundError):
pass # File doesn't exist on all systems
@@ -1123,14 +1135,16 @@ class SpeechRecognitionManager:
)
self.model_size = "base"
# Check if model is downloaded
# Only this directory counts: load_model() gets it as download_root
# and looks nowhere else, so a copy in ~/.cache/whisper saves nothing.
whisper_cache_dir = os.path.join(MODELS_DIR, "whisper")
os.makedirs(whisper_cache_dir, exist_ok=True)
model_file = os.path.join(whisper_cache_dir, f"{self.model_size}.pt")
default_cache = os.path.expanduser("~/.cache/whisper")
default_model_file = os.path.join(default_cache, f"{self.model_size}.pt")
model_exists = os.path.exists(model_file) or os.path.exists(default_model_file)
# Before asking whether the model is here: an earlier release saved it
# under the catalog name, which for "large" is not what load_model
# looks for. Without this it refetches 2.9GB and orphans the old file.
migrate_legacy_checkpoint_names(whisper_cache_dir)
model_file = os.path.join(whisper_cache_dir, whisper_model_file(self.model_size))
model_exists = os.path.exists(model_file)
if not model_exists and self._defer_download:
logger.info(
@@ -1139,7 +1153,6 @@ class SpeechRecognitionManager:
self._model_initialized = False
return # Don't block startup
# If model doesn't exist and we're not deferring, download it with progress
if not model_exists:
logger.info(f"Downloading Whisper '{self.model_size}' model...")
self._download_whisper_model(whisper_cache_dir)
@@ -1152,7 +1165,9 @@ class SpeechRecognitionManager:
# Ensure previous model is released if re-initializing
self.model = None
# Load model with device and custom cache directory
# load_model() verifies the checkpoint against the sha256 in its URL,
# for a file already on disk as well as a fresh download, so Vocalinux
# never hashes Whisper checkpoints itself.
self.model = whisper.load_model(
self.model_size, device=device, download_root=whisper_cache_dir
)
@@ -1255,6 +1270,16 @@ class SpeechRecognitionManager:
# Check if model is downloaded
model_path = get_model_path(self.model_size)
# A file that is merely present did not necessarily come through the
# download path: releases before checksum verification fetched ggml
# models from `resolve/main` with no check at all, and install.sh
# re-hashes only ggml-tiny.bin. whisper.cpp maps this straight through
# ctypes, so hash it here; a failure demotes it to "not downloaded".
if os.path.exists(model_path) and not self._whispercpp_model_is_verified(model_path):
if self._defer_download:
self._model_initialized = False
return
if not os.path.exists(model_path):
if self._defer_download:
logger.info(
@@ -1280,6 +1305,29 @@ class SpeechRecognitionManager:
self.state = RecognitionState.ERROR
raise
@staticmethod
def _whispercpp_model_is_verified(model_path: str) -> bool:
"""Hash the model against its pin, or delete it and report False.
Cheap enough to do every time: sha256 runs at GB/s and whisper.cpp is
about to read the whole file anyway, so this is CPU over pages that are
being fetched regardless. Deleting rather than raising keeps the caller
simple, since an unverifiable model is indistinguishable from a missing
one as far as what happens next.
"""
try:
verify_model_file(model_path)
return True
except ChecksumError as error:
logger.error("whisper.cpp model at %s is not trustworthy: %s", model_path, error)
try:
os.remove(model_path)
except OSError as remove_error:
logger.error("Could not remove %s: %s", model_path, remove_error)
return False
logger.info("Removed the unverified model; it will be downloaded again")
return False
def _build_whispercpp_model_kwargs(self, n_threads: int) -> dict:
model_kwargs = {
"n_threads": n_threads,
@@ -1945,7 +1993,7 @@ class SpeechRecognitionManager:
raise RuntimeError(
f"Model URL returned HTML instead of a binary file "
f"(status {response.status_code}, content-type {content_type}). "
f"Hugging Face may be unavailable; try again later."
f"The server may be unavailable; try again later."
)
total_size = int(response.headers.get("content-length", 0))
@@ -2003,9 +2051,40 @@ class SpeechRecognitionManager:
os.remove(dest_path)
raise RuntimeError(
f"Downloaded 0 bytes from {url}. "
"Hugging Face may be down or blocked; try again later."
"The server may be down or blocked; try again later."
)
def _download_whisper_model(self, cache_dir: str) -> None:
"""Fetch an OpenAI Whisper checkpoint into ``cache_dir``.
No digest check here: whisper.load_model() verifies whatever it loads.
This exists so the UI gets progress and a working cancel, which
load_model does not expose.
"""
import requests
self._download_cancelled = False
url = whisper_model_url(self.model_size)
model_file = os.path.join(cache_dir, whisper_model_file(self.model_size))
temp_file = model_file + ".tmp"
os.makedirs(cache_dir, exist_ok=True)
logger.info(f"Downloading Whisper {self.model_size} model to {model_file}")
try:
self._stream_model_download(url, temp_file)
os.rename(temp_file, model_file)
if self._download_progress_callback:
self._download_progress_callback(1.0, 0, "Complete!")
except requests.exceptions.RequestException as e:
logger.error(f"Failed to download Whisper model from {url}: {e}")
if os.path.exists(temp_file):
os.remove(temp_file)
raise RuntimeError(f"Failed to download Whisper model: {e}") from e
except (OSError, RuntimeError, ValueError):
if os.path.exists(temp_file):
os.remove(temp_file)
raise
def _download_whispercpp_model(self):
"""Download a whisper.cpp model with progress tracking."""
import requests
@@ -2029,6 +2108,10 @@ class SpeechRecognitionManager:
try:
self._stream_model_download(url, temp_file)
# Verify before the rename: whisper.cpp loads ggml files through
# ctypes, so an unverified file must never reach its final path
# where is_model_downloaded() would treat it as good.
verify_model_file(temp_file, os.path.basename(model_path))
os.rename(temp_file, model_path)
logger.info("whisper.cpp model downloaded successfully")
@@ -2049,7 +2132,7 @@ class SpeechRecognitionManager:
"Check your network and try again."
) from e
raise RuntimeError(f"Failed to download whisper.cpp model: {e}") from e
except (OSError, RuntimeError, ValueError) as e:
except (ChecksumError, OSError, RuntimeError, ValueError) as e:
logger.error(f"An error occurred during whisper.cpp model download: {e}")
if os.path.exists(temp_file):
os.remove(temp_file)
@@ -2101,245 +2184,111 @@ class SpeechRecognitionManager:
logger.info("Download cancellation requested")
def _download_vosk_model(self):
"""Download the VOSK model if it doesn't exist."""
"""Download and extract the VOSK model if it doesn't exist."""
import zipfile
import requests
self._download_cancelled = False
model_urls = {
"small": f"https://alphacephei.com/vosk/models/{self.vosk_model_map['small']}.zip",
"medium": f"https://alphacephei.com/vosk/models/{self.vosk_model_map['medium']}.zip",
"large": f"https://alphacephei.com/vosk/models/{self.vosk_model_map['large']}.zip",
}
url = model_urls.get(self.model_size)
if not url:
model_name = self.vosk_model_map.get(self.model_size)
if not model_name:
raise ValueError(f"Unknown model size: {self.model_size}")
url = f"https://alphacephei.com/vosk/models/{model_name}.zip"
model_name = os.path.basename(url).replace(".zip", "")
# Always download to user's local directory
model_path = os.path.join(MODELS_DIR, model_name)
zip_path = os.path.join(MODELS_DIR, os.path.basename(url))
# Create models directory if it doesn't exist
zip_path = os.path.join(MODELS_DIR, f"{model_name}.zip")
os.makedirs(MODELS_DIR, exist_ok=True)
logger.info(f"Downloading VOSK {self.model_size} model to user directory: {model_path}")
# Download the model
logger.info(f"Downloading VOSK model from {url}")
logger.info(f"Downloading VOSK {self.model_size} model to {model_path}")
try:
response = requests.get(url, stream=True, timeout=self._MODEL_DOWNLOAD_TIMEOUT)
response.raise_for_status() # Raise an exception for bad status codes (4xx or 5xx)
self._stream_model_download(url, zip_path)
total_size = int(response.headers.get("content-length", 0))
downloaded_size = 0
start_time = time.time()
last_update_time = start_time
chunk_size = 8192 # 8KB chunks for smoother progress
# Verify before extracting, so a zip that fails its pinned digest
# never writes files into MODELS_DIR.
if self._download_progress_callback:
self._download_progress_callback(1.0, 0, "Verifying model...")
verify_model_file(zip_path)
with open(zip_path, "wb") as f:
for data in response.iter_content(chunk_size=chunk_size):
if self._download_cancelled:
logger.info("Download cancelled by user")
f.close()
if os.path.exists(zip_path):
os.remove(zip_path)
raise RuntimeError("Download cancelled")
f.write(data)
downloaded_size += len(data)
# Update progress callback
current_time = time.time()
if (
self._download_progress_callback
and (current_time - last_update_time) >= 0.1
):
elapsed = current_time - start_time
if elapsed > 0:
speed_mbps = (downloaded_size / (1024 * 1024)) / elapsed
else:
speed_mbps = 0
if total_size > 0:
progress = downloaded_size / total_size
remaining_mb = (total_size - downloaded_size) / (1024 * 1024)
if speed_mbps > 0:
eta_seconds = remaining_mb / speed_mbps
eta_str = (
f"{int(eta_seconds)}s"
if eta_seconds < 60
else f"{int(eta_seconds / 60)}m {int(eta_seconds % 60)}s"
)
else:
eta_str = "--"
status = f"{downloaded_size / (1024 * 1024):.1f} / {total_size / (1024 * 1024):.1f} MB • {speed_mbps:.1f} MB/s • ETA: {eta_str}"
else:
progress = 0
status = (
f"{downloaded_size / (1024 * 1024):.1f} MB • {speed_mbps:.1f} MB/s"
)
self._download_progress_callback(progress, speed_mbps, status)
last_update_time = current_time
# Also log progress periodically
logger.info(f"Download progress: {progress * 100:.1f}% - {status}")
# Update status for extraction phase
if self._download_progress_callback:
self._download_progress_callback(1.0, 0, "Extracting model...")
# Extract the model
logger.info(f"Extracting VOSK model to {model_path}")
with zipfile.ZipFile(zip_path, "r") as zip_ref:
zip_ref.extractall(MODELS_DIR)
self._unpack_vosk_model(zipfile, zip_path, model_name, model_path)
# Remove the zip file
os.remove(zip_path)
logger.info("VOSK model downloaded and extracted successfully")
# Final status
if self._download_progress_callback:
self._download_progress_callback(1.0, 0, "Complete!")
except requests.exceptions.RequestException as e:
logger.error(f"Failed to download VOSK model from {url}: {e}")
# Clean up potentially incomplete download
if os.path.exists(zip_path):
os.remove(zip_path)
self._discard_vosk_scratch(zip_path, model_name)
raise RuntimeError(f"Failed to download VOSK model: {e}") from e
except zipfile.BadZipFile:
logger.error(f"Downloaded file from {url} is not a valid zip file.")
# Clean up corrupted download
if os.path.exists(zip_path):
os.remove(zip_path)
self._discard_vosk_scratch(zip_path, model_name)
raise RuntimeError("Downloaded VOSK model file is corrupted.")
except (OSError, RuntimeError, ValueError) as e:
except (ChecksumError, OSError, RuntimeError, ValueError) as e:
logger.error(f"An error occurred during VOSK model download/extraction: {e}")
# Clean up potentially corrupted extraction
if os.path.exists(zip_path):
os.remove(zip_path)
# Consider removing partially extracted model dir if needed
# if os.path.exists(model_path): shutil.rmtree(model_path)
self._discard_vosk_scratch(zip_path, model_name)
raise
def _download_whisper_model(self, cache_dir: str):
"""Download a Whisper model with progress tracking."""
import requests
@staticmethod
def _vosk_scratch_paths(model_name: str) -> tuple:
"""Where a download unpacks (staging) and parks the tree it replaces."""
return (
os.path.join(MODELS_DIR, f".{model_name}.incoming"),
os.path.join(MODELS_DIR, f".{model_name}.replaced"),
)
self._download_cancelled = False
def _discard_vosk_scratch(self, zip_path: str, model_name: str) -> None:
"""Leave nothing behind that a later run would mistake for a model."""
import shutil
# Whisper model URLs (from openai-whisper package)
model_urls = {
"tiny": "https://openaipublic.azureedge.net/main/whisper/models/"
"65147644a518d12f04e32d6f3b26facc3f8dd46e5390956a9424a650c0ce22b9/"
"tiny.pt",
"base": "https://openaipublic.azureedge.net/main/whisper/models/"
"ed3a0b6b1c0edf879ad9b11b1af5a0e6ab5db9205f891f668f8b0e6c6326e34e/"
"base.pt",
"small": "https://openaipublic.azureedge.net/main/whisper/models/"
"9ecf779972d90ba49c06d968637d720dd632c55bbf19d441fb42bf17a411e794/"
"small.pt",
"medium": "https://openaipublic.azureedge.net/main/whisper/models/"
"345ae4da62f9b3d59415adc60127b97c714f32e89e936602e85993674d08dcb1/"
"medium.pt",
"large": "https://openaipublic.azureedge.net/main/whisper/models/"
"e5b1a55b89c1367dacf97e3e19bfd829a01529dbfdeefa8caeb59b3f1b81dadb/"
"large-v3.pt",
}
if os.path.exists(zip_path):
os.remove(zip_path)
for path in self._vosk_scratch_paths(model_name):
shutil.rmtree(path, ignore_errors=True)
url = model_urls.get(self.model_size)
if not url:
raise ValueError(f"Unknown Whisper model size: {self.model_size}")
def _unpack_vosk_model(self, zipfile, zip_path: str, model_name: str, model_path: str) -> None:
"""Unpack the verified zip beside the model, stamp it, then swap it in.
model_file = os.path.join(cache_dir, f"{self.model_size}.pt")
temp_file = model_file + ".tmp"
The stamp is the only evidence the tree was ever checked, so a tree must
not become visible without one: _init_vosk would treat it as installed
while install.sh, finding no stamp, refetched it on every run. Unpacking
into a staging directory and renaming afterwards makes "extracted" and
"stamped" a single step, the way install_vosk_models already does it, and
keeps whatever was already there until the replacement is complete.
"""
import shutil
os.makedirs(cache_dir, exist_ok=True)
staging, replaced = self._vosk_scratch_paths(model_name)
shutil.rmtree(staging, ignore_errors=True)
shutil.rmtree(replaced, ignore_errors=True)
os.makedirs(staging, exist_ok=True)
logger.info(f"Downloading Whisper {self.model_size} model to {model_file}")
logger.info(f"Downloading from {url}")
with zipfile.ZipFile(zip_path, "r") as zip_ref:
zip_ref.extractall(staging)
extracted = os.path.join(staging, model_name)
if not os.path.isdir(extracted):
raise RuntimeError(f"{model_name}.zip does not contain a {model_name} directory")
write_verification_stamp(extracted, f"{model_name}.zip")
# Same filesystem, so both renames are cheap; the old tree is parked
# rather than deleted, so a failed swap can put it back.
if os.path.isdir(model_path):
os.rename(model_path, replaced)
try:
response = requests.get(url, stream=True)
response.raise_for_status()
total_size = int(response.headers.get("content-length", 0))
downloaded_size = 0
start_time = time.time()
last_update_time = start_time
chunk_size = 8192 # 8KB chunks
with open(temp_file, "wb") as f:
for data in response.iter_content(chunk_size=chunk_size):
if self._download_cancelled:
logger.info("Download cancelled by user")
f.close()
if os.path.exists(temp_file):
os.remove(temp_file)
raise RuntimeError("Download cancelled")
f.write(data)
downloaded_size += len(data)
# Update progress callback
current_time = time.time()
if (
self._download_progress_callback
and (current_time - last_update_time) >= 0.1
):
elapsed = current_time - start_time
if elapsed > 0:
speed_mbps = (downloaded_size / (1024 * 1024)) / elapsed
else:
speed_mbps = 0
if total_size > 0:
progress = downloaded_size / total_size
remaining_mb = (total_size - downloaded_size) / (1024 * 1024)
if speed_mbps > 0:
eta_seconds = remaining_mb / speed_mbps
eta_str = (
f"{int(eta_seconds)}s"
if eta_seconds < 60
else f"{int(eta_seconds / 60)}m {int(eta_seconds % 60)}s"
)
else:
eta_str = "--"
status = f"{downloaded_size / (1024 * 1024):.1f} / {total_size / (1024 * 1024):.1f} MB • {speed_mbps:.1f} MB/s • ETA: {eta_str}"
else:
progress = 0
status = (
f"{downloaded_size / (1024 * 1024):.1f} MB • {speed_mbps:.1f} MB/s"
)
self._download_progress_callback(progress, speed_mbps, status)
last_update_time = current_time
logger.info(f"Download progress: {progress * 100:.1f}% - {status}")
# Rename temp file to final
os.rename(temp_file, model_file)
logger.info("Whisper model downloaded successfully")
if self._download_progress_callback:
self._download_progress_callback(1.0, 0, "Complete!")
except requests.exceptions.RequestException as e:
logger.error(f"Failed to download Whisper model from {url}: {e}")
if os.path.exists(temp_file):
os.remove(temp_file)
raise RuntimeError(f"Failed to download Whisper model: {e}") from e
except (OSError, RuntimeError, ValueError) as e:
logger.error(f"An error occurred during Whisper model download: {e}")
if os.path.exists(temp_file):
os.remove(temp_file)
os.rename(extracted, model_path)
except OSError:
if os.path.isdir(replaced):
os.rename(replaced, model_path)
raise
shutil.rmtree(replaced, ignore_errors=True)
shutil.rmtree(staging, ignore_errors=True)
def register_text_callback(self, callback: Callable[[str], None]):
"""
src/vocalinux/ui/settings_dialog.py
+28 -10
View File
@@ -50,6 +50,10 @@ from ..utils.vosk_model_info import ( # noqa: E402
list_downloaded_vosk_models,
vosk_model_dirname,
)
from ..utils.whisper_model_info import ( # noqa: E402
migrate_legacy_checkpoint_names,
whisper_model_file,
)
from ..utils.whispercpp_model_info import MODEL_SIZES as WHISPERCPP_MODEL_SIZES
from ..utils.whispercpp_model_info import (
WHISPERCPP_MODEL_INFO,
@@ -841,7 +845,7 @@ def _whisper_model_files(model_name: str) -> list[str]:
if model_name not in WHISPER_MODEL_INFO:
return []
filename = f"{model_name}.pt"
filename = whisper_model_file(model_name)
candidates = [
os.path.join(_get_whisper_cache_dir(), filename),
os.path.join(os.path.expanduser("~/.cache/whisper"), filename),
@@ -867,18 +871,32 @@ def _whisper_model_files(model_name: str) -> list[str]:
def _is_whisper_model_downloaded(model_name: str) -> bool:
"""Check if a Whisper model is downloaded."""
cache_dir = _get_whisper_cache_dir()
model_file = os.path.join(cache_dir, f"{model_name}.pt")
if os.path.exists(model_file):
return True
# Also check default whisper cache
default_cache = os.path.expanduser("~/.cache/whisper")
return os.path.exists(os.path.join(default_cache, f"{model_name}.pt"))
"""Check if a Whisper model is downloaded.
Only the directory the engine passes to ``whisper.load_model`` as
``download_root`` counts. A copy in the default ~/.cache/whisper is not used
by the engine, so reporting it as downloaded would promise the user a model
that selecting it then spends up to 2.9GB fetching.
That has a corollary worth stating plainly, because this function is what
``_list_downloaded_whisper_models`` filters on and the dialog can only delete
what it lists: deleting a model here also removes a copy in
~/.cache/whisper, since ``_whisper_model_files`` returns both, but a model
that exists *only* there is not listed and so cannot be reclaimed from this
dialog at all.
"""
model_file = os.path.join(_get_whisper_cache_dir(), whisper_model_file(model_name))
return os.path.exists(model_file)
def _list_downloaded_whisper_models() -> list[str]:
"""Return OpenAI Whisper catalog names that are present on disk."""
"""Return OpenAI Whisper catalog names that are present on disk.
Checkpoints an earlier release stored under the catalog name are renamed
first, so a "large" downloaded back then is listed here (and can therefore be
deleted) instead of sitting on disk unreachable from this dialog.
"""
migrate_legacy_checkpoint_names(_get_whisper_cache_dir())
return [name for name in WHISPER_MODEL_INFO if _is_whisper_model_downloaded(name)]
src/vocalinux/utils/model_checksums.py
+224
View File
@@ -0,0 +1,224 @@
"""Integrity verification for downloaded speech models.
Vocalinux pulls 40MB-2GB model files from Hugging Face, Alphacephei and
openaipublic.azureedge.net, then feeds them to native code -- whisper.cpp maps
its ggml files through ``ctypes.cdll``. A tampered or truncated model is
therefore not merely a bad transcription, so every download is checked against a
digest pinned in this repository before it is moved into place, and a model that
is merely found on disk is hashed before it is handed to native code.
One source of truth, consulted without a network lookup: ``model_checksums.txt``
next to this module pins a sha256 per file name, for whisper.cpp (the
``lfs.oid`` at the pinned Hugging Face revision) and for VOSK (the hash of the
bytes the generator downloaded; the md5 Alphacephei publishes beside a model
pins nothing they could not change alongside it). Regenerate it with
``scripts/generate-model-checksums.py``.
OpenAI Whisper ``.pt`` checkpoints are absent from it: ``openai-whisper``
downloads and verifies those itself, against the sha256 it reads out of the URL
path, so Vocalinux neither fetches nor hashes them. ``install.sh`` does preinstall
the tiny checkpoint, and verifies it there with its own bash implementation.
Verification fails closed: a file with no pinned digest is an error, not a
warning. ``tests/test_model_checksums.py`` asserts the manifest covers every
model the application can download, so that strictness is caught in CI rather
than on a user's machine.
"""
from __future__ import annotations
import hashlib
import logging
import os
import re
from functools import lru_cache
from typing import Dict, NamedTuple, Optional
logger = logging.getLogger(__name__)
_MANIFEST_NAME = "model_checksums.txt"
# Written into an extracted model tree to record the digest of the archive it
# came out of. A directory has no hash of its own and the zip is deleted after
# unpacking, so this file is the only evidence the tree was ever verified.
# install.sh reads it under the same name; keep the two in step.
VERIFICATION_STAMP_NAME = ".vocalinux_verified"
# Hash of a 2GB model read in one go would sit in RAM; stream it instead.
_CHUNK_SIZE = 1024 * 1024
_REVISION_HEADER = re.compile(r"^#\s*whispercpp-revision:\s*([0-9a-f]{7,40})\s*$")
class ChecksumError(Exception):
"""A downloaded model did not match its pinned digest, or has no pin at all."""
class Expected(NamedTuple):
"""The pinned identity of one artifact."""
algo: str
digest: str
size: int
def _manifest_text() -> str:
"""Read the manifest from the installed package, falling back to the source tree.
A missing manifest yields an empty one rather than an import-time crash: the
consequence is that every model then lacks a pin and is refused by
:func:`verify_model_file`, which is the safe direction to fail.
"""
try:
from importlib.resources import files
return files(__package__).joinpath(_MANIFEST_NAME).read_text(encoding="utf-8")
except (ImportError, TypeError, ModuleNotFoundError):
pass
except (FileNotFoundError, OSError):
return ""
path = os.path.join(os.path.dirname(os.path.abspath(__file__)), _MANIFEST_NAME)
try:
with open(path, encoding="utf-8") as handle:
return handle.read()
except OSError:
logger.error("%s is missing; no model download can be verified", _MANIFEST_NAME)
return ""
@lru_cache(maxsize=1)
def _parse_manifest() -> tuple[Dict[str, Expected], str]:
entries: Dict[str, Expected] = {}
revision = ""
for line in _manifest_text().splitlines():
stripped = line.strip()
if not stripped:
continue
if stripped.startswith("#"):
match = _REVISION_HEADER.match(stripped)
if match:
revision = match.group(1)
continue
fields = stripped.split()
if len(fields) != 4:
logger.warning("Ignoring malformed line in %s: %r", _MANIFEST_NAME, line)
continue
filename, algo, digest, size = fields
entries[filename] = Expected(algo=algo, digest=digest.lower(), size=int(size))
return entries, revision
#: Branch to build URLs from when the manifest pins no revision. Only reachable
#: with a missing or hand-damaged manifest, and deliberately not fatal: URL
#: construction runs at import time, whereas refusing an unverifiable file is
#: :func:`verify_model_file`'s job. A download made from this fallback still has
#: no pinned digest, so it is rejected before it is installed.
_UNPINNED_REVISION = "main"
def whispercpp_revision() -> str:
"""The pinned Hugging Face commit the whisper.cpp digests were taken at."""
_, revision = _parse_manifest()
if not revision:
logger.warning(
"%s pins no whisper.cpp revision; falling back to %r. Downloads will "
"fail verification until it is regenerated with "
"scripts/generate-model-checksums.py.",
_MANIFEST_NAME,
_UNPINNED_REVISION,
)
return _UNPINNED_REVISION
return revision
def expected_for(filename: str) -> Optional[Expected]:
"""Return the pinned digest for a model file name, or None when unpinned."""
entries, _ = _parse_manifest()
return entries.get(os.path.basename(filename))
def write_verification_stamp(tree_path: str, archive_name: str) -> None:
"""Record inside ``tree_path`` the pinned digest of the archive it came from.
Only archives get a digest, so an extracted tree is trustworthy later only if
something writes down what was checked. ``install.sh`` re-downloads any VOSK
model whose stamp does not match the digest pinned today, which means a tree
left unstamped here — a first-run or Settings download — is refetched on the
next ``./install.sh``. That makes a failed write an error, not a warning.
"""
expected = expected_for(archive_name)
if expected is None:
raise ChecksumError(f"Cannot stamp {tree_path}: {archive_name} has no pinned digest")
stamp = os.path.join(tree_path, VERIFICATION_STAMP_NAME)
with open(stamp, "w", encoding="utf-8") as handle:
handle.write(f"{expected.digest}\n")
logger.debug("Recorded verified digest for %s in %s", archive_name, stamp)
def pinned_filenames() -> frozenset:
"""Every file name the manifest pins. Used by the coverage test."""
entries, _ = _parse_manifest()
return frozenset(entries)
def file_digest(path: str, algo: str) -> str:
"""Stream ``path`` through ``algo`` and return the lowercase hex digest."""
try:
digest = hashlib.new(algo)
except ValueError as exc:
raise ChecksumError(f"Unsupported checksum algorithm {algo!r}") from exc
with open(path, "rb") as handle:
for chunk in iter(lambda: handle.read(_CHUNK_SIZE), b""):
digest.update(chunk)
return digest.hexdigest()
def verify_file(path: str, expected: Expected, *, description: str = "") -> None:
"""Raise :class:`ChecksumError` unless ``path`` matches ``expected``.
The size is checked first: it is free, and a truncated download (the common
case) is then reported as a truncated download rather than as a hash
mismatch that reads like tampering.
"""
label = description or os.path.basename(path)
if expected.size:
actual_size = os.path.getsize(path)
if actual_size != expected.size:
raise ChecksumError(
f"{label} is {actual_size} bytes, expected {expected.size}. "
"The download was truncated or the file was replaced upstream."
)
actual = file_digest(path, expected.algo)
if actual != expected.digest:
raise ChecksumError(
f"{label} failed {expected.algo} verification.\n"
f" expected: {expected.digest}\n"
f" actual: {actual}\n"
"The file does not match the digest pinned in this release. Delete it "
"and retry; if it keeps failing, do not use it."
)
logger.info("%s verified (%s)", label, expected.algo)
def verify_model_file(path: str, filename: Optional[str] = None) -> None:
"""Verify a downloaded model against the manifest.
``filename`` overrides the lookup key when ``path`` is still a temporary
name such as ``ggml-tiny.bin.tmp``.
"""
key = filename or os.path.basename(path)
expected = expected_for(key)
if expected is None:
raise ChecksumError(
f"No checksum is pinned for {key}. Regenerate "
f"{_MANIFEST_NAME} with scripts/generate-model-checksums.py."
)
verify_file(path, expected, description=key)
src/vocalinux/utils/model_checksums.txt
+86
View File
@@ -0,0 +1,86 @@
# Checksums for every speech model Vocalinux downloads.
#
# Generated by scripts/generate-model-checksums.py -- do not edit by hand.
# Regenerate with `just model-checksums` after adding a model to
# vosk_model_info.py or whispercpp_model_info.py.
#
# Read by src/vocalinux/utils/model_checksums.py and by install.sh; keep the
# format (filename, algorithm, digest, size in bytes) parseable by awk.
#
# whisper.cpp models are pinned to this Hugging Face commit so upstream
# replacing a file cannot invalidate the digests below.
# whispercpp-revision: 5359861c739e955e79d9a303bcbc70fb988958b1
#
# OpenAI Whisper (.pt) models are absent on purpose: their download URLs embed
# the sha256 as a path segment, so the expected digest travels with the URL.
#
# VOSK digests are the sha256 of bytes this script downloaded, not the md5
# Alphacephei publishes; that md5 is only checked against them at that point.
ggml-base-q5_1.bin sha256 422f1ae452ade6f30a004d7e5c6a43195e4433bc370bf23fac9cc591f01a8898 59707625
ggml-base-q8_0.bin sha256 c577b9a86e7e048a0b7eada054f4dd79a56bbfa911fbdacf900ac5b567cbb7d9 81768585
ggml-base.bin sha256 60ed5bc3dd14eea856493d334349b405782ddcaf0028d4b5df4088345fba2efe 147951465
ggml-base.en-q5_1.bin sha256 4baf70dd0d7c4247ba2b81fafd9c01005ac77c2f9ef064e00dcf195d0e2fdd2f 59721011
ggml-base.en.bin sha256 a03779c86df3323075f5e796cb2ce5029f00ec8869eee3fdfb897afe36c6d002 147964211
ggml-large-v1.bin sha256 7d99f41a10525d0206bddadd86760181fa920438b6b33237e3118ff6c83bb53d 3094623691
ggml-large-v2-q5_0.bin sha256 3a214837221e4530dbc1fe8d734f302af393eb30bd0ed046042ebf4baf70f6f2 1080732091
ggml-large-v2-q8_0.bin sha256 fef54e6d898246a65c8285bfa83bd1807e27fadf54d5d4e81754c47634737e8c 1656129691
ggml-large-v2.bin sha256 9a423fe4d40c82774b6af34115b8b935f34152246eb19e80e376071d3f999487 3094623691
ggml-large-v3-q5_0.bin sha256 d75795ecff3f83b5faa89d1900604ad8c780abd5739fae406de19f23ecd98ad1 1081140203
ggml-large-v3-turbo-q5_0.bin sha256 394221709cd5ad1f40c46e6031ca61bce88931e6e088c188294c6d5a55ffa7e2 574041195
ggml-large-v3-turbo-q8_0.bin sha256 317eb69c11673c9de1e1f0d459b253999804ec71ac4c23c17ecf5fbe24e259a1 874188075
ggml-large-v3-turbo.bin sha256 1fc70f774d38eb169993ac391eea357ef47c88757ef72ee5943879b7e8e2bc69 1624555275
ggml-large-v3.bin sha256 64d182b440b98d5203c4f9bd541544d84c605196c4f7b845dfa11fb23594d1e2 3095033483
ggml-medium-q5_0.bin sha256 19fea4b380c3a618ec4723c3eef2eb785ffba0d0538cf43f8f235e7b3b34220f 539212467
ggml-medium-q8_0.bin sha256 42a1ffcbe4167d224232443396968db4d02d4e8e87e213d3ee2e03095dea6502 823369779
ggml-medium.bin sha256 6c14d5adee5f86394037b4e4e8b59f1673b6cee10e3cf0b11bbdbee79c156208 1533763059
ggml-medium.en-q5_0.bin sha256 76733e26ad8fe1c7a5bf7531a9d41917b2adc0f20f2e4f5531688a8c6cd88eb0 539225533
ggml-medium.en.bin sha256 cc37e93478338ec7700281a7ac30a10128929eb8f427dda2e865faa8f6da4356 1533774781
ggml-small-q5_1.bin sha256 ae85e4a935d7a567bd102fe55afc16bb595bdb618e11b2fc7591bc08120411bb 190085487
ggml-small-q8_0.bin sha256 49c8fb02b65e6049d5fa6c04f81f53b867b5ec9540406812c643f177317f779f 264464607
ggml-small.bin sha256 1be3a9b2063867b937e64e2ec7483364a79917e157fa98c5d94b5c1fffea987b 487601967
ggml-small.en-q5_1.bin sha256 bfdff4894dcb76bbf647d56263ea2a96645423f1669176f4844a1bf8e478ad30 190098681
ggml-small.en.bin sha256 c6138d6d58ecc8322097e0f987c32f1be8bb0a18532a3f88f734d1bbf9c41e5d 487614201
ggml-tiny-q5_1.bin sha256 818710568da3ca15689e31a743197b520007872ff9576237bda97bd1b469c3d7 32152673
ggml-tiny-q8_0.bin sha256 c2085835d3f50733e2ff6e4b41ae8a2b8d8110461e18821b09a15c40c42d1cca 43537433
ggml-tiny.bin sha256 be07e048e1e599ad46341c8d2a135645097a538221678b7acdd1b1919c6e1b21 77691713
ggml-tiny.en-q5_1.bin sha256 c77c5766f1cef09b6b7d47f21b546cbddd4157886b3b5d6d4f709e91e66c7c2b 32166155
ggml-tiny.en.bin sha256 921e4cf8686fdd993dcd081a5da5b6c365bfde1162e72b08d75ac75289920b1f 77704715
vosk-model-ar-0.22-linto-1.1.0.zip sha256 6fc103417517203a5503e176963e3a7e20d32da4c37f4ff21181e9d64e1f02bf 1377395170
vosk-model-cn-0.22.zip sha256 7f5580bf7ca3d9e8ce8be68337378950ed78ef974df7a7d4ead8dbe32bec2fa1 1358736686
vosk-model-de-0.21.zip sha256 245060756f8d8394fc5b13639cf220b7620205795f30f37b6823878d4f603b2a 2031717803
vosk-model-en-in-0.5.zip sha256 0365c52f79308d81c2de832cf934ce42da6bf7e04dd3acd4203cacf50ab70f5c 971186187
vosk-model-en-us-0.22.zip sha256 47f9a81ebb039dbb0bd319175c36ac393c0893b796c2b6303e64cf58c27b69f6 1913365522
vosk-model-es-0.42.zip sha256 bbbfaf9239bed5dcb0d140886b5834fcc13c13c7fb2f2799ae790f03ff20bfce 1484681703
vosk-model-fa-0.42.zip sha256 5f03187838a187cd3f8766e3d329da37bf5ddd81a861081132743b76d8a8d14a 1667089770
vosk-model-fr-0.22.zip sha256 b26c34e5ea1b9f9dee6cef5e910a1d93e9adb60f1a0a3f85c069976aec481e46 1523026348
vosk-model-hi-0.22.zip sha256 eaf9bdbd703c79e957f7ea893502e12b336f98d7cd420138bcb224b36ccf86ac 1561909274
vosk-model-it-0.22.zip sha256 5cdc9f0f596e3db270b92fc4c1fd0f1a3e3d5d19ae0d30b69fc008639257de09 1306765292
vosk-model-ja-0.22.zip sha256 af620db815ef716f90355f911092ba0c7454d91d793ecdd8dee2df6342e0e9cc 1045975323
vosk-model-nl-spraakherkenning-0.6.zip sha256 088d40014c43684d65f3cc15f4b06ad764b4407b47854812e8364427702c8b56 1004206965
vosk-model-pt-fb-v0.1.1-pruned.zip sha256 f197db8d66a53e1ae4332dc599b4c035271a0ffe0c6ac0636f341184cea8d830 1223922169
vosk-model-ru-0.22.zip sha256 84d9e2cf6324480e4e768da7d75122fd1250df5449b89d8f9d9417ec7a27fb23 1559419132
vosk-model-small-ar-0.3.zip sha256 8696c91f39ec67a08c513845cfac26dbd4e81937be4fba25233e6c5435c18ca9 104351896
vosk-model-small-ca-0.4.zip sha256 99f90bfff5c2b187c705ddbc20aaa2600eddc359466f404404f1db6029fba5d4 43405881
vosk-model-small-cn-0.22.zip sha256 3af8b0e7e0f835ae9d414ce5df580237a3cfb08d586c9fbbb0f7ff29ad5b14ba 43898754
vosk-model-small-cs-0.4-rhasspy.zip sha256 287c3bbefc8ad67b4ab9636eecef3d62acc3719990777d03e226db5a7f19fbda 46088666
vosk-model-small-de-0.15.zip sha256 b7e53c90b1f0a38456f4cd62b366ecd58803cd97cd42b06438e2c131713d5e43 46499967
vosk-model-small-en-in-0.4.zip sha256 20663dcac4d5cb783a579c54d98339344a688e4ec6e1b4a4b059fd1235454cc7 37573330
vosk-model-small-en-us-0.15.zip sha256 30f26242c4eb449f948e42cb302dd7a686cb29a3423a8367f99ff41780942498 41205931
vosk-model-small-es-0.42.zip sha256 09b239888f633ef2f0b4e09736e3d9936acfd810bc65d53fad45261762c6511f 39817833
vosk-model-small-fa-0.42.zip sha256 977cb5faa538f3a835ccfd35f5f6d8284b5c450b89c700b9bd4736b66536ad46 53431220
vosk-model-small-fr-0.22.zip sha256 cabf6180e177eb9b3a9a9d43a437bd5e549f3a7d09525e5d69a3fed787be12ad 42233323
vosk-model-small-hi-0.22.zip sha256 7c50a10866889f0ac21d912c20537a055a597ed09fc1d3e5bcd798f9f0017e48 44458845
vosk-model-small-it-0.22.zip sha256 9ec65e75861d1c6c2e457cccd932705340dcdf233f5b239f00733b4de0bf3267 49665141
vosk-model-small-ja-0.22.zip sha256 efa092d280153a77615e9e0c7d7283e93e600de3d19d3bec686c57ef19d52eac 49704573
vosk-model-small-ko-0.22.zip sha256 eea36124087fed26c59996a4761519458e3bd185e8ea9d9865ad8760c4a1d989 86914329
vosk-model-small-nl-0.22.zip sha256 039811c3b829de64e4f123a9f684a53784005b212a346ac0b899dc7efce2ed0a 40441176
vosk-model-small-pl-0.22.zip sha256 c4cd16498ea544f446f9e9a55cbd602b71cfe5a2b6f2b0834d81e1b6fce15f0d 52979372
vosk-model-small-pt-0.3.zip sha256 6e1ce909032e1afa7a88e68a3d628ecafff302bdf195befab308826c395e93b7 32453112
vosk-model-small-ru-0.22.zip sha256 961d5ff98a17f4aa6de69864d0aa71fa5bac682301d2b5d17a3f24c5c99a46d4 46236750
vosk-model-small-sv-rhasspy-0.15.zip sha256 5b54f2ca8cacf9766588d05f6c8ca7e8921f054f050726e29d3516e78aefe054 303504931
vosk-model-small-tr-0.3.zip sha256 8c8d07cec1bce31add14967c16891c84152cdf76d391e33c08278119b9ea96e5 36855784
vosk-model-small-uk-v3-small.zip sha256 ee05b6a8c796c53c61cd1c821d38e897e5bd67b4d691d8797ec8f9e5100d4247 143914407
vosk-model-small-vn-0.4.zip sha256 efe5c8494212110471a79befc48c79da679e5b1fc52a4ffb500222ff86d622e5 33656337
vosk-model-uk-v3.zip sha256 b1a0dbb9a19bfb6cdbad5bb4d43b71281ab57ad4a8c2a372656c83b994b72b1f 371048965
vosk-model-vn-0.4.zip sha256 a8ce6ee52e369b8f23f56e87e037db10322e6d94779b08d5e1ab21cb4b737380 74423562
src/vocalinux/utils/vosk_model_info.py
+2 -2
View File
@@ -242,7 +242,7 @@ VOSK_MODEL_INFO = {
"ko": "vosk-model-small-ko-0.22",
"fa": "vosk-model-fa-0.42",
"pl": "vosk-model-small-pl-0.22",
"pt": "vosk-model-pt-0.4",
"pt": "vosk-model-pt-fb-v0.1.1-pruned",
"ru": "vosk-model-ru-0.22",
"es": "vosk-model-es-0.42",
"sv": "vosk-model-small-sv-rhasspy-0.15",
@@ -270,7 +270,7 @@ VOSK_MODEL_INFO = {
"ko": "vosk-model-small-ko-0.22",
"fa": "vosk-model-fa-0.42",
"pl": "vosk-model-small-pl-0.22",
"pt": "vosk-model-pt-0.4",
"pt": "vosk-model-pt-fb-v0.1.1-pruned",
"ru": "vosk-model-ru-0.22",
"es": "vosk-model-es-0.42",
"sv": "vosk-model-small-sv-rhasspy-0.15",
src/vocalinux/utils/whisper_model_info.py
+79
View File
@@ -0,0 +1,79 @@
"""Download URLs and file names for OpenAI Whisper checkpoints.
The file name is derived from the URL rather than from the catalog name, because
that is what `openai-whisper` does. Deriving it as f"{size}.pt" instead is what
made "large" (stored as large-v3.pt) download twice and never show as present.
"""
from __future__ import annotations
import logging
import os
logger = logging.getLogger(__name__)
_BASE = "https://openaipublic.azureedge.net/main/whisper/models"
#: Catalog name -> upstream URL. The path segment before the file name is the
#: checkpoint's sha256; openai-whisper verifies against it on load.
WHISPER_MODEL_URLS = {
"tiny": f"{_BASE}/65147644a518d12f04e32d6f3b26facc3f8dd46e5390956a9424a650c0ce22b9/tiny.pt",
"base": f"{_BASE}/ed3a0b6b1c0edf879ad9b11b1af5a0e6ab5db9205f891f668f8b0e6c6326e34e/base.pt",
"small": f"{_BASE}/9ecf779972d90ba49c06d968637d720dd632c55bbf19d441fb42bf17a411e794/small.pt",
"medium": f"{_BASE}/345ae4da62f9b3d59415adc60127b97c714f32e89e936602e85993674d08dcb1/medium.pt",
"large": f"{_BASE}/e5b1a55b89c1367dacf97e3e19bfd829a01529dbfdeefa8caeb59b3f1b81dadb/large-v3.pt",
}
#: Catalog names the UI offers for the OpenAI Whisper engine.
WHISPER_MODEL_SIZES = list(WHISPER_MODEL_URLS)
def whisper_model_url(model_size: str) -> str:
"""Return the upstream URL for ``model_size``."""
try:
return WHISPER_MODEL_URLS[model_size]
except KeyError:
raise ValueError(f"Unknown Whisper model size: {model_size}") from None
def whisper_model_file(model_size: str) -> str:
"""Return the file name ``openai-whisper`` stores for ``model_size``."""
return os.path.basename(whisper_model_url(model_size))
def migrate_legacy_checkpoint_names(cache_dir: str) -> list[str]:
"""Rename checkpoints saved under the old f"{size}.pt" name. Returns the new names.
Vocalinux's own downloader used to name every checkpoint after the catalog
entry, which is wrong wherever upstream publishes a versioned file: "large"
is stored as large-v3.pt. A file left under the old name is invisible three
times over. load_model() looks for the upstream name and refetches 2.9GB,
Settings does not list it, and because it is not listed it cannot be deleted
from there either, so the disk stays occupied with no way to reclaim it from
the UI.
Renaming is safe rather than lucky: the old downloader fetched "large" from
the same large-v3 URL it uses today, so the bytes already are that
checkpoint. And ``openai-whisper`` verifies whatever it loads against the
sha256 in the URL, so even a wrong guess here costs at most the refetch that
would have happened anyway.
"""
renamed = []
for size in WHISPER_MODEL_URLS:
upstream = whisper_model_file(size)
legacy = f"{size}.pt"
if legacy == upstream:
continue
legacy_path = os.path.join(cache_dir, legacy)
upstream_path = os.path.join(cache_dir, upstream)
if not os.path.isfile(legacy_path) or os.path.exists(upstream_path):
continue
try:
os.rename(legacy_path, upstream_path)
except OSError as error:
# Not fatal: the only cost is the refetch this would have avoided.
logger.warning("Could not rename %s to %s: %s", legacy_path, upstream, error)
continue
logger.info("Renamed legacy Whisper checkpoint %s to %s", legacy, upstream)
renamed.append(upstream)
return renamed
src/vocalinux/utils/whispercpp_model_info.py
+18 -4
View File
@@ -12,19 +12,33 @@ import subprocess
from functools import lru_cache
from typing import Optional
from .model_checksums import whispercpp_revision
from .paths import is_within_directory, models_dir
logger = logging.getLogger(__name__)
# Whisper.cpp model information
# Models are downloaded from Hugging Face (ggml format)
_WHISPERCPP_REPO_URL = "https://huggingface.co/ggerganov/whisper.cpp/resolve/main"
# Models are downloaded from Hugging Face (ggml format).
#
# The revision is pinned rather than tracking `main`: model_checksums.txt holds a
# sha256 per file, and a file replaced upstream would turn every download into a
# checksum failure. Both the pin and the digests are refreshed together by
# scripts/generate-model-checksums.py.
_WHISPERCPP_REPO = "https://huggingface.co/ggerganov/whisper.cpp/resolve"
def whispercpp_model_file(model_name: str) -> str:
"""Return the ggml file name upstream publishes for ``model_name``.
"large" is an alias the UI offers; upstream only ships the versioned file.
"""
file_model_name = "large-v3" if model_name == "large" else model_name
return f"ggml-{file_model_name}.bin"
def _model_url(model_name: str) -> str:
"""Build the Hugging Face URL for a ggml whisper.cpp model."""
file_model_name = "large-v3" if model_name == "large" else model_name
return f"{_WHISPERCPP_REPO_URL}/ggml-{file_model_name}.bin"
return f"{_WHISPERCPP_REPO}/{whispercpp_revision()}/{whispercpp_model_file(model_name)}"
_WHISPERCPP_MODEL_SPECS = [