Files
voculinux/src/vocalinux/custom_dictionary.py
T
dca937fc21 feat(dictionary): combined custom dictionary with corrections (#890)
* feat(dictionary): combined custom dictionary with corrections

Rebased onto latest main (conflict-resolution squash for PR #818).

* fix(dictionary): stabilize the terms and corrections file contracts

- corrections_path() stays at the fixed config_dir() location promised by the file contract instead of following a custom or --dictionary-file terms path, which could strand or duplicate corrections; a leftover file beside the terms file is still read once and copied over.

- _atomic_write() preserves the replaced file's permission mode instead of resetting it to the process umask.

- build_initial_prompt() streams terms and stops at the configured cap instead of reading and processing the whole file for every segment.

* fix(settings): restore the selected dictionary pane after a search

Searching can auto-switch the Custom Dictionary management pane to reveal a match, but clearing the query only restored widget visibility. Record the pane selection in the search baseline snapshot and restore it alongside the previous page so users return to the pane they chose.

* fix(dictionary): close the remaining custom dictionary file-contract gaps

- A --dictionary-file session override no longer imports a corrections
  file found beside the override into the standard config directory;
  the override scopes to terms only.
- Terms reads validate the whole file before yielding: corrupt UTF-8
  mid-file now yields no terms at all, matching the invalid status
  Settings reports instead of leaking a partial dictionary to
  recognition.
- The terms read is bounded by MAX_TERMS_FILE_BYTES (1 MiB); a larger
  file is invalid everywhere with a dedicated status instead of being
  read on every segment.

* fix(dictionary): keep usable terms prefix on oversized files and legacy corrections under overrides

* fix(dictionary): cap yielded terms and keep added terms visible on oversized files

* fix(dictionary): keep added terms visible without dropping scanner-owned lines

* fix(dictionary): keep oversized terms manageable and honor mid-search pane switches

- terms list gains a Show all / Show first 500 toggle; adding a term
  beyond the cap auto-expands so the new row renders with a Remove
  button instead of being invisible until the file is hand-edited
- a manual dictionary pane switch during an active search drops the
  saved pre-search pane so clearing the query no longer rolls back the
  user's newer selection
- annotate the new settings tests and type model_info so the parakeet
  branch stops adding fresh mypy errors

* fix(settings): do not auto-expand terms on add past display cap

Rebuilding up to thousands of term rows on the UI thread stalls the
dialog; the overflow button already shows the true count, so the added
term stays one click away.

* test(dictionary): stub segment_callbacks on bare manager for history dispatch

* fix(dictionary): stream oversized terms files when building the prompt

build_initial_prompt decoded the whole 1 MiB read window before every
transcription segment even though it only needs the first max_words
terms. _iter_terms_for_prompt now reads oversized files line by line and
stops at the prompt limit, keeping the same contract: complete leading
lines only, invalid UTF-8 inside the usable window yields nothing.

* fix(history): only the ended session's worker may extend its snippet

A late segment could extend the newest history entry regardless of which
session produced it, so a worker finishing after the next dictation
session leaked its text into that session's entry. Track the worker that
owned the ended session and require late segments to arrive on the same
thread before merging; anything else forms its own snippet.

* fix(speech): transcribe queued segments the reload worker never drained

The recognition worker can exit during stop_recognition's bounded
audio-thread join, so the released key's final buffer lands in the queue
after the worker is gone and was silently dropped when the reload
teardown replaced the queue. Drain whatever remains before swapping it
— except on capture failure or an explicit cancel, which still discard
the session's leftovers.

* fix(settings): build term rows in idle slices and pin hidden additions

Two problems shared one cause. Rebuilding up to 4096 rows synchronously
stalled the dialog, and a term appended past the 500-row cap reported
success while staying invisible. Rows now build in bounded GLib idle
slices guarded by a refresh token, and a saved term outside the visible
window is pinned to the top of the capped list until it disappears or
becomes visible on its own. Adding under "show all" scrolls to the new
term instead.

* test(engine): annotate new test method return types

* fix(ci): match a candidate row that leads the latest track block

snapcraft status normally prints candidate as a continuation line under
the latest track (channel in column 1). If candidate ever leads the
track block it rides the track line instead (channel in column 3), and
the resolver missed it and refused to promote. Match both layouts in
the resolve and the pre-release re-verify.

* fix(dictionary): bound prompt reads and scroll to landed terms

- _iter_terms_for_prompt now reads each line via readline() capped by the
  remaining byte window, so an overlong line can never pull more than
  MAX_TERMS_FILE_BYTES into memory
- the prompt path keeps decoding every complete line inside the usable
  window after the term limit is collected, so invalid UTF-8 anywhere in
  the window still yields no terms and matches the reader's invalid status
- under "show all" Settings scrolls to the index where an added term
  landed (prepended or appended), verified against the row's label,
  instead of always scrolling to the end

* fix(history,reload): keep late-segment ownership and drain stamped tuples

- Track session_workers_seen so a worker from an earlier session can
  never merge into a newer snippet while a session is open
- Unpack (segment, language) queue tuples in _transcribe_queued_segments;
  passing the tuple as audio raised TypeError and lost final speech
- Refuse 'delete that' when a manual pad edit blurred segment boundaries
- Translate term-row coordinates into the scrolled card's space so the
  scroll actually reaches rows below the fold
- Type _on_dictionary_pane_changed's varargs

* style: drop unneeded nonlocal for mutated worker set

---------

Co-authored-by: Graham <graham@thegeeky.ninja>
2026-09-30 02:33:49 -07:00

735 lines
32 KiB
Python

"""File-backed custom dictionary support for recognition bias and transcript fixes."""
import itertools
import json
import logging
import os
import re
import shutil
import unicodedata
import uuid
from pathlib import Path
from typing import TYPE_CHECKING, Any, Iterator, Optional
from .utils.paths import config_dir
if TYPE_CHECKING:
from .ui.config_manager import ConfigManager
logger = logging.getLogger(__name__)
TERMS_FILENAME = "dictionary.txt"
DEFAULT_TERMS_PATH = str(Path(config_dir()) / TERMS_FILENAME)
# Pre-XDG default persisted by older builds; keep as a configured path.
LEGACY_DEFAULT_TERMS_PATH = "~/.config/vocalinux/dictionary.txt"
CORRECTIONS_FILENAME = "custom-dictionary-corrections.json"
CORRECTIONS_VERSION = 1
CORRECTIONS_TOP_LEVEL_KEYS = frozenset({"version", "corrections"})
CORRECTIONS_ENTRY_KEYS = frozenset({"heard", "replacement"})
DEFAULT_MAX_TERMS = 200
MAX_TERM_CHARACTERS = 200
MAX_PROMPT_CHARACTERS = 2_000
MAX_CORRECTIONS = 500
MAX_CORRECTION_CHARACTERS = 500
# A terms file beyond this bound is invalid everywhere rather than read in
# part: the read itself stays bounded and no partial dictionary can reach
# recognition while Settings reports the file unusable.
MAX_TERMS_FILE_BYTES = 1_048_576
# Even within the byte bound a scanner-managed file can hold ~130k terms; the
# dictionary is a prompt vocabulary, so yields stop here — Settings would
# otherwise build a row per term.
MAX_TERMS_YIELDED = 4_096
class _DuplicateJsonKeyError(ValueError):
"""Raised when a JSON object contains the same key more than once."""
def _object_pairs_without_duplicates(pairs: list[tuple[Any, Any]]) -> dict[Any, Any]:
"""Return a dict, raising if any key repeats at this object level."""
payload: dict[Any, Any] = {}
for key, value in pairs:
if key in payload:
raise _DuplicateJsonKeyError(key)
payload[key] = value
return payload
def normalize_corrections(raw_entries: Any) -> list[dict[str, str]]:
"""Validate correction entries, retaining the first case-insensitive source.
Input order is meaningful for equal-length overlapping entries. Keeping the
first duplicate makes hand-edited JSON deterministic and agrees with the
order used when building the matching expression.
"""
if not isinstance(raw_entries, list):
return []
entries: list[dict[str, str]] = []
seen: set[str] = set()
for index, entry in enumerate(raw_entries):
if len(entries) >= MAX_CORRECTIONS:
logger.warning("Ignoring corrections after the supported limit of %d", MAX_CORRECTIONS)
break
if not isinstance(entry, dict):
logger.warning("Ignoring correction %d: expected an object", index)
continue
heard = entry.get("heard")
replacement = entry.get("replacement")
if not isinstance(heard, str) or not isinstance(replacement, str):
logger.warning("Ignoring correction %d: both fields must be strings", index)
continue
heard = unicodedata.normalize("NFC", heard.strip())
replacement = replacement.strip()
if not heard or not replacement:
logger.warning("Ignoring correction %d: both fields must be non-empty", index)
continue
if len(heard) > MAX_CORRECTION_CHARACTERS or len(replacement) > MAX_CORRECTION_CHARACTERS:
logger.warning("Ignoring correction %d: phrase is too long", index)
continue
normalized_heard = heard.casefold()
if normalized_heard in seen:
logger.warning("Ignoring duplicate correction source %r", heard)
continue
seen.add(normalized_heard)
entries.append({"heard": heard, "replacement": replacement})
return entries
def apply_corrections(text: str, entries: list[dict[str, str]]) -> str:
"""Replace configured phrases in *text* using the custom-dictionary policy.
Sources are matched case-insensitively only when neither adjacent character
is a Unicode word character. Replacements retain the exact user-provided
spelling. Longer source phrases take precedence; ties retain JSON order.
"""
if not text or not entries:
return text
normalized_text = unicodedata.normalize("NFC", text)
ordered = sorted(
entries,
key=lambda entry: (len(entry["heard"].split()), len(entry["heard"])),
reverse=True,
)
alternatives = "|".join(
f"(?P<correction_{index}>{re.escape(entry['heard'])})"
for index, entry in enumerate(ordered)
)
pattern = re.compile(
r"(?<![\w\u0300-\u036f])(?:" + alternatives + r")(?![\w\u0300-\u036f])",
re.IGNORECASE,
)
def replace(match: re.Match[str]) -> str:
"""Return the replacement associated with the matched alternative."""
for index, entry in enumerate(ordered):
if match.group(f"correction_{index}") is not None:
return entry["replacement"]
return match.group(0)
corrected = pattern.sub(replace, normalized_text)
if corrected != text:
logger.debug("Applied custom dictionary corrections to final transcript")
return corrected
class CustomDictionaryManager:
"""Read, write, and live-reload the two custom dictionary file contracts."""
def __init__(
self, config_manager: "ConfigManager", transient_terms_path: Optional[str] = None
) -> None:
self.config = config_manager
self._transient_terms_path = transient_terms_path
@property
def is_transient_terms(self) -> bool:
"""Return whether a CLI session-only terms file supersedes the saved file."""
return self._transient_terms_path is not None
def terms_enabled(self) -> bool:
"""Return whether custom terms should be offered to Whisper-family engines."""
return self.is_transient_terms or bool(self.config.get("dictionary", "enabled", False))
def set_terms_enabled(self, enabled: bool) -> bool:
"""Persist terms enablement, rolling back the in-memory setting on failure."""
if self.is_transient_terms:
logger.info("Ignoring saved terms enablement while a CLI override is active")
return False
old_value = self.config.get("dictionary", "enabled", False)
if not self.config.set("dictionary", "enabled", bool(enabled)):
return False
if self.config.save_config():
return True
self.config.set("dictionary", "enabled", old_value)
logger.warning("Could not save custom terms enablement; keeping previous setting")
return False
def terms_path_text(self) -> str:
"""Return the configured or session-only terms path without expansion.
A missing or blank saved path uses the live XDG default. Historical
leftover pre-XDG default strings are kept, not rewritten to XDG.
"""
if self._transient_terms_path is not None:
return self._transient_terms_path
default_path = str(Path(config_dir()) / TERMS_FILENAME)
configured = self.config.get("dictionary", "file_path", default_path)
if not isinstance(configured, str) or not configured.strip():
return default_path
configured = configured.strip()
if (
self._is_legacy_default_terms_path(configured, default_path)
and not self._file_path_is_explicit()
):
self._stamp_legacy_default_terms_path_explicit()
return configured
def _file_path_is_explicit(self) -> bool:
"""Return whether Settings persisted the current terms path on purpose."""
return bool(self.config.get("dictionary", "file_path_explicit", False))
@staticmethod
def _is_legacy_default_terms_path(configured: str, default_path: str) -> bool:
"""Return whether a persisted path matches the pre-XDG default string."""
if configured == LEGACY_DEFAULT_TERMS_PATH:
return True
try:
expanded_legacy = str(Path.home() / ".config" / "vocalinux" / TERMS_FILENAME)
except RuntimeError:
return False
return configured == expanded_legacy and expanded_legacy != default_path
def _stamp_legacy_default_terms_path_explicit(self) -> None:
"""Best-effort persist that a leftover pre-XDG path is an explicit choice.
Historical configs cannot distinguish leftover defaults from Settings
picks. Keep the stored path and mark it explicit so later reads stay
stable. Failures leave both the path and the marker unchanged.
"""
if self.is_transient_terms:
return
old_explicit = self.config.get("dictionary", "file_path_explicit", False)
if not self.config.set("dictionary", "file_path_explicit", True):
return
if self.config.save_config():
return
self.config.set("dictionary", "file_path_explicit", old_explicit)
logger.warning(
"Could not persist custom terms path explicit marker; keeping the configured path"
)
def terms_path(self) -> Optional[Path]:
"""Return the active expanded terms path, or None when it is invalid."""
configured = self.terms_path_text()
try:
return Path(configured).expanduser()
except RuntimeError as error:
logger.warning("Could not expand custom terms path %r: %s", configured, error)
return None
def _saved_terms_path(self) -> Optional[Path]:
"""Return the persisted terms path, ignoring any CLI override."""
default_path = str(Path(config_dir()) / TERMS_FILENAME)
configured = self.config.get("dictionary", "file_path", default_path)
if not isinstance(configured, str) or not configured.strip():
configured = default_path
try:
return Path(configured.strip()).expanduser()
except RuntimeError as error:
logger.warning("Could not expand saved terms path %r: %s", configured, error)
return None
def set_terms_path(self, path: str) -> bool:
"""Persist a usable terms path, retaining the prior setting on save failure."""
if self.is_transient_terms:
logger.info("Ignoring terms path change while a CLI override is active")
return False
if not isinstance(path, str) or not path.strip():
logger.warning("Ignoring empty custom terms path")
return False
configured = path.strip()
try:
candidate = Path(configured).expanduser()
if candidate.exists() and (not candidate.is_file() or not self._is_readable(candidate)):
logger.warning("Ignoring unusable custom terms path %s", candidate)
return False
except (OSError, RuntimeError) as error:
logger.warning("Ignoring invalid custom terms path %r: %s", configured, error)
return False
old_value = self.config.get(
"dictionary", "file_path", str(Path(config_dir()) / TERMS_FILENAME)
)
old_explicit = self.config.get("dictionary", "file_path_explicit", False)
if not self.config.set("dictionary", "file_path", configured):
return False
if not self.config.set("dictionary", "file_path_explicit", True):
self.config.set("dictionary", "file_path", old_value)
return False
if self.config.save_config():
return True
self.config.set("dictionary", "file_path", old_value)
self.config.set("dictionary", "file_path_explicit", old_explicit)
logger.warning("Could not save custom terms path; keeping previous setting")
return False
def corrections_path(self) -> Path:
"""Return the fixed structured corrections path.
Corrections always live under config_dir(), the location promised by
the file contract, so a custom or session-only terms path cannot
strand or duplicate them.
"""
return Path(config_dir()) / CORRECTIONS_FILENAME
def get_terms(self) -> list[str]:
"""Read the current UTF-8 line file; external edits apply next segment."""
path = self.terms_path()
if path is None:
return []
return list(self._iter_terms(path))
@staticmethod
def _iter_terms(path: Path) -> Iterator[str]:
"""Yield normalized, de-duplicated terms from a bounded read.
The bounded read validates the whole file when it fits the limit, so
a partly invalid file yields nothing and recognition never consumes
terms a status check would call invalid. Files past the limit still
supply their complete leading lines — a scanner-managed file can grow
past the bound without losing its usable prefix. Iterating over the
decoded lines still lets callers stop processing early.
"""
try:
with path.open("rb") as terms_file:
raw = terms_file.read(MAX_TERMS_FILE_BYTES + 1)
except FileNotFoundError:
return
except OSError as error:
logger.warning("Could not read custom terms file: %s", error)
return
if len(raw) > MAX_TERMS_FILE_BYTES:
logger.warning(
"Custom terms file exceeds %d bytes; reading complete leading lines only",
MAX_TERMS_FILE_BYTES,
)
# Keep the valid leading lines: a scanner-managed file can grow
# past the bound, but its usable prefix still supplies terms. A
# prefix with no complete line supplies nothing.
last_newline = raw.rfind(b"\n")
if last_newline == -1:
return
raw = raw[:last_newline]
try:
text = raw.decode("utf-8-sig")
except UnicodeError as error:
logger.warning("Could not read custom terms file: %s", error)
return
seen: set[str] = set()
for line in text.splitlines():
term = unicodedata.normalize("NFC", line.strip())
normalized_term = term.casefold()
if not term or term.startswith("#") or normalized_term in seen:
continue
seen.add(normalized_term)
yield term
if len(seen) >= MAX_TERMS_YIELDED:
logger.warning(
"Custom terms file supplies more than %d terms; trailing entries ignored",
MAX_TERMS_YIELDED,
)
return
def _iter_terms_for_prompt(self, path: Path, limit: int) -> Iterator[str]:
"""Yield up to *limit* terms, streaming oversized files line by line.
Files inside the read bound keep the same whole-file-validated
iteration as ``_iter_terms``. Past it — where the usable-prefix
policy already accepts complete leading lines — the prompt only
collects the first *limit* terms, but every complete line in the
bounded window is still decoded so an invalid file supplies no
terms at all. Each read is capped by the remaining window, so an
overlong line can never pull more than the byte bound into memory.
"""
if limit <= 0:
return
try:
oversized = path.stat().st_size > MAX_TERMS_FILE_BYTES
except OSError:
oversized = False
if not oversized:
yield from itertools.islice(self._iter_terms(path), limit)
return
terms: list[str] = []
seen: set[str] = set()
consumed = 0
try:
with path.open("rb") as terms_file:
index = 0
while True:
raw_line = terms_file.readline(MAX_TERMS_FILE_BYTES + 2 - consumed)
consumed += len(raw_line)
if (
not raw_line
or consumed > MAX_TERMS_FILE_BYTES + 1
or not raw_line.endswith(b"\n")
):
# Lines complete inside the byte bound are usable; a
# line straddling it is not a complete leading line.
break
try:
line = raw_line.decode("utf-8-sig" if index == 0 else "utf-8")
except UnicodeError as error:
logger.warning("Could not read custom terms file: %s", error)
return
index += 1
# Collection stops at the prompt limit, but every
# complete leading line is still decoded: the prompt
# fails closed on the same invalid files the terms
# reader rejects.
if len(terms) >= limit:
continue
term = unicodedata.normalize("NFC", line.strip())
normalized_term = term.casefold()
if not term or term.startswith("#") or normalized_term in seen:
continue
seen.add(normalized_term)
terms.append(term)
except FileNotFoundError:
return
except OSError as error:
logger.warning("Could not read custom terms file: %s", error)
return
yield from terms
def save_terms(self, terms: list[str]) -> bool:
"""Safely replace the standard terms file with normalized line entries."""
if self.is_transient_terms:
logger.info("Ignoring terms edit while a CLI override is active")
return False
path = self.terms_path()
if path is None:
logger.warning("Cannot save custom terms because the configured path is invalid")
return False
normalized_terms = self._normalize_terms(terms)
contents = "\n".join(normalized_terms)
if contents:
contents += "\n"
return self._atomic_write(path, contents)
def add_term(self, term: str) -> bool:
"""Add one valid term without dropping or rewriting unrelated lines."""
if self.is_transient_terms:
logger.info("Ignoring terms edit while a CLI override is active")
return False
cleaned = self._single_term(term)
if cleaned is None:
return False
path = self.terms_path()
if path is None:
logger.warning("Cannot add custom term because the configured path is invalid")
return False
contents = self._read_terms_contents(path, missing_value="")
if contents is None:
return False
existing_terms = self.get_terms()
if any(existing.casefold() == cleaned.casefold() for existing in existing_terms):
logger.warning("Ignoring duplicate custom term %r", cleaned)
return False
if (
len(contents.encode("utf-8")) > MAX_TERMS_FILE_BYTES
or len(existing_terms) >= MAX_TERMS_YIELDED
):
# A tail append would land past the read window where the new term
# stays invisible, while rewriting from a capped read would drop
# scanner-owned lines. Prepending keeps the addition readable and
# leaves every existing line, comment, and blank line intact.
return self._atomic_write(path, f"{cleaned}\n{contents}")
separator = "" if not contents or contents.endswith(("\n", "\r")) else "\n"
return self._atomic_write(path, f"{contents}{separator}{cleaned}\n")
def remove_term(self, term: str) -> bool:
"""Remove matching term lines without changing unrelated file content."""
if self.is_transient_terms:
logger.info("Ignoring terms edit while a CLI override is active")
return False
cleaned = self._single_term(term)
if cleaned is None:
return False
path = self.terms_path()
if path is None:
logger.warning("Cannot remove custom term because the configured path is invalid")
return False
contents = self._read_terms_contents(path)
if contents is None:
return False
remaining_lines: list[str] = []
removed = False
for line in contents.splitlines(keepends=True):
line_term = unicodedata.normalize("NFC", line.rstrip("\r\n").strip())
if not line_term.startswith("#") and line_term.casefold() == cleaned.casefold():
removed = True
continue
remaining_lines.append(line)
if not removed:
logger.warning("Custom term %r was not present in the terms file", cleaned)
return False
return self._atomic_write(path, "".join(remaining_lines))
def build_initial_prompt(self) -> Optional[str]:
"""Build the current Whisper prompt from enabled terms, if any."""
if not self.terms_enabled():
return None
max_terms = self.config.get("dictionary", "max_words", DEFAULT_MAX_TERMS)
try:
max_terms = max(0, int(max_terms))
except (TypeError, ValueError):
logger.warning("Invalid custom terms limit %r; using %d", max_terms, DEFAULT_MAX_TERMS)
max_terms = DEFAULT_MAX_TERMS
path = self.terms_path()
if path is None:
return None
prompt_terms: list[str] = []
prompt_characters = 0
for term in self._iter_terms_for_prompt(path, max_terms):
additional_characters = len(term) + (1 if prompt_terms else 0)
if prompt_characters + additional_characters > MAX_PROMPT_CHARACTERS:
logger.warning("Custom terms prompt reached %d characters", MAX_PROMPT_CHARACTERS)
break
prompt_terms.append(term)
prompt_characters += additional_characters
return " ".join(prompt_terms) or None
def get_corrections(self) -> list[dict[str, str]]:
"""Read corrections from JSON for every segment, failing closed on errors."""
entries = self._read_corrections(for_edit=False)
return entries if entries is not None else []
def get_corrections_for_edit(self) -> Optional[list[dict[str, str]]]:
"""Return losslessly editable entries, or None for an unsafe source file.
Runtime correction reads may safely ignore malformed individual entries,
but a UI edit must never rewrite the file from that filtered view.
"""
return self._read_corrections(for_edit=True)
def _read_corrections_text(self, path: Path) -> Optional[str]:
"""Return corrections JSON text, or None when no file exists.
If *path* is missing, copy once from beside the terms file when a
leftover file from the earlier co-located contract exists there so
those entries survive the move to the fixed location. Copy failure
still returns the old text so entries are not lost.
"""
try:
return path.read_text(encoding="utf-8")
except FileNotFoundError:
pass
# Legacy corrections lived beside the persisted terms file. Resolve
# that path from the saved setting — never the CLI override — so an
# override's sibling is not adopted while a saved file's corrections
# still apply.
terms = self._saved_terms_path()
old_path = terms.parent / CORRECTIONS_FILENAME if terms is not None else None
if old_path is None or old_path == path or not old_path.is_file():
return None
try:
contents = old_path.read_text(encoding="utf-8")
except FileNotFoundError:
return None
self._atomic_write(path, contents)
return contents
def _read_corrections(self, *, for_edit: bool) -> Optional[list[dict[str, str]]]:
"""Read corrections with stricter validation for write-back workflows."""
path = self.corrections_path()
try:
contents = self._read_corrections_text(path)
if contents is None:
return self._legacy_corrections()
payload = json.loads(
contents,
object_pairs_hook=_object_pairs_without_duplicates if for_edit else None,
)
except _DuplicateJsonKeyError:
logger.warning(
"Refusing to edit custom corrections because the source has duplicate JSON keys"
)
return None
except (OSError, UnicodeError, json.JSONDecodeError) as error:
logger.warning("Could not read custom corrections file %s: %s", path, error)
return None if for_edit else []
version = payload.get("version") if isinstance(payload, dict) else None
if (
not isinstance(payload, dict)
or not isinstance(version, int)
or isinstance(version, bool)
or version != CORRECTIONS_VERSION
):
logger.warning("Ignoring custom corrections file with an unsupported schema")
return None if for_edit else []
raw_entries = payload.get("corrections")
entries = normalize_corrections(raw_entries)
if for_edit:
if set(payload) - CORRECTIONS_TOP_LEVEL_KEYS:
logger.warning(
"Refusing to edit custom corrections because the source has extra top-level fields"
)
return None
if not isinstance(raw_entries, list) or len(entries) != len(raw_entries):
logger.warning(
"Refusing to edit custom corrections because some source entries are invalid"
)
return None
if any(
not isinstance(entry, dict) or set(entry) - CORRECTIONS_ENTRY_KEYS
for entry in raw_entries
):
logger.warning(
"Refusing to edit custom corrections because some source entries have extra fields"
)
return None
if any(
entry.get("heard") != normalized["heard"]
or entry.get("replacement") != normalized["replacement"]
for entry, normalized in zip(raw_entries, entries)
):
logger.warning(
"Refusing to edit custom corrections because normalization would change values"
)
return None
return entries
def save_corrections(self, entries: list[dict[str, str]]) -> bool:
"""Safely write validated corrections in the versioned JSON contract."""
payload = {
"version": CORRECTIONS_VERSION,
"corrections": normalize_corrections(entries),
}
return self._atomic_write(
self.corrections_path(), json.dumps(payload, ensure_ascii=False, indent=2) + "\n"
)
def apply_corrections(self, text: str) -> str:
"""Live-reload and apply corrections to a completed transcript."""
return apply_corrections(text, self.get_corrections())
def terms_status(self) -> str:
"""Return a concise, user-facing description of the custom terms file."""
path = self.terms_path()
if path is None:
return "Configured terms path is invalid or cannot be expanded."
try:
if not path.exists():
return "Terms file does not exist yet; add a term to create it."
if not path.is_file():
return "Terms path is not a regular file."
if path.stat().st_size > MAX_TERMS_FILE_BYTES:
count = len(self.get_terms())
return f"{count} term(s) available from the file's leading lines only."
if len(list(self._iter_terms(path))) >= MAX_TERMS_YIELDED:
return f"Only the first {MAX_TERMS_YIELDED} terms are used."
path.read_text(encoding="utf-8-sig")
except UnicodeError:
return "Terms file is not valid UTF-8."
except OSError:
return "Terms file cannot be inspected."
return f"{len(self.get_terms())} term(s) available from the live file."
@staticmethod
def _is_readable(path: Path) -> bool:
"""Return whether a path can be opened for reading."""
try:
with path.open("r", encoding="utf-8"):
return True
except OSError:
return False
@staticmethod
def _normalize_terms(terms: list[str]) -> list[str]:
"""Return trimmed, de-duplicated terms suitable for the line-file contract."""
normalized: list[str] = []
seen: set[str] = set()
for term in terms:
if not isinstance(term, str):
continue
cleaned = unicodedata.normalize("NFC", term.strip())
if len(cleaned) > MAX_TERM_CHARACTERS:
logger.warning(
"Ignoring custom term longer than %d characters", MAX_TERM_CHARACTERS
)
continue
key = cleaned.casefold()
if not cleaned or cleaned.startswith("#") or key in seen:
continue
seen.add(key)
normalized.append(cleaned)
return normalized
@staticmethod
def _single_term(term: str) -> Optional[str]:
"""Validate a single line-file term for an incremental edit."""
if not isinstance(term, str) or "\n" in term or "\r" in term:
logger.warning("Ignoring invalid custom term")
return None
normalized = CustomDictionaryManager._normalize_terms([term])
return normalized[0] if normalized else None
@staticmethod
def _read_terms_contents(path: Path, missing_value: Optional[str] = None) -> Optional[str]:
"""Read terms text for an edit while reporting invalid source files safely."""
try:
return path.read_text(encoding="utf-8-sig")
except FileNotFoundError:
return missing_value
except (OSError, UnicodeError) as error:
logger.warning("Could not read custom terms file %s: %s", path, error)
return None
def _legacy_corrections(self) -> list[dict[str, str]]:
"""Read the never-released #768 config shape without migrating it silently."""
raw_entries = self.config.get("text_injection", "custom_dictionary", [])
entries = normalize_corrections(
[
{"heard": entry.get("spoken"), "replacement": entry.get("replacement")}
for entry in raw_entries
if isinstance(entry, dict)
]
)
if entries:
logger.warning("Using legacy config entries until %s is saved", CORRECTIONS_FILENAME)
return entries
@staticmethod
def _atomic_write(path: Path, contents: str) -> bool:
"""Atomically write UTF-8 user data and leave the old file intact on failure."""
temporary_path: Optional[Path] = None
try:
path.parent.mkdir(parents=True, exist_ok=True)
temporary_path = path.with_name(f".{path.name}.{uuid.uuid4().hex}")
with temporary_path.open("x", encoding="utf-8") as temporary_file:
temporary_file.write(contents)
temporary_file.flush()
os.fsync(temporary_file.fileno())
try:
# A replacement keeps the existing file's permissions; a new
# file keeps the umask-derived mode of the temporary file.
shutil.copymode(path, temporary_path)
except OSError:
pass
os.replace(temporary_path, path)
return True
except OSError as error:
logger.error("Could not save custom dictionary file %s: %s", path, error)
if temporary_path is not None:
try:
temporary_path.unlink(missing_ok=True)
except OSError as cleanup_error:
logger.warning("Could not remove temporary dictionary file: %s", cleanup_error)
return False