perf: faster Settings, typing and dictation; mypy in CI, engine interface, architecture docs (#968)

* fix(perf): reduce startup and inference overhead

* fix(perf): report effective whisper.cpp fallback settings

* fix(injection): add opt-in X11 paste for long dictations

* perf(vad): lower default silence timeout to 1.2s

* fix(types): resolve mypy errors across application source

* ci: run mypy in the lint job

* ci: include mypy in lint dependency group

* fix(types): narrow evdev timestamp comparison

* refactor(recognition): add SpeechEngine protocol and engine dispatch table

* refactor(settings): move About section and engine tables out of settings_dialog

* docs: add ARCHITECTURE.md and refresh docs index and AGENTS layout

* fix(settings): preserve GTK dialog compatibility after extraction

* style: format touched Python modules

* fix(types): keep local typecheck from syncing dependencies

* test(vad): align main startup expectation with new default

* style: sort engine registry and protocol test imports

* test(settings): follow extracted About builder attribute alias

* test(settings): isolate GTK dialog base in About test

* test(settings): isolate About color scheme build test

* test(settings): restore GTK base after About scheme test

* test(settings): isolate About scheme check from GTK mocks

* test(settings): support extracted dialog mixin methods

* docs: add before/after performance benchmark charts

* fix(injection): type instead of pasting when the clipboard holds non-text data
This commit is contained in:
Jatin K Malik
2026-10-20 20:23:08 -07:00
committed by GitHub
parent 4799f06bd2
commit 65de3427fb
48 changed files with 7354 additions and 905 deletions
+111
View File
@@ -0,0 +1,111 @@
# Architecture
Vocalinux is a GTK 3 desktop application. `main.py` wires together the
configuration manager, recognition manager, tray UI, D-Bus service, and text
injector. Audio capture and recognition run outside the GTK event loop.
## Module map
- `main.py` composes the application services and callbacks.
- `audio/capture.py` and `audio/pipewire.py` provide capture sources.
- `speech_recognition/` owns VAD, segment processing, command handling, and
engine implementations under `engines/`.
- `post_processor.py` runs configured transcript processing scripts.
- `text_injection/` contains `TextInjector`, focused-window detection, IBus,
and Remote Desktop Portal support.
- `ui/` contains the tray, Settings, dictation pad, history, and keyboard
backend modules.
- `dbus_service.py` exposes desktop control methods. `gateway_embed/` manages
the optional local VocaGateway integration.
## Dictation data flow
```text
PortAudio or PipeWire
|
v
audio/capture.py and audio/pipewire.py
|
v
recognition_manager.py: VAD and silence segmentation
| |
| Silero VAD, when available
| amplitude VAD otherwise
v
bounded segment queue
|
v
recognition worker -> selected speech engine -> corrections and commands
|
v
text callback -> post_processor.py -> TextInjector backend
```
`PortAudioCaptureSource` and the PipeWire source provide mono 16 kHz PCM chunks.
The recognition manager owns dictation segmentation. It uses Silero VAD when
available and amplitude thresholds otherwise, then places finalized segments
on a bounded queue. A recognition worker consumes the queue, dispatches each
segment to the selected engine, applies dictionary corrections and voice
command parsing, and invokes callbacks.
The text callback submits work to one ordered post-processing worker. That
worker runs the configured post-processing script, then injects text or places
it in the in-app dictation pad. `TextInjector` selects a backend from the
detected `DesktopEnvironment`: X11 uses xdotool, Wayland can use wtype or
ydotool, and the IBus and Remote Desktop Portal paths support their respective
sessions.
## Threads and state ownership
- The GTK main thread owns widgets, dialog state, tray state, and the GLib main
loop. UI-facing state callbacks use `GLib.idle_add` where needed; the D-Bus
control service also marshals Toggle, Start, and Stop calls to this loop.
- The audio capture thread reads the capture source, tracks the current audio
buffer and VAD state, and enqueues completed segments. The recognition worker
owns queue consumption and transcription callback execution.
- The recognition manager's model lock protects engine and recognizer access
during transcription, initialization, and cleanup. The segment queue is
thread-safe; capture-buffer handoff uses its buffer lock.
- Text callbacks enqueue work on a single post-processing worker so script
execution and injection stay in spoken order. Clipboard restoration can run
on a separate worker. `TextInjector` protects its injection and clipboard
restore bookkeeping with locks and generation state.
- Keyboard backends use listener or monitor threads for pynput and evdev.
IBus and Remote Desktop Portal injection also use worker threads for their
input and D-Bus work.
- GLib timeouts drive UI polling and progress updates. Recognition and capture
workers use their own queue and event waits rather than GTK timers.
## Speech engines
`common_types.SpeechEngine` defines the runtime-checkable engine contract.
`speech_recognition/engines/available_engines()` reports optional engines in
Settings display order without importing the large Whisper packages just to
probe availability. `SettingsDialog` retains a compatibility wrapper for its
legacy engine-to-availability mapping.
`SpeechRecognitionManager._process_audio_buffer()` uses an engine-name dispatch
table to call the existing per-engine transcription methods. The manager owns
engine selection, model lifecycle, segment language snapshots, and the model
lock. Optional engine packages are loaded only when their engine is selected.
## Configuration and Settings
`ui/config_manager.py` defines `DEFAULT_CONFIG`, loads the user's JSON config,
and recursively overlays saved values on those defaults. It persists changes
through `save_config()`. Settings controls generally apply changes as they are
made and use callbacks to update the running engine or related services; values
that require a model reload take effect during that reload.
The Settings UI is split across `settings_dialog.py`, `settings_about.py`, and
`settings_engines.py`. Other UI responsibilities live in focused modules such
as `tray_indicator.py`, `dictation_overlay.py`, `transcription_history.py`,
`keyboard_shortcuts.py`, and `keyboard_backends/`.
## Services and optional integrations
`dbus_service.py` owns the `com.vocalinux.Vocalinux` session-bus name and
provides Toggle, Start, and Stop methods for external controls.
`gateway_embed/` manages the optional local VocaGateway container lifecycle,
status, pairing, and configuration. It is separate from the default
on-device whisper.cpp engine.
+5
View File
@@ -11,6 +11,11 @@ User and maintainer documentation for Vocalinux.
| [UPDATE.md](UPDATE.md) | Users | How to upgrade; release notes by version |
| [DISTRO_COMPATIBILITY.md](DISTRO_COMPATIBILITY.md) | Users | Distro matrix, session caveats |
| [HTTP_REMOTE.md](HTTP_REMOTE.md) | Power users | Remote HTTP transcription |
| [FLATPAK_REMOTE.md](FLATPAK_REMOTE.md) | Users / maintainers | Self-hosted Flatpak remote and publishing |
| [GATEWAY_EMBED.md](GATEWAY_EMBED.md) | Users | Optional local VocaGateway integration |
| [CUSTOM_DICTIONARY_DESIGN.md](CUSTOM_DICTIONARY_DESIGN.md) | Contributors | Custom terms and correction design |
| [ARCHITECTURE.md](ARCHITECTURE.md) | Contributors | Application modules, data flow, and threads |
| [Performance benchmarks](benchmarks/README.md) | Contributors | Before and after charts for the performance pass |
| [AUR.md](AUR.md) | Arch users / maintainers | AUR packages and publish secrets |
| [RELEASE_PROCESS.md](RELEASE_PROCESS.md) | Maintainers | Version bump, checklist, tagging |
+2
View File
@@ -34,6 +34,8 @@ When Vocalinux injects through the clipboard (the usual Wayland / ydotool path),
On non-US layouts such as German Neo, that chord uses the key that types **v** on the active layout (not physical KEY_V). Nested terminal panels inside an IDE are often invisible to window-class detection. If paste lands as a literal `^V` or does nothing, open **Settings → Dictation → Clipboard Paste Shortcut** and choose **Ctrl+Shift+V**.
On X11, **Paste long dictations** opts in to clipboard-pasting dictations of 30 or more characters instead of typing them key by key. It is off by default. Vocalinux restores the previous clipboard contents after pasting unless **Copy to Clipboard** is enabled. If the clipboard contains an image or other non-text content, Vocalinux types the dictation and leaves that content alone.
**Other audio.** Settings → Audio → Other audio can lower speakers and headphones while the microphone is open, then put that volume back. It stays off until you turn it on. Level while dictating is a percent of the current volume (0 is silent). Quitting mid-dictation puts the volume back; if the app crashes first, the next launch does.
## Shortcuts
+56
View File
@@ -0,0 +1,56 @@
# Performance benchmarks
Before and after numbers for the performance pass in
[#968](https://github.com/VocaHQ/vocalinux/pull/968). "main" is the code before
that PR. All runs used the same box: 8 vCPU Intel Xeon Platinum 8559C, Python
3.13, pywhispercpp 1.5.1, X11.
## Settings first open
![Settings first open](settings.svg)
The Settings dialog used to import every installed engine package (Vosk,
OpenAI Whisper, faster-whisper) just to see whether it was installed. With all
optional engines installed, the first open took 1.38 s and the process kept an
extra 606 MB for the rest of its life. Probing with `importlib.util.find_spec`
brings that down to 214 ms and 50 MB.
## Text injection on X11
![Text injection](typing.svg)
Measured by injecting text into KWrite through `TextInjector.inject_text`,
mean of 3 runs. Dropping the 0.1 s sleep after every 20 characters saves
about 2 s per 420 characters. Turning on Settings > "Paste long dictations"
sends text of 30 or more characters through the clipboard, which takes about
40 ms at any length; the previous clipboard content is restored afterwards.
Typing speed itself is set by xdotool's per-key delay, so absolute typed
times vary by machine.
## whisper.cpp after a GPU init failure
![GPU fallback](gpu_fallback.svg)
When Vulkan or CUDA fails to initialise, the CPU fallback used to keep the
GPU thread count (cores / 4, so 2 here) instead of the CPU policy
(cores - 2, capped at 8, so 6 here). Times are for an 11 s clip.
## Silence timeout
![Silence timeout](silence_timeout.svg)
200 LibriSpeech dev-clean utterances (5 per speaker, 26 min of audio) were
replayed through the same segmentation `_record_audio` uses: 1024-sample
chunks, Silero VAD at the default sensitivity, and a split whenever silence
exceeds the timeout. Below 1.2 s some sentences get cut in two at a natural
pause; at 1.2 s none were. The default is now 1.2 s instead of 2.0 s, so each
dictation finishes 0.8 s sooner. Read speech pauses less than live dictation,
so raise the timeout in Settings if you pause mid-sentence to think.
## Where the time goes
On an 11 s utterance with the tiny model, the Python code around the
transcription took 0.3 ms of a 389 ms call; the rest is native whisper.cpp.
A standalone `whisper-cli` built from the same whisper.cpp commit was no
faster, so rewriting the app in another language would not speed up
transcription.
File diff suppressed because it is too large Load Diff

After

Width:  |  Height:  |  Size: 39 KiB

File diff suppressed because it is too large Load Diff

After

Width:  |  Height:  |  Size: 37 KiB

File diff suppressed because it is too large Load Diff

After

Width:  |  Height:  |  Size: 39 KiB

File diff suppressed because it is too large Load Diff

After

Width:  |  Height:  |  Size: 38 KiB