perf: faster Settings, typing and dictation; mypy in CI, engine interface, architecture docs (#968)
* fix(perf): reduce startup and inference overhead * fix(perf): report effective whisper.cpp fallback settings * fix(injection): add opt-in X11 paste for long dictations * perf(vad): lower default silence timeout to 1.2s * fix(types): resolve mypy errors across application source * ci: run mypy in the lint job * ci: include mypy in lint dependency group * fix(types): narrow evdev timestamp comparison * refactor(recognition): add SpeechEngine protocol and engine dispatch table * refactor(settings): move About section and engine tables out of settings_dialog * docs: add ARCHITECTURE.md and refresh docs index and AGENTS layout * fix(settings): preserve GTK dialog compatibility after extraction * style: format touched Python modules * fix(types): keep local typecheck from syncing dependencies * test(vad): align main startup expectation with new default * style: sort engine registry and protocol test imports * test(settings): follow extracted About builder attribute alias * test(settings): isolate GTK dialog base in About test * test(settings): isolate About color scheme build test * test(settings): restore GTK base after About scheme test * test(settings): isolate About scheme check from GTK mocks * test(settings): support extracted dialog mixin methods * docs: add before/after performance benchmark charts * fix(injection): type instead of pasting when the clipboard holds non-text data
This commit is contained in:
@@ -0,0 +1,111 @@
|
||||
# Architecture
|
||||
|
||||
Vocalinux is a GTK 3 desktop application. `main.py` wires together the
|
||||
configuration manager, recognition manager, tray UI, D-Bus service, and text
|
||||
injector. Audio capture and recognition run outside the GTK event loop.
|
||||
|
||||
## Module map
|
||||
|
||||
- `main.py` composes the application services and callbacks.
|
||||
- `audio/capture.py` and `audio/pipewire.py` provide capture sources.
|
||||
- `speech_recognition/` owns VAD, segment processing, command handling, and
|
||||
engine implementations under `engines/`.
|
||||
- `post_processor.py` runs configured transcript processing scripts.
|
||||
- `text_injection/` contains `TextInjector`, focused-window detection, IBus,
|
||||
and Remote Desktop Portal support.
|
||||
- `ui/` contains the tray, Settings, dictation pad, history, and keyboard
|
||||
backend modules.
|
||||
- `dbus_service.py` exposes desktop control methods. `gateway_embed/` manages
|
||||
the optional local VocaGateway integration.
|
||||
|
||||
## Dictation data flow
|
||||
|
||||
```text
|
||||
PortAudio or PipeWire
|
||||
|
|
||||
v
|
||||
audio/capture.py and audio/pipewire.py
|
||||
|
|
||||
v
|
||||
recognition_manager.py: VAD and silence segmentation
|
||||
| |
|
||||
| Silero VAD, when available
|
||||
| amplitude VAD otherwise
|
||||
v
|
||||
bounded segment queue
|
||||
|
|
||||
v
|
||||
recognition worker -> selected speech engine -> corrections and commands
|
||||
|
|
||||
v
|
||||
text callback -> post_processor.py -> TextInjector backend
|
||||
```
|
||||
|
||||
`PortAudioCaptureSource` and the PipeWire source provide mono 16 kHz PCM chunks.
|
||||
The recognition manager owns dictation segmentation. It uses Silero VAD when
|
||||
available and amplitude thresholds otherwise, then places finalized segments
|
||||
on a bounded queue. A recognition worker consumes the queue, dispatches each
|
||||
segment to the selected engine, applies dictionary corrections and voice
|
||||
command parsing, and invokes callbacks.
|
||||
|
||||
The text callback submits work to one ordered post-processing worker. That
|
||||
worker runs the configured post-processing script, then injects text or places
|
||||
it in the in-app dictation pad. `TextInjector` selects a backend from the
|
||||
detected `DesktopEnvironment`: X11 uses xdotool, Wayland can use wtype or
|
||||
ydotool, and the IBus and Remote Desktop Portal paths support their respective
|
||||
sessions.
|
||||
|
||||
## Threads and state ownership
|
||||
|
||||
- The GTK main thread owns widgets, dialog state, tray state, and the GLib main
|
||||
loop. UI-facing state callbacks use `GLib.idle_add` where needed; the D-Bus
|
||||
control service also marshals Toggle, Start, and Stop calls to this loop.
|
||||
- The audio capture thread reads the capture source, tracks the current audio
|
||||
buffer and VAD state, and enqueues completed segments. The recognition worker
|
||||
owns queue consumption and transcription callback execution.
|
||||
- The recognition manager's model lock protects engine and recognizer access
|
||||
during transcription, initialization, and cleanup. The segment queue is
|
||||
thread-safe; capture-buffer handoff uses its buffer lock.
|
||||
- Text callbacks enqueue work on a single post-processing worker so script
|
||||
execution and injection stay in spoken order. Clipboard restoration can run
|
||||
on a separate worker. `TextInjector` protects its injection and clipboard
|
||||
restore bookkeeping with locks and generation state.
|
||||
- Keyboard backends use listener or monitor threads for pynput and evdev.
|
||||
IBus and Remote Desktop Portal injection also use worker threads for their
|
||||
input and D-Bus work.
|
||||
- GLib timeouts drive UI polling and progress updates. Recognition and capture
|
||||
workers use their own queue and event waits rather than GTK timers.
|
||||
|
||||
## Speech engines
|
||||
|
||||
`common_types.SpeechEngine` defines the runtime-checkable engine contract.
|
||||
`speech_recognition/engines/available_engines()` reports optional engines in
|
||||
Settings display order without importing the large Whisper packages just to
|
||||
probe availability. `SettingsDialog` retains a compatibility wrapper for its
|
||||
legacy engine-to-availability mapping.
|
||||
|
||||
`SpeechRecognitionManager._process_audio_buffer()` uses an engine-name dispatch
|
||||
table to call the existing per-engine transcription methods. The manager owns
|
||||
engine selection, model lifecycle, segment language snapshots, and the model
|
||||
lock. Optional engine packages are loaded only when their engine is selected.
|
||||
|
||||
## Configuration and Settings
|
||||
|
||||
`ui/config_manager.py` defines `DEFAULT_CONFIG`, loads the user's JSON config,
|
||||
and recursively overlays saved values on those defaults. It persists changes
|
||||
through `save_config()`. Settings controls generally apply changes as they are
|
||||
made and use callbacks to update the running engine or related services; values
|
||||
that require a model reload take effect during that reload.
|
||||
|
||||
The Settings UI is split across `settings_dialog.py`, `settings_about.py`, and
|
||||
`settings_engines.py`. Other UI responsibilities live in focused modules such
|
||||
as `tray_indicator.py`, `dictation_overlay.py`, `transcription_history.py`,
|
||||
`keyboard_shortcuts.py`, and `keyboard_backends/`.
|
||||
|
||||
## Services and optional integrations
|
||||
|
||||
`dbus_service.py` owns the `com.vocalinux.Vocalinux` session-bus name and
|
||||
provides Toggle, Start, and Stop methods for external controls.
|
||||
`gateway_embed/` manages the optional local VocaGateway container lifecycle,
|
||||
status, pairing, and configuration. It is separate from the default
|
||||
on-device whisper.cpp engine.
|
||||
@@ -11,6 +11,11 @@ User and maintainer documentation for Vocalinux.
|
||||
| [UPDATE.md](UPDATE.md) | Users | How to upgrade; release notes by version |
|
||||
| [DISTRO_COMPATIBILITY.md](DISTRO_COMPATIBILITY.md) | Users | Distro matrix, session caveats |
|
||||
| [HTTP_REMOTE.md](HTTP_REMOTE.md) | Power users | Remote HTTP transcription |
|
||||
| [FLATPAK_REMOTE.md](FLATPAK_REMOTE.md) | Users / maintainers | Self-hosted Flatpak remote and publishing |
|
||||
| [GATEWAY_EMBED.md](GATEWAY_EMBED.md) | Users | Optional local VocaGateway integration |
|
||||
| [CUSTOM_DICTIONARY_DESIGN.md](CUSTOM_DICTIONARY_DESIGN.md) | Contributors | Custom terms and correction design |
|
||||
| [ARCHITECTURE.md](ARCHITECTURE.md) | Contributors | Application modules, data flow, and threads |
|
||||
| [Performance benchmarks](benchmarks/README.md) | Contributors | Before and after charts for the performance pass |
|
||||
| [AUR.md](AUR.md) | Arch users / maintainers | AUR packages and publish secrets |
|
||||
| [RELEASE_PROCESS.md](RELEASE_PROCESS.md) | Maintainers | Version bump, checklist, tagging |
|
||||
|
||||
|
||||
@@ -34,6 +34,8 @@ When Vocalinux injects through the clipboard (the usual Wayland / ydotool path),
|
||||
|
||||
On non-US layouts such as German Neo, that chord uses the key that types **v** on the active layout (not physical KEY_V). Nested terminal panels inside an IDE are often invisible to window-class detection. If paste lands as a literal `^V` or does nothing, open **Settings → Dictation → Clipboard Paste Shortcut** and choose **Ctrl+Shift+V**.
|
||||
|
||||
On X11, **Paste long dictations** opts in to clipboard-pasting dictations of 30 or more characters instead of typing them key by key. It is off by default. Vocalinux restores the previous clipboard contents after pasting unless **Copy to Clipboard** is enabled. If the clipboard contains an image or other non-text content, Vocalinux types the dictation and leaves that content alone.
|
||||
|
||||
**Other audio.** Settings → Audio → Other audio can lower speakers and headphones while the microphone is open, then put that volume back. It stays off until you turn it on. Level while dictating is a percent of the current volume (0 is silent). Quitting mid-dictation puts the volume back; if the app crashes first, the next launch does.
|
||||
|
||||
## Shortcuts
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
# Performance benchmarks
|
||||
|
||||
Before and after numbers for the performance pass in
|
||||
[#968](https://github.com/VocaHQ/vocalinux/pull/968). "main" is the code before
|
||||
that PR. All runs used the same box: 8 vCPU Intel Xeon Platinum 8559C, Python
|
||||
3.13, pywhispercpp 1.5.1, X11.
|
||||
|
||||
## Settings first open
|
||||
|
||||

|
||||
|
||||
The Settings dialog used to import every installed engine package (Vosk,
|
||||
OpenAI Whisper, faster-whisper) just to see whether it was installed. With all
|
||||
optional engines installed, the first open took 1.38 s and the process kept an
|
||||
extra 606 MB for the rest of its life. Probing with `importlib.util.find_spec`
|
||||
brings that down to 214 ms and 50 MB.
|
||||
|
||||
## Text injection on X11
|
||||
|
||||

|
||||
|
||||
Measured by injecting text into KWrite through `TextInjector.inject_text`,
|
||||
mean of 3 runs. Dropping the 0.1 s sleep after every 20 characters saves
|
||||
about 2 s per 420 characters. Turning on Settings > "Paste long dictations"
|
||||
sends text of 30 or more characters through the clipboard, which takes about
|
||||
40 ms at any length; the previous clipboard content is restored afterwards.
|
||||
Typing speed itself is set by xdotool's per-key delay, so absolute typed
|
||||
times vary by machine.
|
||||
|
||||
## whisper.cpp after a GPU init failure
|
||||
|
||||

|
||||
|
||||
When Vulkan or CUDA fails to initialise, the CPU fallback used to keep the
|
||||
GPU thread count (cores / 4, so 2 here) instead of the CPU policy
|
||||
(cores - 2, capped at 8, so 6 here). Times are for an 11 s clip.
|
||||
|
||||
## Silence timeout
|
||||
|
||||

|
||||
|
||||
200 LibriSpeech dev-clean utterances (5 per speaker, 26 min of audio) were
|
||||
replayed through the same segmentation `_record_audio` uses: 1024-sample
|
||||
chunks, Silero VAD at the default sensitivity, and a split whenever silence
|
||||
exceeds the timeout. Below 1.2 s some sentences get cut in two at a natural
|
||||
pause; at 1.2 s none were. The default is now 1.2 s instead of 2.0 s, so each
|
||||
dictation finishes 0.8 s sooner. Read speech pauses less than live dictation,
|
||||
so raise the timeout in Settings if you pause mid-sentence to think.
|
||||
|
||||
## Where the time goes
|
||||
|
||||
On an 11 s utterance with the tiny model, the Python code around the
|
||||
transcription took 0.3 ms of a 389 ms call; the rest is native whisper.cpp.
|
||||
A standalone `whisper-cli` built from the same whisper.cpp commit was no
|
||||
faster, so rewriting the app in another language would not speed up
|
||||
transcription.
|
||||
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 39 KiB |
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 37 KiB |
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 39 KiB |
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 38 KiB |
Reference in New Issue
Block a user