feat(asr): FunASR/SenseVoice support for the Remote API engine (#468)

* feat(asr): add chat audio remote benchmarking * style(asr): apply black formatting * test(main): include remote API model kwarg * test(asr): add coverage for chat-completions error handling and parsing Add 10 new tests to cover uncovered lines in PR #468: - ConnectionError handler in _try_chat_completions_audio_api (lines 1601-1602) - _parse_chat_completion_transcription edge cases: None, list content, empty/whitespace strings, JSON without text field, plain text, and single-line language metadata (lines 1610, 1613-1621, 1625, 1629-1640) Coverage for recognition_manager.py improved from 85% to 86%. * feat(asr): support FunASR remote responses * refactor(asr): scope remote FunASR support to the #469 path Drop chat-completions/Qwen client, evaluation notes, and the remote ASR benchmark harness. Keep model field, tolerant OpenAI response parsing, SenseVoice label stripping, FunASR docs, and tests.
2026-07-13 15:03:19 -07:00
parent 9d9638e854
commit 85927be4b4
11 changed files with 236 additions and 25 deletions
@@ -4,12 +4,12 @@ Vocalinux can offload speech recognition to a remote HTTP server instead of runn

 Vocalinux speaks two wire formats out of the box:

- **OpenAI-compatible** — `POST /v1/audio/transcriptions` (e.g. OpenAI, [Speaches](https://github.com/speaches-ai/speaches), LocalAI, vLLM with Whisper)
+- **OpenAI-compatible** — `POST /v1/audio/transcriptions` (e.g. OpenAI, [Speaches](https://github.com/speaches-ai/speaches), LocalAI, FunASR/SenseVoice)
 - **whisper.cpp server** — `POST /inference` (the binary shipped with [whisper.cpp](https://github.com/ggerganov/whisper.cpp))

 Pick whichever your server exposes — the rest of this guide applies to both.

-> **Tip — share the server with your phone.** Both formats are open standards, so the same self-hosted server can also back mobile dictation apps. On Android, apps like *Dictate* and *Transcribro* speak OpenAI-compatible Whisper; on iOS, Shortcuts-based dictation clients can hit the same endpoint. Run one Whisper server, point your laptop and phone at it, and you get consistent dictation everywhere without uploading audio to a third party.
+> **Tip — share the server with your phone.** These HTTP formats are open enough that the same self-hosted server can also back mobile dictation apps. On Android, apps like *Dictate* and *Transcribro* speak OpenAI-compatible Whisper; on iOS, Shortcuts-based dictation clients can hit the same endpoint. Run one Whisper server, point your laptop and phone at it, and you get consistent dictation everywhere without uploading audio to a third party.

 ## How It Works

@@ -17,14 +17,14 @@ When the **Remote API** engine is active, Vocalinux:

 1. Captures microphone audio locally (16 kHz, mono, 16-bit PCM).
 2. On end-of-utterance, packages the buffer as a WAV file in memory.
-3. Uploads it via `multipart/form-data` to the configured server.
-4. Reads the transcribed text from the JSON response (`{"text": "..."}`) and types it into the focused window.
+3. Uploads it using the configured HTTP format.
+4. Reads the transcribed text from the JSON response and types it into the focused window.

 No audio is ever written to disk. The local machine still runs the VAD/segmentation logic — only the heavy ASR step is remote.

 ## Server-Side Setup

-You need a server that accepts audio uploads and returns `{"text": "..."}`. Two common options:
+You need a server that accepts audio uploads and returns text. Three common setups:

 ### Option A: whisper.cpp server

@@ -53,6 +53,29 @@ docker run --rm -p 8000:8000 ghcr.io/speaches-ai/speaches:latest

 The endpoint is `/v1/audio/transcriptions`.

+### Option C: FunASR/SenseVoice local server
+
+[FunASR](https://github.com/modelscope/FunASR) can run a local
+OpenAI-compatible transcription service, which keeps audio on your own machine
+or LAN while letting Vocalinux use SenseVoice or Paraformer-family models:
+
+```bash
+pip install torch torchaudio
+pip install funasr vllm fastapi uvicorn python-multipart
+funasr-server --device cuda
+```
+
+The server listens on `http://localhost:8000` by default and exposes
+`POST /v1/audio/transcriptions`. In Vocalinux, set:
+
+- **Server URL**: `http://localhost:8000`
+- **API Endpoint**: `OpenAI/FunASR (/v1/audio/transcriptions)`
+- **Model**: `sensevoice`
+
+Use `--device cpu` for CPU-only testing if your FunASR install supports it. A
+remote GPU host works too: bind the server to the LAN interface, open the port,
+and point Vocalinux at that host.
+
 ### Network checklist

 - The server must be reachable from the client over HTTP or HTTPS.
@@ -64,14 +87,15 @@ The endpoint is `/v1/audio/transcriptions`.

 ### Via the Settings dialog

-The remote engine is a power-user override and lives on the **Advanced** tab so casual users aren't confronted with it.
+The remote engine is a power-user option in the **Speech Engine** settings.

-1. Launch Vocalinux and open **Settings → Advanced → Remote Server**.
-2. Toggle **Use remote server** on. This overrides whatever local engine is selected on the Speech Engine tab.
+1. Launch Vocalinux and open **Settings → Speech Engine → Remote Server**.
+2. Toggle **Use remote server** on. This overrides whatever local engine is selected above it.
 3. Fill in the fields:
   - **Server URL**: base URL of the server, e.g. `http://192.168.1.100:8080` (no trailing slash needed; one is stripped automatically).
   - **API Key** (optional): sent as `Authorization: Bearer <key>`. Leave blank if your server doesn't require auth.
-   - **API Endpoint**: pick **Whisper.cpp (`/inference`)** or **OpenAI (`/v1/audio/transcriptions`)** to match your server.
+   - **API Endpoint**: pick **Whisper.cpp (`/inference`)** or **OpenAI/FunASR (`/v1/audio/transcriptions`)** to match your server.
+   - **Model**: model identifier sent to OpenAI-compatible servers. Use `whisper-1` for classic Whisper servers, or `sensevoice` for FunASR/SenseVoice.
 4. Click **Test Connection** — a successful test means the URL is reachable and credentials (if any) are accepted. A failure here is just a warning; Vocalinux will still try again on the first transcription.

 Settings auto-save and re-initialise the engine immediately, so you can start dictating as soon as the test passes. Toggling the switch off restores the local engine selected on the Speech Engine tab.
@@ -86,13 +110,28 @@ The same options live in `~/.config/vocalinux/config.json`:
    "engine": "remote_api",
    "remote_api_url": "http://192.168.1.100:8080",
    "remote_api_key": "",
-    "remote_api_endpoint": "/inference"
+    "remote_api_endpoint": "/inference",
+    "remote_api_model": "whisper-1"
  }
 }
 ```

 Restart Vocalinux after editing the file by hand.

+FunASR/SenseVoice example:
+
+```json
+{
+  "speech_recognition": {
+    "engine": "remote_api",
+    "remote_api_url": "http://localhost:8000",
+    "remote_api_key": "",
+    "remote_api_endpoint": "/v1/audio/transcriptions",
+    "remote_api_model": "sensevoice"
+  }
+}
+```
+
 ## Wire Protocol Reference

 If you're writing a custom server, here's exactly what the client sends.
@@ -151,12 +190,31 @@ en
 --boundary--
 ```

-### Response (both formats)
+### Response (`/inference` and `/v1/audio/transcriptions`)

 ```json
 { "text": "the transcribed utterance" }
 ```

+For OpenAI-compatible FunASR/SenseVoice wrappers, extra metadata is allowed and
+ignored by Vocalinux:
+
+```json
+{
+  "text": "the transcribed utterance",
+  "language": "en",
+  "emotion": "neutral",
+  "segments": [
+    { "text": "the transcribed utterance", "start": 0.0, "end": 1.4 }
+  ]
+}
+```
+
+If `text` is not present, Vocalinux also accepts `transcript`, `transcription`,
+or a `segments` list with per-segment `text` fields. If a SenseVoice wrapper
+returns rich labels inline, for example `<|en|><|NEUTRAL|><|Speech|>hello`,
+Vocalinux strips the leading labels before text injection.
+
 Empty utterances should return `{"text": ""}`. HTTP 4xx/5xx responses are surfaced as transcription errors in the Vocalinux log.

 ### Language handling
@@ -188,9 +246,15 @@ curl -H "Authorization: Bearer $API_KEY" \
     -F file=@sample.wav \
     -F model=whisper-1 \
     http://192.168.1.100:8000/v1/audio/transcriptions
+
+# FunASR/SenseVoice
+curl -F file=@sample.wav \
+     -F model=sensevoice \
+     http://localhost:8000/v1/audio/transcriptions
 ```

-Both should return JSON with a `text` field.
+The multipart endpoints should return JSON with a text field or one of the
+compatible response shapes above.

 ## Troubleshooting

@@ -199,6 +263,8 @@ Both should return JSON with a `text` field.
 | Connection test reports "failed" | Wrong host/port, firewall, or server not running | Check `curl <url>` from the client; check server logs |
 | HTTP 401/403 | API key missing or wrong | Set the **API Key** field; ensure the server is configured for that key |
 | HTTP 404 on first send | Endpoint format mismatch | Switch the **API Endpoint** dropdown to the format your server actually exposes |
+| FunASR/SenseVoice server returns model errors | Model field does not match the served model | Try `sensevoice`, `paraformer`, or the exact model configured on the server |
+| SenseVoice labels appear in injected text | Wrapper returned labels somewhere other than the leading transcript prefix | Return plain text in `text` and put language/emotion/speech labels in JSON metadata |
 | Transcription always empty | Server rejected the WAV (sample rate, channels) | Confirm server accepts 16 kHz mono 16-bit WAV; check server logs |
 | First transcription is slow | Cold model load on the server | Warm the server before dictating, or increase the client timeout server-side |
 | Garbled text in foreign language | Wrong `language` setting | Set the language explicitly in Settings instead of `auto` |