Skip to content

How it works

A high-level map of what runs where and how a single voice command travels end-to-end. For source-level detail, read the code itself in the companion repo and the watch repo — the top-level layout in each is summarised in the section below.

Source layout

Companion (Wristotle-companion) — the Android app + supporting modules:

app/             main Android app — handlers, NLU wiring, UI, transport
speech/          recognition service + audio sources (shared library module)
speech-nlu/      intent classifier + slot extractors + embedding model storage
speech-whisper/  whisper.cpp JNI bridge + on-device Whisper recognizer
tools/           codegen + dev scripts (sync_features.sh, regen_help_timeline)

Watch (Wristotle) — the Pebble C app + PKJS companion:

src/c/           watch app (Pebble C SDK) — windows, input, persistence
src/pkjs/        PKJS-side JavaScript (Settings, timeline, find-phone)
appstore/        appstore listing assets (icons, screenshots)

The end-to-end flow

sequenceDiagram
    autonumber
    participant W as Pebble watch<br/>(Wristotle.pbw)
    participant B as BLE companion<br/>(microPebble)
    participant S as Wristotle<br/>WhisperRecognitionService
    participant R as Wristotle Recognizer<br/>(whisper.cpp or HTTP endpoint)
    participant N as Wristotle NLU<br/>(MiniLM)
    participant H as Wristotle Handler<br/>(Call / Sms / Media …)
    participant L as Wristotle<br/>PebbleListenerService

    W->>B: Press select → DictationSession
    B->>S: SpeechRecognizer.startListening<br/>+ EXTRA_AUDIO_SOURCE pipe FD
    Note over B,S: Audio bytes streamed<br/>over the pipe
    S->>R: PipeAudioSource.samples()
    R->>R: whisper_full inference (on-device)
    R-->>S: transcript: "call mom"
    S-->>B: SpeechRecognizer.Results
    B-->>W: DictationResult
    W->>B: AppMessage COMPANION_QUERY = "call mom"
    B->>L: AppMessage delivered
    L->>N: classifier.classify("call mom")
    N-->>L: IntentResult(Intent.Call, slots={contact:"mom"})
    L->>H: HandlerRegistry.dispatch(IntentResult)
    H->>H: TelecomManager.placeCall(...)
    H-->>L: "Calling Mom"
    L-->>B: AppMessage COMPANION_RESPONSE
    B-->>W: Display in chat

The two layers of "Wristotle" are deliberately separate apps:

  • Watch app (Wristotle.pbw) — owns the user-facing chat, dictation triggers, vibration feedback, local commands (time/battery/steps), and the on-watch storage of the most recent 5 chat entries.
  • Companion (Wristotle Companion APK) — owns recognition, intent classification, handler dispatch, conversation history (full retention), and all phone-side capabilities (calls, SMS, reminders, media control, app launching).

Why three layers between watch and recognizer

The dictation pipeline crosses three process boundaries before audio reaches our recognizer:

  1. Watch → BLE companion (microPebble) — over Bluetooth LE, audio frames encoded as Speex.
  2. BLE companion → SpeechRecognitionService (Wristotle, in our case) — via Android's standard SpeechRecognizer API. Audio handed off as a pipe file descriptor in the EXTRA_AUDIO_SOURCE intent extra.
  3. SpeechRecognitionService → Whisper — within our app, audio piped to a PipeAudioSource that feeds WhisperRecognizer.transcribe(...).

This design comes from Pebble's original architecture — the watch SDK has no notion of "Wristotle". It just calls dictation_session_start_session() and asks Android for transcription. The BLE companion is whoever handles that, and they delegate to whichever Android SpeechRecognizer is the system default — but only if the companion is actually using Android's standard recognizer API.

microPebble does, so by setting Wristotle as the system default voice provider, we slot ourselves into that handoff. The watch doesn't know; microPebble doesn't know; Android does the routing.

Core Devices (formerly rePebble) doesn't — they implement their own dictation pipeline (local, cloud, or hybrid, configurable inside Core Devices' settings) and never call Android's SpeechRecognizer. Wristotle's WhisperRecognitionService is dead code in the watch-dictation path under Core Devices; the watch's transcribed text reaches us only via the AppMessage layer below.

Speech recognition

:speech-whisper wraps whisper.cpp via JNI. Models are downloaded on demand from HuggingFace to app-private storage, then loaded into a single shared native handle (WhisperNative). The handle is cached across sessions — model load is the slow part (~600 ms), inference is fast (~700 ms for a 3-second utterance on a Pixel 10a using the quantized base.en model).

The recognizer is swappable via a service-locator pattern: Recognizers.provider returns the active implementation per session. The factory consults Speech provider settings and returns one of:

  • WhisperRecognizer (local default).
  • HttpRecognizer — POSTs the utterance's WAV to any OpenAI-compatible /v1/audio/transcriptions endpoint. Same base URL + bearer + model config shape as the Ask Agent feature; covers OpenAI, Groq, Cloudflare Workers AI, self-hosted Speaches, whisper.cpp's bundled HTTP server, LiteLLM proxies, … with one HTTP client. Vanilla HttpURLConnection multipart upload — no Ktor / OkHttp dep in :speech.
  • CompositeRecognizer(primary, secondary) — drain-first audio buffering then primary, with secondary as fallback on terminal error. Used in the two "primary + fallback" modes.

Any class implementing the Recognizer interface plugs in the same way.

Same swap-pattern applies to the intent classifier: IntentClassifiers.provider returns either the EmbeddingIntentClassifier (MiniLM) or a StubIntentClassifier (always-Unknown fallback on low-RAM devices).

NLU: intent + slots

Transcripts that arrive on COMPANION_QUERY (everything except reminder/cancel — which use dedicated message keys) go through three steps:

  1. Classify — EmbeddingIntentClassifier embeds the query via MiniLM, compares cosine similarity against a per-intent centroid (mean of seed phrasings + learned examples), and returns ranked intent candidates.
  2. Resolve — PebbleListenerService.resolveIntent applies a confidence floor + margin check. Sub-threshold or ambiguous predictions consult PrefixHints (an opening-verb regex map) as a deterministic tie-breaker. Below threshold and no prefix hint → Intent.Unknown.
  3. Extract slots — the per-intent SlotExtractor populates structured parameters (e.g. Call extracts contact; MediaPlay extracts optional app).

The handler then receives an IntentResult(intent, slots, confidence, alternates, rawQuery) and runs the action.

Implicit learning is on by default: every successful dispatch adds the query → intent pair to a Room-backed example bank, debounced centroid rebuild. Toggle off via Settings → Learning → Intent.

Cross-app media control

The most surprising part of Wristotle. When you say "play Audible":

  1. Slot extractor pulls app=audible from the query.
  2. AppIndex.lookup("audible") matches against the locally-indexed launcher apps → returns com.audible.application.
  3. MediaPlayHandler.launchAndPlay(packageId):
    • pauseOthers(keepPackage=audible) — pauses any currently-playing session in a different app so the previous one doesn't fight Audible for audio focus when the new activity launches off-screen.
    • launchApp(audible) via PackageManager.getLaunchIntentForPackage.
    • ActiveMediaSession.playForPackage(audible) — three-tier strategy:
      1. Quick poll for Audible's MediaController to register (Spotify-style fast activation).
      2. Targeted MEDIA_BUTTON broadcast to Audible's MediaButtonReceiver (catches Flutter audio_service-style apps that register a session but keep it active=false until first user play).
      3. System-wide MEDIA_PLAY key event as last resort.

pause, next, previous work the same way: tier 1 if the session is visible to getActiveSessions, tier 2 (targeted MEDIA_BUTTON broadcast) otherwise. System-wide key events are skipped for non-play commands — they'd land on the previously-active app, which is exactly the bug per-package targeting fixes.

Conversation history

Every interaction — watch-initiated voice commands, watch-local commands (time/battery/etc.) that bypass the phone — is persisted in a Room DB on the phone (wristotle_conversation.db). Surfaced on the Conversation tab.

Retention is user-controlled (1 / 10 / 20 / 30 days). Pruning runs on app startup and after every insert.

Optional audio capture: when on, each dictation's WAV is saved to filesDir/conversation-audio/ (capped at 5 most-recent, FIFO eviction). Off by default — audio is more sensitive than transcripts.

Backup & restore

Settings → Backup & Restore lets you write the user's on-device data to a ZIP via the Storage Access Framework (CreateDocument("application/zip")), and re-import a previously-exported ZIP via OpenDocument().

The ZIP layout is intentionally per-entity JSON, not a copy of the Room database files:

manifest.json                    # app/device metadata + settings + pins +
                                 # app aliases + contact aliases
data/notes.json                  # { "schema": N, "rows": [ ... ] }
data/tasks.json
data/conversations.json
data/nlu.json                    # learned NLU examples only (seeds ship with the app)
audio/notes/<basename>.wav       # optional
audio/conversation/<basename>.wav

Each data/*.json file carries its own per-entity schema: integer — independent of Room's @Database(version). The encoder always emits CURRENT_SCHEMA; the decoder accepts every version from 1 up. A column rename / addition in a future schema is absorbed by the decoder's version branch, so old backups keep loading without registering Room Migration chains anywhere.

Encryption is via zip4j's AES-256 (key strength 256). When the user enters a password on export, every entry in the ZIP (including the manifest) is encrypted; the importer detects this via ZipFile.isEncrypted before prompting for a password. Empty password = plain ZIP.

Restore is merge-by-ID: existing local rows are never destroyed. Per domain, the importer dedupes by natural identity and skips collisions (notes by (createdAtEpochMs, body), tasks by (createdAtEpochMs, text), conversations by (timestampEpochMs, query, response), NLU learned examples by normalizedText). Settings prefs and aliases (app + contact) overwrite on collision — scalars and user-defined phrase → target overrides can't meaningfully merge, and the restore intent is to ship the export's version of those. Contact aliases additionally go through a relink pass post-merge: dead lookup keys on the new device are re-bound by exact-name then by phone number against the local Contacts so aliases survive moving phones. The result dialog reports per-entity Total · Imported · Duplicates · Failed counts.

For multi-clip notes (separate-append mode stores N audio paths joined by U+001F in a single column), the encoder emits a JSON array of basenames and the decoder rejoins after audio extraction, so every clip survives the round-trip.

Diagnostics

The in-app diagnostics export (Settings → Diagnostics) bundles app + device + permission + model + AppIndex + conversation + log-buffer state into a markdown report, copies to clipboard, opens the Codeberg new-issue page. See the Bug reports page for the full surface.

Logging routes through WristotleLog (in :speech/util/) — a process-scoped in-memory ring buffer (~500 lines) that mirrors android.util.Log calls. Android 10+ blocks third-party apps from reading their own logcat, so capture has to happen at the call site.

Source layout

Wristotle-companion/
├── app/                # The companion app — services, handlers, slot extractors, UI
├── speech/             # System-wide RecognitionService + audio sources +
│                       # shared model plumbing (ResumableDownloader, ModelFileStorage)
├── speech-whisper/     # whisper.cpp JNI backend + WhisperRecognizer
└── speech-nlu/         # Intent classifier interface + MiniLM embedding implementation +
                        # learnable example bank

Wristotle/  (watch)
├── src/c/              # Pebble C SDK source
│   ├── Wristotle.c     # Entry point — settings inbox, lifecycle, click handling
│   ├── dictation/      # DictationSession + vibration + retry-dialog policy
│   ├── reminder/       # "remind"/"cancel" detection + 15s response timeout
│   ├── input/          # Local command table + dispatch processor
│   ├── quick_launch/   # Long-press button → auto-dictation flow
│   └── ui/chat_ui.c    # ScrollLayer chat (last 5 entries on watch)
└── src/pkjs/           # PebbleKit JS modules (Clay settings, reminders, find-phone)

Full per-module breakdown lives in the repos' README.md files.