Hold-to-talk dictation on an electronic dictionary
How a 27-million-parameter speech model, moonstt, fits on a SHARP Brain: hold HOME, speak, let go, and the words appear at the cursor. Nothing leaves the device.
The boxed figures are live. Click the pipeline stages, drag the sliders, and hold the HOME button in the last figure. Numbers come from the repository's README and source; where a number is not measured, the page says so. Numbered notes open on tap.
SHARP Brain is a line of Japanese electronic dictionaries. Under the hood they are small ARM computers, and the Brainux project runs Linux on them. This project teaches one of them to take dictation: hold HOME, speak, release. A red pill appears at the top of the screen while you talk, turns amber while the device thinks, and then the text is typed where your cursor is.“Typed” is literal. The daemon presses keys on a virtual keyboard, so any program that accepts typing accepts dictation.
1. Why this is hard
Speech recognition is usually a job for a phone-class CPU or a GPU. The PW-SA3 and PW-SH3 have neither.
| Part | What it means for us |
|---|---|
| i.MX28, ARM926EJ-S core at ~454 MHz, ARMv5TE | One instruction at a time, in order |
| No FPU | Every float operation is a software routine |
| No SIMD, no NEON | One number at a time, apart from a few 16-bit multiply-accumulate instructions |
| 16 KB data cache | The working set has to be tiny, or the CPU waits for memory |
| 128 MB of RAM | The model is memory-mapped; the README asks for about 64 MB free and 28 MB of storage |
| A 27M-parameter model | Each second of audio costs millions of multiply-adds |
So the whole project is one question asked many times: what can we do with integers, a cache and a multiplier that takes two 16-bit pairs?
2. Choosing a model
We use Moonshine Tiny (Useful Sensors, MIT licence): 27M parameters, English only, an encoder-decoder Transformer with 6 layers each side, width 288, 8 attention heads. The weights are pinned to one Hugging Face revision and converted by tools/convert.py into a single 27.5 MB model.bin plus a small tokenizer file.
Two alternatives were looked at and put aside. These figures come from the comparison made while choosing, not from a file in this repository, so treat them as notes rather than benchmarks:
- Whistle (Cactus). An unusual architecture, no Japanese, and only about 1.4× fewer multiply-accumulates. Not enough gain to justify porting something exotic.
- Moonshine v2 Tiny. About 1.3× more compute than v1 Tiny. Slower, on a machine that is already the bottleneck.
Moonshine v1 Tiny is also friendly to a port: plain attention, GELU, layer norm, and a convolutional front end that turns raw 16 kHz audio into frames, with no spectrogram step.
3. The pipeline
Hold the key and arecord records 44.1 kHz stereo from the codec. On release the daemon runs moonstt on the WAV file. Click a stage.
moonstt -v.The README does not record per-stage times, so none are shown here. Run moonstt -v clip.wav on your own device to see them: it prints the audio length, encoder time, decoder time, token count and real-time factor, plus one line per profiler counter.
4. Weights as int8, packed for the multiplier
Each row of every weight matrix gets one scale: its largest absolute value divided by 127. Weights are rounded to whole numbers from −127 to 127. Activations stay in 16-bit integers (12-bit values with a dynamic power-of-two scale). The ARMv5TE instructions SMLABB and SMLATT multiply the bottom or top 16-bit halves of two registers and add to an accumulator.
To feed them without unpacking, convert.py stores each weight offset-binary (add 128, so it is 1 to 255) and packs four weights into one 32-bit word in the byte order k, k+2, k+1, k+3. Then x & 0x00ff00ff leaves weights k and k+1 in two 16-bit lanes, and (x >> 8) & 0x00ff00ff leaves k+2 and k+3. The extra 128 is cancelled afterwards by subtracting 128 × the sum of the activations.In moonstt.c: ar[j] - 128 * sm[t].
5. How fast is it?
The matrix kernel processes 2 weight rows × 2 frames per pass, so every loaded word is used twice and the accumulators stay in registers. Weights are tiled in K-chunks so the 16 KB data cache stays hot. In a cache-friendly loop that reaches roughly 230 million multiply-accumulates per second; on real data it is about 140. The remaining gap is memory, plus the parts still in soft-float: layer norm, rotary embeddings and residuals.
6. Things that did not help
Two textbook ideas for squeezing a memory-bound loop were measured and rejected:
pld prefetch
Hinting the cache to fetch ahead. On the ARM926 it was slower.
ldm bursts
Loading several registers per instruction. Also slower.
The README doesn't say why, and neither will we. The code still carries a PLD() macro from that experiment; the lesson is to measure on the real chip.
7. The hold-to-talk daemon
moonstt-dictate watches the input device for the key (default 172, KEY_HOMEPAGE). The rules, from dictate.c and the README: taps under 0.4 s are ignored; recording stops at 25 s; leading and trailing silence is trimmed, and pauses over 0.5 s are shortened, so a full stop can turn into a comma. The text is typed with a trailing space unless you pass -n.
Typing backends
uinput(preferred, automatic when/dev/uinputexists): a virtual US-layout keyboard, so text goes to whatever has focus, console or X. NeedsCONFIG_INPUT_UINPUT=y, whichbrain_defconfighas off.tty(fallback):TIOCSTIinto the active virtual console. Works on the stock kernel but only reaches the console, and kernels 6.2+ may disable it (dev.tty.legacy_tiocsti).
8. Installing it
| Need | Why |
|---|---|
| Microphone capture in ALSA | Provided by linux-brain PR #41. The codec puts the mono mic in the left slot; moonstt picks the louder channel and resamples to 16 kHz. |
| A kernel that reports HOME | The PW-SH3 device tree leaves it unmapped (see below), or choose another key with -k. |
arecord (alsa-utils) | Recording. |
| ~64 MB free RAM, ~28 MB storage | The model is memory-mapped. |
- Debian / Brainux:
make deb(needs Docker), thendpkg -i dist/moonstt-model_*.deb dist/moonstt_*.deb. The service starts on install. - Buildroot: this repository is a Buildroot external tree. Add it to
BR2_EXTERNAL, enableBR2_PACKAGE_MOONSTT, build the model once withtools/build-model.sh model. - By hand:
makefor static armel binaries,tools/build-model.sh model, then copy the files over ssh (Dropbear has no sftp, so pipe throughcat).
Full steps and options are in the README.
The HOME key
The matrix position exists in imx28-pwsh3.dts but is commented out. Replace the two /* <... 6 Home>, */ comments (in keymap and keymap-symbol) with:
<0 6 KEY_HOMEPAGE>, /* Home */
KEY_HOMEPAGE does nothing on the console, so holding it types nothing. This is the whole kernel change, and it is not merged upstream yet.
9. Honest limitations
- About 5× real time. A 5 s sentence takes about 25 s. This is for note-taking, not live captions.
- English only. Moonshine Tiny v1 is English, and the output vocabulary is cut to ASCII tokens.
- Kernel support needed. Mic capture (PR #41), a mapped HOME key, and for the nice typing path,
CONFIG_INPUT_UINPUT. TIOCSTIonly reaches the console. Without uinput, a GUI app gets nothing.- Not bit-exact. It matches the Hugging Face float reference token-for-token on most test clips; a near-tie word can differ. Background music hurts a lot.
Code and licences: tslmy/sharp-brain-push-to-dictate. Moonshine weights are MIT (Useful Sensors).