sharp-brain-push-to-dictate

Hold-to-talk dictation on an electronic dictionary

How a 27-million-parameter speech model, moonstt, fits on a SHARP Brain: hold HOME, speak, let go, and the words appear at the cursor. Nothing leaves the device.

The boxed figures are live. Click the pipeline stages, drag the sliders, and hold the HOME button in the last figure. Numbers come from the repository's README and source; where a number is not measured, the page says so. Numbered notes open on tap.

SHARP Brain is a line of Japanese electronic dictionaries. Under the hood they are small ARM computers, and the Brainux project runs Linux on them. This project teaches one of them to take dictation: hold HOME, speak, release. A red pill appears at the top of the screen while you talk, turns amber while the device thinks, and then the text is typed where your cursor is.“Typed” is literal. The daemon presses keys on a virtual keyboard, so any program that accepts typing accepts dictation.

1. Why this is hard

Speech recognition is usually a job for a phone-class CPU or a GPU. The PW-SA3 and PW-SH3 have neither.

PartWhat it means for us
i.MX28, ARM926EJ-S core at ~454 MHz, ARMv5TEOne instruction at a time, in order
No FPUEvery float operation is a software routine
No SIMD, no NEONOne number at a time, apart from a few 16-bit multiply-accumulate instructions
16 KB data cacheThe working set has to be tiny, or the CPU waits for memory
128 MB of RAMThe model is memory-mapped; the README asks for about 64 MB free and 28 MB of storage
A 27M-parameter modelEach second of audio costs millions of multiply-adds

So the whole project is one question asked many times: what can we do with integers, a cache and a multiplier that takes two 16-bit pairs?

2. Choosing a model

We use Moonshine Tiny (Useful Sensors, MIT licence): 27M parameters, English only, an encoder-decoder Transformer with 6 layers each side, width 288, 8 attention heads. The weights are pinned to one Hugging Face revision and converted by tools/convert.py into a single 27.5 MB model.bin plus a small tokenizer file.

Two alternatives were looked at and put aside. These figures come from the comparison made while choosing, not from a file in this repository, so treat them as notes rather than benchmarks:

  • Whistle (Cactus). An unusual architecture, no Japanese, and only about 1.4× fewer multiply-accumulates. Not enough gain to justify porting something exotic.
  • Moonshine v2 Tiny. About 1.3× more compute than v1 Tiny. Slower, on a machine that is already the bottleneck.

Moonshine v1 Tiny is also friendly to a port: plain attention, GELU, layer norm, and a convolutional front end that turns raw 16 kHz audio into frames, with no spectrogram step.

3. The pipeline

Hold the key and arecord records 44.1 kHz stereo from the codec. On release the daemon runs moonstt on the WAV file. Click a stage.

Try it: from microphone to typed text. Click or tab to a stage. Stage names under the boxes match the profiler counters printed by moonstt -v.

The README does not record per-stage times, so none are shown here. Run moonstt -v clip.wav on your own device to see them: it prints the audio length, encoder time, decoder time, token count and real-time factor, plus one line per profiler counter.

4. Weights as int8, packed for the multiplier

Each row of every weight matrix gets one scale: its largest absolute value divided by 127. Weights are rounded to whole numbers from −127 to 127. Activations stay in 16-bit integers (12-bit values with a dynamic power-of-two scale). The ARMv5TE instructions SMLABB and SMLATT multiply the bottom or top 16-bit halves of two registers and add to an accumulator.

To feed them without unpacking, convert.py stores each weight offset-binary (add 128, so it is 1 to 255) and packs four weights into one 32-bit word in the byte order k, k+2, k+1, k+3. Then x & 0x00ff00ff leaves weights k and k+1 in two 16-bit lanes, and (x >> 8) & 0x00ff00ff leaves k+2 and k+3. The extra 128 is cancelled afterwards by subtracting 128 × the sum of the activations.In moonstt.c: ar[j] - 128 * sm[t].

Try it: one row of eight weights. Move the outlier slider and watch the scale stretch to fit it: every other weight gets coarser. The bits slider is a what-if; the real format is 8-bit.

5. How fast is it?

The matrix kernel processes 2 weight rows × 2 frames per pass, so every loaded word is used twice and the accumulators stay in registers. Weights are tiled in K-chunks so the 16 KB data cache stays hot. In a cache-friendly loop that reaches roughly 230 million multiply-accumulates per second; on real data it is about 140. The remaining gap is memory, plus the parts still in soft-float: layer norm, rotary embeddings and residuals.

Try it: how long will I wait? The curve is drawn through the README's measurements (2 s ≈ 4.8×, 4–6 s ≈ 5.0–5.3×, 8 s ≈ 5.5×).

6. Things that did not help

Two textbook ideas for squeezing a memory-bound loop were measured and rejected:

pld prefetch

Hinting the cache to fetch ahead. On the ARM926 it was slower.

ldm bursts

Loading several registers per instruction. Also slower.

The README doesn't say why, and neither will we. The code still carries a PLD() macro from that experiment; the lesson is to measure on the real chip.

7. The hold-to-talk daemon

moonstt-dictate watches the input device for the key (default 172, KEY_HOMEPAGE). The rules, from dictate.c and the README: taps under 0.4 s are ignored; recording stops at 25 s; leading and trailing silence is trimmed, and pauses over 0.5 s are shortened, so a full stop can turn into a comma. The text is typed with a trailing space unless you pass -n.

Try it: the HUD, a to-scale mock of the 854×480 screen. Press and hold the HOME button (mouse, touch, or Space/Enter) and let go. Choose a typing backend and where the cursor is.

Typing backends

  • uinput (preferred, automatic when /dev/uinput exists): a virtual US-layout keyboard, so text goes to whatever has focus, console or X. Needs CONFIG_INPUT_UINPUT=y, which brain_defconfig has off.
  • tty (fallback): TIOCSTI into the active virtual console. Works on the stock kernel but only reaches the console, and kernels 6.2+ may disable it (dev.tty.legacy_tiocsti).

8. Installing it

NeedWhy
Microphone capture in ALSAProvided by linux-brain PR #41. The codec puts the mono mic in the left slot; moonstt picks the louder channel and resamples to 16 kHz.
A kernel that reports HOMEThe PW-SH3 device tree leaves it unmapped (see below), or choose another key with -k.
arecord (alsa-utils)Recording.
~64 MB free RAM, ~28 MB storageThe model is memory-mapped.
  1. Debian / Brainux: make deb (needs Docker), then dpkg -i dist/moonstt-model_*.deb dist/moonstt_*.deb. The service starts on install.
  2. Buildroot: this repository is a Buildroot external tree. Add it to BR2_EXTERNAL, enable BR2_PACKAGE_MOONSTT, build the model once with tools/build-model.sh model.
  3. By hand: make for static armel binaries, tools/build-model.sh model, then copy the files over ssh (Dropbear has no sftp, so pipe through cat).

Full steps and options are in the README.

The HOME key

The matrix position exists in imx28-pwsh3.dts but is commented out. Replace the two /* <... 6 Home>, */ comments (in keymap and keymap-symbol) with:

<0 6 KEY_HOMEPAGE>,   /* Home */

KEY_HOMEPAGE does nothing on the console, so holding it types nothing. This is the whole kernel change, and it is not merged upstream yet.

9. Honest limitations

  • About 5× real time. A 5 s sentence takes about 25 s. This is for note-taking, not live captions.
  • English only. Moonshine Tiny v1 is English, and the output vocabulary is cut to ASCII tokens.
  • Kernel support needed. Mic capture (PR #41), a mapped HOME key, and for the nice typing path, CONFIG_INPUT_UINPUT.
  • TIOCSTI only reaches the console. Without uinput, a GUI app gets nothing.
  • Not bit-exact. It matches the Hugging Face float reference token-for-token on most test clips; a near-tie word can differ. Background music hurts a lot.

Code and licences: tslmy/sharp-brain-push-to-dictate. Moonshine weights are MIT (Useful Sensors).