Microduck
Microduck Speaks
Adding voice to this cute little robot, you forget you’re talking to a model.
Highlights
Microduck
In this “Simon Says” game, Microduck uses motion, vision, and plenty of cheeky banter to lead an engaging multi-modal game.
-
Open source, start to finish. Hugging Face’s speech-to-speech, open models from the Hub and Pollen’s browser simulator, both work locally on your laptop. Build and tune your robot’s character even before the hardware arrives.
-
Deterministic and Dynamic. Simon Says are confined to a deterministic set that is encoded in the game. This keeps game play predictable and fair. But everything else is dynmaic and the model improvises from what just happened, so no two games sound the same.
-
Selecting Speech Models The choice was between Qwen-TTS - which is more expressive and Kokoro - which is faster. We chose the more expressive voice of Aiden and pre-recorded common game commands to speed up response times.
-
Webcam Referee. With a camera and pose tracking Simon can "see" whether you looked left, turned or raised both arms. Move without the keywords "Simon says" and you lose. Make the wrong move and he'll catch that too.
-
Bi-directional Banter The chat interface works alongside the game and provides two-way communication. This means, you can chime in whenever ready - even if it cuts Simon off mid-gloat. He recovers from interruptions just like humans in natural conversation.
Tech stack
What Makes it Tick?
Four open models running locally on a laptop, connected to a browser game and a simulated robot.
18GB
MacBook runs every model locally. No API keys.
4
open models from the Hugging Face Hub
11
robot moves exposed to the model as tools
57
unit tests covering the game and the webcam referee
Speech-to-speech pipeline. Hugging Face’s speech-to-speech runs as a local Python server with four models from the Hub: Parakeet for speech to text, Smart Turn for end-of-turn detection, Qwen3 4B as the language model and Qwen3-TTS for the voice. All running locally.
Voice and latency. Qwen3-TTS is slower than Kokoro, so we cache the audio for fixed game lines. After the first take, every command and score line plays instantly.
Game logic in plain JavaScript. The browser runs the game: rounds, timing, scoring and the nine commands. Game lines are sent as out-of-band responses, so they never grow the model’s conversation history.
Turn-taking. The mic stays open for the whole session. Smart Turn decides when you’ve finished speaking from how you stopped, not just from silence. If you speak while Simon is busy, his reply pauses rather than being dropped.
Webcam referee. MediaPipe pose tracking runs in the browser and compares each move to your starting stance. A move only counts if you hold it for a second, and 57 unit tests cover the rules, including mirrored left and right. Video never leaves your laptop.
Upstream fixes. Building on speech-to-speech surfaced tool-call bugs, including one where Simon read his own tool calls out loud. We sent two fixes, with tests, to Hugging Face in PR #721; the third was already fixed upstream.
