Back to projectsFP.

Personal prototype / Doly

Doly — Voice Agent

Outcome & decision

Connected real-time conversation to a physical robot’s expressions, with MCP tools and device completion receipts. A personal prototype exploring reliable AI-to-device interaction.

Doly simulation model derived from the official CAD geometry
Official CAD-derived model preview. The voice tests used a physical Doly; this image is a simulation rendering.
Offline tests passed
43822 skipped in the saved Windows test report.
Recorded voice session
21 turnsApproximately 10 minutes, using half-duplex audio.
Text-only emotion cases
16 / 16Model → MCP → eyes / LEDs; no voice sessions in this batch.
My responsibility
Independent Developer / AI Solutions Architect · AI-assisted development
Project scope
Voice and expression prototype validated; half-duplex operation

Evidence basis. Saved device sessions, offline tests, and a separate text-only emotion matrix

Goal & system design

Make conversation produce observable robot behavior. Agora carries device audio, while a Windows host coordinates the model and MCP expression tools. The model receives a completion result only after the device provides supporting evidence.

  1. Device audioLinux runtime captures and plays speech.
  2. Cloud conversationASR, configured GLM endpoint, and TTS.
  3. MCP controlHost validates and dispatches expressions.
  4. Completion receiptDevice outcome returns to the model.

My contribution

Organized architecture, cloud-to-device integration, Windows migration checks, and failure analysis through AI-assisted development. Added tool validation, duplicate-call protection, session limits, and cleanup handling to make execution outcomes explicit.

What the tests showed

A saved Windows device session recorded two successful model responses and one happy expression, with audible speech and expression confirmed by the user. The longer 21-turn session continued through recorded disconnects; it does not establish long-term reliability.

Current boundaries

Users wait for playback to finish before speaking. Echo-cancellation compatibility blocked natural interruption testing, and end-to-end latency remains unmeasured. Full fault recovery and timed shutdown need further device testing. Voice-driven navigation is a separate, planned integration.

Tools & methods

  • Agora RTSA
  • Conversational AI
  • GLM
  • MCP
  • Python / C++

Visuals from the project archive. Doly CAD: DOLY / Levent Erenler, CC BY-NC-SA 4.0. Scene tree: OSRF, CC BY 3.0. Image credits & licenses ↗