Skip to main content
Stream audio from an ESP32-S3 microphone to a voice agent that listens, thinks, and talks back through a speaker, using Elato and AssemblyAI. Products used: Voice Agent API + Real-time STT + LLM Gateway + Text-to-Speech Model selection: The Voice Agent API manages the models for you, so it’s the fastest way to get talking. For the optional pipeline we use universal-3-6-pro for streaming speech-to-text, claude-haiku-4-5-20251001 through LLM Gateway for the reply, and the aai-kyla voice for text-to-speech.

Overview

In this guide we’ll build a voice agent on an ESP32-S3 with an INMP441 MEMS microphone and a MAX98357A amplifier and speaker. Mic audio is sampled as PCM16 at 16 kHz and relayed over a secure WebSocket to a Deno edge server, which talks to AssemblyAI. The reply plays out of the speaker at 24 kHz, and you can interrupt it at any time with a button or touchpad. The voice architecture design comes from ElatoAI, an MIT-licensed, open-source project that ships the ESP32 firmware, the Deno edge server, and a Next.js webapp with a Supabase DB. The AssemblyAI Deno server files are live in the same repo. Let’s dive in!
We’ll set ASSEMBLYAI_API_KEY on the edge server only, so the key never touches the device. You can get one from the AssemblyAI dashboard. The Voice Agent API also needs an account with Voice Agent API access.

Architecture


Hardware

Here’s what we need:

Wiring

Let’s wire the boards to the ESP32-S3. The pin numbers below are the defaults in firmware-arduino/src/Config.cpp, so if you use other pins, change them there.
Wiring diagram: INMP441 microphone, MAX98357A amplifier with speaker, button and LED connected to an ESP32-S3 board

Wiring for the ElatoAI firmware with a starter conversational AI board. Red is power, black is ground, green is signal.

The mic and the amp use separate I2S ports, so they don’t share clock pins. Keep the jumper wires short, and make sure every module shares a ground with the ESP32.

Build the voice agent

Before you start, you need:
  • The hardware above, wired up.
  • PlatformIO to flash the firmware, and Deno to run the edge server.
  • A Supabase project and the Elato web app running locally. The ElatoAI quickstart covers both.
  • An AssemblyAI API key with Voice Agent API access.

Step 1: Clone the repo

From here on, we’ll run everything from the root of this repo.

Step 2: Set your environment variables

The edge server holds your API key, so it never touches the device. You can get one from the AssemblyAI dashboard. The Voice Agent API needs an account with Voice Agent API access.
Set these in server/deno/.env. The server stores conversations in Supabase, so it needs your Supabase project URL and anon key too:
server/deno/.env

Step 3: Run the Deno edge server

Next up is the Deno edge server in server/deno. It relays audio between the ESP32 and AssemblyAI’s Voice Agent API over a single WebSocket. When a device connects, the server opens that WebSocket and sets up the session with one session.update event. We pass in the system prompt, the audio format going in, the voice, and the audio format coming out. The voice and the output format can’t change after this first update. The agent speaks English, Italian, Spanish, German, Portuguese and French.
server/deno/models/assemblyai.ts
Once the session is ready, the AI agent speaks first. We send a reply.create event and the model composes the greeting itself. From there AssemblyAI handles turn-taking, so it decides when you’re done talking. The agent’s voice comes back as reply.audio events, which we encode to Opus and send to the ESP32.
server/deno/models/assemblyai.ts
When the call ends, we send session.end so you aren’t billed for the reconnect grace window. Time to run it. Start the server from server/deno:
Locally the server speaks plain HTTP on port 8000, so the ESP32 reaches it at ws://<your-computer-ip>:8000. There’s no TLS on this hop yet. We’ll fix that in Step 6 when we deploy the server to a remote host. Last piece: in the web app, create a character, pick AssemblyAI as its voice provider, and choose a voice like alba, eve or george. The server reads the provider every time a device connects, so you can switch a character’s provider without restarting the server or reflashing the ESP32.

Step 4: Flash the firmware

The firmware ships in ELATO_MODE, which connects to Elato’s hosted server. To use your own server, open firmware-arduino/src/Config.h and switch to DEV_MODE with the Deno server:
firmware-arduino/src/Config.h
In firmware-arduino/src/Config.cpp, set both addresses in the DEV_MODE block to your computer’s LAN IP. Don’t use localhost, because the ESP32 is a separate device on your network. backend_server is where the Elato web app runs:
firmware-arduino/src/Config.cpp
Plug in the board. From the root of the repo, build, flash and open the serial monitor:
On first boot the board has no Wi-Fi credentials, so it starts a hotspot called ELATO-DEVICE. Join it from your phone or laptop, open http://192.168.4.1/wifi, and enter your 2.4 GHz network name and password. The board then restarts and joins your network.

Step 5: Talk to it

Once the serial monitor shows the device is connected, the speaker should start playing out the AI agent’s voice. Start talking to the agent and you should hear it respond. Depending on your setup, you can press the button or touchpad to interrupt it mid-speech.

Step 6: Deploy and switch to WSS

So far the ESP32 has been talking to your computer over ws:// and http://, and both are unencrypted. Anyone on the same network can read the audio and the auth token. That’s fine on your own home Wi-Fi while we develop, but not once the traffic crosses the internet. For a remote server, we’ll use wss://. It’s the same WebSocket protocol inside TLS, so the traffic is encrypted and can’t be read or changed on the way. Let’s deploy server/deno to Deno Deploy or another Deno-compatible host close to your users, and set ASSEMBLYAI_API_KEY and your Supabase settings as environment variables there. Then, in firmware-arduino/src/Config.h, we switch to PROD_MODE, and in firmware-arduino/src/Config.cpp we fill in the production block:
firmware-arduino/src/Config.cpp
The firmware picks its connection call from the mode:
firmware-arduino/src/Audio.cpp
TLS only protects us from a man-in-the-middle attack if the device checks the server’s certificate. In PROD_MODE the firmware calls beginSslWithCA with the CA_cert you paste into Config.cpp, and it refuses any server that can’t prove its identity. Paste your host’s root CA certificate there. The hops from the edge server to AssemblyAI always use wss:// and https://, even in local development.

Optional: STT, LLM Gateway and Text-to-Speech pipeline

The Voice Agent API is the quickest way to get going. But if you want to choose each model yourself, say to pin the speech-to-text model, use any LLM Gateway model, or pick a specific text-to-speech voice, we can run the same AssemblyAI products as three separate connections instead. The architecture stays the same. The device still talks to the same edge server, and a provider is just one file in server/deno/models/ that the server picks based on the character’s provider. To use the pipeline, we add one more provider next to assemblyai.ts.

Add the provider file

Create server/deno/models/assemblyai-pipeline.ts. It connects to three AssemblyAI endpoints, and each one authenticates with your API key.
The STT socket stays open for the whole session. When a final turn arrives, we queue it.
Each reply gets a new text-to-speech socket. We send Generate and Flush, encode the PCM we get back to Opus for the device, and send Cancel if you interrupt.
We handle replies one at a time: ask the LLM, then speak the answer.

Register the provider

In server/deno/main.ts, we import the provider and add a case to the provider switch:
In server/deno/utils.ts, we add the settings the provider imports. The voice is a text-to-speech voice slug like aai-kyla, which is AssemblyAI’s default voice (you can browse the rest in the voice catalog), and the model is an LLM Gateway model:
It reuses ASSEMBLYAI_API_KEY. To change the LLM, set ASSEMBLYAI_LLM_MODEL in server/deno/.env.

How it works

  • Listening. Once the device is awake and connected, it streams your mic to the server while it’s in the listening state, and AssemblyAI’s turn detection decides when you’re done talking, so you don’t hold a button to talk. The device stops streaming the mic while the agent is speaking.
  • One reply at a time. Each finished turn gets one reply, and we log both sides of the conversation to Supabase.
  • Interrupting. The device doesn’t stream your mic while the agent is speaking, so you interrupt with a short tap on the button or touchpad. We cancel the current reply, the device stops playing, and it goes back to listening.
  • Opus on the way back. Mic audio goes up as PCM16, which is about 32 KB/s. We encode the reply to Opus on the server, so the downstream traffic stays small even on a weak Wi-Fi link and decode it back on the ESP32.

Limitations

After running this design and the framework on hundreds of ESP32 devices globally, here are the limits Akash Deb, founder of Elato, and his team have documented:
  • Cold starts. Connecting to a freshly started edge server takes about 2 to 3 seconds.
  • Session length. Edge hosts end a connection when their wall-clock limit is reached. The team has tested up to 17 minutes of uninterrupted voice AI conversation.
  • Interruptions. For ESP32 boards without echo cancellation, we cannot detect barge-in while the AI agent is speaking. It only stops playing when the server tells it to, or when the user explicitly sends a button/touchpad interrupt signal.

Troubleshooting

Most problems come down to Wi-Fi, wiring or power. Keep the serial monitor open with pio device monitor, because it shows how far the device got.

Wi-Fi never connects

  • Use a 2.4 GHz network. The ESP32-S3 can’t see 5 GHz Wi-Fi. If your router puts both bands under one name and the device won’t join, create a separate 2.4 GHz network. On an iPhone hotspot, turn on Maximize Compatibility.
  • Check the credentials. Wi-Fi names and passwords are case-sensitive. Networks with a login page or a username, like hotel and office Wi-Fi, can be a problem for a device without a screen.
  • Power-cycle the board after you change Wi-Fi credentials.

Server never connects

The device joins Wi-Fi, but it never reaches the voice server.
  • ws:// and wss:// are mixed up. DEV_MODE connects with ws:// and expects your local server on port 8000. Any other mode connects with wss:// on port 443. A local server only accepts ws://, and a hosted one only accepts wss://.
  • The device can’t reach your computer. ws_server and backend_server must be your computer’s LAN IP, not localhost. To check, open http://<your-computer-ip>:8000 from a phone on the same Wi-Fi. If it doesn’t load, allow incoming connections for Deno in your firewall, and avoid guest networks, which often stop devices from reaching each other.
  • The device never got a token. Look for [Auth] Registration HTTP in the serial monitor. Anything other than 200 means the device couldn’t reach the web app at backend_server and backend_port.
  • The TLS handshake fails in PROD_MODE. CA_cert must be your host’s root CA certificate. If it’s wrong or empty, the firmware rejects the server.

No transcripts from the mic

The server connects, but the agent never reacts to anything you say.
  • Recheck the three I2S wires. SCK, WS and SD must go to the pins in the wiring table. A single wire in the wrong place gives you silence, not an error.
  • Tie L/R to GND. The firmware reads the left channel, and a floating L/R pin gives silence or noise.
  • Power the mic from 3V3, and check that its ground reaches the ESP32.
  • Shorten the wires. Long or loose jumper wires corrupt the I2S clock, so reseat them or solder the connections.

No sound from the speaker

You can see the agent’s reply in the web app, but you hear nothing, or only static.
  • Recheck BCLK, LRC and DIN. Swapping BCLK and LRC is the most common mistake.
  • Wire the amp’s SD pin to its GPIO. The firmware mutes the amp by driving SD low between replies, so if SD isn’t connected to the pin in the wiring table, the amp can stay muted.
  • Test the speaker on its own. Get a small internet-radio sketch playing through the MAX98357A. If the radio plays, your wiring and I2S pins are fine, and the problem is in the audio stream. If it doesn’t, recheck the pins in Config.cpp.

Board resets or audio crackles

The board reboots when Wi-Fi connects or when the agent starts speaking, and the serial monitor may print Brownout detector was triggered. That’s a USB brownout. Wi-Fi and the amplifier draw current in bursts, and a weak USB port or a thin cable can’t keep the voltage up.
  • Change the cable and the port. Use a short, good-quality USB cable plugged straight into your computer, or a 5 V supply rated for 1 A or more. Avoid unpowered hubs.
  • Add a capacitor. 470 µF or more across the amp’s VIN and GND soaks up the current spikes.
  • Lower the load. An 8 Ω speaker draws less current than a 4 Ω one.

Customize and extend

Here are a few ways to make it your own:
  • Change the persona and the voice in the web app.
  • Add tool calls like timers or smart-home actions. We declare them in session.update, run them in the server, and send the results back.
  • Deploy the server near your users. End-to-end latency comes from Wi-Fi, network distance, model time and playback, so measure it on your own hardware and region.

Next steps