Skip to content
Abhijeet Maharana, home
All posts

Building a Real-Time Voice AI Agent with Gemini Live and Node.js

How the parts of an open-source voice agent fit together: 16 kHz and 24 kHz AudioWorklets over a WebSocket to Gemini Live, tool calls run on a Node.js server, and Claude grading the call into a Zod-checked report.

Abhijeet Maharana · 9 min read

  • AI agents
  • Voice AI
  • Node.js
  • TypeScript

Short version: A voice agent has three jobs that pull in different directions. The audio has to be fast, the tools have to be trusted, and the score at the end has to be something your code can read. In voice-agent-eval each job lives in a different place: two AudioWorklets in the browser move 16 kHz audio up and 24 kHz audio down over a WebSocket to Gemini Live, a Node.js server runs every tool the agent calls, and Claude grades the transcript into a report that is checked against a Zod schema before it is saved. This post walks through each part with the real code, and the trade-offs I accepted.

What are the moving parts?

PartRuns whereIts job
Mic workletBrowser, audio threadTurns the microphone into 16 kHz, 16-bit chunks of 100 ms
Player workletBrowser, audio threadPlays Gemini's 24 kHz reply, and drops it at once when the person talks over it
Live socketBrowser to GoogleCarries audio, transcripts and tool calls, with no server in the middle
Tool routeNode.js serverRuns each tool call against the session and saves it with its timing
GraderNode.js server, after the callSends the transcript to Claude and parses a typed report

The browser opens the socket itself with a one-use token, so the API key stays on the server. I covered that part in an earlier post. This one is about what happens once the socket is open.

Why AudioWorklets?

The audio has to be processed in small pieces, all the time, without gaps. Code on the main thread shares its time with React renders and network callbacks, so a slow render would mean a click in the audio. An AudioWorklet is a small processor that runs on the browser's audio thread, gets 128 samples at a time and hands results back through a message port.

Gemini Live wants 16 kHz mono PCM going up and sends 24 kHz PCM back. Most microphones run at 44.1 or 48 kHz. So the page builds two audio contexts:

this.outCtx = new AudioContext({ sampleRate: 24000 });
this.micCtx = new AudioContext();
const stream = navigator.mediaDevices.getUserMedia({ audio: { echoCancellation: true, noiseSuppression: true, autoGainControl: true, channelCount: 1 } });

The speaker context runs at 24 kHz, which matches Gemini's output, so the reply needs no resampling in my code; the browser converts it for the sound card. The mic context runs at whatever the device uses, and the mic worklet brings it down to 16 kHz.

How does the microphone get to 16 kHz?

Here is the mic worklet. In the repo it is one string loaded as a Blob URL, so there is no extra build step; it is formatted here so you can read it:

class Mic extends AudioWorkletProcessor {
  constructor() {
    super();
    this.step = sampleRate / 16000;
    this.t = 0;
    this.prev = 0;
    this.buf = new Int16Array(1600);
    this.n = 0;
    this.sq = 0;
  }
  process(inputs) {
    const x = inputs[0][0];
    if (!x) return true;
    for (let k = 0; k < x.length; k++) {
      const s = x[k];
      while (this.t <= 1) {
        const v = Math.max(-1, Math.min(1, this.prev + (s - this.prev) * this.t));
        this.buf[this.n++] = v * 32767;
        this.sq += v * v;
        if (this.n === this.buf.length) {
          this.port.postMessage({ pcm: this.buf.buffer, rms: Math.sqrt(this.sq / this.n) }, [this.buf.buffer]);
          this.buf = new Int16Array(1600);
          this.n = 0;
          this.sq = 0;
        }
        this.t += this.step;
      }
      this.t -= 1;
      this.prev = s;
    }
    return true;
  }
}
registerProcessor("mic", Mic);

What it does, step by step:

  1. step is how many input samples make one output sample. At 48 kHz it is 3.
  2. For each input sample, it walks t between the previous sample and this one and takes a point on the straight line between them. That is linear interpolation.
  3. Each point is clamped to the range -1 to 1 and stored as a 16-bit integer, the format Gemini expects.
  4. Every 1600 samples (100 ms at 16 kHz) it posts the chunk and its RMS loudness to the main thread. The buffer is transferred, not copied, so no memory is duplicated.

On the main thread, each chunk goes straight into the socket as base64:

this.send({ realtimeInput: { audio: { mimeType: 'audio/pcm;rate=16000', data: toBase64(e.data.pcm) } } });

One detail that is easy to miss: a worklet only runs when something downstream pulls audio from it. The mic node is connected to a gain node set to 0, then to the speakers. The graph is pulled, and you do not hear yourself.

The trade-off: linear interpolation with no low-pass filter first lets sound above 8 kHz fold back as faint noise. For speech going to a speech model, with the browser's noise suppression on, that is an acceptable cost. For music it would not be.

How does the reply play without gaps?

Gemini sends audio as base64 parts inside serverContent. The page decodes each one and hands it to the player worklet, again as a transferred buffer:

for (const part of c.modelTurn?.parts ?? []) {
  const mime = String(part.inlineData?.mimeType ?? '');
  if (part.inlineData?.data && (!mime || mime.startsWith('audio/'))) {
    const pcm = fromBase64(part.inlineData.data);
    this.player?.port.postMessage(pcm, [pcm.buffer]);
  }
}
if (c.interrupted) {
  this.player?.port.postMessage('flush');
}

The player keeps a queue of chunks. On every 128-sample tick it copies from the front of the queue into the output and pads with silence when the queue is empty. It also posts true when playback starts and false when it stops, which drives the "speaking" and "listening" state on screen.

The flush message is how interruption works. When Gemini detects the person talking over the agent, it sends interrupted. The page empties the queue, and the agent goes quiet within one tick instead of finishing a sentence nobody wants.

How do you stop the agent interrupting itself?

Without headphones, the agent's own voice comes back through the microphone. Echo cancellation catches most of it, but not all, and Gemini then hears "the person" talking and stops. The fix is a loudness gate while the agent speaks:

const BARGE_IN_RMS = 0.04;

// in the mic handler
} else if (this.playing && e.data.rms < BARGE_IN_RMS) return;

This is why the mic worklet computes RMS. While the player is playing, only chunks louder than the leaked echo go up, so a real interruption still gets through.

The trade-off: the threshold is fixed. A quiet speaker in a loud room may need to speak up to cut in. For those cases the page has a button mode: the person presses Done, the page sends activityEnd, and no audio goes up while the agent talks.

How do tool calls run on the server?

The setup message lists each tool's name, description and parameters, with no code. When the agent wants one, Gemini sends a toolCall to the browser, and the page posts it to one Express route:

router.post(
  '/api/sessions/:id/tool',
  handle((req) => runTool(id(req), String(req.body?.name ?? '').slice(0, 60), req.body?.args && typeof req.body.args === 'object' ? req.body.args : {}, new Date(), Number(req.body?.t))),
);

runTool is a plain switch over the 7 tools. Here is the one that moves the session forward:

case 'next_question': {
  const i = s.plan.findIndex((p) => !p.asked);
  if (i < 0) {
    result = { done: true, message: 'All planned questions are done. Close the interview in one or two sentences, then call end_interview.' };
    break;
  }
  s.plan[i].asked = true;
  result = { done: false, number: i + 1, of: s.plan.length, question_id: s.plan[i].id, question: s.plan[i].question };
  break;
}

Three choices made the agent much steadier:

  • The server owns the state. The model never decides which question is next. It asks, and the plan on the server answers. A model that loses track mid-call cannot skip or repeat a question.
  • Errors go back as results. If the agent passes a question_id that is not in the plan, the tool returns { error: 'question_id is not in this interview plan' } instead of failing the HTTP call. The model reads it and corrects itself on the next turn.
  • Inputs are clamped, not trusted. record_note rounds the rating and keeps it between 1 and 10, and cuts the note to 500 characters.

Every call is saved with its arguments, result, time taken and the second of the call it happened in. The page shows each one as it runs, and the grader sees them all later.

How does Claude grade the call?

When the call ends, the server builds one message from the transcript with timestamps, the tool calls, the question files and delivery numbers measured in code (words per minute, fillers, longest answer). I count those in code because a model is not a reliable counter.

The output is a Zod schema:

const AnswerSchema = z.object({
  question_id: z.string(),
  question: z.string(),
  score: z.number().int(),
  technical: z.string(),
  structure: z.string(),
  delivery: z.string(),
  rewrite: z.string(),
  fix: z.string(),
});

The full ReviewOutput adds an overall score, a summary, next steps and the notes that shape the next session. The same schema goes to the Anthropic SDK as the structured output format, and the reply is parsed against it:

const response = await client.beta.messages.parse({
  model: reviewModel(),
  max_tokens: 16000,
  betas: ['server-side-fallback-2026-07-01'],
  fallbacks: 'default',
  output_config: { effort: 'high', format: betaZodOutputFormat(ReviewOutput) },
  system: systemPrompt(),
  messages: [{ role: 'user', content: reviewInput(s) }],
});
if (response.stop_reason === 'refusal') throw new Error('Claude declined to grade this session');
if (response.stop_reason === 'max_tokens') throw new Error('The review was cut off (max_tokens). Try again, or grade by hand with the review prompt.');
const out = response.parsed_output;

Why this shape works:

  • One schema, two jobs. It tells the model what to return and checks what came back. If a field is missing, nothing half-written is saved.
  • Stop reasons are checked first. A refusal or a reply cut off at max_tokens becomes a clear error message instead of a parse failure nobody can read.
  • Failures are kept. The error is saved on the session, and the page offers a retry. The grade route is safe to call again.
  • Fallback on refusal. With fallbacks: 'default', a request a safety check declines is retried on another model inside the same call.

The trade-off: grading happens after the call, not during it. A full review at high effort takes a while, so the page shows a grading state instead of an instant score. I chose that over a live score, because a grade is only useful if it is careful.

What would I change?

  • Worklets as strings keep the build simple, but that code gets no type checks or lint. If the worklets grow, they should move to their own files in the build.
  • The barge-in gate should adapt to the room: measure the echo level in the first seconds of agent speech and set the threshold from that.
  • The resampler is fine for speech. A product that also streams music or very quiet voices should add a low-pass filter before it.

When does this design fit?

Use it when people talk to an agent in the browser and you need three things at once: low delay, tools you control, and a result your code can store and query. Voice support agents, spoken onboarding checks and training tools all fit. If every audio packet must be recorded or changed on the server, put a proxy in the middle instead and accept the extra hop.

The code is MIT licensed on GitHub, and the case study covers the whole engine, including spaced repetition and how the agent learns rules from its own mistakes.