Skip to content
Abhijeet Maharana, home
All projects

Case study: AI agents and automation

Real-Time Voice AI Agent with an LLM Grading Engine

One engine runs any spoken evaluation from markdown files: live voice, visible tool calls, and Claude grading against your own reference answers.

An open-source voice agent on Gemini Live that runs structured spoken sessions, calls server tools mid-conversation, and hands the transcript to Claude for scored, typed reports.

Where
Open source (MIT) on GitHub
Role
Author and maintainer
When
2026
Team
Solo, end to end
7
server tools the agent calls mid-conversation, each shown on screen
0
API keys in the browser; every call gets a one-use Gemini token
30
tests, run with no network and no API keys
Browsermic 16 kHz, voice 24 kHzGemini Livevoice agentNode.js servertools, one-use tokenMarkdown contentquestions, referencesClaude graderZod-typed reportvoice both waystool callstranscript
The browser streams voice both ways to Gemini Live with a one-use token; the agent's tool calls run on the Node.js server, which owns the markdown content and sends the transcript to Claude for a typed report

TL;DR: voice-agent-eval is an open-source real-time voice AI agent with an evaluation engine behind it. A person talks to it out loud on Gemini Live. While it talks, it calls tools on the server to fetch the next question, check a reference answer or look up background, and every call shows on screen. When the call ends, Claude grades the transcript against your own reference answers and writes a report. Source: github.com/abhijeet-maharana/voice-agent-eval.

What problem does it solve?

Text chatbots are easy to build. Voice agents that hold a structured conversation are harder, and checking how well a spoken session went is harder still. Teams that want spoken technical evaluations, design reviews or teach-back training usually end up with one of two things:

  • A scripted bot that reads questions and cannot react to what the person actually said.
  • A free-form voice chat with no structure, no grounding in the team's own material, and no score at the end.

I wanted one engine that does both halves: a voice agent that follows up like a person, grounded in plain markdown content, and a grader that scores every answer against a written reference.

How does it work?

One Node.js server and one React page. The server never streams audio; the browser talks to Gemini Live directly, and the server owns the content, the tools and the grading.

  1. Start a round. The server picks questions from the markdown bank (ones due for a repeat first, then new ones by priority) and mints a one-use Gemini token that expires shortly after the round. The API key never reaches the browser.
  2. Talk. The page streams microphone audio at 16 kHz and plays the agent's 24 kHz voice through AudioWorklets, over a WebSocket to Gemini Live.
  3. Live tool execution. When the agent needs something, it sends a tool call. The page runs it on the server and returns the result, and the call appears on screen with its arguments.
  4. Grade. When the call ends, the server measures delivery (pace, filler words, longest answer) and sends the transcript to Claude, which returns a typed report.

What can the agent do mid-conversation?

In evaluation mode the agent has 7 server tools. It decides when to call them, the server decides what they return:

ToolWhat it does
next_questionMoves to the next planned question
reference_answerReturns the short reference answer, so the agent can judge the answer and pick a follow up without reading it out
candidate_backgroundLooks up a line from the participant's profile to ask about real work
job_contextReads the context pack the round is grounded in (a system, a team, a product)
related_questionFinds a bank question on a topic the person raised, at most twice per session
record_noteSilently records its own 1 to 10 rating and a one-line note per main question, never said aloud
end_interviewEnds the session cleanly

A learn mode swaps in a tutor with its own tool set: it teaches the reference answer, takes questions, then hides the answer and asks the person to explain it back. Only the explain-back is graded.

How does the grading engine work?

  • Grounded in written references. Each question file has a short answer (all the live agent sees) and a deeper section that only the grader reads.
  • Typed output. Claude's reply comes back through structured outputs and is parsed against a Zod schema before anything is saved. A failed grade is kept, and the page can retry it.
  • Scores per answer. Every main question gets 1 to 10 on technical depth, structure and delivery, with what was missing and a stronger answer built only from facts in the profile.
  • Fallback on refusal. The grading request uses server-side fallbacks, so if a safety check declines it, the same call retries on another model.
  • Works without a key. A command prints the grading prompt to paste into any Claude chat, and the JSON reply can be posted back.

How does it improve between sessions?

  • Spaced repetition. Low scores bring a question back after 1, 3, then 7 days; high scores push it out to three weeks.
  • Gap notes. After each review, recurring gaps go into a notes file that the next session probes.
  • Agent self-correction. The grader also checks the agent itself: two questions in one turn, a follow up not based on what was said, giving the answer away. Each mistake it actually saw becomes a rule (10 at most) that the agent reads at the start of every new call.

How is it kept safe to run?

  • No keys in the browser. Each call gets its own one-use, short-lived Gemini token.
  • Private words. Any line of a profile or context pack that contains a listed word never reaches the live agent or the grader.
  • Local by default. The standalone server binds to 127.0.0.1 with no login.
  • Embeddable behind your login. The whole app is one Express router with a guard hook before every write and an onReview hook (or a webhook) after each grade, so an existing product can serve it under its own path and auth.
  • Path checks. Context pack names are validated so no request can read outside its folder.

What were the results?

MetricResult
StatusOpen source, MIT licensed, runs locally or embedded in another app
CodebaseAbout 5,500 lines of TypeScript across server, page, scripts and tests
Tests30 tests over question picking, spaced repetition, sessions, tools and the markdown renderer, with no network or API keys needed
Live checkA smoke script opens a real Gemini Live session the way the page does, has the model call a tool, and checks it uses the result