Case study: AI agents and automation
Real-Time Voice AI Agent with an LLM Grading Engine
One engine runs any spoken evaluation from markdown files: live voice, visible tool calls, and Claude grading against your own reference answers.
An open-source voice agent on Gemini Live that runs structured spoken sessions, calls server tools mid-conversation, and hands the transcript to Claude for scored, typed reports.
- Where
- Open source (MIT) on GitHub
- Role
- Author and maintainer
- When
- 2026
- Team
- Solo, end to end
- 7
- server tools the agent calls mid-conversation, each shown on screen
- 0
- API keys in the browser; every call gets a one-use Gemini token
- 30
- tests, run with no network and no API keys
- TypeScript
- Node.js
- Express
- React
- Gemini Live API
- Claude API
- Zod
- AudioWorklets
- WebSockets
TL;DR: voice-agent-eval is an open-source real-time voice AI agent with an evaluation engine behind it. A person talks to it out loud on Gemini Live. While it talks, it calls tools on the server to fetch the next question, check a reference answer or look up background, and every call shows on screen. When the call ends, Claude grades the transcript against your own reference answers and writes a report. Source: github.com/abhijeet-maharana/voice-agent-eval.
What problem does it solve?
Text chatbots are easy to build. Voice agents that hold a structured conversation are harder, and checking how well a spoken session went is harder still. Teams that want spoken technical evaluations, design reviews or teach-back training usually end up with one of two things:
- A scripted bot that reads questions and cannot react to what the person actually said.
- A free-form voice chat with no structure, no grounding in the team's own material, and no score at the end.
I wanted one engine that does both halves: a voice agent that follows up like a person, grounded in plain markdown content, and a grader that scores every answer against a written reference.
How does it work?
One Node.js server and one React page. The server never streams audio; the browser talks to Gemini Live directly, and the server owns the content, the tools and the grading.
- Start a round. The server picks questions from the markdown bank (ones due for a repeat first, then new ones by priority) and mints a one-use Gemini token that expires shortly after the round. The API key never reaches the browser.
- Talk. The page streams microphone audio at 16 kHz and plays the agent's 24 kHz voice through AudioWorklets, over a WebSocket to Gemini Live.
- Live tool execution. When the agent needs something, it sends a tool call. The page runs it on the server and returns the result, and the call appears on screen with its arguments.
- Grade. When the call ends, the server measures delivery (pace, filler words, longest answer) and sends the transcript to Claude, which returns a typed report.
What can the agent do mid-conversation?
In evaluation mode the agent has 7 server tools. It decides when to call them, the server decides what they return:
| Tool | What it does |
|---|---|
next_question | Moves to the next planned question |
reference_answer | Returns the short reference answer, so the agent can judge the answer and pick a follow up without reading it out |
candidate_background | Looks up a line from the participant's profile to ask about real work |
job_context | Reads the context pack the round is grounded in (a system, a team, a product) |
related_question | Finds a bank question on a topic the person raised, at most twice per session |
record_note | Silently records its own 1 to 10 rating and a one-line note per main question, never said aloud |
end_interview | Ends the session cleanly |
A learn mode swaps in a tutor with its own tool set: it teaches the reference answer, takes questions, then hides the answer and asks the person to explain it back. Only the explain-back is graded.
How does the grading engine work?
- Grounded in written references. Each question file has a short answer (all the live agent sees) and a deeper section that only the grader reads.
- Typed output. Claude's reply comes back through structured outputs and is parsed against a Zod schema before anything is saved. A failed grade is kept, and the page can retry it.
- Scores per answer. Every main question gets 1 to 10 on technical depth, structure and delivery, with what was missing and a stronger answer built only from facts in the profile.
- Fallback on refusal. The grading request uses server-side fallbacks, so if a safety check declines it, the same call retries on another model.
- Works without a key. A command prints the grading prompt to paste into any Claude chat, and the JSON reply can be posted back.
How does it improve between sessions?
- Spaced repetition. Low scores bring a question back after 1, 3, then 7 days; high scores push it out to three weeks.
- Gap notes. After each review, recurring gaps go into a notes file that the next session probes.
- Agent self-correction. The grader also checks the agent itself: two questions in one turn, a follow up not based on what was said, giving the answer away. Each mistake it actually saw becomes a rule (10 at most) that the agent reads at the start of every new call.
How is it kept safe to run?
- No keys in the browser. Each call gets its own one-use, short-lived Gemini token.
- Private words. Any line of a profile or context pack that contains a listed word never reaches the live agent or the grader.
- Local by default. The standalone server binds to
127.0.0.1with no login. - Embeddable behind your login. The whole app is one Express router with a
guardhook before every write and anonReviewhook (or a webhook) after each grade, so an existing product can serve it under its own path and auth. - Path checks. Context pack names are validated so no request can read outside its folder.
What were the results?
| Metric | Result |
|---|---|
| Status | Open source, MIT licensed, runs locally or embedded in another app |
| Codebase | About 5,500 lines of TypeScript across server, page, scripts and tests |
| Tests | 30 tests over question picking, spaced repetition, sessions, tools and the markdown renderer, with no network or API keys needed |
| Live check | A smoke script opens a real Gemini Live session the way the page does, has the model call a tool, and checks it uses the result |