A Gemini Live Voice Agent in the Browser, Without Exposing Your API Key
How to stream voice straight from the browser to Gemini Live while the key, the prompt and the tools stay on your server: one-use tokens, server-built setup, tool calls relayed over HTTP, and session resumption.
Abhijeet Maharana · 8 min read
- AI agents
- Node.js
- TypeScript
Short version: A voice agent feels natural only when the audio goes straight from the browser to the model, with no server relaying every packet. But the browser is the one place your API key must never go. In voice-agent-eval I solved this with three pieces: the server mints a one-use Gemini Live token per session, the page streams audio to Gemini directly, and every tool the agent calls runs on the server. This post walks through each piece, with the code and the limits I hit along the way.
What is the problem?
Gemini Live is a WebSocket API. You open a socket, send a setup message (model, voice, system prompt, tools), then stream microphone audio up and get spoken audio and tool calls back.
There are two obvious ways to build it, and both are wrong for a product:
- Proxy everything through your server. The key stays safe, but every audio chunk makes an extra hop in both directions, and your server now holds one long-lived socket per user.
- Open the socket from the browser with your API key. Low delay and no server load, but anyone can open dev tools and copy the key.
What I wanted was the middle: audio goes browser to Google directly, while the key, the content and the tools stay on my server.
How does the browser connect without the key?
Google's Live API supports ephemeral tokens: short-lived credentials your backend creates with the real key and hands to the client. The client uses the token in place of the key, only for Live API, only on v1beta.
My server mints one when a session starts:
const expireTime = new Date(Date.now() + (minutes + 15) * 60_000).toISOString();
const res = await fetchImpl('https://generativelanguage.googleapis.com/v1beta/auth_tokens', {
method: 'POST',
headers: { 'content-type': 'application/json', 'x-goog-api-key': key },
body: JSON.stringify({ uses: 1, expireTime, newSessionExpireTime: expireTime }),
signal: AbortSignal.timeout(15_000),
});
Three settings matter:
| Field | What it controls | What I set |
|---|---|---|
uses | How many sessions the token can start | 1: one token, one session |
expireTime | How long the connection may keep sending | Planned session length plus 15 minutes |
newSessionExpireTime | How long the token can open a new session | Same as expireTime |
Google's default for newSessionExpireTime is one minute. I set it to the full length on purpose: a Live connection lasts only about 10 minutes, and resuming it later needs the same token to still be accepted. The docs say a resume works with the same token even when uses is 1.
The browser then opens the socket with the token as a query parameter:
const ws = new WebSocket(`${this.wsUrl}${sep}access_token=${encodeURIComponent(this.token)}`);
If someone copies that token, they get one session that ends shortly after mine. I can live with that risk; a leaked API key works for anyone until I notice and rotate it.
Who decides what the agent says?
The server, even though it never touches the audio. When a session starts, the server returns three things to the page: the socket URL, the token and a ready-made setup message. The page sends that setup as its first message, adding only a resume handle when it reconnects.
So the system prompt, the voice, the model and the tool list are all chosen on the server, from files the page has no say over: the question plan, the persona and the context packs. Someone could still edit the setup in dev tools before it goes out. With a one-use token, the most they get is one changed session on their own screen, which is why the token is kept short and narrow.
How do tools run on the server?
The tool list in the setup has names, descriptions and parameter schemas, but no code. When the agent wants one, Gemini sends a toolCall message down the socket to the browser. The page forwards each call to the server over plain HTTP, waits for the result and sends it back to Gemini:
void this.h
.onToolCalls(list)
.catch((err: Error) => list.map(() => ({ error: err.message })))
.then((responses) => {
this.send({ toolResponse: { functionResponses: list.map((call, i) => ({ id: call.id, name: call.name, response: responses[i] ?? { error: 'no result' } })) } });
});
On the server it is one Express route, POST /api/sessions/:id/tool, which looks up the session and runs the tool. The agent in voice-agent-eval has 7 tools in evaluation mode, for example next_question (the next item from the plan), reference_answer (a short written answer the agent judges against, without reading it out) and record_note (a silent 1 to 10 rating).
This split has two benefits:
- The agent's context stays small. A reference answer enters the conversation only when the agent asks for that one question, and only its short version is returned.
- Every call is visible. The page relays each call, so it shows the name, arguments and result on screen as the conversation goes on. For a voice agent, that is the best debugging tool I have.
One caution: a tool route is a write route that anyone with the page can call. In voice-agent-eval the whole app is an Express router with a guard hook that runs before every write, so an app that embeds it puts its own login in front.
Should tools block the conversation?
Gemini Live lets you mark each tool BLOCKING or NON_BLOCKING. A blocking call pauses the model until the result comes back. A non-blocking one runs in the background while the model keeps talking.
For a structured session, blocking is what you want. The agent should not invent the next question while next_question is still running. So the quick model gets blocking tools.
The extended thinking model was the surprise. It accepts only non-blocking tools and closes the connection with code 1007 if you send a blocking one. So the setup builder picks the behaviour per model and adds one rule to the prompt for the thinking model: say at most one short filler line, then wait for the result and never guess it.
What keeps a long session alive?
Two limits from Google's session docs will end your call if you ignore them:
- Audio-only sessions stop after 15 minutes unless you turn on context window compression. One line in the setup fixes it:
contextWindowCompression: { slidingWindow: {} }. - A single connection lasts about 10 minutes. Before it closes, the server sends a
goAwaymessage. WithsessionResumptionon, Gemini keeps sending a resume handle; the page saves the latest one, opens a new socket with the same token and passes the handle in the setup. The conversation carries on where it was.
The page allows up to 5 resumes per session, and gives up with a clear reason after that.
What about the audio itself?
Gemini Live wants 16 kHz PCM going up and sends 24 kHz PCM back. Microphones usually run at 44.1 or 48 kHz, so the page resamples in an AudioWorklet (a small audio processor that runs off the main thread) and posts 100 ms chunks with their loudness. A second worklet queues and plays the reply, and drops the queue instantly when Gemini says the person interrupted.
What I learned getting it to work on real machines:
- Start audio inside the click. Safari in particular only allows audio while the user's click still counts, so the page opens the speaker and microphone first, before any network call.
- Your own speaker will interrupt the agent. Without headphones, the agent hears itself and stops talking. While it speaks, the page only sends microphone chunks that are clearly louder than the echo.
- Log a heartbeat. Every few seconds the page records socket state, messages received, chunks sent and the loudest mic level. "The agent went quiet" turns into a clear cause: a closed socket, a muted mic, or Gemini simply not answering.
- Test against the real API. The repo has 30 tests that run with no network, plus a smoke script that opens a real Live session the same way the page does, has the model call a tool and checks it uses the result.
When does this pattern fit?
Use it when users talk to an agent in the browser and you care about delay, cost and keeping secrets on the server: voice support agents, spoken onboarding checks, training or evaluation tools. The rules carry over to any provider with short-lived client tokens:
- The server mints a token per session, as short and as narrow as the session allows.
- The server builds the setup; the page only forwards it.
- Tools are names in the setup and code on the server, behind your own auth.
If the server must record or transform every audio packet, a full proxy is simpler.
The code is MIT licensed on GitHub, and the case study covers the grading engine that scores each session afterwards.