ElevenLabs

The ElevenLabs source connects to a Conversational AI agent, plays its voice, and reports state and audio to the visual, including interruptions.

import { ElevenLabsVoiceSource } from "@sinua/voice/elevenlabs";

// A public agent: its agent_id. A private agent: a signed wss:// URL from your backend.
// The agent's user input format must be PCM (e.g. pcm_16000), set in the agent's settings.
export const publicAgent = new ElevenLabsVoiceSource({ credential: "agent_…" });

export async function privateAgent() {
  const { signedUrl } = await (await fetch("/api/elevenlabs-signed-url")).json();
  return new ElevenLabsVoiceSource({ credential: signedUrl });
}

Credentials

• A public agent needs only its agent id. There's no secret anywhere.
• A private agent needs a signed URL, minted by your server with your API key. It's valid for 15 minutes.

Packages

Good to know

• The agent's input format must be PCM. In the agent's settings, set user input audio format to a PCM format (pcm_16000 and so on). With μ-law input, the source refuses to connect and says why.
• How a turn ends. The agent's turn ends when ElevenLabs signals the reply is complete and its audio has finished playing. If the network stalls mid-reply, the visual stays in thinking rather than flipping to listening. thinking shows between the end of the user's speech and the reply's first audio, so with a fast agent it can be short.
• A conversation can't be resumed. When it closes, the source goes back to idle; connect again to start a new one.
• Start from a user gesture on the web, since the browser only allows the microphone and audio playback after one. On iOS, add NSMicrophoneUsageDescription; on Android, declare RECORD_AUDIO. Say where the audio goes: the user's voice is sent to ElevenLabs, so your iOS purpose string (and, on Android, the explanation you show before the system prompt, which carries no app text) should say so. For example: "Your voice is sent to ElevenLabs so the assistant can hear you." App Store review expects the purpose string to explain the use.