Daisy Global

At a glance

Pick it when: Your callers speak English or one of the world languages below.

• One voice for the whole call, in every language it speaks.
• It follows the caller from one language to another. A language it cannot speak is answered in English, and the agent says once which languages it can help in.
• A short “mhm” or “okay” while the agent talks does not interrupt it. Only a real interruption stops the reply.

Need other languages? Daisy India: English and ten Indian languages, in one voice.

Two ways to use it

Set it up in the console

1. Open Agents and create an agent. On the Voice tab, pick the Daisy Global card. That sets the speech engines and the reply model for you.
2. Fill in the settings below, then talk to the agent from its page.
3. Click Publish. On the Share tab, click Add an endpoint and keep Authentication on Deployment key.
4. Copy the key under Issued deployment key. It is shown once. Store it on your server as VOICE_ORCH_DEPLOYMENT_KEY.

Your server mints a short-lived token for each call, and the page or app connects with it. The deployment key never leaves your server.

POST https://voice-orch-backend-100366934313.northamerica-northeast1.run.app/api/public/deployments/{publicId}/token
Authorization: Bearer <VOICE_ORCH_DEPLOYMENT_KEY>
Content-Type: application/json

{
  "endUserId": "customer-123",
  "sessionConfig": {
    "voice": { "languageCode": "en", "vad": { "silenceDurationMs": 600 } }
  }
}

# answer
{ "connectToken": "...", "websocketUrl": "wss://voice-orch-backend-100366934313.northamerica-northeast1.run.app/v1/agents/{publicId}/ws",
  "expiresAt": 1791234567, "deploymentId": "...", "publicId": "..." }

Then drop in the widget (Quickstart, Widget reference), or open websocketUrl?token=<connectToken> yourself and speak the deployment protocol. The token is single use.

sessionConfig may change, for one call: voice.languageCode, voice.voiceId, voice.ttsOutputFormat and the voice.vad values (threshold, prefixPaddingMs, silenceDurationMs, interruptResponse, bargeInThreshold, bargeInFrames). Nothing else: the token reaches a browser, and a browser must not be able to rewrite your agent’s prompt or model.

Get a key

For a direct connection, open API keys and create a key with Realtime voice ticked under Services included. It expires in 30 days, 90 days (the default), a year, or never. The key (voa_live_…) is shown once and stored only as a hash: keep it on your server. Revoking it stops new connections at once; calls already connected run on until they end.

Connect from your server

Daisy speaks OpenAI’s Realtime protocol. A client written for OpenAI Realtime moves here by changing three things: the URL, the key and the model.

import WebSocket from 'ws';

const ws = new WebSocket('wss://voice-orch-backend-100366934313.northamerica-northeast1.run.app/v1/realtime?model=daisy-global', {
  headers: { Authorization: `Bearer ${process.env.DAISY_API_KEY}` },
});

ws.on('message', (data) => {
  const event = JSON.parse(data.toString());
  console.log(event.type, event);
});

from openai import AsyncOpenAI

client = AsyncOpenAI(api_key="voa_live_...", base_url="https://voice-orch-backend-100366934313.northamerica-northeast1.run.app/v1")

async with client.realtime.connect(model="daisy-global") as connection:
    await connection.session.update(session={
        "type": "realtime",
        "instructions": "You are the booking assistant for Sunrise Clinic. Keep answers short.",
        "audio": {
            "input": {"transcription": {"language": "en"}},
            "output": {"voice": "sarah"},
        },
    })
    async for event in connection:
        print(event.type)

The OpenAI Agents SDK takes the same URL in model_config: {"url": "wss://voice-orch-backend-100366934313.northamerica-northeast1.run.app/v1/realtime?model=daisy-global", "api_key": "voa_live_..."}.

Connect from a browser

Never put the API key in a web page. Your server asks for a short-lived client secret and hands that to the browser, exactly as with OpenAI. client.realtime.client_secrets.create() in the OpenAI SDK sends this request.

POST https://voice-orch-backend-100366934313.northamerica-northeast1.run.app/v1/realtime/client_secrets
Authorization: Bearer voa_live_...
Content-Type: application/json

{
  "expires_after": { "anchor": "created_at", "seconds": 600 },
  "session": {
    "type": "realtime",
    "model": "daisy-global",
    "instructions": "You are the booking assistant for Sunrise Clinic. Keep answers short.",
    "audio": { "output": { "voice": "sarah" } }
  }
}

# answer
{ "value": "ek_daisy_...", "expires_at": 1791234567, "session": { ... } }

// With the OpenAI Agents SDK
import { RealtimeAgent, RealtimeSession } from '@openai/agents-realtime';

const session = new RealtimeSession(new RealtimeAgent({ name: 'Assistant' }), {
  transport: 'websocket',
  model: 'daisy-global',
});
await session.connect({ apiKey: clientSecret, url: 'wss://voice-orch-backend-100366934313.northamerica-northeast1.run.app/v1/realtime?model=daisy-global' });

// Or a plain WebSocket: the secret goes in a subprotocol
const ws = new WebSocket('wss://voice-orch-backend-100366934313.northamerica-northeast1.run.app/v1/realtime?model=daisy-global', ['realtime', `openai-insecure-api-key.${clientSecret}`]);

• A secret works once, for the model it names, and must be used before it expires: 10 seconds to 2 hours, 10 minutes by default. The call then lasts as long as any other.
• The session in it applies before anything the browser sends; the browser can still change it with session.update.
• Errors: 400 for another model or a bad expires_after, 401 for a missing, invalid or expired key, 403 for a key without Realtime voice, 413 for a session too large.
• A long-lived key sent from a browser still connects, but an error of type warning (code insecure_api_key) follows session.created.

Configure the session

session.created arrives as soon as the socket opens. Send one session.update; session.updated answers with the whole session as it is now in effect.

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "instructions": "You are the booking assistant for Sunrise Clinic. Keep answers short.",
    "audio": {
      "input": {
        "format": { "type": "audio/pcm", "rate": 24000 },
        "transcription": { "language": "en" },
        "turn_detection": { "type": "server_vad", "silence_duration_ms": 300 }
      },
      "output": {
        "format": { "type": "audio/pcm", "rate": 24000 },
        "voice": "sarah"
      }
    },
    "tools": [
      {
        "type": "function",
        "name": "find_slot",
        "description": "Find free appointment slots for a doctor on a date.",
        "parameters": {
          "type": "object",
          "properties": {
            "doctor": { "type": "string" },
            "date": { "type": "string", "description": "YYYY-MM-DD" }
          },
          "required": ["doctor", "date"]
        }
      }
    ],
    "tool_choice": "auto"
  }
}

• Fixed once the caller first speaks: sending the same value again is fine; a different one gets an error.
• Accepted, not applied: transcription.model, transcription.prompt, noise_reduction, eagerness, create_response, idle_timeout_ms, speed, tracing, include, truncation, prompt.
• Refused with an error: turn_detection: null (push to talk), a voice or audio format this model does not have, and tools that are not functions (MCP). The rest of the same update still applies.
• Tools are flat: name, description and parameters at the top level. Flatten Chat Completions tools ({"type": "function", "function": {...}}).
• The older beta shape (flat voice, input_audio_format, turn_detection, modalities) is accepted too.

Keep instructions and tools byte-identical across sessions. The model’s cache is keyed on them, so the first reply of a new session is fast (0.5 to 0.7 s to first text). Each request must fit in 8,192 tokens: keep instructions plus tools under about 3,500, and put anything per caller in a system item.

Languages

audio.input.transcription.language takes one of these codes to hear and answer in that language. Leave it out, or send "auto", to follow the caller. Regional spellings (hi-IN, en-US) mean the same. Pin it when your callers speak one language: very short replies are easy to mishear as another.

Voices

audio.output.voice. Voices by name, or by voice id. Ask us if you need a voice that is not listed. The default is jessica. The others: sarah, laura, alice, matilda, bella, lily, river, roger, charlie, george, callum, harry, liam, will, eric, chris, brian, daniel, adam, bill.

OpenAI voice names (alloy, marin, cedar and the rest) are accepted and use the default voice. Any other name gets an error, and the voice stays as it was.

Audio

• Mono, 16-bit little-endian, base64 in JSON text frames. We resample on our side.
• Send 20 to 100 ms per message, as it is captured. At most 256 KB per message.
• No faster than 4 times real time over 10 s: pace by the clock. A file pushed faster ends the call.
• Keep sending audio, silence included, for the whole call.

Turns and interruptions

Turns are decided on our server: it finds the caller's speech, then judges whether they have finished, so a pause mid-thought does not always end the turn. speech_started arrives within about 15 ms of the audio_start_ms it reports, speech_stopped within about 60 ms.

While the agent speaks, a short acknowledgement sends no speech_started and does not interrupt. A real interruption looks like this:

input_audio_buffer.speech_started        stop playback now
  ... a few more audio deltas of that reply may follow: drop them
response.output_audio_transcript.done
response.output_audio.done
response.content_part.done
response.output_item.done                status "incomplete"
conversation.item.done
response.done                            status "cancelled", reason "turn_detected"
input_audio_buffer.speech_stopped
input_audio_buffer.committed
conversation.item.added                  the caller's turn
conversation.item.input_audio_transcription.delta / .completed
conversation.item.done
response.created                         the next reply

• Drop audio after speech_started for the reply in progress, until its response.done.
• A reply can resume. If the interruption was nothing (a cough), the unheard part is spoken again as a new response.
• We trim the interrupted reply in our history to what we estimate the caller heard. conversation.item.truncate is acknowledged, but its audio_end_ms is not applied yet.
• response.cancel stops the reply at once (response.done, reason client_cancelled). interrupt_response: false lets every reply finish.
• Push to talk (turn_detection: null) is not supported yet. input_audio_buffer.commit ends the caller’s speech early, but the server still decides when the turn is over.

Transcripts

The caller’s come complete: one conversation.item.input_audio_transcription.delta with the whole text, then .completed, about 0.8 to 1.5 s after they stop. The recognisers revise their partials, so partials would garble. Custom vocabulary is not supported yet.

The agent’s stream: one response.output_audio_transcript.delta per phrase, sent before its audio (by 0.2 to 0.8 s). The deltas join up to the .done transcript.

Tools

Your tools run on your side. When the model calls one, the response ends with a function_call item. Run it, send the result, and the answer comes as a new response.

response.created                         resp_A
response.output_item.added               function_call, call_id "chatcmpl-tool-94ec..."
response.function_call_arguments.delta   the whole arguments, once
response.function_call_arguments.done   name "find_slot", arguments "{...}"
response.output_item.done
response.done                            status "completed", output [function_call]
  -> you send the function_call_output (and response.create)
conversation.item.added / .done          your function_call_output
response.created                         resp_B, the answer
response.output_audio_transcript.delta, response.output_audio.delta ...
response.done                            status "completed", output [message]

if (event.type === 'response.function_call_arguments.done') {
  const args = JSON.parse(event.arguments || '{}');
  const result = await runMyTool(event.name, args); // your code

  ws.send(JSON.stringify({
    type: 'conversation.item.create',
    item: {
      type: 'function_call_output',
      call_id: event.call_id,         // echo it exactly
      output: JSON.stringify(result), // a string
    },
  }));
  ws.send(JSON.stringify({ type: 'response.create' })); // optional
}

• call_id is opaque (ours look like chatcmpl-tool-…). Echo it back exactly.
• response.create after the result is optional: the model continues as soon as it has the result, and we absorb the extra one OpenAI clients send.
• Several calls in one turn (look up, then book): each ends its own response. Handle every call as it arrives.
• No time limit. If the caller speaks while you work, the turn moves on and a result sent after that gets an error.
• Do not count on a spoken “let me check” before a slow tool: the model often calls it straight away, and we add no filler of our own.
• Tool results are not kept in the history after the reply. If a later turn needs an id from a result, restate it in a system item.

Text, and speaking first

• output_modalities: ["text"] replies in response.output_text.* instead of audio.
• Send the caller’s words as text: conversation.item.create with a user message, then response.create.
• Agent speaks first: send response.create right after session.update, optionally with "response": {"instructions": "Greet the caller."}.
• response.create can set instructions, output_modalities and max_output_tokens for one reply. Tools, voice and temperature there are not applied, and a warning says so.

Steering a live call

Push guidance into a live call with a system item. It starts, cancels and delays nothing, and costs nothing.

{"type": "conversation.item.create", "item": {"type": "message", "role": "system",
 "content": [{"type": "input_text", "text": "The caller is an existing patient of Dr. Rao."}]}}

Changing instructions instead makes the model reread its whole context. Measured with a 2,700-token prompt and 19 tools, time to the reply’s first text:

• Each system item stays in the history (the last 48 items, inside 8,192 tokens). Keep them to a sentence or two. Items cannot be deleted.
• A note sent while the caller is still speaking shapes the reply to that turn; one sent after they stop shapes the next.
• After a reconnect, send the earlier conversation as user, assistant and system items before the first turn. None of them triggers a reply.

Events

Every event carries an event_id; events about a reply also carry response_id, item_id, output_index and content_index. Server to client:

Client to server. Anything else is answered with an error:

• Not sent: conversation.item.deleted, .retrieved, input_audio_transcription.failed, input_audio_buffer.timeout_triggered, output_audio_buffer.*, rate_limits.updated, MCP events, and response.cancelled (a cancelled reply ends with response.done).
• Our diagnostics: add &daisy_events=1 to the URL for our platform.* events (usage per reply and per turn, barge-in, warnings). Off by default, because the OpenAI SDK misreads types it does not know.
• Beta clients (header OpenAI-Beta: realtime=v1) get the beta names: response.audio.delta, response.audio_transcript.delta, response.text.delta, conversation.item.created.

Errors and close codes

{"type": "error", "event_id": "event_...", "error": {"type": "invalid_request_error",
 "code": "unknown_event", "message": "Unknown event type 'made.up'.", "param": "type",
 "event_id": "<your event_id, if you sent one>"}}

error.type is invalid_request_error, server_error or warning (log it and carry on). Codes: invalid_json, invalid_event, unknown_event, unsupported_event, unsupported_value, invalid_value, conversation_already_has_active_response, model_not_available, server_error, insecure_api_key.

Limits

Measured end of speech to first audio, on our test setup and without your network: English 2.12 s median (2.42 s p90).

Moving from OpenAI Realtime

1. Change the URL to wss://voice-orch-backend-100366934313.northamerica-northeast1.run.app/v1/realtime (or the SDK base_url to https://voice-orch-backend-100366934313.northamerica-northeast1.run.app/v1), the key to your API key, and the model to daisy-global.
2. Move a WebRTC browser client to the WebSocket transport, with client secrets.
3. Set the language if your callers speak one, and pick a voice from the list above.
4. Flatten Chat Completions tools, and keep server turn detection.

Everything else (instructions, tools, event names, ids, interruption, text in and out) works as on OpenAI. From Gemini Live, move the client to this protocol: setup becomes session.update, 16 kHz input is accepted as {"type": "audio/pcm", "rate": 16000}, toolCall is response.function_call_arguments.done, and interrupted is input_audio_buffer.speech_started.