Deploy Your Calendly AI Receptionist as a Voice Agent with ElevenLabs

Deploy Your Calendly AI Receptionist as a Voice Agent with ElevenLabs

Hey there,

You’ve got the Calendly AI Receptionist — now let’s give it a voice. ElevenLabs lets you create a natural-sounding AI phone receptionist that books, reschedules, and cancels appointments over the phone. Your customers call in, talk to the voice agent, and the appointment booking flows into your CRM automatically.

This guide walks you through connecting your Calendly AI Receptionist to ElevenLabs via the Second Brain OS MCP server.

Note: This guide uses the Calendly AI Receptionist as the example, but the ElevenLabs deployment process is identical regardless of which AI agent you purchased. Whether your agent handles appointment booking, lead qualification, product inquiries, or anything else — the steps for connecting it to ElevenLabs are exactly the same. Your agent’s tools and knowledge are configured in Second Brain OS; ElevenLabs is just the voice interface your callers speak through.

What You’ll Need

• An ElevenLabs account (elevenlabs.io — $5/mo starter plan)
• Your Second Brain OS account and API key
• Your SBOS user ID

Step 1: Create Your Voice Agent in ElevenLabs

1. Sign up or log in at elevenlabs.io
2. Navigate to Agents
3. Create a new agent
4. Set the first message — this is what the caller hears first, e.g. “Hi, this is [Name] from [Business]. How can I help you today?”

Step 2: Configure the System Prompt & Knowledge Base

The system prompt controls your AI receptionist’s personality and behaviour. A good voice agent prompt should cover:

• A friendly, professional persona (virtual receptionist, scheduling assistant, etc.)
• Conversational fillers to sound natural (“basically”, “I mean”, “well…”)
• Booking logic — don’t over-qualify, lock in the appointment quickly
• Never interrupt the caller
• Guardrails for edge cases and off-topic requests

For the knowledge base, you have two options:

• Option A: Append your business info directly after the behaviour section in the system prompt
• Option B: Go to Agents → Knowledge Base → Create Text and paste your content there

Then set your LLM settings:

• Model: GPT-4o — you’re welcome to experiment with newer models, but GPT-4o is the sweet spot for voice calls: fast enough for real-time conversation and smart enough for the scheduling tools
• Temperature: 1
• Timezone: Include Time: {{system__time}} in your system prompt — this is a dynamic variable that ElevenLabs injects automatically with the current time, so your agent always knows what “today” and “tomorrow” mean

Step 3: Choose a Voice

1. Navigate to Voice settings in your agent
2. Select Mark – ConvoAI (recommended for its stability and natural conversational tone). You can also pick another voice from the library, or clone your own (Voices → Create → Instant Voice Clone — record 30-60 seconds)
3. Set TTS model family to Flash (fastest response time for real-time calls)
4. Adjust voice parameters:
   • Stability: 70%
   • Speed: 1.10
   • Similarity: 90%
5. Under Advanced settings:
   • Eagerness: Patient (prevents the agent from interrupting callers)
   • Turn timeout: Increase slightly to give callers time to think

Update: Feel free to try the new TTS Mode called “V3 Conversational”. While it adds a tiny bit of delay, the calls feel way more natural. My verdict? Worth it!

Step 4: Connect the Second Brain OS MCP Server

This is where your Calendly AI Receptionist comes to life inside ElevenLabs.

First, add your SBOS API key as a workspace secret:

1. Go to elevenlabs.io/app/agents/settings
2. Under Workspace Secrets, click Add secret
3. Name it secondbrainos-api-key
4. For the value, paste your full SBOS API key with the Bearer prefix — e.g. Bearer rec7kxm83YNpqFw2a:f49e6b3c-a871-4d2f-9c44-518274610

Then, create an environment variable that references the secret:

1. Go to Developers → Environment Variables
2. Click Add environment variable
3. Name it SECONDBRAINOS_API_KEY (or any name you’ll remember)
4. Set the type to Secret and select the secondbrainos-api-key secret you just created
5. Click Save

Then, add the MCP server:

1. Go to elevenlabs.io/app/agents/integrations
2. Click Add Integration → Custom MCP Server
3. Configure the server:
   • Name: Second Brain OS MCP
   • Type: Streamable HTTP
   • URL: https://api.secondbrainos.com/mcp?user_id=YOUR_USER_ID

   Your user ID is available from your Second Brain OS dashboard. It’s also the first part of your API key — everything before the : (e.g. if your API key is rec7kxm83YNpqFw2a:f49e6b3c..., your user ID is rec7kxm83YNpqFw2a).

4. Under Request Headers, add a header:
   • Key: Authorization
   • Type: Environment Variable
   • Value: Select SECONDBRAINOS_API_KEY (the environment variable you created above)
5. Set Tool Approval to No Approval — the assistant can use any tool without asking
6. Configure Tool Settings:
   • Force pre-tool speech: On
   • Disable interruptions: On
   • Execution mode: Post-Speech
   • Tool call sound: Typing
   • Sound behavior: Always play
7. Check “I trust this MCP server” and click Add Server
8. Refresh the page and look for the green connection icon

Your Calendly tools will load automatically once connected.

Step 5: Make Sure Your Tools Are Visible

Your Calendly AI Receptionist comes with 7 scheduling tools. By default, 4 are already public and active:

• createCalendlyContactMeeting — book appointments
• cancelCalendlyMeeting — cancel with reason tracking
• updateCalendlyMeeting — reschedule appointments
• getCalendlyAvailability — check open time slots

These 4 tools handle the entire appointment booking lifecycle your AI phone receptionist needs. For most setups, this is all you need.

To verify or enable additional tools:

1. Log in at secondbrainos.com
2. Find each tool you want ElevenLabs to access
3. Set visibility to public
4. Set status to active

There are 3 more advanced tools available if you need them — we cover those in a separate guide.

Step 6: Test & Optimize

1. Click Preview / Test agent in ElevenLabs
2. Make a live test call
3. Try the full flow — ask about availability, book an appointment, reschedule it, cancel it

Common issues and fixes:

• AI interrupts the caller — add “never talk over the caller” to your prompt, and set Eagerness to Patient
• Weird tonality or phrasing — increase Stability (e.g. 0.55), adjust Similarity
• Over-qualifying before booking — update your prompt to lock in the appointment with minimal questions

After testing, verify your CRM data:

• Was the contact created?
• Is the appointment linked to the right contact?
• Did rescheduling and cancellation flow through correctly?

Once everything looks good, publish your agent.

How It Works Under the Hood

1. A customer calls your AI phone receptionist
2. ElevenLabs handles the conversation using your system prompt and LLM
3. When the caller wants to book, reschedule, or cancel — ElevenLabs calls your Calendly appointment booking tools via the SBOS MCP server
4. The tools execute the action in Calendly, which syncs with your calendar, which syncs with your CRM
5. The voice AI receptionist confirms the action to the caller in natural speech

All connector logic is rule-based and deterministic — no AI guessing on the backend. The data flows are resilient and zero-maintenance.

That’s it — your Calendly AI Receptionist now handles phone calls. Customers call in, book appointments by voice, and everything lands in your CRM properly linked.

Next up: check out the ChatKit guide if you want to add a website chat widget, or the Advanced Scenarios guide to unlock the remaining 3 tools and the Calendly workflow for multi-event-type setups.

This is part of a Crash Course on Calendly AI Agent. Access all the Automations to build a web & voice ai receptionist for booking, rescheduling & cancelling appointments by clicking the button below

Umair Kamil

Install Calendly AI Agent