AI LABS thumbnail

AI LABS

I Just Talk to My MCP AI Agents — And They Get Stuff DONE!

How to Set Up Advanced Voice Agents with MCP Integration: A Step-by-Step Guide

Imagine having a voice assistant that not only listens to you in real time and responds naturally but also interrupts when you speak, pulls in live data, and performs tasks on your behalf. Sounds futuristic? Thanks to recent developments in open-source voice agents and MCP (Multi-Channel Processing) servers, this is now achievable with relative ease. In this blog post, we’ll walk you through setting up such a voice agent, integrating it with MCP servers to access external knowledge, and customizing it to suit your needs.

What You’ll Learn

  • How to install and run a cutting-edge voice agent
  • How to set up necessary API keys for OpenAI, Cartisia, and LiveKit
  • How the voice agent works under the hood
  • How to customize the assistant’s personality and responses
  • How to integrate MCP servers for enhanced functionality, such as WhatsApp messaging
  • Insights into pricing and cost-effective options for voice synthesis APIs

Getting Started: Installation Made Simple

The voice agent project is open source and hosted on GitHub (link in the description). To get started:

  1. Clone the repository: Open your terminal and run
    git clone <repository-url>
    Replace <repository-url> with the actual URL of the project.

  2. Set up your Python environment:
    Navigate into the cloned folder, open the project in your preferred code editor (the video uses Cursor), and create a Python environment by running the command palette shortcut (command+shift+p) and selecting "Create Python Environment." Choose any available interpreter.

  3. Install dependencies:
    This project uses uv instead of pip for package management. Run the specific command provided in the project readme or video to install all dependencies.

Configuring API Keys: What You Need and How to Get Them

The voice agent requires three main API keys:

  • OpenAI API Key: Easily obtained by signing up on OpenAI’s platform.
  • Cartisia API Key: Register at Cartisia’s website and find your API key in your account dashboard.
  • LiveKit API Credentials:
  • Log into the LiveKit dashboard.
  • Navigate to the Settings > Keys section.
  • Copy the Server URL, API Key, and reveal the API Secret.
  • Paste all three into your environment file.

Create a .env file in your project directory containing all these keys in the specified format.

Running the Voice Agent

With dependencies installed and your .env configured:

  1. Run the command to start the voice agent server.
  2. The terminal will output a URL. Open this link in your browser.
  3. Ensure you have created a LiveKit project matching your API credentials.
  4. Connect to the LiveKit project via the browser link.
  5. You should hear a greeting from the voice agent, confirming it’s ready to interact.

Try speaking to it – the agent listens, responds promptly, and even allows you to interrupt naturally.

Under the Hood: How the Voice Agent Works

The core of the voice agent is a Python script that includes:

  • Voice Activity Detection (VA): This component detects when you speak and automatically pauses the assistant, making interactions smooth and natural.
  • Language Model: Powered by GPT-4o Mini, though you can swap it out for more powerful models if you need longer or more nuanced responses.
  • System Prompt: Defines how the assistant behaves — currently set to be witty, concise, and avoids complex words or emojis to keep text-to-speech smooth.
  • Text-to-Speech: Uses Cartisia’s Sonic Preview model, which is affordable and flexible, supporting customizable voice IDs.

Pricing Considerations

Compared to alternatives like 11 Labs, Cartisia offers a generous free tier with 20,000 credits. For example, transcribing 25 minutes of audio costs about 15 credits per second, which fits comfortably within the free tier for experimentation. Paid upgrades are available at competitive prices, making Cartisia a cost-effective choice for developers.

Customization: Make the Agent Your Own

You can easily tailor the assistant’s personality and behavior by:

  • Editing the greeting message directly in the code.
  • Modifying the system prompt to change tone, style, or even impersonate characters by adding descriptive prompts.
  • Selecting different voices through Cartisia’s API by changing the voice ID.

This level of customization lets you build anything from a professional assistant to a fun, character-driven chatbot.

MCP Integration: Extending the Voice Agent’s Capabilities

One of the most powerful features is integrating MCP servers. For example, the video demonstrates connecting a WhatsApp MCP server locally:

  • Voice inputs are transcribed and routed through the WhatsApp MCP server.
  • The server processes messages, sends replies, and interacts with your WhatsApp messages.
  • The voice agent interacts with any MCP-registered tool, allowing you to add multiple services seamlessly.

This setup supports building specialized voice agents, like an Airbnb booking assistant or an internet search assistant via Brave Search MCP, enabling voice-first workflows.

Real-Time Usage and Testing

The voice agent works with minimal latency (2-3 seconds from speaking to response), offering a near real-time conversation experience. Although there are minor transcription errors due to the underlying speech-to-text model (OpenAI’s Whisper), the overall performance is impressive.

Final Thoughts and Next Steps

This open-source voice agent project is an excellent foundation for anyone interested in voice-first applications with powerful integrations. With easy setup, flexible customization, and expandable MCP integration, you can build assistants tailored to your needs.

What’s Next?

  • Keep an eye on updates addressing interruption handling improvements.
  • Explore different MCP integrations to unlock new capabilities.
  • Clone the repository and experiment with the code.

Support and Stay Connected

If you find this guide helpful and want to dive deeper into voice agents, consider subscribing to related channels or supporting the creators. They regularly share tutorials that can help you build sophisticated voice solutions.

Links Mentioned

  • GitHub Repository (voice agent project)
  • Cartisia API Sign-up
  • LiveKit Dashboard
  • WhatsApp MCP Setup Video
  • MCP Use Library Documentation

Ready to build your own voice assistant that listens, talks, and acts on your behalf? Get started today and transform how you interact with technology!

← Back to AI LABS Blog