Stateful AgentsVoice Agents

Connecting with Livekit Agents

You can build an end-to-end stateful voice agent using Letta and Livekit. You can see a full example in the letta-voice repository.

For this example, you will need accounts with the following providers:

You will also need to set up the following environment variables (or create a .env file):

1LETTA_API_KEY=... # Letta Cloud API key (if using cloud)
2
3LIVEKIT_URL=wss://<YOUR-ROOM>.livekit.cloud # Livekit URL
4LIVEKIT_API_KEY=... # Livekit API key
5LIVEKIT_API_SECRET=... # Livekit API secret
6
7DEEPGRAM_API_KEY=... # Deepgram API key
8CARTESIA_API_KEY=... # Cartesia API key

Connecting to Letta Cloud

To connect to LiveKit, you can use the Letta connector openai.LLM.with_letta and pass in the agent_id of your voice agent.

Below is an example defining an entrypoint for a Livekit agent with Letta:

1import os
2from dotenv import load_dotenv
3from livekit import agents
4from livekit.agents import AgentSession, Agent, AutoSubscribe
5from livekit.plugins import (
6    openai,
7    cartesia,
8    deepgram,
9)
10load_dotenv()
11
12async def entrypoint(ctx: agents.JobContext):
13    agent_id = os.environ.get('LETTA_AGENT_ID')
14    print(f"Agent id: {agent_id}")
15    session = AgentSession(
16        llm=openai.LLM.with_letta(
17            agent_id=agent_id,
18        ),
19        stt=deepgram.STT(),
20        tts=cartesia.TTS(),
21    )
22
23    await session.start(
24        room=ctx.room,
25        agent=Agent(instructions=""), # instructions should be set in the Letta agent
26    )
27
28    session.say("Hi, what's your name?")
29    await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)

You can see the full script here.

Connecting to a self-hosted Letta server

You can also connect to a self-hosted server by specifying a base_url. To use LiveKit, your Letta sever needs to run with HTTPs. The easiest way to do this is by connecting ngrok to your Letta server.

Setting up ngrok

If you are self-hosting the Letta server locally (at localhost), you will need to use ngrok to expose your Letta server to the internet:

  1. Create an account on ngrok
  2. Create an auth token and add it into your CLI
ngrok config add-authtoken <YOUR_AUTH_TOKEN>
  1. Point your ngrok server to your Letta server:
ngrok http http://localhost:8283

Now, you should have a forwarding URL like https://<YOUR_FORWARDING_URL>.ngrok.app.

Connecting LiveKit to a self-hosted Letta server

To connect a LiveKit agent to a self-hosted Letta server, you can use the same code as above, but with the base_url parameter set to the forwarding URL you got from ngrok (or whatever HTTPS URL the Letta server is running on).

1import os
2from dotenv import load_dotenv
3from livekit import agents
4from livekit.agents import AgentSession, Agent, AutoSubscribe
5from livekit.plugins import (
6    openai,
7    cartesia,
8    deepgram,
9)
10load_dotenv()
11
12async def entrypoint(ctx: agents.JobContext):
13    agent_id = os.environ.get('LETTA_AGENT_ID')
14    print(f"Agent id: {agent_id}")
15    session = AgentSession(
16        llm=openai.LLM.with_letta(
17            agent_id=agent_id,
18            base_url="https://<YOUR_FORWARDING_URL>.ngrok.app", # point to your Letta server
19        ),
20        stt=deepgram.STT(),
21        tts=cartesia.TTS(),
22    )
23
24    await session.start(
25        room=ctx.room,
26        agent=Agent(instructions=""), # instructions should be set in the Letta agent
27    )
28
29    session.say("Hi, what's your name?")
30    await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)

You can see the full script here. `