OpenAI-compatible endpoint

OpenAI proxy endpoints are not officially supported and you are likely to encounter errors. We strongly recommend using providers directly instead of via proxy endpoints (for example, using the Anthropic API directly instead of Claude through OpenRouter). For questions and support you can chat with the dev team and community on our Discord server.

To use OpenAI-compatible (/v1/chat/completions) endpoints with Letta, those endpoints must support function/tool calling.

You can configure Letta to use OpenAI-compatible ChatCompletions endpoints by setting OPENAI_API_BASE in your environment variables (in addition to setting OPENAI_API_KEY).

OpenRouter example

Create an account on OpenRouter, then create an API key.

Once you have your API key, set both OPENAI_API_KEY and OPENAI_API_BASE in your environment variables.

Using Letta Server via Docker

Simply set the environment variables when you use docker run:

$# replace `~/.letta/.persist/pgdata` with wherever you want to store your agent data
>docker run \
>  -v ~/.letta/.persist/pgdata:/var/lib/postgresql/data \
>  -p 8283:8283 \
>  -e OPENAI_API_BASE="https://openrouter.ai/api/v1" \
>  -e OPENAI_API_KEY="your_openai_api_key" \
>  letta/letta:latest

Using the Letta CLI

First we need to export the variables into our environment:

1export OPENAI_API_KEY="sk-..."  # your OpenRouter API key
2export OPENAI_API_BASE="https://openrouter.ai/api/v1"  # the OpenRouter OpenAI-compatible endpoint URL

Now, when we run letta run in the CLI, we can select OpenRouter models from the list of available models:

% letta run
? Would you like to select an existing agent? No
🧬 Creating new agent...
? Select LLM model: (Use arrow keys)
 » letta-free [type=openai] [ip=https://inference.memgpt.ai]
   google/gemini-pro-1.5-exp [type=openai] [ip=https://openrouter.ai/api/v1]
   google/gemini-flash-1.5-exp [type=openai] [ip=https://openrouter.ai/api/v1]
   google/gemini-flash-1.5-8b-exp [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.2-11b-vision-instruct:free [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.2-1b-instruct:free [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.2-3b-instruct:free [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.1-8b-instruct:free [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.2-1b-instruct [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.2-3b-instruct [type=openai] [ip=https://openrouter.ai/api/v1]
   google/gemini-flash-1.5-8b [type=openai] [ip=https://openrouter.ai/api/v1]
   mistralai/mistral-7b-instruct [type=openai] [ip=https://openrouter.ai/api/v1]
   mistralai/mistral-7b-instruct-v0.3 [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3-8b-instruct [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.1-8b-instruct [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.2-11b-vision-instruct [type=openai] [ip=https://openrouter.ai/api/v1]
   google/gemini-flash-1.5 [type=openai] [ip=https://openrouter.ai/api/v1]
   deepseek/deepseek-chat [type=openai] [ip=https://openrouter.ai/api/v1]
   cohere/command-r-08-2024 [type=openai] [ip=https://openrouter.ai/api/v1]
   openai/gpt-4o-mini [type=openai] [ip=https://openrouter.ai/api/v1]
   openai/gpt-4o-mini-2024-07-18 [type=openai] [ip=https://openrouter.ai/api/v1]
   mistralai/mistral-nemo [type=openai] [ip=https://openrouter.ai/api/v1]
   ...

For information on how to configure the Letta server or Letta Python SDK to use OpenRouter or other OpenAI-compatible endpoints providers, refer to our guide on using OpenAI.

Built with