OpenAI-compatible endpoint

OpenAI proxy endpoints are not officially supported and you are likely to encounter errors. We strongly recommend using providers directly instead of via proxy endpoints (for example, using the Anthropic API directly instead of Claude through OpenRouter). For questions and support you can chat with the dev team and community on our Discord server.

To use OpenAI-compatible (/v1/chat/completions) endpoints with Letta, those endpoints must support function/tool calling.

You can configure Letta to use OpenAI-compatible ChatCompletions endpoints by setting OPENAI_API_BASE in your environment variables (in addition to setting OPENAI_API_KEY).

OpenRouter example

Create an account on OpenRouter, then create an API key.

Once you have your API key, set both OPENAI_API_KEY and OPENAI_API_BASE in your environment variables:

1export OPENAI_API_KEY="sk-..."  # your OpenRouter API key
2export OPENAI_API_BASE="https://openrouter.ai/api/v1"  # the OpenRouter OpenAI-compatible endpoint URL

Now, when we run letta run in the CLI, we can select OpenRouter models from the list of available models:

% letta run
? Would you like to select an existing agent? No
🧬 Creating new agent...
? Select LLM model: (Use arrow keys)
 » letta-free [type=openai] [ip=https://inference.memgpt.ai]
   google/gemini-pro-1.5-exp [type=openai] [ip=https://openrouter.ai/api/v1]
   google/gemini-flash-1.5-exp [type=openai] [ip=https://openrouter.ai/api/v1]
   google/gemini-flash-1.5-8b-exp [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.2-11b-vision-instruct:free [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.2-1b-instruct:free [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.2-3b-instruct:free [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.1-8b-instruct:free [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.2-1b-instruct [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.2-3b-instruct [type=openai] [ip=https://openrouter.ai/api/v1]
   google/gemini-flash-1.5-8b [type=openai] [ip=https://openrouter.ai/api/v1]
   mistralai/mistral-7b-instruct [type=openai] [ip=https://openrouter.ai/api/v1]
   mistralai/mistral-7b-instruct-v0.3 [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3-8b-instruct [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.1-8b-instruct [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.2-11b-vision-instruct [type=openai] [ip=https://openrouter.ai/api/v1]
   google/gemini-flash-1.5 [type=openai] [ip=https://openrouter.ai/api/v1]
   deepseek/deepseek-chat [type=openai] [ip=https://openrouter.ai/api/v1]
   cohere/command-r-08-2024 [type=openai] [ip=https://openrouter.ai/api/v1]
   openai/gpt-4o-mini [type=openai] [ip=https://openrouter.ai/api/v1]
   openai/gpt-4o-mini-2024-07-18 [type=openai] [ip=https://openrouter.ai/api/v1]
   mistralai/mistral-nemo [type=openai] [ip=https://openrouter.ai/api/v1]
   ...

For information on how to configure the Letta server or Letta Python SDK to use OpenRouter or other OpenAI-compatible endpoints providers, refer to our guide on using OpenAI.

Built with