To use OpenAI-compatible (/v1/chat/completions) endpoints with Letta, those endpoints must support function/tool calling.

You can configure Letta to use OpenAI-compatible ChatCompletions endpoints by setting OPENAI_API_BASE in your environment variables (in addition to setting OPENAI_API_KEY).

​
OpenRouter example

Create an account on OpenRouter, then create an API key.

Once you have your API key, set both OPENAI_API_KEY and OPENAI_API_BASE in your environment variables:

export OPENAI_API_KEY="sk-..."  # your OpenRouter API key
export OPENAI_API_BASE="https://openrouter.ai/api/v1"  # the OpenRouter OpenAI-compatible endpoint URL

Now, when we run letta run in the CLI, we can select OpenRouter models from the list of available models:

% letta run

? Would you like to select an existing agent? No

🧬 Creating new agent...
? Select LLM model: (Use arrow keys)
 » letta-free [type=openai] [ip=https://inference.memgpt.ai]
   google/gemini-pro-1.5-exp [type=openai] [ip=https://openrouter.ai/api/v1]
   google/gemini-flash-1.5-exp [type=openai] [ip=https://openrouter.ai/api/v1]
   google/gemini-flash-1.5-8b-exp [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.2-11b-vision-instruct:free [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.2-1b-instruct:free [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.2-3b-instruct:free [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.1-8b-instruct:free [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.2-1b-instruct [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.2-3b-instruct [type=openai] [ip=https://openrouter.ai/api/v1]
   google/gemini-flash-1.5-8b [type=openai] [ip=https://openrouter.ai/api/v1]
   mistralai/mistral-7b-instruct [type=openai] [ip=https://openrouter.ai/api/v1]
   mistralai/mistral-7b-instruct-v0.3 [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3-8b-instruct [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.1-8b-instruct [type=openai] [ip=https://openrouter.ai/api/v1]
   meta-llama/llama-3.2-11b-vision-instruct [type=openai] [ip=https://openrouter.ai/api/v1]
   google/gemini-flash-1.5 [type=openai] [ip=https://openrouter.ai/api/v1]
   deepseek/deepseek-chat [type=openai] [ip=https://openrouter.ai/api/v1]
   cohere/command-r-08-2024 [type=openai] [ip=https://openrouter.ai/api/v1]
   openai/gpt-4o-mini [type=openai] [ip=https://openrouter.ai/api/v1]
   openai/gpt-4o-mini-2024-07-18 [type=openai] [ip=https://openrouter.ai/api/v1]
   mistralai/mistral-nemo [type=openai] [ip=https://openrouter.ai/api/v1]
   ...

For information on how to configure the Letta server or Letta Python SDK to use OpenRouter or other OpenAI-compatible endpoints providers, refer to our guide on using OpenAI.

OpenAIAnthropic