Image inputs

The Letta API supports image inputs, enabling LLMs that have vision capabilities to see images natively.

Letta Code supports image inputs via the CLI. Simply use CTRL-V to copy-paste an image from your clipboard into your input message. Letta Code can also read images on your computer using the Read tool.

Not all LLMs support image inputs. Ensure your agent is configured with a multi-modal capable LLM when using image inputs.

Model Support

Multi-modal capabilities depend on the underlying language model. You can check which models from the API providers support image inputs by checking their individual model pages:

• OpenAI: Most models beyond GPT-4.1, o1/3/4, GPT-4o
• Anthropic: Most models beyond Claude Opus 4, Claude Sonnet 4
• Gemini: Most models beyond Gemini 2.5 Pro, Gemini 2.5 Flash

If the provider you’re using doesn’t support image inputs, your images will still appear in the context window, but as a text message telling the agent that an image exists.

Sending an Image via URL

TypeScript

import Letta from "@letta-ai/letta-client";

const client = new Letta({ apiKey: process.env.LETTA_API_KEY });

const response = await client.agents.messages.create(agentState.id, {
  messages: [
    {
      role: "user",
      content: [
        {
          type: "text",
          text: "Describe this image.",
        },
        {
          type: "image",
          source: {
            type: "url",
            url: "https://upload.wikimedia.org/wikipedia/commons/a/a7/Camponotus_flavomarginatus_ant.jpg",
          },
        },
      ],
    },
  ],
});

Python

from letta_client import Letta
import os

client = Letta(api_key=os.getenv("LETTA_API_KEY"))

response = client.agents.messages.create(
    agent_id=agent_state.id,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe this image."
                },
                {
                    "type": "image",
                    "source": {
                        "type": "url",
                        "url": "https://upload.wikimedia.org/wikipedia/commons/a/a7/Camponotus_flavomarginatus_ant.jpg",
                    },
                }
            ],
        }
    ],
)

Sending an Image via Base64

Different LLM providers have different max support images (count and resolution). If you encounter an error, check the LLM provider’s documentation to make sure your images are the right size, and you are not sending more images than supported.

TypeScript

import Letta from "@letta-ai/letta-client";

const client = new Letta({ apiKey: process.env.LETTA_API_KEY });

const imageUrl =
  "https://upload.wikimedia.org/wikipedia/commons/a/a7/Camponotus_flavomarginatus_ant.jpg";
const imageResponse = await fetch(imageUrl);
const imageBuffer = await imageResponse.arrayBuffer();
const imageData = Buffer.from(imageBuffer).toString("base64");

const response = await client.agents.messages.create(agentState.id, {
  messages: [
    {
      role: "user",
      content: [
        {
          type: "text",
          text: "Describe this image.",
        },
        {
          type: "image",
          source: {
            type: "base64",
            mediaType: "image/jpeg",
            data: imageData,
          },
        },
      ],
    },
  ],
});

Python

import base64
import httpx
from letta_client import Letta

client = Letta(api_key="LETTA_API_KEY")

image_url = "https://upload.wikimedia.org/wikipedia/commons/a/a7/Camponotus_flavomarginatus_ant.jpg"
image_data = base64.standard_b64encode(httpx.get(image_url).content).decode("utf-8")

response = client.agents.messages.create(
    agent_id=agent_state.id,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe this image."
                },
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/jpeg",
                        "data": image_data,
                    },
                }
            ],
        }
    ],
)