Skip to content
My Docs My Docs Letta Docs
  • Auto
  • Light
  • Dark
DiscordForumGitHubSign up
RAG
Agentic RAG with Letta
View as Markdown
Copy Markdown
Open in Claude
Open in ChatGPT

Agentic RAG with Letta

In the agentic RAG approach, we delegate the retrieval process to the agent itself. Instead of your application deciding what to search for, we provide the agent with a custom tool that allows it to query your vector database directly. This makes the agent more autonomous and your client-side code much simpler.

By the end of this tutorial, you’ll have a research assistant that autonomously decides when to search your vector database and what queries to use.

Prerequisites

Section titled “Prerequisites”

To follow along, you need free accounts for:

  • Letta: To access the agent development platform
  • Hugging Face: To generate embeddings (MongoDB and Qdrant users only)
  • One of the following vector database platforms:

You also need a code editor and either Python (version 3.8 or later) or Node.js (version 18 or later).

Getting your API keys

Section titled “Getting your API keys”

You need API keys for Letta and your chosen vector database.

Get your Letta API key

  1. Create a Letta account

    If you don’t have one, sign up for a free account at letta.com.

  2. Navigate to API keys

    Once logged in, click API keys in the sidebar.

  3. Create and copy your key

    Click + Create API key, give it a descriptive name, and click Confirm. Copy the key and save it somewhere safe. Letta API key navigation

Get your vector database credentials

  1. Create a ChromaDB Cloud account

    Sign up for a free account on the ChromaDB Cloud website.

  2. Create a new database

    From your dashboard, create a new database. ChromaDB new project

  3. Get your API key and host

    In your project settings, find your API Key, Tenant, Database, and Host URL. You’ll need all four values for your scripts. ChromaDB keys
Get your Hugging Face API token (MongoDB and Qdrant users)

  1. Create a Hugging Face account

    Sign up for a free account at huggingface.co.

  2. Create an access token

    Click the profile icon in the top right. Navigate to Settings > Access Tokens (or go directly to huggingface.co/settings/tokens).

  3. Generate a new token

    Click New token, give it a name (such as Letta RAG Demo), select the Read role, and click Create token. Copy the token and save it securely. Hugging Face token

Once you have these credentials, create a .env file in your project directory. Add the credentials for your chosen database:

Terminal window
LETTA_API_KEY="..."
CHROMA_API_KEY="..."
CHROMA_TENANT="..."
CHROMA_DATABASE="..."

Step 1: Set up the vector database

Section titled “Step 1: Set up the vector database”

First, you need to populate your chosen vector database with the content of the research papers. We’ll use two papers for this demo:

Before we begin, let’s set up our development environment:

Terminal window
# Create a Python virtual environment to keep dependencies isolated
python -m venv venv
source venv/bin/activate  # On Windows, use: venv\Scripts\activate

Typescript users must update package.json to use ES modules:

"type": "module"

Download the research papers using curl with the -L flag to follow redirects:

curl -L -o 1706.03762.pdf https://arxiv.org/pdf/1706.03762.pdf
curl -L -o 1810.04805.pdf https://arxiv.org/pdf/1810.04805.pdf

Verify that the PDFs downloaded correctly:

file 1706.03762.pdf 1810.04805.pdf

You should see output indicating these are PDF documents, not HTML files.

Install the necessary packages for your chosen database:

requirements.txt
letta-client
chromadb
pypdf
python-dotenv

For Python, install the packages with the following command:

Terminal window
pip install -r requirements.txt

Now create a setup.py file (Python) or a setup.ts file (TypeScript) to load the PDFs, split them into chunks, and ingest them into your database:

import os
import chromadb
import pypdf
from dotenv import load_dotenv


load_dotenv()


def main():
    # Connect to ChromaDB Cloud
    client = chromadb.CloudClient(
        tenant=os.getenv("CHROMA_TENANT"),
        database=os.getenv("CHROMA_DATABASE"),
        api_key=os.getenv("CHROMA_API_KEY")
    )


    # Create or get the collection
    collection = client.get_or_create_collection("rag_collection")


    # Ingest PDFs
    pdf_files = ["1706.03762.pdf", "1810.04805.pdf"]
    for pdf_file in pdf_files:
        print(f"Ingesting {pdf_file}...")
        reader = pypdf.PdfReader(pdf_file)
        for i, page in enumerate(reader.pages):
            collection.add(
                ids=[f"{pdf_file}-{i}"],
                documents=[page.extract_text()]
            )


    print("\nIngestion complete!")
    print(f"Total documents in collection: {collection.count()}")


if __name__ == "__main__":
    main()

Run the script from your terminal:

Terminal window
python setup.py

If you’re using MongoDB Atlas, manually create a vector search index by following the steps below.

Create the vector search index (MongoDB Atlas Only)

  1. Navigate to Atlas Search

    Log in to your MongoDB Atlas dashboard, navigate to your cluster, and click on the Atlas Search tab.

  2. Create a search index

    Click Create Search Index, then choose JSON Editor (not Visual Editor).

  3. Select the database and collection

    • Database: Select rag_demo (or whatever you set as MONGODB_DB_NAME).
    • Collection: Select rag_collection.

  4. Name and configure the index

    • Index Name: Enter vector_index (this exact name is required by the code).
    • Copy and paste this JSON definition as the configuration:
    {
      "fields": [
        {
          "type": "vector",
          "path": "embedding",
          "numDimensions": 384,
          "similarity": "cosine"
        }
      ]
    }

    Note: The 384 dimensions are for Hugging Face’s BAAI/bge-small-en-v1.5 model.

  5. Create and wait

    Click Create Search Index. The index takes a few minutes to build. Wait until it displays an Active status before proceeding.

Your vector database is now populated with research paper content and ready to query.

Step 2: Create a custom search tool

Section titled “Step 2: Create a custom search tool”

A Letta tool is a Python function that your agent can call. We’ll create a function that searches your vector database and returns the results. Letta handles the complexities of exposing this function to the agent securely.

Create a new file named tools.py (Python) or tools.ts (TypeScript) with the appropriate implementation for your database:

def search_research_papers(query_text: str, n_results: int = 1) -> str:
    """
    Searches the research paper collection for a given query.


    Args:
        query_text (str): The text to search for.
        n_results (int): The number of results to return.


    Returns:
        str: The most relevant document found.
    """
    import chromadb
    import os


    # ChromaDB Cloud Client
    # This tool code is executed on the Letta server. It expects the ChromaDB
    # credentials to be passed as environment variables.
    api_key = os.getenv("CHROMA_API_KEY")
    tenant = os.getenv("CHROMA_TENANT")
    database = os.getenv("CHROMA_DATABASE")


    if not all([api_key, tenant, database]):
        raise ValueError("CHROMA_API_KEY, CHROMA_TENANT, and CHROMA_DATABASE must be set as environment variables.")


    client = chromadb.CloudClient(
        tenant=tenant,
        database=database,
        api_key=api_key
    )


    collection = client.get_or_create_collection("rag_collection")


    try:
        results = collection.query(
            query_texts=[query_text],
            n_results=n_results
        )


        document = results['documents'][0][0]
        return document
    except Exception as e:
        return f"Tool failed with error: {e}"

This function takes a query, connects to your database, retrieves the most relevant documents, and returns them as a single string.

Step 3: Configure an agentic research assistant

Section titled “Step 3: Configure an agentic research assistant”

Next, we’ll create a new agent. This agent will have a specific persona that determines how it behaves. We’ll equip the agent with our new search tool, with dependencies configured programmatically.

Create a file named create_agentic_agent.py (Python) or create_agentic_agent.ts (TypeScript):

import os
from letta_client import Letta
from dotenv import load_dotenv
from tools import search_research_papers


load_dotenv()


# Initialize the Letta client
client = Letta(api_key=os.getenv("LETTA_API_KEY"))


# Create a tool from our Python function with dependencies configured
search_tool = client.tools.create_from_function(
    func=search_research_papers,
    pip_requirements=[{"name": "chromadb"}]
)


# Define the agent's persona
persona = """You are a world-class research assistant. Your goal is to answer questions accurately by searching through a database of research papers. When a user asks a question, first use the `search_research_papers` tool to find relevant information. Then, answer the user's question based on the information returned by the tool."""


# Create the agent with the tool and environment variables
agent = client.agents.create(
    name="Agentic RAG Assistant",
    model="openai/gpt-4o-mini",
    embedding="openai/text-embedding-3-small",
    description="A smart agent that can search a vector database to answer questions.",
    memory_blocks=[
        {
            "label": "persona",
            "value": persona
        }
    ],
    tools=[search_tool.name],
    secrets={
        "CHROMA_API_KEY": os.getenv("CHROMA_API_KEY"),
        "CHROMA_TENANT": os.getenv("CHROMA_TENANT"),
        "CHROMA_DATABASE": os.getenv("CHROMA_DATABASE")
    }
)


print(f"Agent '{agent.name}' created with ID: {agent.id}")

Run this script once to create the agent in your Letta project:

Terminal window
python create_agentic_agent.py

Your agent is now fully configured! You’ve set both the tool dependencies and environment variables programmatically.

Step 4: Let the agent lead the conversation

Section titled “Step 4: Let the agent lead the conversation”

With the agentic setup, our client-side code becomes incredibly simple. We no longer need to worry about retrieving context. We just send the user’s raw question to the agent and let it handle the rest.

Create the agentic_rag.py or agentic_rag.ts script:

import os
from letta_client import Letta
from dotenv import load_dotenv


load_dotenv()


# Initialize client
letta_client = Letta(api_key=os.getenv("LETTA_API_KEY"))


AGENT_ID = "your-agentic-agent-id"  # Replace with your new agent ID


def main():
    while True:
        user_query = input("\nAsk a question about the research papers: ")
        if user_query.lower() in ['exit', 'quit']:
            break


        response = letta_client.agents.messages.create(
            agent_id=AGENT_ID,
            messages=[{"role": "user", "content": user_query}]
        )


        for message in response.messages:
            if message.message_type == 'assistant_message':
                print(f"\nAgent: {message.content}")


if __name__ == "__main__":
    main()

When you run this script, the agent receives the question, understands from its persona that it needs to search for information, calls the search_research_papers tool, gets the context, and then formulates an answer. All the RAG logic is handled by the agent, not your application.

Next steps

Section titled “Next steps”

Now that you’ve integrated agentic RAG with Letta, you can expand on this foundation.

Simple RAG

Learn how to manage retrieval on the client-side for complete control.

Custom tools

Explore creating more advanced custom tools for your agents.