Ollama

⚠️ Make sure to use tags when downloading Ollama models!

Don’t do ollama pull dolphin2.2-mistral, instead do ollama pull dolphin2.2-mistral:7b-q6_K.

If you don’t specify a tag, Ollama may default to using a highly compressed model variant (e.g. Q4). We highly recommend NOT using a compression level below Q5 when using GGUF (stick to Q6 or Q8 if possible). In our testing, certain models start to become extremely unstable (when used with MemGPT) below Q6.

Setup Ollama

Download + install Ollama and the model you want to test with
Download a model to test with by running ollama pull <MODEL_NAME> in the terminal (check the Ollama model library for available models)

For example, if we want to use Dolphin 2.2.1 Mistral, we can download it by running:

# Let's use the q6_K variant
ollama pull dolphin2.2-mistral:7b-q6_K

pulling manifest
pulling d8a5ee4aba09... 100% |█████████████████████████████████████████████████████████████████████████| (4.1/4.1 GB, 20 MB/s)
pulling a47b02e00552... 100% |██████████████████████████████████████████████████████████████████████████████| (106/106 B, 77 B/s)
pulling 9640c2212a51... 100% |████████████████████████████████████████████████████████████████████████████████| (41/41 B, 22 B/s)
pulling de6bcd73f9b4... 100% |████████████████████████████████████████████████████████████████████████████████| (58/58 B, 28 B/s)
pulling 95c3d8d4429f... 100% |█████████████████████████████████████████████████████████████████████████████| (455/455 B, 330 B/s)
verifying sha256 digest
writing manifest
removing any unused layers
success

Configuring with Ollama

First, make sure you install Ollama dependencies with:

pip install 'letta[ollama]'

To configure your LLM model, modify the [model] section of your ~/.letta/config file:

~/.letta/config

[model]
model_endpoint_type = ollama
model = dolphin2.2-mistral:7b-q6_K
model_endpoint = http://localhost:11434
context_window = 8192

To configure your embedding model, modify the [embedding] section of your ~/.letta/config file:

~/.letta/config

[embedding]
embedding_endpoint_type = ollama
embedding_endpoint = http://localhost:11434
embedding_model = mxbai-embed-large 
embedding_dim = 512
embedding_chunk_size = 300

Configuring with `docker compose`

The compose.yaml file uses enviornment variables to configure the LLM and embedding models. You can place these enviornment variables in either a .env file or set the variables.

Configuring LLM endpoints:

LETTA_LLM_ENDPOINT=http://host.docker.internal:11434
LETTA_LLM_ENDPOINT_TYPE=ollama
LETTA_LLM_MODEL=dolphin2.2-mistral:7b-q6_K
LETTA_LLM_CONTEXT_WINDOW=8192

Configuring embeddings:

LETTA_EMBEDDING_ENDPOINT=http://host.docker.internal:11434
LETTA_EMBEDDING_ENDPOINT_TYPE=ollama
LETTA_EMBEDDING_MODEL=mxbai-embed-large
LETTA_EMBEDDING_DIM=512

from memgpt import LLMConfig, EmbeddingConfig

# override the default LLM config
client.set_default_llm_config(
    LLMConfig(
        model="dolphin2.2-mistral:7b-q6_K",
        model_endpoint_type="ollama",
        model_endpoint="http://localhost:11434",
        context_window=8192
    )
)

# override the default embedding config
client.set_default_embedding_config( 
    EmbeddingConfig(
        embedding_endpoint_type="ollama",
        embedding_endpoint="http://localhost:11434",
        embedding_model="mxbai-embed-large",
        embedding_dim=512,
    )
)

Get Started

Key concepts

Agents

Deployment

Advanced

Setup Ollama

Configuring with Ollama

Configuring with `docker compose`

Get Started

Key concepts

Agents

Deployment

Advanced

​Setup Ollama

​Configuring with Ollama

Configuring with docker compose

Setup Ollama

Configuring with Ollama

Configuring with `docker compose`