Data sources allow you to load files into an agent’s archival memory. Agents’ archival memory is a set of Passage objects, which consist of a chunk of text and a corresponding embedding. Data sources consist of a group of Passage objects which can be copied into an agent’s archival memory.

Creating a data source

To create a data source, you will need to specify a unique name as well as an EmbeddingConfig:

1
# get an available embedding_config
2
embedding_configs = client.models.list_embedding_models()
3
embedding_config = embedding_configs[0]
4
5
# create the source
6
source = client.sources.create(
7
    name="my_source", 
8
    embedding_config=embedding_config
9
)

Now that you’ve created the source, you can start loading data into the source.

Uploading a file into a data source

Uploading a file to a source will create an async job for processing the file, which will split the file into chunks and embed them.

1
# upload a file into the source
2
job = client.sources.files.upload(
3
    source_id=source.id, 
4
    file=open("my_file.txt", "rb")
5
)
6
# wait until the job is completed
7
while True:
8
    job = client.jobs.retrieve(job.id)
9
    if job.status == "completed":
10
        break
11
    elif job.status == "failed":
12
        raise ValueError(f"Job failed: {job.metadata}")
13
    print(f"Job status: {job.status}")
14
    time.sleep(1)

Once the job is completed, you can list the files and the generated passages in the source:

1
# list files in the source
2
files = client.sources.files.list(source_id=source.id)
3
print(f"Files in source: {files}")
4
5
# list passages in the source
6
passages = client.sources.passages.list(source_id=source.id)
7
print(f"Passages in source: {passages}")

Listing available data sources

You can view available data sources by listing them:

1
# list sources
2
sources = client.sources.list()

Connecting a data source to an agent

When you attach a data source to an agent, the passages in that data source will be copied into the agent’s archival memory. Note that if you load new data into your data source, it will not be copied into the agent’s archival memory unless you re-attach the data sources (detach then attach it again)

Attaching the data source

You can attach a source to an agent by specifying both the source and agent IDs:

1
client.agents.sources.attach(agent_id=agent.id, source_id=source.id)

Note that your agent and source must be configured with the same embedding model, to ensure that the agent is able to search accross a common embedding space for archival memory.

Detaching the data source

Detaching a data source will remove the passages in an agent’s archival memory that was loaded from that source:

1
client.agents.sources.detach(agent_id=agent.id, source_id=source.id)