Built-in Extractors

Letta Evals provides a set of built-in extractors that cover the most common extraction needs.

What are extractors? Extractors determine what part of an agent’s response gets evaluated. They take the full conversation trajectory and extract just the piece you want to grade.

Common Extractors

last_assistant

Extracts the last assistant message content.

1 extractor: last_assistant  # Most common - gets final response

first_assistant

Extracts the first assistant message content.

1 extractor: first_assistant

all_assistant

Concatenates all assistant messages with a separator.

1 extractor: all_assistant
2 extractor_config:
3   separator: "\n\n"  # Join messages with double newline

pattern

Extracts content matching a regex pattern.

1 extractor: pattern
2 extractor_config:
3   pattern: 'Result: (\d+)'  # Regex pattern to match
4   group: 1  # Extract capture group 1

tool_arguments

Extracts arguments from a specific tool call.

1 extractor: tool_arguments
2 extractor_config:
3   tool_name: search  # Which tool to extract from

tool_output

Extracts the return value from a specific tool call.

1 extractor: tool_output
2 extractor_config:
3   tool_name: search

memory_block

Extracts content from a specific memory block.

1 extractor: memory_block
2 extractor_config:
3   block_label: human  # Which memory block to extract

Important: This extractor requires the agent’s final state, which adds overhead.

after_marker

Extracts content after a specific marker string.

1 extractor: after_marker
2 extractor_config:
3   marker: "ANSWER:"
4   include_marker: false

Next Steps

Custom Extractors - Write your own extractors
Extractors Concept - Understanding extractors