Extractors

Custom Extractors

Create your own extractors to pull exactly what you need from agent trajectories.

While built-in extractors cover common cases, custom extractors let you implement specialized extraction logic for your specific use case.

Why Custom Extractors?

Use custom extractors when you need to:

  • Extract structured data: Parse JSON fields from agent responses
  • Filter specific patterns: Extract code blocks, URLs, or formatted content
  • Combine data sources: Merge information from multiple messages or memory blocks
  • Count occurrences: Track how many times something happened
  • Complex logic: Implement domain-specific extraction

Basic Structure

1from letta_evals.decorators import extractor
2from letta_client import LettaMessageUnion
3from typing import List
4
5@extractor
6def my_extractor(trajectory: List[List[LettaMessageUnion]], config: dict) -> str:
7    """Extract custom content from trajectory."""
8    # Your extraction logic here
9    return extracted_text

Example: Extract Memory Insert

1from letta_evals.decorators import extractor
2
3@extractor
4def memory_insert_args(trajectory, config):
5    """Extract arguments from memory_insert tool calls."""
6    for turn in trajectory:
7        for message in turn:
8            if hasattr(message, 'tool_call') and message.tool_call:
9                if message.tool_call.name == "memory_insert":
10                    return str(message.tool_call.arguments)
11    return ""

Registration

Custom extractors are automatically registered when you import them in your suite’s setup script or custom evaluators file.

Next Steps