🏄♀️ REPORT → A Ride Through AI Evals & Benchmarks
Beginner to Advanced AI Scientists and Engineers | Today's word: Focus · Fokus · 集中 (しゅうちゅう) · מיקוד
🎧 "The successful warrior is the average man, with laser-like focus." — Bruce Lee

Hello there, my anon friend!
This is the final installment of our Technical Debt Festival (check out Part I here and Part II here). As we continue our research for NULLSTAR's PleiadesOS: ALCYONE module, we'll take a deep dive into the current landscape of agentic evaluations and benchmarks. We'll explore a selection of relevant articles while examining some of the most prominent projects in the field.

This will also be my final post of 2026 on this blog. As I close a personal chapter in my life and shift my focus entirely toward building NULLSTAR, I'll be taking a short break from this format of outreach and external interactions to dedicate more time to the work ahead. It's been a remarkable year, and I'm deeply grateful for the people, conversations, and opportunities along the way. I'm looking forward to an even more productive and exciting 2027.
As I read through the material, below are my highlights — with a moderate attempt at coherence — for my personal inventory (and perhaps as a way to connect with other friendly AI scientists and builders out there who love talking about the same things). As always, please remember our rules:
🤖 If these notes look cool to you, it's your ✦moral duty✦ to read the original resources.
🤖 Just another blog, but not your usual blog: they're living notes that I change as I fancy.
0️⃣🦋 Agentic Evals and Benchmarks TL;DR
✨ Traditional model benchmarks measure isolated capabilities, whereas agentic evaluations must assess systems operating over multiple turns/steps, managing context, and pursuing objectives. The evaluation of the model is wrapped in a harness with tools, memory, skills, and a sandbox, working through long tasks that end in a verifier. The common measurements are: task success, correctness, tool-use accuracy, reliability, efficiency, robustness, safety, and cost. ✨
The Pieces Around a Benchmark
A benchmark is the task set and its expected outcomes. A runner or task format packages tasks and executes trials. An evaluation framework helps define, run, and score experiments. Observability records what happened during a run so failures can be diagnosed. Training data is used to change a system; it is not itself evidence that the changed system works.
benchmark tasks + environment
│
▼
runner / harness ── configures model, tools, memory, budgets
│
▼
agent trial ── traces and artifacts ──▶ observability
│
▼
verifier / judge ──▶ scores and failure analysis
│
└──▶ new regression tasks
Which Eval Answers Which Question?
| Evaluation family | Examples in this guide | What a result can tell you | A useful caution |
|---|---|---|---|
| Software engineering and terminal work | Terminal-Bench 2 and Pro, SWE-bench and SWE-bench Pro, DeepSWE, SWE-Atlas, CompileBench | How well an agent handles real repositories and terminal tasks; most are graded with tests | Results depend on repository snapshot, tools, budget, and test quality |
| Memory and personalization | LongMemEval, LoCoMo, BEAM, ConvoMem, BABILong | Whether a system recalls and applies information across long conversations; newer evaluations probe personalization, over-personalization, and sycophancy | Recall alone does not measure whether memory is relevant, current, or appropriately used |
| Tool use and APIs | BFCL, Toolathlon, MCP-Atlas | Whether an agent selects and uses tools to complete tasks | A valid call may still be a poor or unsafe action; inspect end-task outcomes too |
| Web and computer interaction | WebArena, OSWorld-Verified, AndroidWorld | Whether an agent can operate interfaces through a sequence of actions | Environment state, visual variation, and reset behavior can change difficulty |
| Safety and robustness | Agent-SafetyBench and task-specific adversarial cases | How the system behaves under risky, ambiguous, or adversarial conditions | A benchmark pass does not establish safety outside its tested threat model |
| Task formats, eval frameworks, and observability | Harbor, SWE-gen; Inspect AI, DeepEval, Promptfoo, LM Evaluation Harness; Langfuse, Phoenix, Opik, OpenTelemetry GenAI conventions | Harbor packages tasks for runs; SWE-gen turns merged pull requests into tasks usable for evaluation and reinforcement learning. Frameworks run and score experiments; observability captures traces. Production failures can become tasks for the next evaluation cycle | These tools serve different roles. Reusing training tasks for evaluation can overstate progress; trace portability depends on what each system records |
Trusting Scores
Agent scores are estimates from a particular set of tasks, environment, harness, and run budget. A leaderboard number is useful evidence, not a universal measure of agent quality.
- Before comparing or relying on a result, check:
1. Task coverage: Do tasks resemble the real work and its difficulty?
2. Verifier quality: Do tests check requested behavior, including regressions and edge cases?
3. Run-to-run variation: Different paths on repeated runs. Report the number of attempts and a confidence interval.
4. Harness disclosure: The model/version, prompt, tools, context and memory setup,, concurrency, retries, time or token limits.
5. Useful trade-offs: Pair task success with latency, token or monetary cost, tool errors, and safety failures.
6. Reproducibility: Publish run configuration and scoring code, and retain traces and artifacts for failures.
- For LLM judges, define a rubric, test agreement against human ratings, and use deterministic checks for facts that can be verified directly.
- For high-impact decisions, report disagreements and failure examples alongside the aggregate score.
A small example
Suppose two agent versions each attempt the same 40 tasks five times under the same model, tools, and budget:
| Measure | Agent A | Agent B |
|---|---|---|
| Successful runs | 156 / 200 (78%) | 164 / 200 (82%) |
| Median cost per run | $0.42 | $0.71 |
| Median completion time | 4.8 min | 6.1 min |
| Safety failures | 0 | 2 |
- The four-point success gap alone is not enough to call B better.
- Show uncertainty for the difference, account for the fact that runs share the same tasks, and inspect whether B's extra successes justify its higher cost and safety failures.
- A useful report would also break results down by task category and include representative failure traces.
- If the uncertainty interval includes no meaningful difference, say the results are inconclusive rather than ranking the systems by the point estimates.
A Practical Eval Loop
- Define the job. Write down the user goal, acceptable completion criteria, and unacceptable outcomes.
- Build a representative task set. Start with real, permissioned examples, include common cases and difficult edge cases, and keep a held-out set for final comparisons.
- Fix the conditions. Pin the model, prompts, harness, tools, environment, and budgets. Log them with every run.
- Run repeatedly. Use enough independent trials to see variability. Capture success, cost, latency, and safety signals rather than one headline score.
- Inspect failures. Review traces and artifacts to distinguish model errors from bad instructions, broken tools, weak tests, and environment problems.
- Turn recurring failures into regression tasks. Re-run those tasks after every meaningful change, then periodically refresh the set so it still represents current work.
1️⃣🦋 Notable Projects
Coordinators and Standards
Harbor
Harbor is a framework from the creators of Terminal-Bench for evaluating and optimizing agents and language models. You can use it to evaluate agents such as Claude Code, OpenHands, and Codex CLI, build and share your own benchmarks and environments, run experiments in thousands of environments in parallel through sandbox providers, and generate rollouts for RL optimization.

- A Harbor task is an instruction, environment, and test script. The task is defined as a directory of files with the following structure:
my-task/
├── instruction.md
├── task.toml
├── environment/
│ ├── Dockerfile # or some other spec
│ └── ...
├── solution/
│ ├── solve.sh
│ └── ...
└── tests/
├── test.sh
└── ...
- A Harbor trial is an agent's attempt at completing a task. The diagram below shows how the components of a task are used in a trial.
- A task defines the logic to verify the completion of the instruction and produce a reward in the tests/test.sh script. The script must produce a numerical reward in the environment at /logs/verifier/reward.txt or /logs/verifier/reward.json (for multi-dimensional or labeled rewards).
Coding Benchmarks
terminal-bench-1 (legacy)
Terminal-Bench 1 is the original benchmark for LLMs on complicated tasks in the terminal, from compiling code and training models to setting up servers. Its authors now point users to Harbor, the framework that runs Terminal-Bench 2.0.
Terminal-Bench 1 (legacy repo)
task prompt ──▶ agent in a terminal ──▶ test checks ──▶ pass / fail
current runner: Harbor, which runs Terminal-Bench 2.0
(use this repo only for the original tasks)
terminal-bench-2
Terminal-Bench is a popular benchmark for measuring how well agents and language models do valuable work in containerized environments. Its tasks include assembling proteins for synthesis, debugging async code, and resolving security vulnerabilities, so it tests end-to-end work rather than isolated questions.
one task = one container
┌──────────────────────────────────────────────────┐
│ instruction.md: "debug this async code ..." │
│ environment/ : Docker image with the repo │
└──────────────────────────────────────────────────┘
│ agent works in a shell (run, edit, install)
▼
tests/test.sh ──▶ /logs/verifier/reward.txt ──▶ score
terminal-bench-pro
Terminal-Bench Pro is a systematic extension of the original Terminal-Bench, designed to address key limitations of existing terminal-agent benchmarks. It has 400 tasks (200 public and 200 private) across eight domains, including data processing, games, debugging, system administration, scientific computing, software engineering, ML, and security.
400 tasks = 200 public + 200 private
┌─ data processing
├─ games
terminal task ─────────┼─ debugging
(one of 8 domains) ├─ system administration
├─ scientific computing
├─ software engineering
├─ ML
└─ security
otel-bench
OTel-Bench is an open-source benchmark for evaluating AI models on OpenTelemetry instrumentation tasks, where models must instrument applications across multiple programming languages. The repository also links a results page and a blog post about the benchmark.
# what a correct answer looks like: the model adds spans like these
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
def checkout(cart):
with tracer.start_as_current_span("checkout") as span:
span.set_attribute("cart.items", len(cart))
return process_payment(cart) # the benchmark checks emitted spans
CompileBench
CompileBench measures how well LLMs handle real open-source projects that suffer from dependency hell, legacy toolchains, and complex build systems. It focuses on the messy build-and-compile work that contest-style and vibe-coding benchmarks usually skip.
old open-source project (deps, legacy toolchain, custom build)
│ agent: read README ─▶ install deps ─▶ pick compiler ─▶ fix Makefile
▼
build ──▶ binary runs? ──▶ score
▲ │
└── missing lib / wrong GCC / broken CMake: loop until it builds
RuneBench
RuneBench evaluates AI coding agents on RuneScape gameplay through rs-sdk. Agents play by writing and executing TypeScript snippets against an emulated game server running at 8x speed, with a folder of wiki-extracted markdown for strategy, and they are scored on their peak XP rate.
agent (LLM) ──writes TypeScript snippet──▶ rs-sdk ──▶ game server (8x speed)
▲ │
└──── game state, XP, wiki notes ◀─────────────┘
score = peak XP rate reached
legacy-bench
Legacy-Bench evaluates AI coding agents on legacy software engineering tasks in COBOL, Fortran, and Java 7, the languages behind systems that handle financial settlements, telephone routing, and insurance claims. It measures how well frontier agents maintain, debug, and modernize that code, which modern Python and JavaScript benchmarks do not cover.
01 WS-TOTAL PIC 9(7)V99 VALUE ZERO.
ADD WS-AMOUNT TO WS-TOTAL.
*> Agents must read code like this (COBOL, Fortran, Java 7), then
*> maintain, debug, or modernize it and pass the hidden tests.
SWE-atlas
SWE Atlas is a benchmark for coding agents across professional software engineering work, beyond simple issue resolution. It covers Codebase QnA, Test Writing, and Refactoring, and this repository provides the tasks and run setup, with results published on the SWE Atlas leaderboard.
repo snapshot
├── Codebase QnA ──▶ answer a question about the code ──▶ graded answer
├── Test writing ──▶ new tests run against hidden checks
└── Refactoring ──▶ behavior must stay the same ──▶ tests still pass
SWE-bench Pro
SWE-bench Pro asks whether AI agents can solve long-horizon software engineering tasks. The repository contains the code and data for the paper, and a public leaderboard hosted by Scale AI tracks results.
issue + repo @ base commit
│
▼
agent writes a patch
│
▼
apply patch ──▶ run FAIL_TO_PASS tests (must now pass)
└──▶ run PASS_TO_PASS tests (must still pass)
│
▼
resolved? yes / no
DeepSWE
DeepSWE is a benchmark for frontier coding agents on original, long-horizon software engineering tasks drawn from active open-source repositories. The tasks are written from scratch to avoid contamination, span 113 tasks across 91 repos in 5 languages, and use hand-written verifiers that test behavior rather than implementation details.

Long-horizon software engineering benchmark with:
- Contamination free: tasks are written from scratch, not adapted from existing commits or PRs, so no model has seen the solution during pretraining.
- High diversity: 113 tasks span a broad pool of 91 repos in 5 programming languages.
- Real-world complexity: prompts are half the length of SWE-bench Pro's, yet solutions require 5.5x more code and ~2x more output tokens.
- Reliable verification: verifiers are hand-written to test software behavior, rather than implementation details.
It uses Harbor task format:
task.toml Metadata (repo, base commit, language, image, limits)
instruction.md The prompt the agent sees
environment/ Dockerfile reproducing the prebuilt image
tests/ Verifier entry point, held-out tests, and grader config
solution/ Reference solution (held out from the agent)
The verifier produces the following outputs for each run:
verifier/
reward.json Structured scores (binary reward + pass fractions)
ctrf.json Machine-readable test report with failure messages
test-stdout.txt Raw suite output and a list of failure reasons
run.log Raw stdout/stderr captured during the run
reports/ Framework-native report/log files from the grader
SWE-bench
SWE-bench evaluates language models on resolving real GitHub issues from popular Python repositories, where the model must produce a patch that passes the project's tests. It has become one of the standard benchmarks for coding agents and publishes a leaderboard on its website.
# a typical model answer: a patch that makes the failing test pass
--- a/pkg/parser.py
+++ b/pkg/parser.py
@@ -41,7 +41,7 @@ def parse_value(value):
- if value == None:
+ if value is None:
return default
CCA-SWE-bench
A minimal harness from Meta Research that runs Confucius Code Agent (CCA) on SWE-bench Pro inside a Docker container. The agent receives the task, reasons about the fix, and executes actions such as shell commands and file reads and writes directly.
docker container
┌──────────────────────────────────────────────┐
│ CCA agent │
│ ├─ reasons about the fix │
│ ├─ runs shell commands ──▶ repo │
│ └─ reads / writes files ──▶ repo │
└──────────────────────────────────────────────┘
│ final diff
▼
SWE-bench Pro grader
SWE-bench Verified
A human-validated subset of SWE-bench, with 500 real GitHub issues that agents must resolve by producing patches that pass the project's tests. It is the version most often reported for coding agents, and results are tracked on the SWE-bench leaderboard.
500 human-validated issues (a subset of SWE-bench)
issue ──▶ agent ──▶ patch ──▶ tests
│
resolved rate = passed / 500 ◀─┘
Memory Benchmarks
LongMemEval
LongMemEval is a comprehensive, challenging, and scalable benchmark for testing the long-term memory of chat assistants. It was introduced by Di Wu and colleagues and tests recall and reasoning across long, multi-session conversations.
chat history: session 1 ─ session 2 ─ ... ─ session N (long, multi-session)
│
question: "What did I say about my trip in March?" (illustrative)
│
memory system or long context ──▶ answer ──▶ judged against the gold answer
LoCoMo
LoCoMo comes from the ACL 2024 paper "Evaluating Very Long-Term Conversational Memory of LLM Agents" and is a benchmark of very long conversations. It has ten conversations, each annotated for question answering and event summarization, and its dialogs can also support multimodal dialog generation.
very long conversation (many sessions, months of dialog)
│
├──▶ QA pairs ──▶ agent answers ──▶ score
└──▶ event summaries ──▶ agent summarizes ──▶ score
ConvoMem
ConvoMem is a benchmark for conversational memory in large language models, with 75,336 question-answer pairs across six evidence categories. It tests how well assistants remember and use information from previous conversations.
past conversations ──▶ memory store
new question ──▶ retrieve evidence from memory ──▶ answer
75,336 QA pairs, grouped into six evidence categories
BEAM
BEAM, from the paper "Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs", benchmarks and tries to improve long-term memory in language models. The repository contains the code and data for the paper.
long dialogue (1M+ tokens)
├── baseline: put everything in the context window
└── memory method: store, index, retrieve only what matters
│
▼
questions ──▶ long-term memory score
BABILong
BABILong is a long-context needle-in-a-haystack benchmark for LLMs, where the relevant facts are hidden inside long distractor text. Its evaluation sets cover many context lengths, from short texts up to 10M tokens, with samples for each task at each length.
[ ...distractor...
FACT: "Maria keeps the keys in the blue box"
...distractor... ]
│
question: "Where are the keys?" ──▶ model answer ──▶ exact match check
context lengths: from short texts up to 10M tokens
SQuAD
SQuAD, the Stanford Question Answering Dataset, is a large reading comprehension dataset. This repository lets people explore the dataset, visualize model predictions, and generate predictions for their own data through the workflow used by the SQuAD leaderboard.
passage: "... the tower was completed in 1889 ..."
question: "When was the tower completed?"
│
model ──▶ answer span "1889" ──▶ compare with gold span ──▶ EM / F1
Agent Memory Benchmark
The Agent Memory Benchmark (AMB) is an open, reproducible leaderboard for evaluating AI agent memory and retrieval systems on real-world long-context tasks. It is meant to show how memory systems compare on the same tasks, with the scoring and results visible.
real-world long-context tasks (same tasks for every system)
│
▼
memory system under test ──▶ answers ──▶ scoring ──▶ public leaderboard
Mem0 Memory Benchmarks
An open-source evaluation suite for running benchmarks on memory-augmented LLM systems. It supports the Mem0 cloud and open-source versions and measures memory recall, extraction quality, and retrieval accuracy, using datasets such as LoCoMo, LongMemEval, and BEAM.
# the loop these benchmarks run against a memory layer (Mem0 OSS API)
from mem0 import Memory
memory = Memory()
messages = [{"role": "user", "content": "I moved to Lisbon last year"}]
memory.add(messages, user_id="user-1")
results = memory.search(query="Where does the user live?",
filters={"user_id": "user-1"}, top_k=3)
# the suite then scores recall, extraction quality, and retrieval accuracy
Honcho Memory Benchmarks
Honcho's evals page reports long-context memory results on LongMem S, LoCoMo, and BEAM 100K. These numbers come from the company itself, so compare them against the original benchmarks before drawing conclusions.
benchmark system result
LongMem S ──┐
LoCoMo ──┼──▶ Honcho (self-reported) ──▶ compare with originals
BEAM 100K ──┘
FrontierMemory
FrontierMemory is a benchmark for continual learning through memory, measuring how well systems keep learning as they accumulate memories over time.
interaction 1 ──▶ memory ──▶ interaction 2 ──▶ ... ──▶ interaction N
▲ │
└──── does accuracy improve as memory grows? ◀┘
Cognee Knowledge Graph Memory Benchmarks
Two benchmark exercises from Cognee, run on HotPotQA, TwoWikiMultiHop, and MuSiQue, plus a head-to-head comparison with Mem0, Graphiti, and LightRAG. The page publishes the numbers, methodology, caveats, and reproduction code.
HotPotQA / TwoWikiMultiHop / MuSiQue documents
│ ingest ──▶ knowledge graph (entities + relations)
▼
multi-hop question ──▶ graph traversal across hops ──▶ answer
compared head-to-head with Mem0, Graphiti, and LightRAG
Vectorize Agent Memory Benchmark
Vectorize's agent memory benchmark was built because LoCoMo and LongMemEval were designed for 32k-token context windows, and modern models can often score well by putting everything in context. The project publishes its datasets, prompts, scoring logic, and results in the open.
LoCoMo / LongMemEval (designed for 32k-token windows)
│
▼
modern model: put the whole history in context ──▶ high score (a shortcut)
│
▼
new benchmark: publishes datasets, prompts, and scoring, aiming to test memory
Zep Evaluation Harness
The evaluation harness in the Zep repository, used to test Zep's memory system. Zep describes itself as a context layer for agents, and this harness is one of the tools for checking how well that memory works.
conversation logs ──▶ Zep (context layer: facts and graph)
eval questions ──▶ Zep retrieval ──▶ LLM answer ──▶ grader ──▶ score
Letta Evals
Letta Evals is an open-source evaluation framework for systematically testing stateful agents, introduced by the Letta team. It is aimed at agents that learn and keep state across interactions, which static benchmarks do not capture well.
stateful agent (keeps memory across turns)
│ turn 1 ──▶ turn 2 ──▶ ... ──▶ turn N (state changes along the way)
▼
check final state and outputs ──▶ graded result
MemoryBench
A pluggable benchmarking framework for memory and context systems. You can mix any provider with any benchmark, bring your own datasets, resume runs from any pipeline stage, compare providers side by side, and inspect results in a web UI.
providers (any) x benchmarks (any)
│ ingest ──▶ index ──▶ search ──▶ answer ──▶ evaluate
│ (resume a run from any stage)
▼
side-by-side comparison table + web UI
CL-Bench 1.0
Continual Learning Bench 1.0 measures an agent's ability to use memory to tackle long-horizon tasks. It checks whether systems adapt and improve from feedback across sequences of task instances.
task 1 ──▶ agent + memory ──▶ feedback ──▶ memory update
task 2 ──▶ agent + memory ──▶ feedback ──▶ memory update
...
task N ──▶ does performance improve across the sequence?
Other Benchmarks
skillsbench
SkillsBench is the first benchmark for evaluating how well AI agents use skills, which are modular folders of instructions, scripts, and resources. It measures both skill effectiveness and agent behavior through gym-style benchmarking.
skill/
├── SKILL.md instructions
├── scripts/ helper code
└── resources/ templates and data
│ agent loads the skill for a task
▼
gym-style run ──▶ score with the skill vs. without it
AgentBench
AgentBench is a comprehensive benchmark for evaluating LLMs as agents, presented at ICLR 2024. The project has since added AgentBench FC, a function-calling version built on AgentRL.
┌──────────────────────────────────────┐
LLM ◀──▶│ environments: OS · DB · knowledge │
agent │ graph · web browsing · games · ... │
└──────────────────────────────────────┘
observation ──▶ action ──▶ observation ... ──▶ task success rate
CommerceAgentBench
CommerceAgentBench, from the Accio team at Alibaba International, benchmarks long-horizon agents in high-fidelity, stateful, and reproducible replicas of real online services. The project includes a live leaderboard, a mock showcase, and reproducibility instructions.
agent ──▶ stateful replica of an online store (catalog, cart, orders)
│ actions: search, add to cart, check out
▼
final state checked against the goal ──▶ reproducible score
Claw-Eval
Claw-Eval is an evaluation harness for LLMs as agents, aimed at trustworthy evaluation of autonomous agents. It has 300 human-verified tasks, 2,159 rubrics, and nine categories covering completion, safety, and robustness.
300 human-verified tasks ──▶ agent run ──▶ trace
│
rubric checks (2,159 rubrics)
├─ completion
├─ safety
└─ robustness
Harvey LAB
Harvey LAB, the Legal Agent Benchmark, is an open-source benchmark for evaluating agents on real legal work in realistic environments. It is designed to measure and improve how LLM agents perform legal tasks.
legal task + workspace (documents, matter files)
│ agent drafts, reviews, or analyzes
▼
checked against task rubric ──▶ LAB score
APEX
Mercor's APEX benchmarks and leaderboards measure AI productivity across software engineering, law, consulting, medicine, and more. The site presents results for frontier models in these professional domains.
professional tasks: software engineering · law · consulting · medicine ...
│ frontier models attempt each task
▼
graded per domain ──▶ leaderboard
Olam Labs — Multi-Agent Mindustry
PvP and cooperative PvE benchmarks built on Mindustry and Harbor Multi-Agent. In the PvP series, teams compete over consecutive games with persistent agent sessions, and in endless survival, one team cooperates to defend its core against waves of enemies.
PvP: team A ◀──▶ Mindustry map ◀──▶ team B (consecutive games)
PvE: team agents defend the core against waves of enemies
built on Harbor Multi-Agent
MCP-Atlas
Scale AI's benchmark for tool-use competency with real Model Context Protocol servers. It has 1,000 tasks across 36 servers and 220 tools, and prompts avoid naming tools, so agents must choose the right tools and chain 3 to 6 calls across servers while ignoring plausible distractors.
user prompt (tool names NOT given)
│
▼
agent picks from 220 tools across 36 MCP servers
│ chains 3-6 calls, while distractor tools are present
▼
final answer checked against the task
BFCL v4
The Berkeley Function-Calling Leaderboard from the Gorilla project, which evaluates how well LLMs select functions, format arguments against schemas, and chain calls. Version 4 is the current release of the leaderboard.
// function schema given to the model
{"name": "get_weather",
"parameters": {"city": "string", "unit": "celsius | fahrenheit"}}
// model output, compared against the gold call
get_weather(city="Lisbon", unit="celsius")
Toolathlon
The Tool Decathlon, presented at ICLR 2026, benchmarks language agents on diverse, realistic, and long-horizon task execution with many tools, testing whether agents can plan and carry out multi-step work across them.
long multi-step task across many apps
│ agent plans ──▶ calls tools (mail, sheets, calendar, files, ...)
▼
final environment state verified ──▶ success
OSWorld-Verified
OSWorld is a NeurIPS 2024 benchmark for multimodal agents on open-ended tasks in real computer environments. OSWorld-Verified is its verified release, where agents click, type, and navigate live desktop applications to finish tasks.
desktop screenshot ──▶ VLM agent ──▶ action (click x,y · type · key)
▲ │
└──── new screen ◀── VM desktop ◀───────┘
task done? ──▶ execution-based check of app / file state
WebArena
The code repository for WebArena, a realistic web environment for building and evaluating autonomous agents. Agents complete tasks in websites such as shopping, forums, and code hosting by reading pages and acting through the browser.
goal: "find X on the shop and add it to the cart"
browser (page DOM / screenshot)
│ click · type · scroll · navigate
▼
sites: shopping · forum · code hosting · wiki
▼
functional check of the final page or site state
AndroidWorld
An environment and benchmark for autonomous agents on Android. Agents must tap, type, and navigate real apps on a live mobile interface to complete tasks.
task: "set an alarm for 7am in Clock"
│ agent sees the screen (UI tree or pixels)
▼
tap · type · swipe · back ──▶ live Android app ──▶ state check
τ²-bench
The τ-bench family from Sierra for tool-agent-user interaction. Agents talk to simulated users, keep state across turns, and follow strict business policies in domains such as airline, retail, and telecom, with τ²-bench extending the original.
agent ◀──── messages ────▶ simulated user (LLM)
│ tool calls
▼
domain (airline / retail / telecom): database + policy rules
│
▼
final database state + tool trace ──▶ pass / fail per trial
BrowseComp
A benchmark of 1,266 questions that require persistently browsing the internet for hard-to-find, entangled information. Answers are short and easy to verify, and in the original paper accuracy ranged from 0.6% for GPT-4o to 51.5% for Deep Research.
question: obscure, multi-constraint clue
│ agent browses: search ──▶ open ──▶ read (many steps)
▼
short answer ──▶ compared with the reference (easy to verify)
Agent-SafetyBench
A benchmark for evaluating the safety of LLM agents, with 349 interaction environments and 2,000 test cases across 8 risk categories and 10 failure modes. When 16 popular agents were tested, none scored above 60% on safety.
349 environments + 2,000 test cases
agent ──▶ actions on simulated tools
▼
risk category (8) x failure mode (10) ──▶ safe / unsafe
AgentX
The AgentX benchmark from SemiAnalysis's InferenceX tests how inference systems serve long-context, multi-turn coding-agent sessions. It replays recorded agentic-coding traces, reproducing real prompt sizes, KV-cache prefix sharing, subagent fan-out, and timing, and its median request has 142,016 input tokens.
recorded agentic-coding trace
├─ median request: 142,016 input tokens
├─ KV-cache prefix shared across turns
└─ subagent fan-out ──▶ parallel requests
replayed against an inference server ──▶ latency and throughput
2️⃣ 🦋 Tools and Resources
Training Datasets
- SWE-gen: An automated pipeline that creates tasks from real bug fixes in open-source GitHub repos, in any programming language. It reverts a merged PR to recreate the buggy state, checks that the tests fail on the baseline and pass after the fix, and containerizes each task.
- SWE-gen-Java: A dataset of 1,000 JVM tasks generated from 16 open-source GitHub repos with SWE-gen. Each task is a merged PR with 1 to 10 edited source files and Fail-to-Pass unit tests, validated against both the baseline and the fix.
- SWE-gen-JS: A dataset of 1,000 JavaScript and TypeScript tasks from 30 open-source repos, generated with SWE-gen. Each task is a merged PR with linked issues, 3 to 10 edited source files, and Fail-to-Pass tests, following the Harbor format.
- SWE-gen-Rust: A dataset of 1,000 Rust tasks from 23 open-source repos, generated with SWE-gen. Each task is a merged PR with 2 to 10 edited source files and Fail-to-Pass unit tests, validated against the baseline and the fix.
- SWE-gen-Go: A dataset of 1,000 Go tasks from 20 open-source repos, generated with SWE-gen. Each task is a merged PR with 2 to 10 edited source files and Fail-to-Pass unit tests, validated against the baseline and the fix.
- SWE-gen-Cpp: A dataset of 1,000 C++ tasks from 17 open-source repos, generated with SWE-gen. Each task is a merged PR with 1 to 10 edited source files and Fail-to-Pass unit tests, validated against the baseline and the fix.
- Nemotron-Terminal-Synthetic-Tasks: A synthetic dataset of terminal tasks from NVIDIA, hosted on Hugging Face, for training terminal agents. It is one of several public datasets for agent training.
- seta-env: The SETA RL dataset from the CAMEL-AI Scaling Environments for Agents project, generated by an automated synthesis and verification pipeline. Each folder is one task with a task.yaml, a Dockerfile, and run-tests.sh, and the tasks are compatible with the Terminal-Bench format.
Relevant Tools
- OpenThoughts-Agent: A research codebase from the OpenThoughts project for building tooling and finding data to train small agentic models. It covers generating Harbor tasks, training with SkyRL, and distilling trajectories with SFT.
- endless-terminals: A fully autonomous pipeline that generates terminal-use tasks without human annotation, for training terminal agents with reinforcement learning. It can generate tasks with a model such as Qwen3-32B and write them to a folder.
- Ares: An RL-first framework for training and evaluating LLM agents, especially coding agents. It works as a modern gym, the environment layer for RL research on Harbor and SkyRL.
- Benchmark Templates: A desktop application for creating, browsing, and managing Terminal-Bench-Science tasks in the Harbor format. Scientists describe a task in plain language, and a Claude Code agent generates and refines its five Harbor files before they are validated and tested locally.
- AutoAgent: Autoresearch-style iteration for agent engineering: an AI agent builds and improves an agent harness overnight, changing the system prompt, tools, configuration, and orchestration. It runs the benchmark, keeps or discards each change based on the score, and repeats.
- Agent Accounting: A local-first dashboard that shows what your coding agents did and what they cost, across projects and clients. It reads local session logs from tools such as Claude Code, Codex, and OpenCode, and nothing leaves your computer.
- shadcn Lint: An agent-first linter for Tailwind design systems, where you define what is allowed. When an agent breaks a rule, the error explains the problem and suggests a fix based on your components, variants, and theme.
- Weights & Biases Weave: A toolkit from Weights & Biases for tracing and evaluating AI agents and LLM applications. Conversations, turns, LLM calls, and tool calls appear in the Agents tab, and the @weave.op decorator traces individual functions.
- EvalAI: An open-source platform for evaluating and comparing machine learning and AI algorithms at scale. It is used to host evaluation challenges where participants submit their results and compare them on leaderboards.
- Future AGI: An open-source platform for shipping self-improving AI agents, combining evaluations, tracing, simulations, guardrails, a gateway, and optimization. It aims to close the feedback loop from prototype to production by turning traces into signal for the next version.
- Inspect AI: A framework for LLM evaluations created by the UK AI Security Institute. It has built-in components for prompt engineering, tool use, multi-turn dialog, and model-graded evaluations, and it can be extended with other Python packages.
- DeepEval: An open-source LLM evaluation framework that works like Pytest, but for LLM apps. It runs evals through metrics such as G-Eval, task completion, answer relevancy, and hallucination, many of which use LLM-as-a-judge.
- Promptfoo: A CLI and library for evaluating and red-teaming LLM-based apps. It is used to test prompts and agents and to find security issues before they reach users.
- EleutherAI LM Evaluation Harness: The widely used harness for running language model benchmarks, with support for many model backends. Recent versions added plugins for custom backends, filters, and metrics, along with a CLI that has subcommands and YAML configuration.
- NVIDIA NeMo Evaluator: An LLM evaluation framework with benchmark environments, pluggable solvers, an interceptor proxy, and multi-format reporting. It has optional integrations, such as Harbor agents and Inspect AI log export.
- NVIDIA NeMo Gym: A library for evaluating and improving models and agents using environments. It provides infrastructure to develop environments, run evaluation and training at scale, and includes a collection of popular benchmarks and training environments.
- Prime Intellect Verifiers: A library for creating environments to train and evaluate LLMs. It integrates with Prime Intellect's Environments Hub, its prime-rl training framework, and its hosted training platform.
- AgentOps: A platform that helps developers build, evaluate, and monitor AI agents, from prototype to production. The app is open source under the MIT license.
- Arize — OpenInference: A set of conventions and plugins that complement OpenTelemetry to enable tracing of AI applications. It works natively with Arize Phoenix and Arize AX, and with any OpenTelemetry-compatible backend.
- Arize — Phoenix: An open-source AI observability platform for experimentation, evaluation, and troubleshooting. Arize also offers Arize AX for managed production workflows.
- Comet — Opik: An open-source LLM observability and evaluation platform for agent tracing, LLM evaluation, prompt management, and production monitoring. It is Apache-2.0 licensed, and the full platform can be self-hosted.
- Helicone: An open-source LLM observability platform with agent tracing, LLM routing, cost and latency tracking, datasets for fine-tuning, and automatic fallbacks. Setup takes one line of code.
- Traceloop — OpenLLMetry: A set of extensions built on OpenTelemetry that gives complete observability over an LLM application. Because it uses OpenTelemetry, it connects to existing tools such as Datadog and Honeycomb.
- Langfuse: An open-source AI engineering platform for agent evals and observability. It tracks usage and costs, traces and evaluates agents, and manages prompts, and it can be used on Langfuse Cloud or self-hosted.
- MLflow: An open-source AI engineering platform for agents, LLMs, and ML models, used to debug, evaluate, monitor, and optimize production AI applications. Teams use it to control costs and manage access to models and data.
- OpenTelemetry — GenAI Semantic Conventions: The OpenTelemetry specification of standard attribute names and span structure for generative AI calls, so traces from different tools look the same. Many of the observability tools in this list build on it.
- OpenTelemetry — Demo: The Astronomy Shop, a microservice-based distributed system that shows how OpenTelemetry works in a near real-world environment. Vendors and tool authors use it to demonstrate their integrations, and contributors use it to test new versions.
- Pydantic — Logfire: An observability platform from the team behind Pydantic, built for production LLM and agent systems. It offers Python-centric insights such as profiling and database query visibility, and lets you query your data with standard SQL.
- OpenObserve: An observability platform for logs, metrics, traces, real user monitoring, and AI/LLM workloads. It emphasizes low storage costs compared with other platforms.
- Braintrust: An AI observability platform for tracing production systems, running evals, and catching regressions before they reach users. It is used to ship quality agents at scale.
- RagaAI Catalyst: A Python SDK for agent observability, monitoring, and evaluation. It covers project, dataset, evaluation, trace, and prompt management, along with synthetic data generation, guardrails, and red-teaming.
- Laminar: An open-source observability platform built for AI agents, with OpenTelemetry-native tracing and signals that flag behaviors such as loops, described in plain English. It also has an SDK and CLI for running evals locally or in CI/CD.
- NVIDIA — NeMo Relay: NeMo Relay manages scopes, middleware, plugins, and lifecycle events for agent systems. Its documentation covers the overview and how to trace agent harness behavior.
- NVIDIA — NeMo Fabric: NeMo Fabric gives applications, evaluation systems, and rollout platforms one interface for running agent harnesses and custom agents. It lets you compose runs, validate configurations before execution, switch harnesses, and compare normalized results.
Notable Readings
- OpenAI's Aligment
- ByteRover — Benchmarking AI Agent Memory
- Letta — Evaluating Memory in Production Agents
- Supermemory — Research
- Mem0 — Why BEAM Is a Good Memory Benchmark
- Mem0 — Understanding Memory Benchmarks for Production AI Agents
- Mem0 — How to Read Memory Benchmarks
- Memory Papers — Benchmarks
- Awesome Evals: curated list of evaluation resources
- OpenAI — Eval Skills: OpenAI blog post on evaluation skills
- LangChain — Evaluating Deep Agents: LangChain's lessons from evaluating deep agents
- LangChain — Agent Evaluation Readiness Checklist: checklist for getting an agent ready to evaluate
- LangChain — Evaluating Skills: how to evaluate agent skills
- LangChain — Agent Observability Powers Agent Evaluation: how observability data feeds agent evaluation
- Anthropic — Demystifying Evals for AI Agents: Anthropic's overview of designing evals for AI agents
- Anthropic — Infrastructure Noise: how infrastructure differences add noise to agentic evals
- Anthropic — Eval Awareness and BrowseComp: eval awareness in models tested on BrowseComp
- Anthropic — AI-Resistant Technical Evaluations: designing technical evaluations that resist AI assistance
- Red Hat — Eval-Driven Development for AI Agents: building and evaluating AI agents with eval-driven development
- Splunk — Agent Evaluation Framework: Splunk's framework for evaluating agents
- Google — Driving the Agent Quality Flywheel: improving agent quality with a feedback loop from coding agents
- OpenAI — How We Detect Frontier Model Misalignment: OpenAI on detecting misalignment in frontier models
- AWS — AgentCore Evaluations: AWS announcement that AgentCore Evaluations is generally available
- NVIDIA — Tracing Agent Harness Behavior with NeMo Relay: tracing agent harness behavior with NeMo Relay
- Temporal — Building Observable AI Agents with Braintrust: building observable agents with Temporal and Braintrust
- Google Cloud — BigQuery Agent Analytics: analyzing agent behavior with BigQuery
- Red Hat — Distributed Tracing for Agentic Workflows: distributed tracing for agentic workflows with OpenTelemetry
- METR — Red Teaming Anthropic Agent Monitoring: METR's red teaming of Anthropic's agent monitoring
- Interconnects — Artificial Analysis V5 and the Benchmarkers: a review of the benchmarking industry and what's missing
- Olam Labs: measures social intelligence in AI models through behavioral evaluations in multi-agent simulations
3️⃣ 🦋 Notable Research Papers
Memory Evaluation
- MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks — Zexue He et al., 2026
- BenchPreS: A Benchmark for Context-Aware Personalized Preference Selectivity of Persistent-Memory LLMs — Sangyeon Yoon et al., 2026
- How Does Personalized Memory Shape LLM Behavior? Benchmarking Rational Preference Utilization in Personalized Assistants — Xueyang Feng et al., 2026
- OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agents — Yulin Hu et al., 2026
- MemSyco-Bench: Benchmarking Sycophancy in Agent Memory — Zhishang Xiang et al., 2026
- MemCalib: Benchmarking and Optimizing Memory Use in LLM Agents — Ruike Cao et al., 2026
- MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents — Haoran Tan et al., 2025
- Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions — Yuanzhe Hu et al., 2025
Coding Agents and Software Engineering
- Agyn: A Multi-Agent System for Team-Based Autonomous Software Engineering — Nikita Benkovich et al., 2026
- AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation — Priyam Sahoo et al., 2026
- StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns — Vlad Sobal et al., 2026
- Repo2Run: Automated Building Executable Environment for Code Repository at Scale — Ruida Hu et al., 2025
- R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents — Naman Jain et al., 2025
- OpenHands: An Open Platform for AI Software Developers as Generalist Agents — Xingyao Wang et al., 2024
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Carlos E. Jimenez et al., 2023
Web, Computer and Workplace Agents
- OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation — Xiaomeng Hu et al., 2026
- VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks — Jing Yu Koh et al., 2024
- WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? — Alexandre Drouin et al., 2024
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments — Tianbao Xie et al., 2024
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — Shunyu Yao et al., 2024
- WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks — Léo Boisvert et al., 2024
- The BrowserGym Ecosystem for Web Agent Research — Thibault Le Sellier De Chezelles et al., 2024
- TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks — Frank F. Xu et al., 2024
- WebArena: A Realistic Web Environment for Building Autonomous Agents — Shuyan Zhou et al., 2023
- AgentBench: Evaluating LLMs as Agents — Xiao Liu et al., 2023
- GAIA: a benchmark for General AI Assistants — Grégoire Mialon et al., 2023
Agent Harnesses and Runtimes
- VeRO: A Harness for Agents to Optimize Agents — Varun Ursekar et al., 2026
- Evaluating Agents Across Runtime Contracts: When Mismatch Costs Efficiency or Quality — Victor May et al., 2026
- Natural-Language Agent Harnesses — Linyue Pan et al., 2026
- Meta-Harness: End-to-End Optimization of Model Harnesses — Yoonho Lee et al., 2026
- AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agents — Hailin Zhong et al., 2026
- Stop Comparing LLM Agents Without Disclosing the Harness — Yunbei Zhang et al., 2026
- Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows — Yilun Yao et al., 2026
- Cache Engineering for Agents — Tianyu Fu et al., 2026
Tools, Skills and Interfaces
- SkillTester: Benchmarking Utility and Security of Agent Skills — Leye Wang et al., 2026
- The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior — Xiangzhe Xu et al., 2026
Observability and Failure Analysis
- Beyond Black-Box Benchmarking: Observability, Analytics, and Optimization of Agentic Systems — Dany Moshkovich et al., 2025
- Why Do Multi-Agent LLM Systems Fail? — Mert Cemri et al., 2025
- SentinelAgent: Graph-based Anomaly Detection in Multi-Agent Systems — Xu He et al., 2025
- Mind the Metrics: Patterns for Telemetry-Aware In-IDE AI Application Development using the Model Context Protocol (MCP) — Vincent Koc et al., 2025
- Taming Uncertainty via Automation: Observing, Analyzing, and Optimizing Agentic AI Systems — Dany Moshkovich et al., 2025
- AgentSight: System-Level Observability for AI Agents Using eBPF — Yusheng Zheng et al., 2025
- Where LLM Agents Fail and How They can Learn From Failures — Kunlun Zhu et al., 2025
- Watson: A Cognitive Observability Framework for the Reasoning of LLM-Powered Agents — Benjamin Rombaut et al., 2024
Evaluation Methodology and Reliability
- The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systems — Leon Staufer et al., 2026
- AgentAssay: Token-Efficient Regression Testing for Non-Deterministic AI Agent Workflows — Varun Pratap Bhardwaj, 2026
- LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications — Alexandre Cristovão Maiorano, 2026
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation — Sayash Kapoor et al., 2025
- Lessons from the Trenches on Reproducible Evaluation of Language Models — Stella Biderman et al., 2024
- A Survey on LLM-as-a-Judge — Jiawei Gu et al., 2024