Why can't I just use SWE-bench scores to choose a coding agent?

SWE-bench is 2,294 problems drawn from 12 popular Python repositories. Your code, your conventions, your test suite and your definition of done are not in it. A published score also belongs to the scaffold that produced it, not to the model alone. A leaderboard measures a research system, not your repository.

How should an internal agent eval define success?

The strongest practical bar is merged and not reverted within N days. It captures human acceptance rather than machine plausibility, it survives code review, and the revert window catches plausible-but-wrong changes. Tests passing alone is weaker: research has shown patches that merely delete functionality can satisfy a validation suite.

How many times do I need to run an eval before the number means anything?

More than once. Published measurements of repeated runs show accuracy varying substantially across identical prompts. A forty-task eval executed a single time cannot distinguish a 55 percent agent from a 65 percent agent, because the difference is only four tasks. Run each task several times, treat per-task success rate as the unit of analysis, and report an interval rather than a point estimate.

Agent Evals - Measuring Agents Against Your Own Repo

Public benchmarks say little about your codebase. How to build evaluations that measure agents on the work you actually do

Your Repository Is Not In The Benchmark

A leaderboard measures somebody else's agent, corpus and harness

SWE-bench is the number most often quoted in agent marketing. Its authors describe it as "an evaluation framework consisting of 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories" (Jimenez et al., ICLR 2024). That is a specification of what the score can mean. If you do not write Python in a library-shaped codebase with a mature test suite, it measures somebody else's job. The paper also shows how fast these figures age: "The best-performing model, Claude 2, is able to solve a mere 1.96% of the issues."

Cross-language generalization remains open. SWE-bench Multimodal added "617 task instances collected from 17 JavaScript libraries" and found "top-performing SWE-bench systems struggle with SWE-bench M, revealing limitations in... cross-language generalization" (arXiv, October 2024). Multi-SWE-bench covers seven more languages with "1,632 high-quality instances... annotated from 2,456 candidates by 68 expert annotators" (arXiv, April 2025). That these had to be built at all is the point.

Your code is not in it

Internal frameworks, legacy modules, generated clients, build quirks - exactly where agents lose the thread.

Your conventions are not in it

House style, error handling, migration rules. A patch can be functionally right and still fail review.

Your test suite is not in it

Benchmark tasks are picked partly for fast, deterministic tests. A slow or thin suite gives a far weaker signal.

Your definition of done is not in it

The benchmark is done when hidden tests pass. You are done when a human merges it and nobody reverts it.

The Public Benchmarks Struggle On Their Own Terms

Not a reason to dismiss them, a reason to stop procuring with them

SWE-bench Verified exists because the original set had measurable quality problems. OpenAI (2024), with the SWE-bench authors, "worked with 93 software developers experienced in Python to manually screen SWE-bench samples for quality." The result: "38.3% of samples were flagged for underspecified problem statements, and 61.1% were flagged for unit tests that may unfairly mark valid solutions as incorrect," and "68.3% of SWE-bench samples" were "filtered out due to underspecification, unfair unit tests, or other issues," leaving 500 validated instances. Read the two numbers carefully: the 68.3 percent is a share of the annotated sample the screeners actually worked through, not of the full 2,294-problem set, so it and the 500 are not two figures over the same denominator. Each cause reappears in internal evals: tests "often overly specific... unrelated to the issue"; environments "difficult to reliably set up", "causing unit tests to fail regardless of the solution."

Contamination is the second layer. The preprint SWE-Bench+ (Aleithan et al., October 2024) reported "32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments" - solution leakage - plus "31.08% of the passed patches are suspicious patches due to weak test cases." Filtering both, "the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%." Also, "over 94% of the issues were created before LLM's knowledge cutoff dates." A July 2025 paper on agentic benchmarks found such issues "can lead to under- or overestimation of agents' performance by up to 100%."

A benchmark is a research instrument; a leaderboard is a marketing artifact

None of this makes benchmarks worthless - they are how the field measures progress, and how solution leakage was found at all. The error is transfer: using an instrument built to compare research systems on a fixed Python corpus to pick the agent for your monorepo.

The Score Belongs To The Harness, Not Just The Model

A published score is model plus scaffold plus excluded tasks

Vendors are open about this in the footnotes. In its Claude 3.7 Sonnet announcement (Anthropic, February 2025), the vendor reports two SWE-bench Verified figures: 63.7% without high compute, 70.3% with it. High compute, per the vendor, samples multiple attempts in parallel, discards patches that break visible regression tests, and ranks what remains. The run also excludes 11 of the 500 tasks as incompatible with its infrastructure, so the denominator is 489.

Taking that vendor-reported result at face value, the gap between the two figures is not intelligence. It is scaffolding: parallel sampling, a filter, a ranker. Your platform has its own scaffold, retry policy and tool definitions. An internal eval holds your scaffold fixed and varies one thing at a time - exactly what a leaderboard cannot do.

Building An Eval On Your Own Repository

Harvest real tasks, define success, hold something back, repeat runs

The raw material already exists: closed issues and merged pull requests. Each task needs the repository, the commit immediately before the fix landed, the issue text as written at the time, and the change eventually accepted. Check out the base commit, hand the agent the issue text and nothing else, then compare what comes back.

Define success first. The strongest practical bar is "merged and not reverted within N days": it captures human acceptance rather than machine plausibility, it survives the review filter, and the revert window catches the plausible-but-wrong change that passed tests on Tuesday and broke something on Thursday. It is slow, so pair it with a cheaper proxy treated as a leading indicator only. Hold out a set that appears nowhere in your prompt surface - no ticket template, no AGENTS.md, no retrieval index. Run every task repeatedly and record cost per success alongside quality; see agent observability for attribution.

An eval task definition

eval:
  name: internal-repo-eval
  runs_per_task: 5            # stochastic agent, repeated trials, report an interval
  budget:
    usd_per_task: 4.00        # hard stop; a run that exceeds this is scored as failed
    wall_clock_minutes: 30

  scaffold:                   # pin the harness - it is part of what you are measuring
    agent: internal-agent
    agent_version: 2026.03.1
    model: pinned-model-id
    tools: [read, write, shell, test-runner]
    retries: 2

  tasks:
    - id: PLAT-4821
      repo: git.internal/platform/billing
      base_commit: 9f2c1ab      # commit immediately BEFORE the accepted fix
      task_text_from: issue     # the issue as written then; no hindsight edits
      holdout: true             # never referenced in prompts, templates or docs
      difficulty: medium        # sample the distribution you actually face
      reference_diff: 4d81e07   # what humans merged; for review only, never shown
      success:
        tests:
          command: make test
          must_pass: true
        static:
          command: make lint typecheck
          must_pass: true
        human_review:
          required: true        # a named reviewer accepts or rejects
        merged_and_not_reverted:
          window_days: 14       # the primary bar
      scoring:
        primary: merged_and_not_reverted
        secondary: [tests, human_review]
        report: [success_rate, ci_95, usd_per_success, tokens_per_success]

Harvesting candidate tasks, in pseudocode

candidates = []
for pr in merged_pull_requests(since="-18 months"):
    if pr.linked_issue is None:        # need a task statement written before the fix
        continue
    if pr.files_changed > 25:          # multi-week refactors are not eval tasks
        continue
    if pr.reverted_within(days=30):    # humans got it wrong; do not grade against it
        continue
    if leaks_solution(pr.linked_issue):  # fix pasted into the issue or its comments
        continue
    candidates.append(Task(
        repo=pr.repo,
        base_commit=pr.parent_commit,
        task_text=pr.linked_issue.body_at_open_time,
        reference_diff=pr.diff,
        difficulty=bucket(pr.files_changed, pr.review_rounds, pr.time_to_merge),
    ))

# Sample to MATCH your real difficulty mix, not to flatter the agent.
eval_set = stratified_sample(candidates, by="difficulty", match=production_mix)
holdout, visible = split(eval_set, holdout_fraction=0.4, seed_locked=True)

Candidate definitions of success, and how each is gamed

Definition	Measures	Gamed by
Tests pass	The assertions you happen to have	Weakening an assertion, deleting the failing behavior
Diff similarity to the human fix	Closeness to one accepted solution	Penalizes correct alternative designs
Human review accepts	Conventions and maintainability	Reviewer fatigue; small diffs approved faster than read
Merged and not reverted in N days	Acceptance plus survival in the real system	Routing agents onto low-risk work; slow to measure
Model-as-judge score	Agreement with a rubric you wrote	Rewards confident-sounding output; correlated failure when judge and agent share a model family

Stochasticity And Sample Size

A single number with no interval is not a measurement

Run-to-run variance is large enough to swamp the effects most teams want to detect. A study of non-determinism in "deterministic" LLM settings (Atil et al., August 2024) took five models across eight tasks with ten repeated runs each and observed "accuracy variations up to 15% across naturally occurring runs with a gap of best possible performance to worst possible performance up to 70%". None of the models "consistently delivers repeatable accuracy across all tasks."

Adding Error Bars to Evals (Miller, November 2024): "Fundamentally, evaluations are experiments; but the literature on evaluations has largely ignored the literature from other sciences on experiment analysis and planning." Quantifying Variance in Evaluation Benchmarks (Madaan et al., June 2024): "we rarely quantify the variance in our evaluation benchmarks, which dictates whether differences in performance are meaningful." Published numbers often come from one pass - OpenAI notes it "ran a single seed... for each scaffold."

Forty tasks, one run, no interval

A 40-task eval run once cannot tell a 55% agent from a 65% agent. The difference is four tasks; noise moves it further than that.

Repeat per task, then aggregate

Treat per-task success rate as the unit of analysis and report an interval. If two configurations' intervals overlap heavily, you have not shown a difference.

Small n is the common case

Internal sets usually start at n < 30 because harvesting is manual. State the sample size in every report so nobody reads a 10-point swing as a trend.

Failure Modes Of Internal Evals

Each has a published analogue in the benchmark literature

Rewarding easy work

Harvest tasks the agent already handles and the score climbs while nothing improves. Stratify so the difficulty mix matches the work you face.

Contaminating your own holdout

Tickets get pasted into prompts, fixes become AGENTS.md examples, and a public repository is contaminated by definition. SWE-Bench+ found the fix in the issue report or comments for 32.67% of successful patches.

A passing suite is not a correct change

This predates LLMs. Qi et al. (ISSTA 2015) found their Kali system, which only deletes functionality, "generates at least as many correct patches as prior GenProg, RSRepair, and AE systems". Also Smith et al. (ESEC/FSE 2015). EvalPlus (Liu et al., NeurIPS 2023) extended HumanEval's tests roughly 80x, "reducing the pass@k by up-to 19.3-28.9%".

Overfitting, weak holdouts, Goodhart

AI Agents That Matter (Kapoor et al., July 2024) observed "inadequate holdout sets, and sometimes none at all... agents that are fragile because they take shortcuts and overfit to the benchmark". Once the score lands in a team's objectives it stops being a measurement. Give the holdout to someone who does not tune the agent.

The biggest one: the eval measures the agent, not the system

METR's randomized controlled trial (blog, July 2025; paper) put 16 experienced developers across 246 tasks on repositories they already maintained: "When developers are allowed to use AI tools, they take 19% longer to complete issues - a significant slowdown." The belief gap is what should worry you. They forecast a 24% speedup, and after experiencing the slowdown still believed they had been sped up by 20%. One trial with experienced maintainers, so do not over-generalize; but a high eval score is compatible with a slower team, because the cost lands downstream in review.

Wiring Evals Into The Pipeline

Trigger on change, and sandbox the runs like production work

Attach the outcome to the run that produced it. If every eval attempt emits the same trace structure as a production run - task identifier, holdout flag, scoring outcome and cost as span attributes - a regression is traceable end to end rather than a number in a spreadsheet. That instrumentation belongs to agent observability; the eval writes into it.

Trigger on change, not on a calendar: a model upgrade, a prompt edit, a scaffold change, a tool-definition change, an index rebuild. A monthly cadence with no change to explain a shift is wasted compute. The execution path is ordinary CI work; see headless agents in CI. Isolate the runs: eval tasks check out historical commits and execute their build tooling, which is untrusted code by any reasonable definition. Use sandboxed environments for execution and egress control for what the run may reach. GUI-shaped tasks need separate treatment - no test suite, screen state hard to assert on - see computer-use agents.

An eval program is organizational capability

The DORA report (Google Cloud, September 2025) states that "AI's primary role is as an amplifier, magnifying an organization's existing strengths and weaknesses." Building an eval requires that you can already articulate what a good change looks like, that your tests mean something, that review is real rather than ceremonial, and that someone owns the holdout. A team with those properties learns something within a quarter. A team without them produces a number, watches it drift, and argues about it.

That is the honest limit: an eval does not teach a team to tell a good change from a bad one, it makes the distinction they already hold measurable. Where that judgment is contested it becomes a governance question - see AI governance.