How do I run my model on it?

You can use their official Docker setup or the SWE-bench CLI. You need to containerize your agent and have it process the issue descriptions to generate patches.

What's the difference between SWE-bench Verified and the original?

Verified is a smaller, human-filtered subset of 500 instances where the test cases are more reliable, reducing false negatives from faulty evaluations.

Why are some scores on the leaderboard lower than I'd expect for strong models?

Scores depend heavily on the agent architecture (e.g., mini-SWE-agent vs. SWE-agent), the model version, and evaluation settings like 'bash-only' mode.

Can I evaluate a model that doesn't use tools or bash commands?

The standard evaluation environment is designed for agents that can execute code and commands. Evaluating a pure text completion model would require a custom setup.

How is the cost calculated in the leaderboard?

It's the estimated API cost for the model to generate the patch, based on token counts and standard API pricing for the specific model used in the run.

Directory/Coding/AI Agents/SWE-bench

SWE-bench

SWE-bench is a comprehensive benchmark suite designed to evaluate the software engineering capabilities of large language models and AI agents. It provides a standardized evaluation framework using real-world GitHub issues from popular Python repositories, measuring an agent's ability to understand, reproduce, and solve complex programming problems. The platform includes multiple benchmark variants like SWE-bench Verified, Lite, Multilingual, and Multimodal to cater to different evaluation needs and resource constraints. The core offering is a leaderboards system that ranks AI models and agent frameworks based on their percentage of resolved issues. It includes detailed metrics like resolution rate, average cost, and trajectory information. SWE-bench provides the infrastructure for running these evaluations, often using Docker containers for reproducible environments. Beyond the main benchmark, the ecosystem includes related tools like SWE-agent, mini-SWE-agent, and SWE-smith for mode

Coding

Visit website

Added on

Sep 19, 2026

AI SWE BenchmarkGitHub Issue ResolverCost-Performance LeaderboardsMulti-Model EvalOpen Source

Product information

Everything worth knowing about SWE-bench

The complete picture — from everyday features to the technical detail builders and IT folks dig for.

Core features

What it actually does

Standardized SWE Benchmarking

Provides a standardized benchmark for evaluating AI models on real-world software engineering tasks from GitHub issues.

Multiple Benchmark Variants

Offers specialized subsets like Verified (human-filtered), Lite (cost-effective), Multilingual (42 repos, 9 languages), and Multimodal (visual elements).

Public Leaderboards

Features transparent leaderboards to compare AI model and agent performance on task resolution percentage and average cost.

Open Source & Community Driven

Core benchmarks and agents like mini-SWE-agent are open source, fostering community research and reproducibility.

Technical capabilities

Cost-Per-Task Tracking

Reports average dollar cost per resolved instance for different model-agent combinations, enabling cost-benefit analysis.

Dockerized Evaluation Environment

Utilizes containerized environments (e.g., Docker) to ensure consistent, reproducible evaluation conditions for all submissions.

Must watch videos

2:43

Tutorial2:43

11 Generate Issue Text

10. Evaluate Predictions on Task Instances

3:59

Tutorial3:59

10. Evaluate Predictions on Task Instances

4:24

Tutorial4:24

9. Validate Task Instances (Pt. 3)

Who uses it

Real use cases, no hype

Benchmarking AI Code Agents

Compare the real-world bug-fixing ability of different LLMs and agent frameworks.

Model Selection for Dev Tools

Choose the most cost-effective model for AI-powered IDEs or automated coding assistants.

Research in Software Engineering

Provide a standardized, reproducible testbed for studying automated program repair.

Agent Framework Development

Iterate on and validate new agent architectures against a challenging, curated dataset.

Cost-Performance Analysis

Analyze the trade-offs between model API cost and task resolution rate for deployments.

What's great

Gold standard for evaluating AI coding agents on real-world tasks.
Comprehensive benchmark suite covering multiple languages and modalities.
Transparent leaderboards with detailed cost and trajectory data.

Technical strengths

Docker-ized evaluation environment for reproducible results.
Human-verified instance set (SWE-bench Verified) reduces evaluation noise.
Supports diverse agent architectures from simple RAG to complex tool-use.

Where it falls short

Evaluation can be expensive and time-consuming for large models.
Focuses narrowly on issue resolution; doesn't assess code quality or style.
Leaderboard saturation makes it hard to distinguish top-tier models.

Technical limitations

Instances are Python-centric originally; multilingual coverage is newer and smaller.
The default 'bash-only' setting may not reflect complex, interactive agent capabilities.
Evaluation metrics are binary (resolved/not), lacking granular partial credit.

For technical folks

The deep-dive specs

FAQ

Includes technical Q&A

It's a benchmark dataset of over 2,000 real GitHub issues and their corresponding pull request solutions, designed to test an AI's ability to write code to fix them.

Traffic Insights

Monthly Visits

—

October 2026

Growth Rate

vs last month

Dominance

in Coding

Top Country

—

Top Countries Breakdown

—

Growth Trend

Stable

Traffic has remained stable.

Pricing overview

Unknown Pricing

SWE-bench Pricing

SWE-bench uses a unknown pricing structure. Real-time rates, custom enterprise quotas, and plan details are managed directly on their official platform.

Check Pricing on SWE-bench

Reviews