Methodology
How we source, evaluate, and rank the world's top autonomous AI agents.
How does The Agentic Leaderboard evaluate AI agents?
The Agentic Leaderboard is an autonomous benchmark that evaluates open-source AI agents on their ability to complete real-world tasks without human intervention. Each agent is scored from 0 to 100 using five weighted criteria:
- • Reliability (35%) — Success rate of end-to-end task completion
- • Tool Selection Quality (20%) — Accuracy of API and tool selection for sub-tasks
- • Autonomous Iteration (15%) — Depth of continuous reasoning-and-action steps
- • Cost Efficiency (10%) — Normalized cost per successful task
- • Community Mindshare (20%) — Real-world adoption via GitHub stars, citations, and usage
The pipeline runs daily, autonomously discovering agents from GitHub, Hugging Face, Papers With Code, curated seed lists, and community awesome lists.
The Philosophy
Most AI benchmarks measure stochastic language prediction — how well a model guesses the next token. That tells you very little about whether an agent can actually do something useful in the real world.
The Agentic Leaderboard measures something different: closed-loop execution and autonomous task completion. Can the agent select the right tools, chain multiple reasoning steps, recover from errors, and deliver a verifiable result — without a human in the loop?
We evaluate every agent across five dimensions that together capture reliability, intelligence, efficiency, and real-world adoption. The result is a single score out of 100 that reflects genuine agentic capability.
Data Sources & Discovery
Our Scraper Agent autonomously discovers new agentic builds from five complementary sources, running twice weekly.
GitHub Search API
We query multiple agentic topic tags and keywords — ai-agents, agentic-workflows, autonomous-agents, llm-agent, and more — filtering for actively maintained repos with 50+ stars. Results are deduplicated across queries.
Hugging Face Spaces & Models
We scan Hugging Face for agent-related Spaces and Models, cross-referencing with GitHub repos when available. This surfaces research-oriented agents that may not appear in standard GitHub searches.
Papers With Code
Academic papers tagged with agent-related topics are queried, and their linked GitHub repositories are extracted. This captures cutting-edge research agents backed by peer-reviewed work.
Curated Seed List
A manually maintained list of 30+ known agentic products and frameworks — Devin, CrewAI, AutoGPT, LangGraph, n8n, and others — ensures that high-profile agents are never missed by automated discovery.
Awesome Lists
Community-curated awesome lists (e2b-dev/awesome-ai-agents, kyrolabs/awesome-agents, and others) are parsed for GitHub links, capturing agents surfaced by the developer community.
The 5 Core Evaluation Criteria
Each agent is evaluated across five dimensions. The Evaluator Agent analyses repository metadata, documentation quality, CI/CD maturity, and community signals to produce proxy scores for each criterion.
Reliability (Success Rate)
35%The percentage of end-to-end tasks an agent completes successfully without human intervention. We measure this through CI/CD presence, test coverage, community health signals, and commit activity — proxies that correlate strongly with production reliability.
Tool Selection Quality (TSQ)
20%How accurately an agent selects and formats the correct API or tool for a given sub-task. We evaluate this through README documentation quality, presence of usage examples, structured docs, and configuration maturity — indicators of well-defined tool interfaces.
Autonomous Iteration
15%The average number of continuous reasoning-and-action steps an agent can execute before failing. We model this as a Gaussian curve peaking at ~15 steps — too few suggests shallow execution, too many suggests looping. Repo structural complexity serves as the proxy.
Cost Efficiency
10%Normalized cost per successful task, combining token usage estimates and infrastructure overhead. Lighter, well-structured repos with minimal dependencies score higher. Docker presence and dependency count factor into the estimate.
Community Mindshare
20%A composite signal of real-world adoption and developer trust. Calculated from GitHub stars, fork count, push recency, Hugging Face likes, and academic citations — log-normalized and weighted to prevent any single metric from dominating.
The Scoring Formula
Our Ranker Agent applies a weighted formula to compute a final score between 0 and 100 for every evaluated agent.
Total Score = (0.35 × R) + (0.20 × T) + (0.15 × Snorm) + (0.10 × Enorm) + (0.20 × M)
Variable Legend
- R
- Reliability Success Rate (0–100)
- T
- Tool Selection Quality (0–100)
- Sₙₒᵣₘ
- Normalized Autonomous Iteration (Gaussian, 0–100)
- Eₙₒᵣₘ
- Normalized Efficiency — inverse of cost + latency (0–100)
- M
- Community Mindshare Score (0–100)
Transparency & Limitations
- • Static analysis only (for now): Scores reflect repository metadata, documentation quality, CI/CD maturity, and community adoption — strong proxies for reliability, but not direct runtime measurements.
- • No live sandbox execution yet: We plan to introduce opt-in sandbox evaluation for agents that expose a standard interface, enabling direct measurement of task completion, tool accuracy, and cost.
- • Fully autonomous pipeline: The Scraper, Evaluator, Ranker, and Publisher agents run daily without human intervention. All scoring weights are documented and reflected in our open-source codebase.
- • Open methodology: Every formula, weight, and data source is published on this page. We believe transparency is essential for a credible benchmark.
- • Deduplication: Agents are deduplicated by normalized repository name (case-insensitive). The highest-scoring entry is kept when duplicates are found.
Momentum Score — how The New 100 is ranked
The five criteria above evaluate established agents and power the all-time Top 100. They do not work on brand-new projects: 35% of that score is Reliability, measured through CI presence, test coverage, and community-health signals that are near-zero on a four-day-old repository. Scoring newcomers that way would compress every project into a narrow band and quietly reduce the ranking to raw star count.
So The New 100 uses a separate metric built for young projects — Momentum Score (0–100), computed over a 30-day rolling window:
| Component | Weight | What it measures |
|---|---|---|
| Star Velocity | 35% | Stars per day since launch — normalised across the cohort. Rewards genuine traction rather than accumulated age. |
| Traction | 20% | Forks and watchers, log-normalised. Forking signals real use, not just bookmarking. |
| Activity | 20% | Commit cadence within the window plus push recency — is it still being built, or was it a one-day dump? |
| Craft | 15% | README depth and structure, code examples, license, description, topics, homepage. |
| Relevance | 10% | Strength of the agentic signal in topics, description, and name — keeps general AI tooling from crowding out actual agents. |
Why a 30-day window?
Seven days is too tight: it yields fewer candidates than board slots, so almost everything found would qualify and the ranking would mean little. Thirty days draws from roughly 1,000–2,000 projects, making a Top 100 genuinely selective — and it lets an agent hold a place for up to four weeks, so week-over-week movement, rank history, and climbers remain meaningful.
- • Discovery: twelve GitHub search queries covering agentic topics and phrasings, paginated, filtered to repositories created inside the window with 5+ stars.
- • Two-phase scoring: every candidate is pre-scored on signals present in search results; the top 150 are then enriched with README and commit data before final scoring. This keeps the pass inside API limits without sampling.
- • Known limitation: star velocity can be gamed, and a viral launch is not the same as a good agent. Momentum measures early traction and craft signals — it is explicitly a discovery ranking, not a claim about long-term quality. Agents that sustain it graduate onto the Top 100 on their own merits.
Data & Downloads
The full dataset behind this methodology is free under CC-BY 4.0 — machine-readable formats for researchers, developers, and LLMs.
Available Formats
- JSON/new100.json
The New 100 — newly launched agents with full Momentum Score breakdowns.
- JSON/leaderboard.json
The all-time Top 100. Machine-readable. All metrics included.
- JSON/history.json
Longitudinal rank history — every agent's weekly rank and score since March 2026.
- TXT/llms.txt
Concise markdown summary (~5KB). LLM-optimized site index.
- TXT/llms-full.txt
Complete markdown dump. Every agent, every metric.
- JSON/weekly/*.json
Historical weekly digests. Movers, new entries, fresh agents.
- RSS/feed.xml
Weekly digest feed (also /atom.xml and /feed.json).
- GITGitHub data mirror
Auto-updated repo with the live Top 100 as a README table + raw JSON. Star it to follow weekly changes.
Embed the live Top 10 on your site
One script tag — auto-updates every Monday. Perfect for blogs, newsletters, and docs.
Get the widget →How to Cite
APA
The Agentic Leaderboard. (2026). Weekly ranking of top open-source AI agents. Retrieved from https://www.theagenticleaderboard.com
BibTeX
@misc{agenticleaderboard2026,
title={The Agentic Leaderboard},
url={https://www.theagenticleaderboard.com},
year={2026},
note={Weekly snapshot}
}Plaintext
The Agentic Leaderboard, weekly snapshot. theagenticleaderboard.com.