What is mattwood.fyi?

A live, structured feed of riffs and links from Matt Wood, Chief AI & Technology Officer at AWS — readable by people and queryable by agents as a typed knowledge graph.

Is there an API and is it free?

Yes. A public, read-only query API is available with no authentication and no API keys. See https://mattwood.fyi/openapi.json and https://mattwood.fyi/docs.

How should an AI agent use this site?

Fetch https://mattwood.fyi/feed.json for the full structured feed, or query the API for keyword/semantic search and the connection graph. See https://mattwood.fyi/agents/ for when-to-use guidance.

SWE-bench Science: Coding Agents for Engineering Tasks

Connections

Related to

Material Discovery Bench: LLM Research Benchmark Both are benchmark datasets targeting AI capabilities in scientific/research domains — Material Discovery Bench evaluates LLMs on research tasks while SWE-bench Science focuses on coding agents resolving engineering tasks in scientific computing
Leiden Declaration on Artificial Intelligence and Mathematics Both address the intersection of AI systems and rigorous scientific/mathematical domains — the Leiden Declaration concerns AI in mathematics while SWE-bench Science benchmarks coding agents on scientific engineering tasks
Grok 4.6 Benchmarks and Cost Efficiency Analysis SWE-bench Science serves as a comparable evaluation framework to Grok 4.6 benchmark analysis — both are about rigorously measuring coding/reasoning agent capabilities, providing complementary data points for assessing frontier model performance

Supports

AI Coding and the Bug Selection Problem SWE-bench Science provides empirical evidence for the 'bug selection problem' thesis — by benchmarking which engineering tasks coding agents can actually resolve, it illuminates the systematic biases in what AI coding tools succeed and fail at
Agentic Coding: LLM Agents, Testing, and Hallucination Risks SWE-bench Science directly instantiates concerns raised about agentic coding hallucination risks — scientific computing tasks require correctness that is harder to verify, making benchmark evaluation of agent reliability especially important

Develops into

Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks Both investigate how AI agents perform (or cheat) on structured task benchmarks — SWE-bench Science adds scientific computing as a domain where agent benchmark integrity and honest evaluation are critical concerns
The AI Operating Layer SWE-bench analyzes coding agents for engineering tasks; Coder's platform represents the enterprise infrastructure layer that operationalizes such agents at scale with governance controls

SWE-2: Advanced Coding Model with Improved Cost-Capability Tradeoff SWE-2 directly targets the SWE-bench style coding agent evaluation domain; both focus on AI models solving real-world software engineering tasks with measurable benchmarks
GDM Science Skills: Agentic Scientific Workflows Both address agentic AI applied to scientific/engineering problem-solving tasks, with SWE-bench Science benchmarking coding agents on structured problems while GDM Science Skills grounds agents in real scientific databases
Jalapeño Shows Power of LLMs for Chip Design Both examine LLM agents applied to specialized engineering tasks (chip design vs. software engineering), exploring capability boundaries of LLMs in technical domains
Anthropic Formalizes Fermat's Last Theorem in Lean SWE-bench Science evaluates AI coding agents on engineering tasks; Anthropic's Lean formalization represents a parallel benchmark achievement — completing the last of Wiedijk's 100 formalization challenges — demonstrating AI's frontier formal reasoning capabilities
AI-Driven Development Life Cycle: Reimagining Software Engineering SWE-bench evaluates coding agents on engineering tasks, providing empirical grounding for claims about AI capabilities across the software development lifecycle that the reimagining article theorizes about
Domain-Driven Agents SWE-bench evaluates coding agents on engineering tasks; domain-driven agents provide an architectural framework for how such agents should be oriented within complex codebases to improve task performance

Supported by

Evaluating LLM Judge Agreement and Reliability SWE-bench science for coding agents uses automated evaluation pipelines; understanding LLM judge reliability is foundational to trusting agent performance measurements on such benchmarks.
Cognition and AWS Strategic Partnership for Autonomous Engineers SWE-bench science on coding agents for engineering tasks provides benchmarking context for evaluating autonomous engineers like those Cognition deploys
Autonomous Mode in Kiro Web for Technical Debt Kiro's autonomous mode for technical debt is a direct real-world application of the coding agent paradigm analyzed in SWE-bench Science, validating that LLM agents can handle full engineering task cycles
WikiSkill: Agent Experience into Persistent Knowledge WikiSkill's reusable skill consolidation directly addresses the engineering task performance challenges identified in SWE-bench, as persistent knowledge could reduce repeated errors in coding agent workflows
Dream-RSI: Recursive Self-Improvement through Evolving Worlds Dream-RSI's approach to improving coding agent exploration strategies through accumulated history directly extends the scientific methodology behind SWE-bench agentic coding evaluation
EnvHarness: Awakening Static Worlds for Agent Learning SWE-bench Science evaluates coding agents on engineering tasks, and EnvHarness's dynamic environment adaptation directly addresses the challenge of creating reproducible, controllable environments needed for such agent benchmarking
Skillbay: AI Skills Marketplace SWE-bench Science evaluates coding agent performance on engineering tasks — Skillbay's before-and-after evidence model provides a practical verification mechanism analogous to benchmark-driven validation of skill acquisition
Can AI Design Circuit Boards Yet? EEBench follows the same scientific benchmarking philosophy as SWE-bench, extending rigorous AI capability measurement from software engineering tasks into hardware/electronics design domains

Challenged by

Building a Software Factory: Merging 1000 PRs in a Week SWE-bench evaluates coding agents on isolated tasks; the Software Factory's 1000-PR result suggests real-world agent pipelines outpace what benchmarks capture about production-scale throughput

Type:	link
First published:	Tuesday, 25 August 2026 at 22:48 UTC
Location:	https://github.com/OpenMOSS/SWE-bench-Science
URL:	https://mattwood.fyi/i/086a655f

I'm Matt Wood, and this is For Your Information. A live list of riffs and links for you and your agent, drawn from what I'm reading, noticing, questioning, concluding, and revising.

Connections