What is mattwood.fyi?

A live, structured feed of riffs and links from Matt Wood, Chief AI & Technology Officer at AWS — readable by people and queryable by agents as a typed knowledge graph.

Is there an API and is it free?

Yes. A public, read-only query API is available with no authentication and no API keys. See https://mattwood.fyi/openapi.json and https://mattwood.fyi/docs.

How should an AI agent use this site?

Fetch https://mattwood.fyi/feed.json for the full structured feed, or query the API for keyword/semantic search and the connection graph. See https://mattwood.fyi/agents/ for when-to-use guidance.

Evaluating LLM Judge Agreement and Reliability

Connections

Related to

Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks Both examine trustworthiness gaps in LLM evaluation contexts — one on cheating behavior, one on judge reliability — revealing systematic ways LLM assessments can be misleading.
Comparing 11 Different AI Models Comparing 11 AI models requires evaluation methodology; LLM judges are a common tool for such comparisons, making judge reliability directly relevant to the validity of those comparisons.

Supports

SWE-bench Science: Coding Agents for Engineering Tasks SWE-bench science for coding agents uses automated evaluation pipelines; understanding LLM judge reliability is foundational to trusting agent performance measurements on such benchmarks.
Trismik: Evidence-Based Model Switching Trismik's evidence-based model switching depends on reliable model evaluation signals; understanding when LLM judges agree and can be trusted underpins evidence-based selection systems.

Challenges

Grok 4.6 Benchmarks and Cost Efficiency Analysis Grok 4.6 benchmark analysis relies on evaluation scores that may themselves come from LLM judges; if judge agreement is misleading, benchmark rankings like these may be less reliable than presented.

Numberwang Neural Network Both deal with classification reliability and evaluation - Numberwang NN satirizes what 'correct' classification even means when the ground truth is arbitrary, mirroring real concerns about LLM judge agreement

Supported by

How To Write With An LLM Both address the reliability problem with LLM outputs—the new item warns against trusting LLM encouragement while the existing item evaluates LLM judge agreement and reliability

Type:	link
First published:	Monday, 14 September 2026 at 19:15 UTC
Location:	https://www.amazon.science/blog/when-llm-judges-agree-should-we-believe-them
URL:	https://mattwood.fyi/i/f59bdda9

I'm Matt Wood, and this is For Your Information. A live list of riffs and links for you and your agent, drawn from what I'm reading, noticing, questioning, concluding, and revising.

Connections