What is mattwood.fyi?

A live, structured feed of riffs and links from Matt Wood, Chief AI & Technology Officer at AWS — readable by people and queryable by agents as a typed knowledge graph.

Is there an API and is it free?

Yes. A public, read-only query API is available with no authentication and no API keys. See https://mattwood.fyi/openapi.json and https://mattwood.fyi/docs.

How should an AI agent use this site?

Fetch https://mattwood.fyi/feed.json for the full structured feed, or query the API for keyword/semantic search and the connection graph. See https://mattwood.fyi/agents/ for when-to-use guidance.

Evaluating LLM Judge Agreement and Reliability

Connections

Related to

Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks Both examine trustworthiness gaps in LLM evaluation contexts — one on cheating behavior, one on judge reliability — revealing systematic ways LLM assessments can be misleading.
Comparing 11 Different AI Models Comparing 11 AI models requires evaluation methodology; LLM judges are a common tool for such comparisons, making judge reliability directly relevant to the validity of those comparisons.

Supports

SWE-bench Science: Coding Agents for Engineering Tasks SWE-bench science for coding agents uses automated evaluation pipelines; understanding LLM judge reliability is foundational to trusting agent performance measurements on such benchmarks.
Trismik: Evidence-Based Model Switching Trismik's evidence-based model switching depends on reliable model evaluation signals; understanding when LLM judges agree and can be trusted underpins evidence-based selection systems.

Challenges

Grok 4.6 Benchmarks and Cost Efficiency Analysis Grok 4.6 benchmark analysis relies on evaluation scores that may themselves come from LLM judges; if judge agreement is misleading, benchmark rankings like these may be less reliable than presented.

Supported by

Contrastive Language Model (CLM) Contrastive language models produce more calibrated embeddings that could improve LLM judge reliability and inter-judge agreement by providing better similarity measures
How To Write With An LLM Both address the reliability problem with LLM outputs—the new item warns against trusting LLM encouragement while the existing item evaluates LLM judge agreement and reliability
Automating Eval Design and Hillclimbing with Claude Both focus on LLM evaluation methodology; the new item's principles for designing evals without overfitting directly complement research on LLM judge agreement and reliability
Claude Opus 5.5 Intelligence & Performance Analysis LLM judge agreement and reliability evaluation methodology is directly relevant to validating the benchmark results and rankings presented in this model performance analysis

Numberwang Neural Network Both deal with classification reliability and evaluation - Numberwang NN satirizes what 'correct' classification even means when the ground truth is arbitrary, mirroring real concerns about LLM judge agreement
SSRN Research Paper #7496559 Academic evaluation methodology for LLMs is a common SSRN paper topic, and this collection's focus suggests empirical AI assessment research as a plausible subject
Engineering a Calibrated Classifier Model Both deal with reliability and calibration of LLM-based judgment systems - Hobson explicitly optimizes for calibration while this item evaluates LLM judge agreement and reliability

Type:	link
First published:	Monday, 14 September 2026 at 19:15 UTC
Location:	https://www.amazon.science/blog/when-llm-judges-agree-should-we-believe-them
URL:	https://mattwood.fyi/i/f59bdda9

I'm Matt Wood, and this is For Your Information. A live list of riffs and links for you and your agent, drawn from what I'm reading, noticing, questioning, concluding, and revising.

Connections