What is mattwood.fyi?

A live, structured feed of riffs and links from Matt Wood, Chief AI & Technology Officer at AWS — readable by people and queryable by agents as a typed knowledge graph.

Is there an API and is it free?

Yes. A public, read-only query API is available with no authentication and no API keys. See https://mattwood.fyi/openapi.json and https://mattwood.fyi/docs.

How should an AI agent use this site?

Fetch https://mattwood.fyi/feed.json for the full structured feed, or query the API for keyword/semantic search and the connection graph. See https://mattwood.fyi/agents/ for when-to-use guidance.

11–16× Faster LLM Inference with llama.cpp

Connections

Related to

TerminalWidget for Mac, iPhone, and iPad Both concern macOS-specific technical infrastructure: TerminalWidget targets Mac/iPhone/iPad native tooling, while the new item addresses Apple Silicon's Virtualization framework for GPU passthrough — both navigate Apple platform constraints
AMD acquires AI chip startup Taalas to boost inference performance by etching models into silicon Both address hardware-level acceleration for LLM inference: the new item via GPU passthrough on Apple Silicon, AMD's Taalas via etching models into silicon — complementary approaches to inference performance bottlenecks

Supports

Introducing Muse Glimmer: An Open Agentic Model That Runs on Your Device Both focus on running capable AI models locally on device; faster llama.cpp inference via Apple Silicon GPU passthrough directly enables the kind of on-device agentic models Muse Glimmer represents
Ways to think about token pricing The 11-16x inference speedup quantified here provides concrete data points for token pricing analysis — faster local inference changes cost structures and the economics of self-hosted vs. API inference

Challenges

Together AI: Research-Driven Inference and Training Platform llama.cpp's local GPU-accelerated inference on consumer Apple Silicon hardware challenges the premise of cloud-based inference platforms like Together AI by reducing latency and cost at the edge
Baseten: High-Performance AI Inference Platform 11-16x faster local inference via llama.cpp on Apple Silicon GPU passthrough directly competes with high-performance cloud inference platforms like Baseten, making on-device inference more viable

Supported by

Nemotron 3.5 Lightning and NeMo Switchyard for Agentic AI Nemotron Lightning's emphasis on faster, more efficient inference directly aligns with and is supported by techniques achieving 11-16x faster LLM inference; both target the same goal of reducing inference latency
Qwen 3.8 2.4T Mixture of Experts Model The 11-16x faster llama.cpp inference work is directly applicable to running large MoE models like Qwen 3 8B 2.4T locally, as MoE architectures benefit especially from sparse activation inference optimizations

GPT-5.6 Sol Ultrafast: Frontier Intelligence at Unprecedented Speed Both focus on dramatically accelerating LLM inference speed (11x for Ultrafast Mode vs 11-16x for llama.cpp), representing parallel approaches to the same performance challenge
Bullet: Fast Coding Agent Both focus on latency/performance optimization in AI systems — Bullet through parallel tool execution and task routing, the other through faster LLM inference — reflecting the same performance-first design philosophy
MiniMax H3 Inference Engine for Mac Both focus on optimizing LLM inference performance on local/consumer hardware, with llama.cpp achieving 11-16x speedups and H3 providing an alternative inference architecture for Mac
Compression is prediction Faster LLM inference via llama.cpp optimizations relates to compression-prediction: understanding that LLMs are fundamentally predictive compressors informs where computational bottlenecks lie in token generation
A simple fix for LLM tail latency Both address LLM inference performance optimization - the duplicate-request strategy and llama.cpp speed improvements are complementary approaches to reducing response latency in production systems
DiffusionGemma Technical Report Both focus on dramatically accelerating LLM inference throughput—DiffusionGemma achieves ~1,500 tokens/sec via parallel diffusion decoding, while llama.cpp achieves 11-16x speedup via quantization and CPU optimization; both attack the same bottleneck from different architectural angles

Type:	link
First published:	Tuesday, 11 August 2026 at 17:55 UTC
Location:	https://github.com/trycua/cua/blob/main/blog/gpu-passthrough-macos-vms.md
URL:	https://mattwood.fyi/i/00ff9707

I'm Matt Wood, and this is For Your Information. A live list of riffs and links for you and your agent, drawn from what I'm reading, noticing, questioning, concluding, and revising.

Connections