Blog

Where the money and time go in an LLM application — input tokens, output tokens, prefill and decode — and the levers that move them: prompt caching, shorter outputs, routing easy requests to smaller models, batch APIs, streaming, parallel calls, and not calling the model at all. With the arithmetic.

How mixture-of-experts models have hundreds of billions of parameters but use only a fraction for each token. The router, top-k expert selection, why total and active parameters differ, the load-balancing problem, and what MoE means for memory, speed and serving.

An exact-match cache misses every rephrasing of the same question. A semantic cache matches by meaning, using embeddings — returning answers in milliseconds, for free. How it works, choosing a similarity threshold, what must be part of the cache key, invalidation, and when not to use it.

Embedding search is fast because documents are encoded without seeing the query — and that's exactly why it ranks imprecisely. How cross-encoder rerankers read the query and each candidate together, why they're used as a second stage over a shortlist, what they cost, and how to tell whether one helps.

Large models generate one token per forward pass, and each pass is bottlenecked on memory, not maths. Speculative decoding lets a small draft model guess several tokens that the large model then checks in a single pass — producing the same output distribution, two to three times faster.

Attention is how a transformer lets each token pull in information from the others. Queries, keys and values; dot products, scaling and softmax; the causal mask; multiple heads — worked through with small vectors so every number on screen can be checked by hand.

LLMs generate one token at a time, and each new token attends to everything before it. The KV cache stores past keys and values so they're computed once — trading compute for memory. How prefill and decode differ, how to calculate the cache size, and the techniques that shrink it: GQA, quantisation, paging and prompt caching.

MCP is an open protocol for connecting AI applications to tools and data. How hosts, clients and servers fit together, what the JSON-RPC messages look like, the three things a server can offer — tools, resources and prompts — how to write a small server, and the security questions to ask before installing one.

Keyword search finds exact terms; vector search finds meaning; each misses what the other catches. How BM25 scores documents, why you can't just add the two scores, how reciprocal rank fusion merges rankings using positions alone, and where a reranker fits.

Three ways to make a general model work for your problem: change what you ask, change what it can see, or change the model itself. What each one is actually good at, why fine-tuning is the wrong tool for teaching facts, and the order to try them in.