Blog

A long model answer can take many seconds to finish, but the first tokens exist almost immediately. How streaming works over server-sent events, how to relay it from your server to the browser, and the details — stop reasons, cancellation, errors — that make it production-ready.

“Respond in JSON” works most of the time, which isn't good enough for software. How schema-constrained generation guarantees the shape of a model's output, how to use it with a Zod schema, and what it still can't guarantee.

A language model doesn't choose words — it produces probabilities, and then something draws from them. How temperature, top-p and top-k reshape that draw, why randomness is useful, and why you can't always count on turning it off.

A model trained to predict plausible text will produce plausible text, whether or not it's true. Where hallucinations come from, why they sound so confident, and the practical defences: grounding, letting the model abstain, citations, and verification.

Models don't see characters or words — they see tokens, produced by byte-pair encoding. How BPE learns its vocabulary, why it explains so many odd LLM behaviours, and what it means for cost and context limits.

Prompt engineering without measurement is guessing with extra steps. How to build an eval set from real traffic, pick a grader that isn't lying to you, and avoid tuning your way to a number that means nothing.

Everything an agent reads is input, and a model has no structural way to separate instructions from data. Why mitigations in the prompt are not a boundary, and where the real one goes.

A list of numbers where distance means similarity. What cosine similarity measures, why the same word can have two vectors, and the queries embeddings are quietly bad at.

Most RAG failures are retrieval failures, not generation failures. How chunking and embeddings decide what the model ever sees, the similarity-is-not-relevance trap, and what actually fixes it.

The window is a budget you refill from scratch every turn. Where the tokens actually go in an agent, why prompt caching is a prefix match, and the difference between clearing context and compacting it.