Part 1: How LLMs work - tokens, vectors, attention

Module 10 · Sat 19 Sep

LLMs aren't magic chat boxes. They're token-based systems whose behaviour depends on context, prompting and product design - so PMs need to know the basics.

• Tokens are the real unit of cost and processing. Token count ≠ word count - one word can split into several tokens.
• Vectors are numbers that represent meaning; more dimensions = richer meaning. Attention weighs the important earlier tokens, turning a plain vector into a contextual one.
• The model writes by next-token prediction, one token at a time.

Product levers: temperature (low for support, enterprise or factual assistants; higher for creative output), max tokens (set per task, not by default), input limits and model choice (bigger = more capable, more expensive). Think cost, latency and user need together.

More context isn't always better - relevant context helps, noisy context hurts. Don't paste huge documents; retrieve the relevant parts (the class linked this to RAG and retrieval systems). Domain context is what makes business AI useful, not just impressive. And prototypes are easy - validation is the hard part.