Transformers and LLM Intuition

Concept

Large language models (LLMs) are transformer-based models trained to predict likely next tokens in text. A token is a piece of text processed by the model; it may be a word, part of a word, or punctuation. Repeating next-token prediction across very large text collections can produce models useful for drafting, classification, extraction, translation, and question answering.

Attention Intuition

In a sentence, not every earlier token matters equally for the next one. Attention lets a transformer weight relevant parts of its current context when forming representations. It is not a claim that a model has human attention or understanding; it is a learned computation over token relationships.

prompt/context → tokens → transformer layers → likely next token → repeat

The context window is the limited amount of input and prior generated text available at a time. Information outside it is not automatically considered. Prompting provides instructions and context, but a clear prompt cannot guarantee factuality or correct reasoning.

Pretraining and Adaptation

Pretraining exposes a model to broad patterns in language and code. Later use may rely on prompting, retrieval, or additional task-specific adaptation. This explains why one model can perform many language tasks, but it does not make it an authority on current facts or a database with guaranteed recall.

Failure Signals

Common Mistakes

Equating next-token prediction with verified reasoning.
Assuming the model has unlimited memory of a conversation.
Treating attention as a complete explanation for a model decision.
Assuming a longer prompt always improves the result.

Interview Perspective

Question: What does an LLM predict during generation?
Answer: A likely next token conditioned on the available context.
What the interviewer is testing: whether you can explain the mechanism without anthropomorphizing it.

Practice Questions

Why is a context window relevant to a long document task?
What is attention intended to help a transformer use?
Why can next-token prediction still generate an incorrect answer?
What does pretraining provide, and what does it not guarantee?

Quick Quiz

Are tokens always whole words? Answer: No.
Does an LLM automatically know current facts? Answer: No.

Key Takeaway

Key Takeaways

Transformers use attention over token context and generate likely continuations. Their language capability is powerful, but context limits and unreliable factuality remain central boundaries.

Next Lesson

Next, see how RAG adds retrieved source context without guaranteeing truth.