Embeddings and Semantic Similarity

Concept

An embedding is a numeric vector representing an item such as text, an image, or a product. Unlike a simple ID, a useful embedding places items with related learned patterns nearer in vector space. It lets a system compare meaning-like similarity rather than only exact keyword overlap.

Intuition

The questions “How do I reset my password?” and “I cannot sign in” use different words but may be close in embedding space. A semantic search system embeds a query, embeds document chunks, then retrieves nearby vectors. This can support search, recommendations, clustering, deduplication, and Retrieval-Augmented Generation.

Cosine Similarity

Cosine similarity compares the direction of two vectors. At an intuitive level, a higher value means vectors point more similarly and are often more semantically related for the embedding model. It is a ranking signal, not proof that two statements are equivalent, correct, or appropriate for a user.

Practical Workflow

documents → chunk text → embed chunks → store vectors
query → embed query → retrieve nearest chunks → inspect/use results

Chunking matters: a whole policy manual can be too broad, while tiny fragments can lose context. The embedding model, text quality, language, domain, similarity threshold, and metadata filters all affect retrieval. Embeddings are learned representations, not a universal measure of truth.

Failure Signals

Common Mistakes

Assuming nearest text is always the correct answer.
Using embeddings as a replacement for access control or source verification.
Confusing embeddings with PCA components or hand-selected features.
Ignoring stale, incomplete, or private source documents.

Interview Perspective

Question: Why use embeddings for search?
Answer: They can retrieve content with related learned meaning even when exact wording differs.
What the interviewer is testing: semantic retrieval intuition and its limits.

Practice Questions

Why can keyword search miss a paraphrased question?
What does a high cosine similarity not guarantee?
Why might chunk size affect retrieval quality?
Give a non-search use for embeddings.

Quick Quiz

Are embeddings human-readable labels? Answer: No, they are numeric vectors.
Do close vectors prove factual agreement? Answer: No.

Key Takeaway

Key Takeaways

Embeddings turn items into comparable vectors. They make semantic retrieval useful, but retrieval quality and source governance still determine whether a result is trustworthy.

Next Lesson

Next, connect tokens, attention, and next-token prediction to LLM behavior.