Embeddings

An embedding is a numerical representation of a piece of text as a list of numbers, built by a model so that texts with similar meaning get vectors that are close to each other.

Timothé Merle

TL;DR

Key Definition

Embeddings convert words, sentences, or passages into vectors with hundreds or thousands of dimensions. The closer two vectors are, the closer the two texts are in meaning. An AI search engine compares the embedding of a question with those of millions of passages, keeps the closest, and builds its answer on them. A passage left out at this step can't be cited.

Importance

Why Does It Matter?

  • In a RAG-style system, embeddings largely decide which passages get retrieved, and so which sources stand a chance of being cited.
  • Two texts with not a single word in common can have close vectors. A page can then answer questions phrased nothing like its title.
  • A passage that mixes several topics yields a blurry vector. It fits each specific question less well than a passage focused on one topic.
  • Clustering content, spotting duplicates, recommending related pages. Embeddings handle these very practical website tasks too.
  • Once you understand embeddings, you can write for AI engines without sliding into keyword stuffing, which doesn't bring a text any closer to the meaning of a question.

How Does It Work?

An embedding model reads a text and outputs a fixed-size vector. Those vectors are then stored, indexed, and compared to find the texts closest to a query.

  1. 1

    From text to vector

    Trained on large volumes of text, the model has learned to place content with similar meaning in the same region of the space. Early well-known work such as word2vec in 2013 encoded single words. It showed that vector(King) minus vector(Man) plus vector(Woman) lands near Queen. Recent models encode whole sentences and passages. OpenAI, for example, states that text-embedding-3-small outputs 1536-dimension vectors by default, and text-embedding-3-large outputs 3072.

  2. 2

    Measuring closeness

    Closeness between two embeddings is most often measured with cosine similarity, which looks at the angle between the two vectors. A score near 1 signals texts that are very close in meaning. The OpenAI documentation puts it this way, a small distance suggests high relatedness and a large distance suggests low relatedness.

  3. 3

    Chunking and vector index

    A whole page makes a poor vector, because it often covers several topics. Systems split it into passages, from a few sentences to a few paragraphs, and compute one embedding per passage. Those vectors go into a vector index that can find the nearest neighbors of a query among millions of entries in milliseconds.

  4. 4

    The role in AI search

    In a RAG architecture, the user's question becomes an embedding that is compared against the index. The closest passages go to the language model, which writes the answer. In 2020, the Dense Passage Retrieval work showed how well this dense retrieval handles open-domain questions. Commercial engines don't publish their exact models and often combine this score with other signals.

Metrics

How Do You Measure It?

  • With an embeddings API, such as OpenAI's, compute the cosine similarity between your important passages and a list of real customer questions.
  • Find the questions whose closest passage on your site scores low. Those are your coverage gaps.
  • Compare a passage's scores before and after a rewrite, to check that it really moves closer to the target questions.
  • The embeddings of your pages surface near-identical content fighting over the same intent.
  • Then check whether the reworked passages show up among the sources AI assistants cite, for example by tracking a prompt set with Hikoo's Reporting agent.

Pitfalls

What Are the Common Mistakes?

  • Long paragraphs that cover several topics dilute the vector of each passage, which then drifts away from every specific question.
  • Starting a paragraph with "it" or "this" without naming the subject gives you a passage that, once split out, no longer says what it's about.
  • Repeating a keyword does not bring a text closer to a query. The embedding encodes overall meaning, not word frequency.
  • Comparing similarity scores from different models makes no sense, since each model has its own vector space.
  • A good embedding score doesn't guarantee a citation. Engines weigh retrieval against other quality and trust signals.

What Should You Do First?

Follow these steps to get started.

  • Break your pages into short sections, each centered on a single question.
  • Start each section with a sentence that answers directly and names the subject.
  • Make sure each paragraph reads fine without the rest of the page.
  • Use your customers' vocabulary, on top of your industry vocabulary.
  • Test your important passages against real questions with an embedding model.
  • Merge the near-identical pages that similarity brings to light.
  • Track whether reworked passages get cited by AI assistants.

Examples

Same meaning, different words

The question "how do I cancel my subscription" and the passage "To end your plan, go to Settings then Billing" share almost no words. Their embeddings are close anyway, because both are about ending a subscription. An exact-word search would have missed it.

Coverage audit on 40 questions

An illustrative scenario. A marketing team gathers 40 questions its prospects ask and computes their similarity with every passage on its site. For about ten of them, no passage scores high. The team writes dedicated sections for those questions, then watches whether they appear among the sources AI assistants cite.

Frequently Asked Questions

  • What is an embedding in simple terms?

    An embedding is a list of numbers that sums up the meaning of a text. The model assigns it so that two texts saying the same thing get similar lists. Texts can then be compared by calculation, for example to find the passage that best answers a question, even when it doesn't use the same words.

  • Why do embeddings matter for AI visibility?

    AI search engines often use them in the retrieval step, the one that picks which passages go to the model before it writes. If your passage isn't among the closest to the question, nobody reads it and it can't be cited. Clear, focused, self-contained passages raise your odds of being retrieved.

  • What is the difference between embeddings and semantic search?

    Embeddings are the tool, semantic search is the use. Semantic search looks for content by meaning, and the most common way to get there is to compare the embeddings of the query and the documents. Embeddings have other uses too, such as clustering content or detecting duplicates.

  • Can I see the embeddings ChatGPT or Perplexity compute for my page?

    No. AI search engines publish neither their exact retrieval models nor the vectors they compute. Nothing stops you, though, from using a public embedding model, such as those in the OpenAI API, to estimate how close your passages are to real questions. The result is an approximation, but a handy one for spotting passages that are too vague.

  • Do embeddings replace keywords?

    Not entirely. Embeddings capture meaning well but can miss proper nouns, product references, or precise codes. Many systems therefore pair a semantic score from embeddings with a lexical score from exact words. Keeping your brand and product names in the text still pays off.

Sources

  1. 1OpenAI API documentation Vector embeddings
  2. 2Google for Developers Embeddings, Machine Learning Crash Course
  3. 3arXiv (Mikolov et al.) Efficient Estimation of Word Representations in Vector Space
  4. 4arXiv (Karpukhin et al.) Dense Passage Retrieval for Open-Domain Question Answering

About the author

Timothé MerleTimothé MerleCo-founder of Hikoo, AEO and GEO expert
View LinkedIn profile

Related Content

Test your site's AI visibility in 30 seconds.

GEO score, technical health, and an overview of your visibility on ChatGPT and Gemini.

Run my free auditStart for free