How Text Embedding Models Are Trained and Served

You send an embeddings API a sentence. It sends back an array like this:

[0.021, -0.008, 0.034, ..., -0.017, 0.006]

That looks almost insultingly simple. The array does not contain words, rules, or a readable summary. Yet it lets a search engine retrieve “change my login credentials” when someone types “reset password,” even though the phrases barely overlap.

The useful mental model is:

An embedding model is a learned coordinate generator. Training designs the coordinate system. Inference places new text inside it.

This article builds that model from the ground up. We will follow one sentence through tokenization, a transformer encoder, pooling, and normalization; train a tiny embedding space with positive and negative pairs; derive the contrastive loss; separate backpropagation from the optimizer update; and finally walk through a production API request.

The diagrams use small, illustrative vectors so the mechanism stays visible. Real systems use far more dimensions and data.

First, separate training from inference

Nearly every confusing explanation of embeddings mixes these two loops together.

Training has examples of what should match, computes an error, and changes model weights. It may run for days or weeks across many accelerators.

Inference receives new text, runs the already-trained model once, and returns a vector. There is no correct partner, no loss, no backpropagation, and no weight update in the request path.

Keep that split in your head for the rest of the article:

TRAINING:   text pairs → vectors → similarity scores → loss → gradients → new weights
INFERENCE:  new text   → vector  → return / store / compare

The encoder and pooling path can be shared. The surrounding loop is different.

1. What an embedding actually is

An embedding is a vector: an ordered list of numbers.

"reset my password"  →  [0.12, -0.41, 0.08, ..., 0.27]

A vector with 1,536 numbers lives in a 1,536-dimensional space. We cannot draw that space, but the geometry follows the same rules as a two-dimensional map.

The model is not normally trained so coordinate 17 literally means “password” and coordinate 94 means “urgency.” Meaning is distributed across coordinates. What matters is the relationship among complete vectors.

Dot product: multiply matching coordinates, then add

For two short vectors:

a = [a₁, a₂, a₃]
b = [b₁, b₂, b₃]
 
a · b = a₁b₁ + a₂b₂ + a₃b₃

The dot product becomes large when the vectors point in compatible directions and/or have large magnitudes.

Cosine similarity: keep the angle, remove the lengths

cos(a, b) = (a · b) / (||a||₂ ||b||₂)

Here, ||a||₂ is the Euclidean length of a. Dividing by both lengths makes cosine similarity care about direction rather than scale.

Use the lab below before moving on. Compare the puppy sentence with all three candidates and watch the dot product, lengths, and cosine change together.

If vectors are normalized to length one, the denominator is one:

||a||₂ = ||b||₂ = 1  ⇒  cos(a, b) = a · b

That is operationally useful: a vector database can rank normalized vectors with a fast dot product. OpenAI’s embeddings guide says its embeddings are normalized to length one, so cosine similarity and Euclidean distance produce identical rankings for those outputs.

The space is learned, not hand-designed

No engineer manually chooses the coordinates for “puppy,” “loan,” or “sourdough.” At initialization, the model does not have the task-specific geometry we want. Training repeatedly shows it relationships such as:

anchor:    "reset my password"
positive:  "change the password for your account"
negative:  "reset a Wi-Fi router"

The loss says, in effect: move the anchor and positive closer relative to the negatives. Backpropagation converts that request into gradients for millions or billions of parameters. After enough varied examples, useful structure emerges across the space.

Checkpoint

At this point, you should be able to explain three things:

• An embedding is a coordinate, not a compressed sentence you can read directly.
• Similarity comes from geometry over the complete vector.
• Training determines which distinctions that geometry preserves.

Now we need the machine that produces the coordinate.

2. The model: tokens in, contextual states out

A common modern text-embedding architecture is a bi-encoder built from a pretrained transformer encoder. “Bi-encoder” means the two texts are encoded independently—usually by the same weight-shared network—then compared as vectors.

That independence is why embeddings scale. You can encode a million documents once, store their vectors, and embed only the live query at request time.

A cross-encoder works differently:

bi-encoder:     encode(query) · encode(document)     → cheap comparison
cross-encoder:  score([query; document] together)    → richer, expensive pair score

A cross-encoder can inspect every query-document token interaction, but it must run again for every pair. Production search often uses a bi-encoder to retrieve a small candidate set, then a cross-encoder to rerank it.

Let us unpack each stage.

2.1 Tokenization creates IDs, not meanings

The tokenizer splits input into reusable pieces and maps each piece to an integer ID. A word may be one token, several subword tokens, or part of a larger token depending on the vocabulary.

"unbelievable" → ["un", "believ", "able"]   # conceptual only

Tokenization helps a fixed vocabulary represent new words, names, punctuation, and many languages. The IDs themselves are arbitrary addresses. A learned embedding table turns each ID into an initial token vector, then positional information tells the model where it appeared.

For a batch, shorter sequences are padded to a common length. An attention mask tells the model which positions are real and which are padding. Forgetting that mask during pooling is a subtle way to contaminate the final vector.

2.2 Self-attention makes token vectors contextual

Before the transformer, the word “bank” might have one generic lookup vector. After contextualization:

"the bank approved my loan"  → financial-bank state
"we sat on the river bank"   → river-bank state

Every transformer layer creates three learned projections from each token state:

Q = H W_Q      # what this position is looking for
K = H W_K      # what each position offers
V = H W_V      # information each position can contribute

Then attention mixes information across positions:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V

Read this from the inside out:

1. QKᵀ computes how strongly every position matches every other position.
2. dividing by √dₖ keeps large dot products from making the softmax too sharp merely because vectors are wide;
3. softmax turns each row into weights that sum to one;
4. multiplying by V forms a weighted mixture of token information.

Multi-head attention performs several such mixtures with different learned projections. A feed-forward network transforms each position. Residual connections and normalization stabilize the stack. Repeating this process lets every final token state carry context from the sentence.

This is the largest conceptual jump in the architecture: the encoder does not retrieve a finished sentence meaning from a table. It constructs contextual token states through repeated learned communication.

2.3 Pooling converts many token states into one text vector

The encoder returns one final hidden state per token:

[h₁, h₂, h₃, ..., hₜ]      each hᵢ has d numbers

An embedding API needs one vector for the entire input. A pooling function reduces the sequence axis:

Mean pooling

z = (Σᵢ maskᵢ hᵢ) / (Σᵢ maskᵢ)

Special-token pooling

z = h_[CLS]

Learned attention pooling

z = Σᵢ αᵢhᵢ,  where the model learns αᵢ

The Sentence-BERT paper compared special-token, mean, and max pooling while adapting BERT/RoBERTa into a siamese sentence encoder. The E5 paper used a shared transformer encoder with average pooling in its published recipe. Those are examples, not universal requirements.

A model must be trained and served with a compatible pooling contract. Replacing mean pooling with “just take the first token” after training is not a harmless implementation detail; it changes the representation.

2.4 Projection and normalization finish the vector

Some systems add a learned linear projection from the encoder width to the desired embedding width:

z_projected = Wz + b

Then L2 normalization produces a unit vector:

e = z_projected / ||z_projected||₂

Normalization does not create semantic quality. It standardizes the geometry used by the objective and downstream similarity function.

3. Training begins with relationships, not isolated text

A language model can begin with broad linguistic knowledge, but that does not automatically make its pooled vectors good retrieval coordinates. Sentence-BERT was motivated by exactly this gap: pairwise BERT scoring was expensive, while naive fixed vectors did not provide the desired sentence-similarity behavior.

Modern embedding work therefore constructs pairs or groups that reveal what should be near what.

These signals are not equally trustworthy. A click can reflect position bias. Adjacent paragraphs may discuss different ideas. Two answers can both be relevant even if the dataset labels only one as positive.

The E5 paper is a useful public example: its authors assembled heterogeneous weakly supervised pairs from sources including community Q&A, Common Crawl, and scientific papers, then applied aggressive consistency filtering before contrastive pretraining. They later fine-tuned on smaller labeled datasets.

The lesson is bigger than E5: data construction defines the semantics of the space. If all positives are paraphrases, the model may learn equivalence but underperform at question-answer retrieval. If every negative is from a random topic, it may never learn the fine boundary between two plausible answers.

4. Negatives are the curriculum

A positive pair says what should be close. A negative says what kind of confusion the model must resolve.

Random negatives

Random negatives are cheap and usually correct. Early in training they create an easy global separation between unrelated topics. Later, many contribute almost no gradient because the model already scores them far below the positive.

In-batch negatives

Suppose a batch contains N anchor-positive pairs. After one forward pass, each anchor can treat the other N - 1 candidates as negatives. The model gets roughly N² pair scores from 2N encoded texts.

That efficiency is why large batches appear throughout contrastive learning. The published E5 recipe explicitly uses large-batch in-batch negatives.

But there is a trap: another pair’s “negative” may actually be relevant to this anchor. That is a false negative. Aggressively pushing false negatives away damages the space.

Common mitigations include:

• deduplicating near-identical examples before batching;
• filtering negatives with labels or a stronger teacher;
• excluding candidates known to be relevant;
• using softer targets instead of forcing every unlabeled candidate to probability zero.

Hard negatives

A hard negative is wrong but scores suspiciously high. It might share keywords, entities, or structure with the anchor:

query:         reset my password
positive:      change my account login password
hard negative: reset my Wi-Fi router

A common mining loop is:

1. Train a first embedding model.
2. Use it to retrieve top candidates for every anchor.
3. Remove known positives and obvious false negatives.
4. Keep plausible wrong candidates as hard negatives.
5. Train again on the sharper dataset.

This makes training self-improving, but also dangerous: a biased first model mines its own blind spots. High-quality systems combine model mining with filters, teacher scores, labels, and data audits.

The public BGE/C-Pack paper describes a staged recipe in which task-specific fine-tuning adds a mined hard negative alongside in-batch negatives. It is another example of a broader pattern, not evidence that every proprietary service uses the same loop.

5. Contrastive loss turns “closer” into a number

We now have a batch of anchors, positives, and negatives. The model encodes all of them and builds a similarity matrix.

For normalized query embeddings Q and document embeddings D:

S = QDᵀ
 
Sᵢⱼ = similarity(query i, document j)

If row i belongs with column i, training should make the diagonal dominate its row.

From scores to probabilities

For one anchor, divide every similarity by a temperature τ, then apply softmax:

P(positive | anchor)
 
          exp(sim(anchor, positive) / τ)
= ------------------------------------------------
  Σ candidates exp(sim(anchor, candidate) / τ)

The per-example InfoNCE loss is the negative log-probability of the positive:

L = -log P(positive | anchor)

similarities:  positive 0.80, hard negative 0.55, easy negative 0.10
temperature:   0.20
logits:        4.00, 2.75, 0.50
softmax:       0.759, 0.218, 0.023
loss:          -log(0.759) = 0.275

The E5 paper uses this family of InfoNCE objective with cosine similarity scaled by a temperature. SimCSE is another influential demonstration of contrastive sentence representation learning; its unsupervised variant uses two dropout-corrupted passes of the same sentence as a positive pair, while its supervised variant uses natural-language-inference relationships.

Temperature controls how unforgiving the comparison is

A small temperature magnifies score differences before softmax. The highest-scoring candidates dominate, and near-misses receive strong attention. Too sharp can make training unstable or amplify mislabeled negatives. Too soft can produce weak separation.

Temperature is not a confidence score returned to users. It is part of the training/scoring recipe.

Symmetric loss

If the relationship works both ways, train in both directions:

L = ½ CrossEntropy(QDᵀ / τ, diagonal_labels)
  + ½ CrossEntropy(DQᵀ / τ, diagonal_labels)

For asymmetric retrieval—short queries against long passages—the two sides can use different instructions, prefixes, or even different towers. E5, for example, published a shared encoder with query: and passage: prefixes to break the symmetry for retrieval tasks.

6. Backpropagation and gradient descent are not the same step

The words are often blurred together, but the distinction is clean.

Forward pass

Current parameters θ produce embeddings, similarities, and loss:

θ → encoder outputs → pooled vectors → scores → L

Backpropagation

Backprop applies the chain rule from the loss backward through every differentiable operation. Its output is a gradient for each parameter:

∂L/∂θ

The gradient says how a tiny change to each weight would change the current loss. Backprop computes gradients; it does not update the weights. The optimizer converts those gradients—together with its algorithm, running state, learning rate, and weight decay—into the actual parameter update.

Optimizer step

An optimizer such as AdamW uses the gradients, learning rate, and running statistics to change parameters:

θ ← optimizer(θ, ∂L/∂θ, learning_rate, state)

Then the next batch starts. A production-scale loop adds mixed precision, gradient accumulation, distributed all-gather for more negatives, checkpointing, learning-rate schedules, validation, and failure recovery. The conceptual order stays the same.

A compact PyTorch-style training step

This is intentionally small, but it contains the real structure:

import torch
import torch.nn.functional as F
 
 
def mean_pool(last_hidden_state, attention_mask):
    mask = attention_mask.unsqueeze(-1).to(last_hidden_state.dtype)
    summed = (last_hidden_state * mask).sum(dim=1)
    counts = mask.sum(dim=1).clamp(min=1e-9)
    return summed / counts
 
 
def embed(encoder, batch):
    output = encoder(**batch)
    pooled = mean_pool(output.last_hidden_state, batch["attention_mask"])
    return F.normalize(pooled, p=2, dim=-1)
 
 
query_vectors = embed(encoder, query_batch)       # [N, d]
document_vectors = embed(encoder, document_batch) # [N, d]
 
# Every other document in the batch acts as a negative.
temperature = 0.05
logits = query_vectors @ document_vectors.T / temperature
labels = torch.arange(logits.size(0), device=logits.device)
 
loss_q_to_d = F.cross_entropy(logits, labels)
loss_d_to_q = F.cross_entropy(logits.T, labels)
loss = 0.5 * (loss_q_to_d + loss_d_to_q)
 
optimizer.zero_grad()
loss.backward()   # compute gradients
optimizer.step()  # apply a weight update

What this omits matters: input curation, hard negatives, false-negative masks, cross-device negatives, projection heads, teacher targets, and evaluation. But there is no mysterious missing “embedding operation.” The embedding is the normalized pooled encoder output; the contrastive objective is what makes that output useful.

7. How a modern model is assembled over multiple stages

“Train an embedding model” rarely means initialize a transformer randomly and run one loss.

A representative stack looks like this:

Stage 0: start with language understanding

Initialize from a pretrained language encoder or perform embedding-oriented pretraining on raw text. This gives the network syntax, vocabulary, and broad world patterns before retrieval-specific supervision.

Contriever showed that retrieval representations can be learned without relevance labels by creating related views of documents and applying contrastive learning. Public BGE work describes a separate pretraining stage before contrastive fine-tuning. The exact pretraining objective varies.

Stage 1: broad weakly supervised contrastive training

Train on a large, diverse mixture of naturally paired or synthetic data. The goal is coverage across domains, languages, lengths, and relationship types.

Scale helps, but weak labels are noisy. Filtering and mixture balance are first-class modeling decisions.

Stage 2: high-quality supervised multitask tuning

Use smaller datasets with explicit relevance, paraphrase, entailment, classification, or ranking judgments. Task instructions can tell one model whether “similar” currently means retrieval relevance, duplicate meaning, classification affinity, or something else.

The GTE paper presents this general multi-stage contrastive pattern; E5 also reports weakly supervised pretraining followed by labeled fine-tuning.

Stage 3: mine hard negatives and distill finer rankings

Retrieve confusing candidates with an earlier checkpoint. A stronger teacher—often a cross-encoder or larger model—can assign graded relevance scores. The embedding model then learns not only a binary positive/negative boundary, but part of the teacher’s ranking structure.

Distillation is powerful because a bi-encoder is asked to compress rich pairwise interactions into independently computed vectors. It cannot preserve everything, so the dataset and teacher decide which information survives.

Stage 4: train multiple usable widths

A wide vector costs more memory and comparison work. Naively chopping off its final coordinates is not guaranteed to preserve quality.

Matryoshka Representation Learning changes the objective: it applies predictive losses to several nested prefixes of the same representation. The early coordinates must work alone; wider prefixes add detail.

OpenAI’s public embeddings guide says its v3 models were trained with a technique that supports shortening through the dimensions parameter. It documents default widths of 1,536 for text-embedding-3-small and 3,072 for text-embedding-3-large, and links directly to the Matryoshka paper. It also recommends using the API parameter rather than manually truncating; if developers manually shorten an output, the guide tells them to renormalize it.

Do not over-interpret this as “dimension 1 is topic and dimension 2 is tone.” Nested usefulness does not make individual coordinates human-readable.

8. Evaluation: loss going down is not the finish line

Contrastive training loss only measures performance against the examples and negatives presented during training. A model can win that game and fail the product.

Evaluation should match downstream behavior:

MTEB packages many embedding tasks into one benchmark. It is useful precisely because one average hides trade-offs: retrieval, clustering, classification, and semantic textual similarity do not demand identical geometry.

A production evaluation set should include the actual query distribution, tail domains, short and long inputs, typos, multilingual cases, temporal drift, and adversarially similar negatives.

Failure modes worth testing explicitly

• Negation: “supports refunds” versus “does not support refunds.”
• Numbers and identifiers: two sentences can be semantically similar but differ in a critical amount or product code.
• Long documents: one vector can blur several topics; chunking strategy becomes part of the model system.
• Domain shift: medical or legal similarity may not match general web similarity.
• Fresh knowledge: a frozen model cannot encode distinctions it never learned reliably.
• Hubness: a few generic vectors can become nearest neighbors for too many queries.
• False-negative damage: relevant alternatives may have been pushed apart during training.

Embedding quality is always conditional: good for which relation, distribution, metric, width, and latency budget?

9. Inference: what happens when you call the API

After training, the model weights are frozen and the runtime is optimized around forward passes.

POST /v1/embeddings
{
  "model": "text-embedding-3-small",
  "input": "reset my password",
  "dimensions": 512
}

For OpenAI’s public API, a request includes an input and model name; v3 models also accept a dimensions parameter. The response contains one embedding per input plus token-usage metadata. The docs currently list an 8,192-token maximum input for the v3 embedding models.

Inside a representative service—not a disclosed OpenAI blueprint—the path looks like this:

1. Validate and tokenize. Reject oversized input, produce token IDs and masks.
2. Queue and batch. Combine compatible requests to use accelerator matrix units efficiently.
3. Run the encoder. Execute the transformer forward pass with dropout disabled.
4. Pool, project, and normalize. Apply the exact serving contract learned during training.
5. Serialize. Return float coordinates or another supported encoding format.

Dynamic batching trades latency for throughput

A GPU running one tiny sequence may be underused. Waiting briefly for more requests increases batch size and throughput, but waiting too long hurts tail latency.

A serving scheduler often groups requests by approximate sequence length. Otherwise one very long item forces heavy padding for every short item in the batch.

bad batch:   lengths [18, 22, 31, 4000] → enormous padding
better:      short requests together; long requests together

Operators tune queue delay, maximum batch tokens, sequence buckets, replicas, and admission control against real p50 and p95 latency targets. There is no universal fastest batch size. As one concrete production reference, NVIDIA Triton’s dynamic batcher exposes preferred batch sizes and a maximum queue delay rather than assuming one fixed optimum.

Cache the side that does not change

For search, documents change slowly and queries arrive live:

offline: documents → embeddings → vector index
online:  query     → embedding  → nearest-neighbor search

This is the core economic advantage of a bi-encoder. The expensive document encoder pass is paid once per document, not once per query-document pair.

At small scale, exact search computes a dot product against every stored vector. At large scale, approximate nearest-neighbor structures such as HNSW or the inverted-file indexes implemented in Faiss trade some recall for dramatically lower search cost.

A strong retrieval system commonly has more than one stage:

query
  ↓
embedding retrieval        fetch 100–1000 candidates cheaply
  ↓
metadata / lexical filters preserve exact constraints
  ↓
cross-encoder reranking    inspect richer query-document interactions
  ↓
top results

Embeddings do not replace exact search, metadata filters, or rerankers. They provide a high-recall semantic candidate generator.

10. The full lifecycle in one pass

We can now describe the entire system without hand-waving.

Creation

1. Start from a transformer with useful language representations.
2. Construct diverse positive pairs that encode the relationships the product needs.
3. Add random, in-batch, and mined hard negatives while filtering false negatives.
4. Encode each text independently, pool token states, optionally project, and normalize.
5. Build a similarity matrix and optimize a contrastive or ranking objective.
6. Backpropagate the loss, then let the optimizer update weights.
7. Repeat across broad weak supervision and sharper labeled or teacher-scored stages.
8. Optionally optimize nested vector prefixes for multiple serving widths.
9. Evaluate on downstream tasks, failures, domains, and operational constraints.

Inference

1. Receive one text or a batch.
2. Tokenize and mask it.
3. Run one forward pass through frozen weights.
4. Pool and normalize according to the learned contract.
5. Return a fixed-length vector.
6. Compare, store, cluster, classify, or retrieve with it.

The array returned by an API is the last visible artifact of a much larger design process. The most important work happened earlier: choosing what should count as related, constructing the comparisons, and forcing one shared geometry to preserve those decisions.

A practical mental model to keep

When you see an embedding vector, do not imagine a bag of interpretable features. Imagine a location produced by a learned map.

When you see cosine similarity, do not imagine that the model “understood” two sentences in every possible sense. It learned to place them in compatible directions under a particular training curriculum.

When you see an embeddings API call, do not imagine online learning. It is a frozen coordinate generator executing one forward pass.

And when retrieval fails, do not look only at the nearest-neighbor index. The error may originate in the tokenizer, truncation, pooling contract, training pairs, negative mining, task mixture, vector width, domain shift, or the fact that one vector was never enough for that document.

That is the deeper picture: an embedding model is not merely a transformer that emits numbers. It is a geometry engineered through data, objectives, optimization, and serving constraints.

Sources and further reading

• Reimers & Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, 2019.
• Gao, Yao & Chen, SimCSE: Simple Contrastive Learning of Sentence Embeddings, 2021.
• Izacard et al., Unsupervised Dense Information Retrieval with Contrastive Learning, 2021/2022.
• Wang et al., Text Embeddings by Weakly-Supervised Contrastive Pre-training, 2022/2024.
• Li et al., Towards General Text Embeddings with Multi-stage Contrastive Learning, 2023.
• Xiao et al., C-Pack: Packed Resources for General Chinese Embeddings, 2023/2024.
• Kusupati et al., Matryoshka Representation Learning, 2022/2024.
• Muennighoff et al., MTEB: Massive Text Embedding Benchmark, 2022/2023.
• Malkov & Yashunin, Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs, 2016/2020.
• Johnson, Douze & Jégou, Billion-scale similarity search with GPUs, 2017.
• NVIDIA, Triton Inference Server: Models and Schedulers.
• OpenAI, Vector embeddings guide and Embeddings API reference.