Vector Databases and Embeddings Explained

What is an embedding, really?

An embedding is a fixed-length array of floating-point numbers that a trained model produces from your input. The model has learned, from huge amounts of text, to place inputs with similar meaning near each other in a geometric space. "Cancel my subscription" and "how do I end my plan" produce vectors that point in nearly the same direction, even though they share almost no words.

The length of the vector is its dimensionality. Common text models output 384 dimensions (small and fast), 768 (a frequent middle ground), or 1,536 and higher (more nuance, more storage and compute). Higher dimensions can capture finer distinctions, but they cost more to store and search, and past a point they add little. Pick based on your accuracy needs and your budget, not on the assumption that bigger is always better.

Embeddings are not limited to text. The same idea works for images, audio, code, and product records. A multimodal model can place a photo of a red sneaker and the phrase "red running shoe" close together. What matters is consistency: you must embed your stored data and your queries with the same model and version. Mix two models and the geometry no longer lines up, so your similarity scores become noise. Re-embedding everything after a model change is one of the most common, and most painful, migrations teams forget to plan for.

How does similarity search with cosine similarity work?

Once your data is a set of vectors, finding relevant results means finding the vectors closest to your query vector. The most common distance metric for text embeddings is cosine similarity, which measures the angle between two vectors rather than the straight-line distance between them. Two vectors that point the same way score close to 1; unrelated ones score near 0; opposite meanings approach -1.

Cosine similarity is popular because it ignores magnitude and focuses on direction, which suits text where the "strength" of a vector is less meaningful than its orientation. Other metrics exist: dot product is faster and is equivalent to cosine when vectors are normalized to unit length, and Euclidean distance (L2) is sometimes used for image or spatial data. The key rule is to match the metric your embedding model was trained or recommended for. Many text models assume cosine or normalized dot product, and using the wrong one quietly degrades quality.

Why you need an ANN index: HNSW and IVF

Comparing a query against every stored vector (an exact, or "brute force," search) is accurate but slow once you have millions of records. Approximate nearest neighbor (ANN) indexes trade a tiny amount of recall for a huge speedup. They organize vectors so the search only inspects a small, promising slice of the data. The two indexes you will meet most often are HNSW and IVF.

HNSW (Hierarchical Navigable Small World) builds a layered graph where each vector links to nearby neighbors. A search starts at the top, sparse layer and walks downhill toward denser ones, hopping closer to the answer at each step. As of 2026 it is the default for most production workloads: it gives an excellent speed-to-recall tradeoff, handles high dimensions well, needs no training step, and supports inserts on the fly. The cost is higher memory use and slower index builds.

IVF (Inverted File index, often IVFFlat in Postgres) takes a different route. It clusters vectors into a set number of lists, then at query time it only searches the lists nearest the query. It builds faster, uses less memory, and is simpler to insert into, which is why it earns its place on very large, mostly static datasets where memory or build time dominates the cost model. The downside is a weaker speed-recall tradeoff and a training step that assumes your data distribution is stable. Two knobs matter: how many lists you create, and how many you probe per query. More probes means better recall but slower queries.

What does a vector database actually do?

A vector database stores embeddings, indexes them for fast similarity search, and runs that search alongside ordinary filters and metadata. That last part is what separates a real database from a raw index. In production you almost never want "the most similar chunk anywhere." You want "the most similar chunk in this customer's documents, from the last 90 days, marked public." The vector store has to combine the semantic search with structured constraints.

Most vector stores give you a few core capabilities. Knowing them helps you evaluate options without getting lost in marketing pages.

• Storage and indexing: it persists vectors and builds an ANN index (usually HNSW) so queries stay fast as data grows, instead of degrading into a full scan.
• Metadata filtering: it stores fields next to each vector (tenant ID, document type, date, ACL tags) and filters on them, because semantic relevance without access control is a security incident waiting to happen.
• CRUD and freshness: it lets you insert, update, and delete vectors as your source data changes, so stale or deleted content does not keep surfacing in results.
• Hybrid search: many stores blend vector similarity with keyword (BM25-style) search, which catches exact terms like SKUs, error codes, and names that pure semantic search can miss.
• Scaling and operations: it handles sharding, replication, and backups so search keeps working as you move from one million to one hundred million vectors.

Chunking and metadata: where retrieval quality is won or lost

Before anything gets embedded, long documents get split into smaller passages called chunks. This matters more than most teams expect. Embed a whole 40-page PDF as one vector and you get a blurry average of every topic in it, useless for pinpoint retrieval. Embed each sentence in isolation and you lose the surrounding context that made it meaningful. The sweet spot is usually a few hundred tokens per chunk with a small overlap between neighbors so a thought that straddles a boundary is not cut in half.

Good chunking respects structure. Splitting on headings, paragraphs, or list items keeps related ideas together far better than slicing every N characters. For a support knowledge base, one article section per chunk often works well. For code, function or class boundaries make better seams than arbitrary line counts.

Metadata is the other half of the equation, and it is what makes a vector store safe for multi-tenant and enterprise use. Every chunk should carry fields like source URL, document ID, owner, tenant, timestamp, and permission tags. At query time you filter on those before or during the similarity search. This is how an AI agent answering a question for one customer never retrieves another customer's data, even if the two documents are semantically close. When OnPilot agents retrieve context to take an action, that retrieval runs inside least-privilege RBAC and leaves an audit trail, so a search can never quietly cross a permission boundary.

How embeddings power RAG and agent memory

Retrieval-augmented generation is the headline use case. A language model only knows what was in its training data plus what you put in the prompt. RAG fills that gap: embed the user's question, retrieve the most relevant chunks from your own content, and hand them to the model as grounding. The model answers from real, current, company-specific material instead of guessing. This is how a support agent cites your actual return policy rather than a plausible-sounding invention.

Agent memory uses the same machinery for a different goal: continuity. When an agent finishes a task or a conversation, the useful facts can be embedded and stored, then recalled later by similarity when a related topic comes up. "What did we decide about the Acme renewal?" pulls back the relevant earlier exchange even across sessions. The vector store becomes the agent's long-term recall layer, while metadata and access controls keep one user's memory from leaking into another's.

A quick scenario ties it together. A support rep asks an AI agent, "Has this customer hit their API rate limit before, and what did we tell them?" The agent embeds that question, runs a similarity search scoped by metadata to that customer's history, retrieves the two relevant past tickets, and drafts a reply grounded in what was actually said. With approvals turned on, the rep reviews and confirms before anything is sent. Embeddings did the recall, metadata enforced the boundary, and a human-in-the-loop step kept control.

How fast is approximate search versus brute force?

The whole reason ANN indexes exist is latency at scale. The numbers below are illustrative and directional, meant to show the shape of the tradeoff rather than promise specific timings on your hardware. The pattern holds in practice: exact search stays accurate but slows down roughly in proportion to data size, while a graph index keeps query latency low by inspecting only a fraction of the vectors, at the cost of a few percent of recall you can tune back.

The lesson is not "HNSW always wins." It is that you should never run brute force in production past a few hundred thousand vectors, and that the recall you give up to ANN is a dial you control. If you need higher recall, search more of the graph or probe more lists and accept slightly higher latency. If you need lower latency, do the reverse. Measure recall against an exact baseline on a sample so you know exactly what you are trading.

When is pgvector enough, and when do you need a dedicated vector database?

Reach for plain Postgres with the pgvector extension first. It puts vector columns and ANN indexes (both HNSW and IVFFlat) right inside the database you probably already run, which means one system to operate, transactional consistency between your vectors and your relational rows, and no separate sync job to keep them aligned. For most teams in 2026, pgvector comfortably handles RAG and agent-memory workloads into the tens of millions of vectors with moderate query volume.

Move to a dedicated vector database (Qdrant, Milvus, Pinecone, Weaviate, and similar) only when you can name the specific bottleneck forcing the move. The usual triggers are real scale and real latency demands, not hype.

• Use pgvector when vectors are a feature inside a larger app, your data fits in the tens of millions, and you value atomic updates between relational and vector data over raw search throughput.
• Use pgvector when your team already runs Postgres and you would rather tune one system than operate two, since fewer systems means fewer sync failures and fewer ways for the two stores to disagree.
• Move to a dedicated vector DB when you push past roughly 50 to 100 million vectors and HNSW rebuild times in Postgres become a real operational drag.
• Move to a dedicated vector DB when vector search is your primary workload and you need single-digit-millisecond p99 latency at high query volume.
• Move to a dedicated vector DB when you need heavy filtered ANN at scale or built-in features like advanced hybrid search, quantization, and horizontal sharding that purpose-built engines ship out of the box.

Common mistakes that quietly wreck retrieval

Most retrieval problems are not exotic. They come from a handful of avoidable errors that show up as "the AI gives wrong or irrelevant answers" long before anyone suspects the vector layer.

• Mismatched embedding models: embedding stored data with one model and queries with another (or after a silent version bump) breaks the geometry, so results look random until you re-embed everything with one consistent model.
• Chunks that are too big or too small: oversized chunks blur meaning and waste the context window, while tiny chunks strip away the context that made a passage useful, both tanking answer quality.
• Skipping metadata and filters: without tenant, permission, and date fields, similarity search will happily return another customer's data or stale content, which is a quality bug and a security problem at once.
• Treating recall as free: ANN trades a little accuracy for speed, and if you never measure recall against an exact baseline you will not notice the day a parameter change drops it from 98 to 85 percent.
• Pure vector search with no keyword fallback: semantic search misses exact tokens like order numbers, SKUs, and error codes, so add hybrid (keyword plus vector) search when precise terms matter.
• Forgetting freshness: if you never update or delete vectors when source documents change, the system keeps confidently retrieving content you already removed.

A practical decision framework

Start with the simplest thing that could work and only add complexity when a real constraint forces it. The order below keeps you from over-engineering on day one and from getting stuck on day two hundred.

First, pick one embedding model and write its name and version down, because that choice touches everything downstream and changing it later means re-embedding your whole corpus. Second, chunk on natural structure (headings, sections, functions) at a few hundred tokens with small overlap, and attach rich metadata to every chunk from the start, including tenant and permission tags. Third, store in pgvector with an HNSW index unless you already know you are past tens of millions of vectors. Fourth, measure recall and latency on a real sample, tune the index knobs, and add hybrid search if exact terms matter for your domain.

Only after all of that, if you hit a named wall (latency at scale, rebuild times, filtered search at one hundred million vectors), graduate to a dedicated vector database. Most teams never need to, and the ones that do will know exactly why. The goal is meaning-aware retrieval that is fast, scoped to the right data, and cheap to operate, not the most impressive infrastructure diagram.