Multi-vector retrieval

Multi-vector retrieval does not sum up a whole document with one vector. It represents the document with many vectors, one made for each token, and compares them with the query in fine detail. It can check which part of the document matches each word of the query, so precision is high.

The price of that precision is that each document produces many vectors, so storage and computing grow. For this reason it is often used together with techniques that cut the number of vectors, such as token pooling.

The query and the document are each turned into several token vectors. For each vector in the query, the method finds the document vector that matches it best, and then adds up those scores. The document vectors are computed ahead of time and stored.

Because the document vectors are stored in advance, the whole document does not have to be computed again at search time. In exchange, the capacity of the vector store and the comparison work get bigger, so the design matters.

It can be used in search services where documents are long and an exact answer must be found. It fits especially well when you need to compare the fine details of the query and the document.

Using it together with techniques that reduce the number of vectors can ease the storage burden. Before you adopt it, it is a good idea to measure storage space, response speed and search quality together.

You should not assume it is always better than summing up with one vector. If the dataset is small or fast responses matter more, a simpler method may be better.

As the number of documents grows, storage and computing costs rise quickly. If you adopt it by looking only at accuracy, you may run into trouble with running costs.

It is like making an index card for each scene in a long book, and then picking and reading the cards that match your question.

A service that wants better search quality needs to know the limits of summing up with a single vector. You also need to understand why multi-vector retrieval raises storage cost, so that you can choose a suitable design.