Token pooling

When a document is represented as many vectors, one per token, token pooling merges vectors that are close to each other into one, so there are fewer of them. With fewer vectors, less storage is needed. But if you merge too many, fine details can get blurred. So you have to decide carefully how much to merge.

As multi-vector embedding models became more common, the amount of storage became a problem, and token pooling drew attention as a way to cut it. If you add pooling to an existing model afterward, quality tends to drop. So research continues on training a model separately so that it handles merged vectors well.

First, a vector is made for each token in the document. Then vectors that are close together are grouped by a method such as hierarchical clustering. A representative vector is made and stored for each group. This vector is only a summary, so it cannot hold all of the original meaning.

Pooling-specific training retrains the model so that search still works well with the vectors after they are merged. Pooling is included in the training process, so the model learns a representation that loses less information.

It can be used when a document search system has large storage needs and you want to cut cost by reducing the number of vectors. It is important to measure search quality at the same time and decide how much reduction is acceptable.

You can consider it for search over company documents or long documents when the number of token vectors is large. But the right level differs for each dataset and model, so you need to test it on your own data.

The more you compress, the more detail disappears and the more likely search quality is to fall. You should not assume that the numbers from one experiment apply as they are to every model and dataset.

Token pooling does not only cut cost. The merging and retraining take time and resources, and how much faster search becomes depends on how the system is built.

It is like organizing many photos by putting similar scenes together into one picture, so the album gets lighter.

Document search services spend a lot to store and compare vectors. If you understand how token pooling works, you can judge the balance between search quality and storage cost.