Mixture of experts (MoE)

Mixture of experts is a neural network design. It puts many expert modules inside one model, and for each input it picks only a few of them to do the computing. The model has a large total number of parameters, but only a small part is used in a single run of inference. That means it needs less computing than a design that uses every parameter every time.

This design took hold as people tried to make large language models bigger while holding down the cost of computing. As a model grows, the computing power needed for training and inference rises sharply. Activating only some experts drew attention because it lets you adjust total capacity and the actual amount of computing separately.

When an input arrives, a small decision module called a router gives each token a score for each expert. Only the few experts with the highest scores run, and the rest are left out of the computation. The outputs of the chosen experts are combined according to their weights and passed on to the next layer.

During training, the experts are tuned to handle different kinds of input, and a mechanism is used to spread the load so that work does not pile up on one expert. So at inference time only the active part is computed, but the full set of weights still has to be held in memory. That is a defining feature of this design.

Many large open-weight models use a mixture of experts design to lower the cost of running them after release. A company should first check whether it can run the model on its own servers with computing power equal to the active parameters.

However, the full set of weights has to be loaded into memory, so you often need several high-end GPUs or a large amount of memory. When you handle long context or many requests at once, it is a good idea to test the hardware requirements and the response speed together.

A large total number of parameters does not mean high performance on every task. If the choice of experts does not match the input, quality can drop, and the design needs a way to prevent work from piling up on particular experts.

A common misunderstanding is that few active parameters mean the model will run lightly anywhere. In practice, the full weights must be stored, so storage and memory demands are heavy. The routing step and the cost of communication between parts can also keep it from being as fast as you hoped.

It is like a school with specialist teachers for many subjects. When a question comes in, only the two or three teachers who know that subject step forward to answer. The school has many staff, but few of them move at any one time.

To read announcements about model size correctly, you need to know why the total number of parameters in an article differs from the number activated at one time. If you judge performance or cost from total capacity alone, it is easy to get the wrong idea.