Mixture of experts (MoE)
A model design that splits the network into many specialised sub-models, or "experts", and uses only the few most relevant ones for each input. This lets a model be very large in total while staying relatively cheap to run.
In a standard model, every part of the network is used to process every input, which is thorough but expensive: the bigger the model, the more computation each answer costs. A mixture-of-experts model breaks this link. It contains many smaller expert sub-networks, and for any given input a lightweight “router” picks just a couple of them to do the work. The rest sit idle.
The clever part is that the model can hold an enormous number of parameters in total — spread across all its experts — while only a fraction are active for any one query. That gives much of the capability of a giant model at a fraction of the running cost, because you are never paying to run the whole thing at once.
It is a mostly behind-the-scenes idea, but a consequential one. Several of the most capable recent models use a mixture-of-experts design, and the term is worth recognising as one of the main tricks for scaling AI up without scaling the bills up just as fast.