Transformer
A transformer is the neural-network design that powers almost every modern language model, including the systems behind popular chatbots. Introduced by Google researchers in a 2017 paper titled "Attention Is All You Need", its key trick is called attention: the ability to weigh up how much each word in a sentence relates to every other word. This lets the model understand context and long-range meaning far better than earlier designs could.
To see why attention is such a big deal, consider the sentence “The trophy would not fit in the suitcase because it was too big.” What does “it” refer to, the trophy or the suitcase? A human knows instantly. Older AI systems read text one word at a time and tended to lose track of these connections over a long sentence. The transformer’s attention mechanism lets the model look at all the words at once and decide, for every word, which other words matter most to its meaning. In this case it learns to link “it” to “trophy”.
Before transformers, language models processed words strictly in order, like reading through a straw. That made them slow to train and forgetful over long passages. Because a transformer can consider a whole passage in parallel, it trains far more efficiently on modern hardware and copes with much longer stretches of text. That efficiency is exactly what made it practical to train the enormous models we see today.
The transformer is arguably the single most important idea in recent AI. Nearly every large language model, and a growing number of image and audio systems, is built on it. When you read that a company has released a new model, the odds are overwhelming that a transformer sits at its core. Understanding this one architecture goes a long way towards understanding the current wave of AI.