For years, if you wanted a computer to understand a picture, you reached for a convolutional network. It was the obvious choice, almost the only choice. Then a group of researchers asked a slightly cheeky question. The transformer had just conquered language by reading sentences as sequences of words. What if we treated an image the same way, as a sequence of little pieces? That question became the vision transformer, and it shook up computer vision more than anyone expected.
A quick reminder of what a transformer does
The transformer earned its fame in language models. Its core trick is attention, a mechanism that lets every word in a sentence look at every other word and decide which ones matter for its meaning. In the sentence "the trophy did not fit in the case because it was too big," attention is what helps the model figure out that "it" refers to the trophy, not the case.
That ability to weigh relationships across an entire input, no matter how far apart the pieces sit, is powerful. The insight behind vision transformers was that images have long range relationships too. The sky in the top corner and the reflection in a puddle at the bottom are related, even though they are far apart in the frame.
Turning a picture into a sentence
A transformer reads a sequence, but an image is a grid. So the first job is translation. The vision transformer slices the image into a set of small square patches, maybe sixteen pixels by sixteen pixels each, and lines them up in order like words in a sentence.
Each patch is flattened into a list of numbers and passed through a small layer that turns it into a token, the same kind of unit a language model uses for a word. To keep track of where each patch came from, the model also adds a position tag, since a shuffled image means nothing. Now the picture is a sentence of patch tokens, and the familiar transformer machinery can take over.
Why attention changes the view
Here is where vision transformers feel genuinely different from convolutional networks. A CNN builds understanding locally. Its little sliding window only ever sees a small neighbourhood at a time, and it takes many stacked layers before information from opposite corners of the image can meet.
A transformer skips that patience. From the very first attention step, every patch can look at every other patch directly. A patch showing a wheel can immediately relate itself to a patch showing a car door on the far side of the image. That global view, available right away, is the vision transformer's signature strength.
The trade you make
Nothing comes for free. Convolutional networks carry a helpful built in assumption, which is that nearby pixels belong together and that a pattern is a pattern wherever it appears. That assumption is baked into their design, so they learn efficiently even from smaller datasets.
Vision transformers throw that assumption away and start almost blank. Attention could connect any patch to any other, which is flexible but also means the model has more to figure out on its own. In practice that means vision transformers are hungry. Give them a modest dataset and a CNN often wins. Give them an enormous one, and the transformer's flexibility starts to pay off, matching or beating convolutional networks on the hardest benchmarks.
There is also a cost to that all sees all design. Since every patch attends to every other, the work grows quickly as the number of patches rises, which makes high resolution images expensive. A lot of recent research is about taming that cost with cleverer, more local forms of attention.
Where things stand now
The interesting outcome is not that one side won. It is that the line between them has blurred. Modern vision systems borrow from both, using convolution to capture local detail cheaply and attention to tie the whole scene together. The best image models today are often hybrids, and the same architecture increasingly handles images, text, and audio side by side, which is a big part of why multimodal AI has moved so fast.
If the convolutional network taught machines to see by scanning for local patterns, the vision transformer taught them to see the whole board at once. Having both in the toolbox is what makes today's computer vision feel less like pattern matching and more like understanding.