There is a big difference between knowing a photo contains a car and knowing exactly which pixels are the car. The first is enough to sort your holiday snaps. The second is what lets a vehicle decide whether it can steer left without hitting anything. That second, far harder task is image segmentation, and it quietly powers some of the most serious uses of computer vision today.
Labels, boxes, and masks
To see why segmentation matters, it helps to line up the family of vision tasks, because they sound similar but ask for very different things.
Classification just names what is in the picture. Two sheep. It says nothing about where they are. Object detection goes a step further and draws a rectangle around each thing it finds. That box is useful, but it is crude. A box around a sheep also contains a lot of grass.
Segmentation throws the boxes away and works at the level of pixels. It colours in the exact outline of each region, so the sheep is only the sheep, right down to the wool at its edges.
Semantic versus instance
Within segmentation there are two flavours worth knowing, and the difference is easy to feel with an example.
Semantic segmentation asks what is this pixel, and answers with a category. Every sheep pixel gets painted the same colour, because they are all sheep. It does not care that there are three separate animals. Road, sky, building, person, each becomes a coloured zone.
Instance segmentation is fussier. It still labels every pixel, but it also keeps the objects apart. Sheep one, sheep two, and sheep three each get their own colour, even though they belong to the same category. When you need to count things or track them one by one, this is the version you reach for.
How a segmentation network is shaped
Most segmentation models share a distinctive shape that looks like the letter U, and once you know why, the design feels almost inevitable.
The first half is an encoder. It behaves much like the convolutional networks used for classification, shrinking the image step by step while pulling out richer and richer features. By the middle, the picture has become small but deeply meaningful. The network now understands what is present, but it has lost track of precisely where, because all that shrinking blurred the fine detail.
So the second half does the opposite. A decoder grows the small, meaning rich map back up toward the original size, rebuilding the fine boundaries as it goes. Along the way it borrows detail from the matching encoder layers, which is how it recovers crisp edges. The output is a full resolution map where every pixel carries a label.
Where segmentation earns its keep
This is not an academic exercise. Segmentation is the difference between a demo and a tool in several fields that matter.
In medicine, it outlines a tumour on a scan so a doctor can measure its exact size and watch it over time. A bounding box would be almost useless here, because treatment depends on the precise shape.
In self driving research, the car needs to know which pixels are drivable road, which are pavement, and which are a child stepping off the kerb. Only a per pixel understanding is safe enough to act on.
It also shows up in quieter places. The portrait mode on your phone segments you from the background before it blurs everything behind you. Farmers use it from the air to tell healthy crops from struggling ones, field by field. Video tools use it to swap or erase backgrounds cleanly.
The honest catch
All of this depends on training data, and segmentation data is painful to make. Someone, somewhere, had to trace the outline of every object in thousands of images by hand so the model had something to learn from. That labour is why good segmentation felt out of reach for so long, and why tools that speed up labelling have been such a big deal.
Still, when you need to know not just what is in an image but exactly where every part of it lives, nothing else will do. Segmentation is computer vision taking the question seriously, one pixel at a time.