Image Classification vs Object Detection vs Segmentation

The short answer

Picture a photo of a store shelf with three bottles of water and a carton of juice.

• Classification says: "The image shows drinks" or "Shelf is full / shelf is empty". One label for the whole image.
• Detection draws bounding boxes around every bottle and carton and labels them: "water", "water", "water", "juice". Now you know how many objects there are and where.
• Segmentation colors every pixel that belongs to an object, following its exact outline, including the bottle neck and label.

Classification

The model takes an image and returns a class with a confidence score: "defect — 0.93". It can be multi-class (one label from a list) or multi-label (several labels at once, such as "scratch" and "chip").

Labeling: each image gets a label. It is fast; often sorting files into folders is enough.

Limitation: the model does not know where the defect is or how many objects are in the frame. If a photo contains several parts, classification is a poor fit.

Object detection

The model finds each object, draws a box around it and names its class. This lets you count objects, track them in video and know where they are.

Labeling: an annotator draws a rectangle around each object and picks a class. Consistent rules matter: how to handle partly hidden objects, how to box objects cut off by the frame edge, the minimum size worth labeling.

Limitation: a box describes shape roughly. For tilted, elongated or overlapping objects, a lot of background ends up inside the box.

Segmentation

There are three flavors:

• Semantic — every pixel gets a class ("road", "person", "background"), but separate objects of the same class are not distinguished.
• Instance segmentation — a separate mask for each object: "bottle 1", "bottle 2".
• Panoptic — a combination of both for the whole scene.

Labeling: tracing outlines with polygons or painting with a brush. This is the most labor-intensive type. Semi-automatic tools help: a model proposes a mask from a click and a human corrects it.

When you need it: when area, shape or exact boundary matter — the size of a corrosion spot, a tumor outline, the planted area in a field image.

Which one fits typical cases

Manufacturing quality control.

• One part per frame, the answer is "pass / fail" — classification.
• You need to show where the defect is or count defects — detection.
• The area or length of a defect decides tolerance — segmentation.

Retail shelves.

• Checking whether a shelf is generally stocked — classification.
• Counting products, finding gaps, checking planogram compliance — detection, the most common choice.
• Precise share-of-shelf calculation — sometimes segmentation, but boxes are usually enough.

Medical images.

• Sorting scans into "normal / needs attention" — classification.
• Finding and counting lesions — detection.
• Measuring sizes and volumes, planning procedures — segmentation.

In medicine, any model supports a doctor's decision rather than replacing it, and labeling must be done or reviewed by qualified specialists.

Common mistakes

• Starting with the hardest task. Segmentation "just in case" multiplies labeling cost when the business only needed counts.
• Vague labeling rules. Two annotators outline the same object differently, and the model learns noise.
• Data from the wrong conditions. The model is trained on perfect photos, but the factory has different lighting and camera angles.
• No held-out test set. Without one you cannot compare options fairly.