Image annotation conventions

Image annotation looks like drawing, but the work that makes it usable is mostly rules: what counts as the object, where the edge of a box goes, what to do when objects overlap. This lesson uses the two most detailed public accounts of large-scale image annotation, the COCO dataset (COCO paper) and Google’s Open Images V4 (Open Images V4 paper), to show those rules. Individual projects set their own, so read this as an example of the method, not as any provider’s guidelines.

What gets drawn

The two projects annotate at different levels of detail. COCO labels each object instance with its own outline, which it calls per-instance segmentation (COCO paper). Open Images draws bounding boxes, and it annotated 90% of them with a technique called extreme clicking (Open Images V4 paper).

The perfect box

Open Images defines a perfect box as the smallest possible box that contains all visible parts of the object (Open Images V4 paper). The authors say that definition hides corner cases that depend on the class, such as whether straps are part of a camera or water is part of a fountain (Open Images V4 paper).

They also report cultural differences they did not expect. In some parts of the world annotators treated a human hand as including the whole arm (Open Images V4 paper).

To keep annotators consistent, the team drew a perfect box by hand on two examples of each of the 600 object classes. For 20% of classes they also collected common mistakes from pilot studies. Annotators worked on one class at a time and saw the correct examples and the common mistakes before each session (Open Images V4 paper).

Extreme clicking

The traditional way to draw a box is to click the corners of an imaginary rectangle around the object. The Open Images authors say this is hard because the corners are often outside the object itself, and several adjustments are needed to get a tight box (Open Images V4 paper).

In extreme clicking the annotator instead clicks four physical points on the object: its top, bottom, leftmost and rightmost points. The authors describe these points as natural and easy to find (Open Images V4 paper). The average time to draw a box was 7.4 seconds, against a median of 42 seconds reported for the older ILSVRC method (Open Images V4 paper).

Overlap, crowds and attributes

Some images contain too many objects to box one by one. Open Images lets an annotator draw one box around five or more heavily overlapping instances and mark it with a GroupOf attribute (Open Images V4 paper).

COCO handles the same problem differently. Up to 10 to 15 instances of a category in an image were segmented individually. After that, the remaining instances were marked as a single crowd region, and crowd regions are ignored when a detector is scored (COCO paper).

Open Images also asked annotators to mark attributes where they applied: partially occluded (hidden behind another object), truncated (extending outside the image), depiction (a cartoon, drawing or statue) and inside (the box covers the inside of an object, such as the inside of a car) (Open Images V4 paper).

To avoid duplicate boxes, a parent label was skipped when a child label was already present. If an image carried the labels animal, zebra and car, boxes were drawn for zebra and car (Open Images V4 paper).

How quality is checked

Both projects trained annotators before letting them work. Open Images trained annotators on 10 objects from each of 20 classes, with automatic feedback after each class showing where the extreme points could validly go. The qualification then required 800 boxes at an overlap (IoU) above 0.84 with the reference and at most 20 seconds per box (Open Images V4 paper).

COCO required a training task for each object category, and workers could not finish it until their outline adequately matched the reference. Roughly 1 in 3 workers passed (COCO paper).

Checking continued after the work was done. In COCO, 3 to 5 workers judged each segmentation, poor ones were discarded and the instance was sent back to be redone, and all work from workers who consistently produced poor outlines was discarded (COCO paper).

Open Images also had annotators verify boxes proposed by a model. They were told to accept a box if it overlapped an imaginary perfect box by more than 0.7 (Open Images V4 paper). In the first training round, 15% of boxes with almost no overlap were accepted. After three rounds, boxes with an overlap between 0.3 and 0.6 were accepted less than 4% of the time, and 80% of annotators were judged qualified (Open Images V4 paper).

Why presence is harder than it looks

Deciding whether an object is in an image is itself ambiguous. COCO compared crowd workers with seven expert workers on that task and found that even the experts often disagreed, because of ambiguity in the image or in the category definitions (COCO paper). To keep recall high, COCO asked 8 workers to label each image at the category stage (COCO paper).

What it costs

Segmenting COCO’s 2.5 million instances took over 22 worker hours per 1,000 segmentations (COCO paper).

Intersection over union (IoU) is the area where your box and the reference box overlap, divided by the area covered by both together. A box that matches the reference exactly scores 1. A box that misses it entirely scores 0. A box that is twice as large as the object, but contains it completely, scores 0.5.