Video and egocentric capture

Egocentric video is footage recorded from the wearer’s own point of view, usually from a camera on the head or glasses. It is collected to teach systems that will work from a first-person view, such as augmented reality and robotics (Ego4D paper). This lesson uses Ego4D, a very large public egocentric dataset, to show how capture and annotation are organised. Providers that pay for this work run their own projects, so read it as a worked example, not as any provider’s rules.

What the dataset contains

Ego4D offers 3,670 hours of daily-life video from 931 camera wearers in 74 locations across 9 countries (Ego4D paper). Some of the video comes with audio, 3D scans of the environment, eye gaze, stereo, or synchronized video from several wearers at the same event (Ego4D paper).

How capture was organised

The team chose a middle path between directing the wearers and giving no guidance. They recruited people whose normal day would cover many scenarios, and asked them to wear the camera for as long as the battery lasted, so that activity would unfold naturally (Ego4D paper). A typical raw clip is about 8 minutes long, much longer than the 10-second clips common in third-person video research (Ego4D paper).

Some teams recruited people whose jobs have interesting visual settings, such as bakers, carpenters, landscapers and mechanics (Ego4D paper).

Seven different head-mounted cameras were used, so that models would not overfit to one device. The choice matters: a GoPro mounted on the head and pointing down gives a clear view of the hands handling objects, while glasses that share the eyes’ vantage point miss interactions close to the body (Ego4D paper).

Narration: the annotation that comes first

Before any other annotation, every video went through narration. An annotator watched a clip of up to 5 minutes and wrote a one to three sentence summary of the overall activity. They then re-watched it, pausing repeatedly to write one sentence for each thing the camera wearer did, with timestamps (Ego4D paper).

Each narration describes a single, atomic action or object interaction, written with the wearer marked by a tag. Interactions with other people are captured with other tags (Ego4D paper).

The annotators were told to imagine describing everything that happens to a friend on the phone who cannot see the video, which is meant to draw out a play-by-play of the action (Ego4D paper).

Every clip was narrated by two different annotators independently, and the team kept both versions rather than merging them, because the narrations were not used as ground truth for any benchmark (Ego4D paper).

The result was on average 13.2 sentences per minute of video. A person watching television was narrated at 0.17 sentences a minute, and a person harvesting crops at 63.6 (Ego4D paper). Individual narrations averaged 7.4 words and the clip summaries 16.8 (Ego4D paper).

Known limits

The authors list biases in the data. Wearers were mostly in urban or college-town areas, and battery life meant the footage leans towards active parts of a day (Ego4D paper). The annotation was done by crowd workers at two sites in Africa, which the authors say means the narrations may reflect local word choices (Ego4D paper).