Spatio-Temporal Event Segmentation and Localization for Wildlife
  Extended Videos
  Using offline training schemes, researchers have tackled the event
segmentation problem by providing full or weak-supervision through manually
annotated labels or self-supervised epoch-based training. Most works consider
videos that are at most 10's of minutes long. We present a self-supervised
perceptual prediction framework capable of temporal event segmentation by
building stable representations of objects over time and demonstrate it on long
videos, spanning several days. The approach is deceptively simple but quite
effective. We rely on predictions of high-level features computed by a standard
deep learning backbone. For prediction, we use an LSTM, augmented with an
attention mechanism, trained in a self-supervised manner using the prediction
error. The self-learned attention maps effectively localize and track the
event-related objects in each frame. The proposed approach does not require
labels. It requires only a single pass through the video, with no separate
training set. Given the lack of datasets of very long videos, we demonstrate
our method on video from 10 days (254 hours) of continuous wildlife monitoring
data that we had collected with required permissions. We find that the approach
is robust to various environmental conditions such as day/night conditions,
rain, sharp shadows, and windy conditions. For the task of temporally locating
events, we had an 80% recall rate at 20% false-positive rate for frame-level
segmentation. At the activity level, we had an 80% activity recall rate for one
false activity detection every 50 minutes. We will make the dataset, which is
the first of its kind, and the code available to the research community.
