Sentence-Guided Activity Recognition In Video.pdf

Sentence-Guided Activity Recognition In Video" presents a system that integrates language understanding with video processing. The system uses the compositional structure of language to guide activity recognition in multi-activity videos. Here's a concise summary:

1. Problem: Current object detection in videos is unreliable, with false positives and negatives. This hinders event recognition, which requires detecting and tracking objects and applying semantic predicates to their tracks.

2. Solution: The authors propose a 'Sentence Tracker' that takes over-generated detections, a sentence, and a lexicon as inputs. It outputs a score and a set of tracks that satisfy the sentence while optimizing a linear combination of detection scores and temporal coherence.

3. Key Features:
- Represents sentence semantics using predicates over tracks, formulated as finite-state recognizers over features extracted from detections.
- Handles nouns, adjectives, verbs, adverbs, and prepositions uniformly using this mechanism.
- Supports three tasks: focus of attention (identifying tracks for different sentences in a video), generation (finding the best sentence to describe a video), and retrieval (searching videos based on a sentential query).

4. Contributions: The paper introduces the Sentence Tracker and demonstrates its utility through these three tasks. It also extends previous work on detection-based tracking by selecting detections that collectively satisfy a complex, sentence-level predicate.