Enhancing Multi-Object Tracking with Segmentation Masks: A Solution for Lost Object Recovery

Tracking by detection is an effective approach to multiple object tracking: detections are extracted and matched across video frames. However, detection errors persist, leading to false negatives that degrade tracker performance. We propose an architecture to overcome detection failures by obtaining and tracking segmentation masks for each object instead of relying only on bounding boxes, which can lack precision in crowded scenes. When a detector fails and a track cannot be associated, the track is sent to a Lost Tracks Recovery (LTR) architecture. LTR combines an off-the-shelf mask generation network, a transformer-based mask selection network, and a segmentation-based tracker to preserve identities through missed detections. Results on the MOT20 crowded dataset show improved performance for state-of-the-art tracking systems.