Lost and Found: Overcoming Detector Failures in Online Multi-Object Tracking

Multi-object tracking (MOT) estimates the positions and identities of multiple objects over time. Tracking-by-detection first detects objects and then links detections, resulting in a simple and effective paradigm, but contemporary detectors may miss objects in some frames and cause trackers to stop tracking prematurely. We propose BUSCA, a framework compatible with any online tracking-by-detection system that persistently tracks objects missed by the detector, primarily due to occlusions. BUSCA generates proposals based on neighboring tracks, motion, and learned tokens, then uses a decision Transformer that integrates visual and spatiotemporal information to address object-proposal association as a multi-choice question-answering task. BUSCA is trained independently of the underlying tracker, solely on synthetic data and without fine-tuning, yielding consistent improvements across multiple trackers and benchmarks.