Publications

Portrait segmentation, isolating a picture’s subject from the background, has become a key step for applications such as video conferencing, augmented reality, and mobile imaging. Deploying these capabilities directly on low-power devices, including smartphones, action cameras, and even within image sensors themselves, eliminates reliance on external processors and reduces system complexity and power consumption. We present SINergy, a scalable and hardware-aware portrait segmentation network specifically designed for embedded deployment. SINergy builds upon efficient backbones such as XiNet and PhiNet, optimizing arithmetic intensity, memory access patterns, and operator compatibility rather than conventional FLOP-centric metrics. We systematically evaluate SINergy across heterogeneous platforms, ranging from consumer microcontrollers to small accelerators, single-board computers, and GPU-equipped edge devices. Experimental results show that SINergy achieves 2x to 5x speedup over existing architectures while preserving accuracy, and delivers the first real-time implementation of portrait segmentation on microcontrollers.

This keynote paper summarizes BUSCA, a work published in the European Conference on Computer Vision 2024. Multi-object tracking (MOT) endeavors to precisely estimate the positions and identities of multiple objects over time. The prevailing approach, tracking-by-detection (TbD), first detects objects and then links detections, resulting in a simple yet effective method. However, contemporary detectors may occasionally miss some objects in certain frames, causing trackers to cease tracking prematurely. To tackle this issue, we propose BUSCA, a versatile framework compatible with any online TbD system, enhancing its ability to persistently track those objects missed by the detector, primarily due to occlusions. Remarkably, this is accomplished without modifying past tracking results or accessing future frames, i.e., in a fully online manner. BUSCA generates proposals based on neighboring tracks, motion, and learned tokens. Utilizing a decision Transformer that integrates multimodal visual and spatiotemporal information, it addresses the object-proposal association as a multi-choice question-answering task. BUSCA is trained independently of the underlying tracker, solely on synthetic data, without requiring fine-tuning. Through BUSCA, we showcase consistent performance enhancements across five different trackers and establish a new state-of-the-art baseline across three different benchmarks. Code available at: https://github.com/lorenzovaquero/BUSCA.

This keynote paper summarizes RAXO, a work published in the IEEE/CVF International Conference on Computer Vision 2025. Open-vocabulary object detection (OvOD) is set to revolutionize security screening by enabling systems to recognize any item in X-ray scans. However, developing effective OvOD models for X-ray imaging presents unique challenges due to data scarcity and the modality gap that prevents direct adoption of RGB-based solutions. To overcome these limitations, we propose RAXO, a training-free framework that repurposes off-the-shelf RGB OvOD detectors for robust X-ray detection. RAXO builds high-quality X-ray class descriptors using a dual-source retrieval strategy. It gathers relevant RGB images from the web and enriches them via a novel X-ray material transfer mechanism, eliminating the need for labeled databases. These visual descriptors replace text-based classification in OvOD, leveraging intra-modal feature distances for robust detection. Extensive experiments demonstrate that RAXO consistently improves OvOD performance, providing an average mAP increase of up to 17.0 points over base detectors. To further support research in this emerging field, we also introduce DET-COMPASS, a new benchmark featuring bounding box annotations for over 300 object categories, enabling large-scale evaluation of OvOD in X-ray.

Human-Object Interaction (HOI) detection is a longstanding computer vision problem concerned with predicting the interaction between humans and objects. Current HOI models rely on a vocabulary of interactions at training and inference time, limiting their applicability to static environments. With the advent of Multimodal Large Language Models (MLLMs), it has become feasible to explore more flexible paradigms for interaction recognition. In this work, we revisit HOI detection through the lens of MLLMs and apply them to in-the-wild HOI detection. We define the Unconstrained HOI (U-HOI) task, a novel HOI domain that removes the requirement for a predefined list of interactions at both training and inference. We evaluate a range of MLLMs on this setting and introduce a pipeline that includes test-time inference and language-to-graph conversion to extract structured interactions from free-form text. Our findings highlight the limitations of current HOI detectors and the value of MLLMs for U-HOI.

Semantic Multi-Object Tracking (SMOT) extends multi-object tracking with semantic outputs such as video summaries, instance-level captions, and interaction labels, aiming to move from trajectories to human-interpretable descriptions of dynamic scenes. Existing SMOT systems are trained end-to-end, coupling progress to expensive supervision and limiting the ability to rapidly adapt to new foundation models and interactions. We propose TF-SMOT, a training-free SMOT pipeline that composes pretrained components for detection, mask-based tracking, and video-language generation. TF-SMOT combines D-FINE and the promptable SAM2 segmentation tracker to produce temporally consistent tracklets, uses contour grounding to generate video summaries and instance captions with InternVideo2.5, and aligns extracted interaction predicates to BenSMOT WordNet synsets via gloss-based semantic retrieval with LLM disambiguation. On BenSMOT, TF-SMOT achieves state-of-the-art tracking performance within the SMOT setting and improves summary and caption quality compared to prior art. Interaction recognition remains challenging under strict exact-match evaluation on the fine-grained and long-tailed WordNet label space; our analysis and ablations indicate that semantic overlap and label granularity substantially affect measured performance.

Tracking by detection is an effective approach to multiple object tracking: detections are extracted and matched across video frames. However, detection errors persist, leading to false negatives that degrade tracker performance. We propose an architecture to overcome detection failures by obtaining and tracking segmentation masks for each object instead of relying only on bounding boxes, which can lack precision in crowded scenes. When a detector fails and a track cannot be associated, the track is sent to a Lost Tracks Recovery (LTR) architecture. LTR combines an off-the-shelf mask generation network, a transformer-based mask selection network, and a segmentation-based tracker to preserve identities through missed detections. Results on the MOT20 crowded dataset show improved performance for state-of-the-art tracking systems.

This live demonstration shows a mixed-signal compute-in-memory macro deep neural network integrated circuit in 180 nm CMOS technology for image recognition. Images are coded as pulse-width modulation signals, while DNN weights are stored as voltages in 6T-SRAM memories that drive current sources inside every multiplier. Multipliers are arranged within processing elements laid down in a 2D mesh suitable for image processing. The macro implements a lightweight CNN for digit recognition on MNIST, using an FPGA and tablet interface so visitors can draw digits and see the predicted probabilities. The power consumption per multiplier of the CIM macro is 0.22 uW, below state-of-the-art competitors following the same multiply-and-accumulate principle.

Depth estimation and all-in-focus image restoration from defocused RGB images are related problems, although most existing methods address them separately. The few approaches that solve both problems use a pipeline to derive a depth or defocus map as an intermediate product that supports image deblurring. We propose 2HDED:NET, an encoder-decoder network for depth from defocus that is extended with a deblurring branch sharing the same encoder. The two branches operate in parallel and attach equal importance to depth estimation and image restoration. The network is tested on NYU-Depth V2 and compared with state-of-the-art methods for both depth estimation and image deblurring.

Many computer vision applications require real-time processing speeds, which prevents them from running an object detector on every frame. In such circumstances, motion estimation techniques are needed to maintain target identities. Instantiating multiple single-object trackers is feasible only for a few targets, while global frame-feature methods can produce features with limited semantic information and inefficient multiscale tests. SiamFAST addresses these problems with a feature-pyramid-based region-of-interest extractor for object exemplars and search areas, a pairwise depthwise region proposal network to compute similarities for several dozen objects, and a multi-object penalization module to suppress distractors. The method is validated on three public benchmarks and achieves leading performance against state-of-the-art trackers.

Visual object tracking preserves the identity of an object throughout a video. Real applications often demand systems capable of tracking multiple objects in real time, but multi-object tracking solutions usually follow tracking-by-detection, depending on costly detectors and category-specific training. SiamMT efficiently applies individual visual tracking techniques to multiple arbitrary objects in real time, making it a deep-learning-based arbitrary multi-object tracker. It uses global frame feature extraction with a fully convolutional neural network, crops and resizes object search areas, and performs the final similarity operation between search areas and target exemplars with optimized pairwise cross-correlation. These components allow the system to track multiple targets in a scalable manner.

This workshop paper addresses object detection and object tracking in video using convolutional networks generated through deep learning. It focuses on three lines of work: small target detection for objects below 16x16 pixels with STDnet, real-time tracking of multiple objects using a convolutional network, and traffic monitoring through an integrated detection and tracking system. STDnet includes an early visual attention mechanism, the Region Context Network, which selects promising regions containing small objects and their context while keeping memory use low. The integrated detection and tracking system is designed to operate in real time with hundreds of objects and remain robust to full occlusions by combining low-level visual tracking with high-level motion modeling, data association, and tracker reinitialization.