Efficient Image Pre-Training with Siamese Cropped Masked Autoencoders

Abstract

Self-supervised pre-training of image encoders has become omnipresent in the literature, especially since the introduction of Masked Autoencoders (MAE). To excel in propagation tasks such as video segmentation, current research focuses on learning object-centric representations from video motion. Notably, SiamMAE introduced a Siamese network that trains a shared-weight encoder from two video frames with a high asymmetric masking ratio (95%), achieving state-of-the-art performance in video object segmentation, human pose propagation, and semantic part propagation.

In this work, we propose CropMAE, an alternative to the Siamese pre-training method introduced by SiamMAE. Unlike SiamMAE that uses pairs of frames from videos, CropMAE exclusively considers pairs of cropped images sourced from the same still image, but cropped differently. This approach eliminates the need for video decoding, enabling training on still image datasets and significantly reducing pre-training time while maintaining competitive performance.

Our empirical results demonstrate that CropMAE can learn object-centric representations without relying on motion, unlike SiamMAE. This discovery indicates that with the appropriate pretext task, it is possible to acquire object-centric features without using videos or motion information. Furthermore, we show that the pretext task in CropMAE is more explicit and accelerates the learning process of object-centric representations compared to SiamMAE. Additionally, CropMAE achieves the highest masking ratio to date (98.5%), allowing image reconstruction with only two visible patches.