Supervisor:
Institutions:
Contrastive Learning is an unsupervised method for learning similarities or differences in a dataset, without the need of labels. The main idea is to provide the machine with similar (so called positive samples) and with very different data (negative or corrupted samples). The task of the machine then is to leverage this information and to pull the positive examples in the embedded space together, while pushing the negative examples further apart. Next to being unsupervised, another major advantage is that the loss is applied on the latent space rather than being pixel-base. This saves computation and memory, because there is no need for a decoder and also delivers more accurate results.
In this work, we will investigate the SetCon model from 'Learning Object-Centric Video Models by Contrasting Sets' by Löwe et al. [1] (Paper) The SetCon model has been published in November 2020 by the Google Brain Team and introduces an attention-based object extraction in combination with contrastive learning. It incorporates a novel slot-attention module [2](Paper), which is an iterative attention mechanism to map the feature maps from the CNN-Encoder to a predefined number of object slots and has been inspired by the transformer models from the NLP world.
We investigate the utility of this architecture when used together with realistic video footage. Therefore, we implemented the SetCon with PyTorch according to its description and build upon it to meet our requirements. We then created two different datasets, in which we film given objects from different angles and distances, similar to Pirk [3] (Github, Paper). However, they relied on a faster-RCNN for the object detection, whereas the goal of the SetCon is to extract the objects solely by leveraging the contrastive loss and the slot attention module. By training a decoder on top of the learned representations, we found that in many cases the model can successfully extract objects from a scene.
This repository contains our PyTorch-implementation of the SetCon-Model from 'Learning Object-Centric Video Models by Contrasting Sets' by Löwe et al. Implementation is based on the description in the article. Note, this is not the official implementation. If you have questions, feel free to reach out to me.
For our work, we have taken two videos, a Three-Object video and a Seven-Object video. In these videos we interacted with the given objects and moved them to different places and constantly changed the view perspective. Both are 30mins long, such that each contains about 54.000 frames.
Figure 1: An example of the object extraction on the test set of the Three-Object dataset.
We trained the contrastive pretext model (SetCon) on the first 80% and then evaluated the learned representations on the remaining 20%. Therefore, we trained a decoder, similar to the evaluation within the SetCon paper and looked into the specialisation of each slot. Figures 1 and 2 display two evaluation examples, from the test-set of the Three-Object Dataset and the Seven-Object Dataset. Bot figures start with the ground truth for three timestamps. During evaluation only the ground truth at t will be used to obtain the reconstructed object slots as well as their alpha masks. The Seven-Object video is itended to be more complex and one can perceive in figure 2 that the model struggles more than on the Three-Obejct dataset to route the objects to slots. On the Three-Object dataset, we achieved 0.0043 ± 0.0029 MSE and on the Seven-Object dataset 0.0154 ± 0.0043 MSE.
Figure 2: An example of the object extraction on the test set of the Seven-Object dataset.
For our work, we have taken two videos, a Three-Object video and Seven-Object video. Both datasets are saved as frames and are then encoded in a h5-files.
To use a different dataset, we further provide a python routine process frames.py, which converts frames to h5 files.
For the contrastive pretext-task, the training can be started by:
python3 train_pretext.py --end 300000 --num-slots 7
--name pretext_model_1 --batch-size 512
--hidden-dim=1024 --learning-rate 1e-5
--feature-dim 512 --data-path ’path/to/h5file’
Further arguments, like the size of the encoder or for an augmentation pipeline, use the flag -h for help.
Afterwards, we froze the weights from the encoder and the slot-attention-module and trained a downstream decoder on top of it.
The following command will train the decoder upon the checkpoint file from the pretext task:
python3 train_decoder.py --end 250000 --num-slots 7
--name downstream_model_1 --batch-size 64
--hidden-dim=1024 --feature-dim 512
--data-path ’path/to/h5file’
--pretext-path "path/to/pretext.pth.tar"
--learning-rate 1e-5
For MSE evaluation on the test-set, use both checkpoints, from the pretext- model for the encoder- and slot-attention-weights and from the downstream- model for the decoder-weights and run:
python3 eval.py --num-slots 7 --name evaluation_1
--batch-size 64 --hidden-dim=1024
--feature-dim 512 --data-path ’path/to/h5file’
--pretext-path "path/to/pretext.pth.tar"
--decoder-path "path/to/decoder.pth.tar"
Instead of many small sequences of artificially created frames, we need to deal with a long video-sequence. Therefore, each element in our batch mirrors a single frame at a given time t, not a sequence. For this single frame at time t, we load its two predecessors, which are then used to predict the frame at t, and thereby create a positive example. Further, we found, that the infoNCE-loss to be numerically unstable in our case, hence we opted for the almost identical but more stable NT-Xent in our implementation.
[1] Löwe, Sindy et al. (2020). Learning object-centric video models by contrasting sets. Google Brain team.
[2] Locatello, Francesco et al. Object-centric learning with slot attention.
[3] Pirk, Sören et al. (2019). Online object representations with contrastive learning. Google Brain team.