A carregar...
Publicação
Multilingual Vision-Language Pre-training for the Remote Sensing Domain
| Nome: | Descrição: | Tamanho: | Formato: | |
|---|---|---|---|---|
| 5.36 MB | Adobe PDF |
Orientador(es)
Resumo(s)
Methods based on Contrastive Language-Image Pre-training (CLIP) are nowadays extensively used in support of vision-and-language tasks involving remote sensing data, such as cross-modal retrieval. The adaptation of CLIP to this specific domain has relied on model fine-tuning with the standard contrastive objective, using existing human-labeled image-caption datasets, or using synthetic data corresponding to image-caption pairs derived from other annotations over remote sensing images (e.g., object classes). The use of different pre-training mechanisms has received less attention, and only a few exceptions have considered multilingual inputs. This work proposes a novel vision-and-language model for the remote sensing domain, exploring the fine-tuning of a multilingual CLIP model and testing the use of a self-supervised method based on aligning local and global representations from individual input images, together with the standard CLIP objective. Model training relied on assembling pre-existing datasets of remote sensing images paired with English captions, followed by the use of automated machine translation into nine additional languages. We show that translated data is indeed helpful, e.g. improving performance also on English. Our resulting model, which we named Remote Sensing Multilingual CLIP (RS-M-CLIP), obtains state-of-the-art results in a variety of vision-and-language tasks, including cross-modal and multilingual image-text retrieval, or zero-shot image classification.
Descrição
Funding information:
This research was supported by the Portuguese Recovery and Resilience Plan through project C645008882-00000055 (i.e., the Center For Responsible AI), and also by the Fundação para a Ciência e Tecnologia (FCT), specifically through the project with reference UIDB/50021/2020 (DOI: 10.54499/UIDB/50021/2020), and the project with reference UIDP/04516/2020 (DOI: 10.54499/UIDB/04516/2020).
Publisher Copyright:
© 2024 Copyright held by the owner/author(s).
Palavras-chave
Contrastive Language-Image Pre-training Cross-Modal Retrieval Remote Sensing Self-Supervised Pre-training Vision and Language Information Systems Earth-Surface Processes Modelling and Simulation Computer Graphics and Computer-Aided Design Computer Science Applications
Contexto Educativo
Citação
Editora
ACM - Association for Computing Machinery