Recent Advances in Vision-and-Language Pre-training

CVPR 2022 Tutorial on "Recent Advances in Vision-and-Language Pre-training"

Humans perceive the world through many channels, such as images viewed by the eyes or voices heard by the ears. Though any individual channel might be incomplete or noisy, humans can naturally align and fuse information collected from multiple channels, in order to grasp the key concepts needed for a better understanding of the world. One of the core aspirations in artificial intelligence is to develop algorithms that endow computers with an ability to effectively learn from multi-modality (or multi-channel) data. This data is similar to sights and sounds attained from vision and language that help humans make sense of the world around us. For example, computers could mimic this ability by searching the most similar images for a text query (or vice versa) and by describing the content of an image using natural language.

Vision-and-Language (VL), a popular research area that sits at the nexus of Computer Vision and Natural Language Processing (NLP), aims to achieve this goal. Inspired by the great success of language model pre-training in NLP, Vision-and-Language Pre-training (VLP) has recently attracted rapidly growing attention from both communities. In this tutorial, we will cover the most recent approaches and principles at the frontier of VLP, including (1) region-feature-based and end-to-end image-text pre-training; (2) unified vision-language modeling; (3) its extension to video-language pre-training; (4) learning visual models from language supervision; and (5) visual synthesis. The tutorial will be a full-day event (9:00 am to 5:00pm) with several middle breaks.