Transformers in Computer Vision – English version

Transformer Networks are the brand new pattern in Deep Studying these days. Transformer fashions have taken the world of NLP by storm since 2017. Since then, they grow to be the mainstream mannequin in nearly ALL NLP duties. Transformers in CV are nonetheless lagging, nonetheless they began to take over since 2020.

We’ll begin by introducing consideration and the transformer networks. Since transformers had been first launched in NLP, they’re simpler to be described with some NLP instance first. From there, we are going to perceive the professionals and cons of this structure. Additionally, we are going to talk about the significance of unsupervised or semi supervised pre-training for the transformer architectures, discussing Massive Scale Language Fashions (LLM) in short, like BERT and GPT.

This may pave the best way to introduce transformers in CV. Right here we are going to attempt to prolong the eye thought into the 2D spatial area of the picture. We’ll talk about how convolution will be generalized utilizing self consideration, inside the encoder-decoder meta structure. We’ll see how this generic structure is sort of the identical in picture as in textual content and NLP, which makes transformers a generic operate approximator. We’ll talk about the channel and spatial consideration, native vs. international consideration amongst different subjects.

Within the subsequent three modules, we are going to talk about the particular networks that resolve the massive issues in CV: classification, object detection and segmentation. We’ll talk about Imaginative and prescient Transformer (ViT) from Google, Shifter Window Transformer (SWIN) from Microsoft, Detection Transformer (DETR) from Fb analysis, Segmentation Transformer (SETR) and plenty of others. Then we are going to talk about the appliance of Transformers in video processing, by way of Spatio-Temporal Transformers with utility to Shifting Object Detection, together with Multi-Process Studying setup.

Lastly, we are going to present how these pre-trained arcthiectures will be simply utilized in apply utilizing the well-known Huggingface library utilizing the Pipeline interface.