The Transformers are a special type of sequence-to-sequence models used for language modeling, machine translation, image captioning and text generation. These models are essentially created of two main components: Encoder and Decoder. These are essentially Recurrent Neural Networks (RNNs) The Encoder is the component which receives each part of the input sequence. Then it encodes it (duh!) into a vector, which is called context. This context carries information about the whole sequence and it is sent over to the Decoder. RNNs are somewhat time traveling neural networks.”]
Source: https://rubikscode.net/2019/07/29/introduction-to-transformers-architecture/