T5: Encoder-only or decoder-only is NOT all you need, though text-to-text is all you need. (Also, pre-training + finetuning 🚀) https://arxiv.org/abs/1910.10683

GPT2: Unsupervised pre-training is all you need?! https://openai.com/research/better-language-models