Sentence and Contextual Word Representations (3/4/2021)
Content:
- Sentence Representations
- Contextual Word Representations
Reading Material
- Highly Recommended Reading: Illustrated BERT (Alammar 2019)
- Highly Recommended Reading: Pre-trained Language Model Papers (Wang and Zhang 2019)
- Reference: Learning Sentence Representations (Hill et al. 2016)
- Reference: Skip-thought Vectors (Kiros et al. 2015)
- Reference: Semi-supervised Sequence Learning (Dai and Le 2015)
- Reference: Paraphrase Detection (Dolan et al. 2005)
- Reference: Semantic Relatedness (Marelli et al. 2014)
- Reference: Recognizing Textual Entailment (Dagan et al. 2006)
- Reference: Paraphrastic Sentence Embeddings (Wieting et al. 2015)
- Reference: Paraphrase Database
- Reference: ParaNMT-50M (Wieting et al. 2018)
- Reference: context2vec (Melamud et al. 2016)
- Reference: Learned in Translation: Contextualized Word Vectors (McMann et al. 2017)
- Reference: ELMo: Deep Contextualized Word Representations (Peters et al. 2018)
- Reference: BERT: Bidirectional Transformers (Devlin et al. 2018)
- Reference: RoBERTa: Robustly Optimized BERT (Liu et al. 2019)
- Reference: XLNet: Autoregressive Training w/ Permutation Objectives (Yang et al. 2019)
- Reference: ELECTRA: Pre-training Text Encoders as Discriminators (Clark et al. 2020)
- Reference: Inference -> Generalization (Conneau et al. 2017)
- Reference: Paraphrase ID -> Generalization (Wieting and Gimple 2018)
- Reference: Comparison of Training Objectives (Zhang and Bowman 2018)
Slides: Sentence Representation Slides
Video: Sentence Representation Video
Sample Code: Sentence Representation Code Examples