Schedule

Date	Topic
27-Aug	Introduction
29-Aug	Representing Words
03-Sep	Language and Sequence Modeling
05-Sep	Attention and Transformers
10-Sep	Pre-training and Pre-trained LLMs
12-Sep	Instruction Tuning Assgn-1 out
17-Sep	Prompting + Complex Reasoning
19-Sep	Reinforcement Learning and RLHF
24-Sep	Experimental Design and Human Annotation
26-Sep	Retrieval and RAG Assgn-1 due Assgn-2 out
01-Oct	Distillation, Quantization, and Pruning
03-Oct	Domain Specific Modeling: Code and Math Assgn-3,4 out
08-Oct	Long Sequence Models
10-Oct	Other LMs (Masked LMs, Encoder-decoders)
15-Oct	Fall Break
17-Oct	Fall Break
22-Oct	Language Agents I - Framework Assgn-2 due
24-Oct	Language Agents II - Applications
29-Oct	Evaluation of LLMs
31-Oct	Multimodal Models
05-Nov	Democracy Day - No Class
07-Nov	Linguistics and Computational Linguistics
12-Nov	Knowledge-based QA and Information Extraction Assgn-3 due
14-Nov	Ensembling and Mixture of Experts
19-Nov	Safety and Security: Bias, Fairness and Privacy
21-Nov	Inference Algorithms - Sean Welleck
26-Nov	Guest Lecture - Beidi Chen
28-Nov	Thanksgiving - No Class
03-Dec	Posters
05-Dec	Posters
09-Dec	Assgn-4 due

Introduction - Overview of NLP (Aug 27)

Content:

• What is natural language processing?
• What are the features of natural language?
• What do we want to do with NLP?
• What makes it hard?
• Building a rule-based classifier
• Training a bag-of-words classifier

Slides: Intro Slides

Code: Simple Text Classifiers

Reading Material

• Reference: Examining Power and Agency in Film (Sap et al. 2017)

Representing Words (Aug 29)

Content:

• Subword models
• Continuous word embeddings
• Training more complex models
• Neural network basics
• Visualizing word embeddings

Recitation (OH): PyTorch and SentencePiece

Slides: Word Representation and Text Classification Slides

Code: Subword Models, Text Classification

Reading Material

• Reference: Neural Machine Translation of Rare Words with Subword Units (Sennrich et al. 2015)
• Reference: Unigram Models for Subword Segmentation (Kudo 2018)
• Software: SentencePiece
• Reference: Exploring BERT’s Vocabulary (Ács 2019)

Language and Sequence Modeling (Sep 03)

Content:

• Language Modeling Problem Definition
• Count-based Language Models
• Measuring Language Model Performance: Accuracy, Likelihood, and Perplexity
• Log-linear Language Models
• Neural Network Basics
• Feed-forward Neural Network Language Models

Recitation (OH): N-Gram Language Model

Slides: Language Modeling Slides

Reading Material

• Highly Recommended Reading: Goldberg Book Chapter 8-9
• Reference: An Empirical Study of Smoothing Techniques for Language Modeling (Goodman 1998)
• Software: kenlm
• Reference: Maximum entropy (log-linear) language models. (Rosenfeld 1996)
• Reference: Lossless Data Compression with Arithmetic Coding. (neptune.ai 2023)
• Reference: Using the Output Embedding. (Press and Wolf 2016)
• Reference: A Neural Probabilistic Language Model. (Bengio et al. 2003)
• Reference: On the Calibration of Modern Neural Networks (Guo et al. 2017)
• Reference: How can we Know when Language Models Know (Jiang et al. 2020)
• Reference: Self-Consistency Improves Chain of Thought Reasoning in Language Models (Wang et al. 2022)
• Reference: Just Ask for Calibration (Tian et al. 2023)
• Reference: Can LLMs Express Their Uncertainty? (Xiong et al. 2023)

Attention and Transformers (Sep 05)

Content:

• Attention
• Transformer Architecture
• Multi-Head Attention
• Positional Encodings
• Layer Normalization
• Optimizers and Training
• LLaMa Architecture

Recitation (OH): Hugging Face Transformers, Annotated Transformer

Slides: Attention and Transformers Slides

Reading Material

• Recommended Reading: Neural Machine Translation and Sequence-to-Sequence Models Chapter 8
• Reference: RNNs (Elman 1990)
• Reference: LSTMs (Hochreiter and Schmidhuber 1997)
• Reference: Attentional NMT (Bahdanau et al. 2015)
• Reference: Effective Approaches to Attention (Luong et al. 2015)
• Reference: Self Attention (Cheng et al. 2016)

• Reference: Attention is All You Need (Vaswani et al. 2017)

• Highly Recommended Reading: The Annotated Transformer
• Reference: Attention is All You Need (Vaswani et al. 2017)
• Reference: Attentional NMT (Bahdanau et al. 2015)
• Reference: Effective Approaches to Attention (Luong et al. 2015)
• Reference: Relative Positional Encodings (Shaw et al. 2018)
• Reference: RoPE (Su et al. 2021)
• Reference: Layer Normalization (Ba et al. 2016)
• Reference: RMSNorm (Zhang and Sennrich 2019)
• Reference: Pre- and Post-LayerNorm (Xiong et al. 2020)
• Reference: SiLU (Hendrycks and Gimpel 2016)
• Reference: AdamW (Loshchilov and Hutter 2017)
• Reference: LLaMa (Touvron et al. 2023)
• Reference: Comparison of Architectures (Gu and Dao 2023)

Pre-training and Pre-trained LLMs (Sep 10)

Content:

• Overview of pre-training
• Pre-training objectives
• Pre-training data
• Open vs. closed models
• Representative pre-trained models

Slides: Pretraining Slides

References

• Levels of Release in LMs (Liang et al. 2022)
• Pythia (Biderman et al. 2023)
• The Pile (Gao et al. 2021)
• OLMo (Groeneveld et al. 2024)
• LLaMa 2 (Touvron et al. 2023)
• Context Distillation (Askell et al. 2021)
• Mistral (Jiang et al. 2023)
• Mixtral (Jiang et al. 2023)
• Qwen (Bai et al. 2023)
• Starcoder (Li et al. 2023)
• Code LLaMA (Rozière et al. 2023)
• Llema (Azerbayev et al. 2023)
• Galactica (Taylor et al. 2022)
• GPT-4 (OpenAI 2023)
• Gemini (Gemini Team 2023)
• Claude (Anthropic 2023)

Instruction Tuning (Sep 12)

Co-Lecturer Xiang Yue

Content:

• Multi-tasking
• Fine-tuning and Instruction Tuning
• Parameter Efficient Fine-tuning
• Instruction Tuning Datasets
• Synthetic Data Generation

Slides: Instruction Tuning Slides

Reading Material

• Recommended Reading: Instructiion Tuning Survey (Zhang et al. 2023)
• Recommended Reading: FLAN Collection (Longpre et al. 2023)
• Recommended Reading: Unified View of PEFT (He et al. 2021)
• Reference: ZeRo (Rajbhandari et al. 2019)
• Reference: Adapters (Houlsby et al. 2019)
• Reference: Adapter Fusion (Pfeiffer et al. 2020)
• Reference: LoRa (Hu et al. 2021)
• Reference: QLoRA (Dettmers et al. 2023)
• Reference: BitFit (Ben Zaken et al. 2023)
• Reference: MMLU (Hendrycks et al. 2020)
• Reference: Natural Questions (Kwiatkowski et al. 2019)
• Reference: HumanEval (Chen et al. 2021)
• Reference: WikiSum (Liu et al. 2018)
• Reference: FLORES (Goyal et al. 2021)
• Reference: OntoNotes (Weischedel et al. 2013)
• Reference: BIGBench (Srivastava et al. 2022)
• Reference: Instruction Tuning (1) (Wei et al. 2021)
• Reference: Instruction Tuning (2) (Sanh et al. 2021)
• Reference: Learning to In-context Learn (Min et al. 2021)
• Reference: Self-instruct (Wang et al. 2022)
• Reference: ORCA (Mukherjee et al. 2023)
• Reference: Evol-Instruct (Xu et al. 2023)

Prompting and Complex Reasoning (Sep 17)

Content:

• Prompting Methods
• Sequence-to-sequence Pre-training
• Prompt Engineering
• Answer Engineering
• Multi-prompt Learning
• Prompt-aware Training Methods
• Types of Reasoning
• Chain-of-thought and Variants
• Supervised Training for Reasoning

Recitation(OH): OpenAI API, LiteLLM

Slides: Prompting Slides

Reading Material

• Recommended Reading: Prompting Survey
• Recommended Reading: Towards Reasoning in Large Language Models: A Survey
• Recommended Reading: Prompt Engineering Guide
• Reference: Unsupervised Prompting (Radford et al. 20)
• Reference: Few-shot Prompting (Brown et al. 2020)
• Reference: LiteLLM Prompt Templates (LiteLLM 2024)
• Reference: How to Format Inputs to ChatGPT Models (OpenAI Cookbook 2024)
• Reference: Prompt Ordering (Lu et al. 2021)
• Reference: Label Balance and Label Coverage (Zhang et al. 2022)
• Reference: What Makes In-context Learning Work (Min et al. 2022)
• Reference: Chain of Thought (Wei et al. 2022)
• Reference: Let’s think Step by Step (Kojima et al. 2022)
• Reference: Structuring Outputs as Programs (Madaan et al. 2022)
• Reference: Program Aided Language Models (Gao et al. 2022)
• Reference: Prompt Paraphrasing (Jiang et al. 2019)
• Reference: Iterative Prompt Paraphrasing (Zhou et al. 2021)
• Reference: AutoPrompt (Shin et al. 2020)
• Reference: Language Model’s Sensitivity to Prompts (Sclar et al. 2023)
• Reference: A Unified View of Parameter-efficient Transfer Learning (He et al. 2021)
• Reference: Adapters (Houlsby et al. 2019)
• Reference: Combining Prompting with Fine-tuning (Schick and Schütze 2020)
• Reference: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Wei et al. 2022
• Reference: Let’s think Step by Step (Kojima et al. 2022)
• Reference: Self Ask (Press et al. 2022)
• Reference: Chain of Thought with Retrieval (He et al. 2023)
• Reference: Multilingual Chain of Thought (Shi et al. 2022)
• Reference: Complexity-based Prompting (Fu et al. 2022)
• Reference: Reliability of Explanations (Ye and Durrett 2022)
• Reference: Emergent Abilities are a Mirage (Schaeffer et al. 2023)
• Reference: Let’s Verify Step-by-step (Lightman et al. 2023)
• Reference: ORCA (Mukherjee et al. 2023)
• Reference: Rule Inference with LLMs (Qiu et al. 2023)
• Reference: LLMs can learn rules
• Reference: Goal-driven Discovery of Distributional Differences (Zhong et al. 2023)

Reinforcement Learning (Sep 19)

Content:

• Methods to Gather Feedback
• Error and Risk
• Reinforcement Learning
• Stabilizing Reinforcement Learning

Slides: Reinforcement Learning and Human Feedback Slides

• Recommended Reading: Deep Reinforcement Learning Tutorial (Karpathy 2016)
• Recommended Reading: Human Feedback Survey (Fernandes et al. 2023)
• Reference: Course in Machine Learning Chapter 17 (Daume)
• Reference: Reinforcement Learning Textbook (Sutton and Barto 2016)
• Reference: TrueSkill (Sakaguchi et al. 2014)
• Reference: Multi-dimensional Quality Metrics
• Reference: Large-scale MQM Annotation (Freitag et al. 2021)
• Reference: BERTScore (Zhang et al. 2019)
• Reference: COMET (Rei et al. 2020)
• Reference: Prometheus 2 (Kim et al. 2023)
• Reference: AutoMQM (Fernandes et al. 2023)
• Reference: WMT Metrics Shared Task (Freitag et al. 2023)
• Reference: SummEval (Fabbri et al. 2020)
• Reference: Summarization Evaluation through QA (Eyal et al. 2019)
• Reference: Minimum Risk Training for NMT (Shen et al. 2015)
• Reference: REINFORCE (Williams 1992)
• Reference: Co-training (Blum and Mitchell 1998)
• Reference: Revisiting Self-training (He et al. 2020)
• Reference: Adding Baselines (Dayan 1990)
• Reference: Sequence-level Training for RNNs (Ranzato et al. 2016)
• Reference: PPO (Schulman et al. 2017)
• Reference: DPO (Rafailov et al. 2023)

Experimental Design and Human Annotation (Sep 24)

Content:

• Experimental Design
• Data Annotation

Slides: Experimental Design Slides

References:

• Recommended Reading: How to Avoid Machine Learning Pitfalls (Lones 2021)
• Recommended Reading: Best Practices for Data Annotation (Tseng et al. 2020)
• Recommended Viewing: How to Write a Great Research Paper (Peyton-Jones 2006)
• Reference: Sentiment Analysis (Pang et al. 2002)
• Reference: Conversational Question Answering (Reddy et al. 2019)
• Reference: Bottom-up Abstractive Summarization (Gehrmann et al. 2018)
• Reference: Unsupervised Word Segmentation (Kudo and Richardson 2018)
• Reference: Analyzing Language of Bias (Rankin et al. 2017)
• Reference: Are All Languages Equally Hard to Language-Model? (Cotterell et al. 2018)
• Reference: Modeling Podcasts (Reddy et al. 2021)
• Reference: BERT Rediscovers the Classical NLP Pipeline (Tenney et al. 2019)
• Reference: When and Why are Word Embeddings Useful in NMT? (Qi et al. 2018)
• Reference: Kappa Statistic (Carletta 1996)
• Reference: Downside of Surveys (Varian 1994)
• Reference: Penn Treebank Annotation Guidelines (Santoroni 1990)
• Reference: Data Statements for NLP (Bender and Friedman 2018)
• Reference: Power Analysis (Card et al. 2020)
• Reference: Active Learning (Settles 2009)
• Reference: Active Learning Curves (Settles and Craven 2008)

Retrieval and RAG (Sep 26)

Content:

• Retrieval Methods
• Retrieval Augmented Generation
• Long-context Transformers

Recitation(OH): LangChain or LlamaIndex

Slides: Retrieval Augmented Generation Slides

References

• Recommended Reading: ACL 2023 RAG Tutorial (Asai et al. 2023)
• Reference: Retrieval-based QA (Chen et al. 2017)
• Reference: Dense Passage Retrieval (Karpukhin et al. 2020)
• Reference: Retro (Borgeaud et al. 2021)
• Reference: Introduction to Information Retrieval (Manning et al. 2009)
• Software: Apache Lucene
• Reference: BehnamGader et al. 2024
• Reference: Echo Embeddings (Springer et al. 2024)
• Reference: DPR (Karpukhin et al. 2020)
• Reference: Contriever (Izacard et al. 2022)
• Software: FAISS
• Software: ChromaDB
• Reference: Instructor Embeddings (Su et al. 2022)
• Reference: Cross-encoder Reranking (Nogueira et al. 2019)
• Reference: Token-level Retrieval (Khattab and Zaharia 2020)
• Reference: Hypothetical Document Embeddings (Gao et al. 2022)
• Reference: Understanding NDCG (Hegde 2022)
• Reference: Mean Average Precision (Tan 2024)
• Reference: End-to-end RAG Training (Lewis et al. 2020)
• Reference: Toolformer (Schick et al. 2023)
• Reference: FLARE (Jiang et al. 2023)
• Reference: kNN-LM (Khandelwal et al. 2019)
• Reference: Unlimiformer (Bertsch et al. 2023)
• Reference: Deciding Whether to Use Passages (Asai et al. 2021)
• Reference: Learning to Filter Context (Wang et al. 2023)

Distillation, Quantization, and Pruning (Oct 01)

Co-Lecturer Vijay Viswanathan

Content:

• Distillation
• Quantization
• Pruning

Slides (Spring 2024 version):: Distillation Slides

References

• Recommended Reading: Theia Vogel’s blog on “How to make LLMs go fast”
• Recommended Reading: Lilian Weng’s blog on “Inference Optimization”
• Reference: Over-parametrization is provably useful in training neural nets (Du and Lee 2018)
• Reference: Model-Aware Quantization: GOBO (Zadeh et al. 2020)
• Reference: Model-Aware Quantization: LLM.int8 (Dettmers et al. 2022)
• Software: Binarized Neural Networks (Courbariaux et al. 2016)
• Reference: Layer-by-Layer Quantization-Aware Distillation (Yao et al. 2020)
• Reference: QLoRA (Dettmers et al. 2023)
• Reference: Magnitude pruning (in general) (Han et al. 2015)
• Reference: An analysis of magnitude pruning for machine translation) (See et al. 2016)
• Reference: The Lottery Ticket Hypothesis) (Frankle et al. 2018)
• Reference: Wanda (Pruning by Weights and Activations) (Frankle et al. 2018)
• Reference: Coarse-to-fine Structured Pruning (Xia et al. 2022)
• Reference: Are Sixteen Heads Really Better than One? (Michel and Neubig 2019)
• Reference: Pruning with Forward Passes (Dery et al 2024)
• Reference: Self-Training (Yarowski 1995)
• Reference: Hard vs Soft Target Distillation (Hinton et al 2015)
• Reference: Sequence-Level Distillation (Kim and Rush 2016)
• Reference: DistilBERT (Sanh et al 2019)
• Reference: Deep Learning is Robust to Massive Label Noise (Furlanello et al 2018)
• Reference: Born Again Neural Networks (Rolnick et al 2018)
• Reference: Self-Instruct (Wang et al 2022)
• Reference: Prompt2Model (Viswanathan et al 2023)
• Reference: SynthIE (Exploiting Asymmetry for Synthetic Training Data Generation) (Josifoski et al 2023)
• Reference: DataDreamer: A Toolkit for Synthetic Data Generation (Patel et al 2024)

Domain Specific Modeling: Code and Math (Oct 03)

Co-Lecturer Xiang Yue Content:

Recitation(OH):

Slides :: References

Long Sequence Models (Oct 08)

Content:

Recitation (OH): Unlimiformer, Mamba

• Reference: Training Transformers with Context (Voita et al. 2018)
• Reference: Transformer-XL (Dai et al. 2019)
• Reference: Mistral (Jiang et al. 2023)
• Reference: Sparse Transformers (Child et al. 2019)
• Reference: Compressive Transformer (Rae et al. 2019)
• Reference: Linformer (Wang et al. 2020)
• Reference: Nystromformer (Xiong et al. 2021)
• Reference: Long Range Arena (Tay et al. 2020)
• Reference: SCROLLS (Shaham et al. 2022)
• Reference: Lost in the Middle (Liu et al. 2023)

Other LMs (Oct 10)

Content:

Recitation(OH):

Slides :: References

Language Agents I - Frameworks (Oct 22)

Content:

Recitation(OH):

Slides :: References

Language Agents II - Applications (Oct 24)

Content:

• Tool Use
• Language Agents

Slides (Spring 2024 version):: Tool Use Slides

Slides (Spring 2024 version):: Language Agents Slides

References

• Recommended Reading: What Are Tools Anyway? A Survey from the Language Model Perspective Wang et al. 2024
• Reference: Toolformer: Language Models Can Teach Themselves to Use Tools Schick et al. 2023
• Reference: ART: Automatic multi-step reasoning and tool-use for large language models Paranjape et al. 2023
• Reference: ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs Qin et al. 2024
• Reference: Gorilla: Large Language Model Connected with Massive APIs Patil et al. 2023
• Reference: HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face Shen et al. 2023
• Reference: VOYAGER: An Open-Ended Embodied Agent with Large Language Models Wang et al. 2023
• Reference: TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Tasks Wang et al. 2024
• Reference: Animal tool behavior: the use and manufacture of tools by animals Shumaker et al. 2011
• Reference: Artificial intelligence: a modern approach Russell, Stuart and Norvig, Peter, 2016
• Reference: A syntactic neural model for general-purpose code generation Yin et al. 2017
• Reference: Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments Chen et al. 2018
• Reference: ALFWorld: Aligning Text and Embodied Environments for Interactive Learning Shridhar et al. 2021
• Reference: MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge Fan et al. 2022
• Reference: Scaling Instructable Agents Across ManySimulated Worlds SIMA Team 2024
• Reference: Devin Devin Team 2024
• Reference: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Wei et al. 2022
• Reference: ReAct: Synergizing Reasoning and Acting in Language Models Yao et al. 2023
• Reference: PAL: Program-aided Language Models Gao et al. 2022
• Reference: Mind2Web: Towards a Generalist Agent for the Web Deng et al. 2023
• Reference: A Data-Driven Approach for Learning to Control Computers Humphreys et al. 2022
• Reference: WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents Yao et al. 2023
• Reference: WebArena: A Realistic Web Environment for Building Autonomous Agents Zhou et al. 2023
• Reference: Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks Gururangan et al. 2020
• Reference: Training language models to follow instructions with human feedback Ouyang et al. 2022
• Reference: Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents Song et al. 2024
• Reference: Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization Yao et al. 2023

Evaluation of LLMs (Oct 29)

Content:

Recitation(OH):

Slides :: References

Multimodal Models (Oct 31)

Co-Lecturer Xiang Yue

Content:

Recitation(OH):

Slides :: References

Linguistics and Computational Linguistics (Nov 07)

Co-Lecturer: Lindia Tjuatja

Content:

• Linguistics and Computational Linguistics

Slides (Spring 2024 version):: Linguistics and Computational Linguistics Slides

References

• Reference Automated reconstruction of ancient languages using probabilistic models of sound change (Bouchard-Côté et al. 2013)
• Reference Articulation GAN: Unsupervised modeling of articulatory learning (Beguš et al. 2023)
• Reference What do phone embeddings learn about Phonology? (Kolachina and Magyar 2019)
• Reference PWESuite: Phonetic Word Embeddings and Tasks They Facilitate (Zouhar et al. 2024)
• Reference Bootstrapping a Neural Morphological Analyzer for St. Lawrence Island Yupik from a Finite-State Transducer (Schwartz et al. 2019)
• Reference Prosodic Structure and Expletive Infixation (McCarthy 1982)
• Reference UCxn: Typologically Informed Annotation of Constructions Atop Universal Dependencies (Weissweiler et al. 2024)
• Reference Distributional Structure (Harris 1954)
• Reference COGS: A Compositional Generalization Challenge Based on Semantic Interpretation (Kim and Linzen 2020)
• Reference The Paradox of the Compositionality of Natural Language: A Neural Machine Translation Case Study (Dankers et al. 2022)
• Reference Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering (Kim et al. 2021)
• Reference Predicting Pragmatic Reasoning in Language Games (Frank and Goodman 2012)

Knowledge Based QA (Nov 12)

Content:

• Knowledge Based QA

Slides (Spring 2024 version):: Knowledge Based QA Slides

References

• Required Reading: Relation Extraction Jurafsky and Martin Chapter 17.2
• Reference: Relation Extraction Survey (Nickel et al. 2016)
• Reference: WordNet (Miller 1995)
• Reference: Cyc (Lenant 1995)
• Reference: DBPedia (Auer et al. 2007)
• Reference: YAGO (Suchanek et al. 2007)
• Reference: Babelnet (Navigli and Ponzetto 2010)
• Reference: Freebase (Bollacker et al. 2008)
• Reference: Wikidata (Vrandečić and Krötzsch 2014)
• Reference: Relation Extraction by Translating Embeddings (Bordes et al. 2013)
• Reference: Relation Extraction with Neural Tensor Networks (Socher et al. 2013)
• Reference: Relation Extraction by Translating on Hyperplanes (Wang et al. 2014)
• Reference: Relation Extraction by Representing Entities and Relations (Lin et al. 2015)
• Reference: Relation Extraction w/ Decomposed Matrices (Xie et al. 2017)
• Reference: Distant Supervision for Relation Extraction (Mintz et al. 2009)
• Reference: Relation Classification w/ Recursive NNs (Socher et al. 2012)
• Reference: Relation Classification w/ CNNs (Zeng et al. 2014)
• Reference: Open IE from the Web (Banko et al. 2007)
• Reference: ReVerb Open IE (Fader et al. 2011)
• Reference: Supervised Open IE (Stanovsky et al. 2018)
• Reference: Universal Schema (Riedel et al. 2013)
• Reference: Joint Entity and Relation Embedding (Toutanova et al. 2015)
• Reference: Distant Supervision for Neural Models (Luo et al. 2017)
• Reference: Relation Extraction w/ Tensor Decomposition (Sutskever et al. 2009)
• Reference: Relation Extraction via. KG Paths (Lao and Cohen 2010)
• Reference: Relation Extraction by Traversing Knowledge Graphs (Guu et al. 2015)
• Reference: Relation Extraction via Differentiable Logic Rules (Yang et al. 2017)
• Reference: Improving Embeddings w/ Semantic Knowledge (Yu et al. 2014)
• Reference: Retrofitting Word Vectors to Semantic Lexicons (Faruqui et al. 2015)
• Reference: Multi-sense Embedding with Semantic Lexicons (Jauhar et al. 2015)
• Reference: Antonymy and Synonym Constraints for Word Embedding (Mrksic et al. 2016)
• Reference: Language Models as Knowledge Bases? (Petroni et al. 2019)
• Reference: How Can We Know What Language Models Know? (Jiang et al. 2019)
• Reference: AUTOPROMPT: Eliciting Knowledge from Language Models with Automatically Generated Prompts (Shin et al. 2020)
• Reference: GPT Understands, Too (Liu et al. 2021)
• Reference: How Much Knowledge Can You Pack Into the Parameters of a Language Model? (Roberts et al. 2020)
• Reference: X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models (Jiang et al. 2020)
• Reference: REALM: Retrieval-Augmented Language Model Pre-Training (Guu et al. 2020)
• Reference: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al. 2020)
• Reference: Multi-hop Reasoning in LMs (Jiang et al. 2022)

Ensembling and Mixture of Experts (Nov 14)

Content:

• Ensembling
• Model Merging
• Sparse Mixture of Experts
• Pipeline Models

Slides (Spring 2024 version):: Multi-model Slides

References:

• Reference: Domain Differential Adaptation (Dou et al. 2019)
• Reference: Dexperts (Liu et al. 2021)
• Reference: Knowledge Distillation (Hinton et al. 2015)
• Reference: cuSPARSE
• Reference: NVIDIA Block Sparsity
• Reference: Sparsely Gated MOE (Shazeer et al. 2017)
• Code Example: Mistral MOE Implementation
• Reference: Weight Averaging for Neural Networks (Utans 1996)
• Reference: Model Soups (Wortsman et al. 2022)
• Software: MergeKit
• Reference: Task Vectors (Ilharco et al. 2022)
• Reference: TIES (Yadav et al. 2023)
• Reference: Stacking (Niehues et al. 2017)
• Reference: Deliberation Networks (Xia et al. 2017)
• Reference: Diffuser (Reid et al. 2022)
• Reference: Self-refine (Madaan et al. 2023)

Safety and Security: Bias, Fairness and Privacy (Nov 19)

Content:

Recitation(OH):

Slides :: References

Inference Algorithms - Sean Welleck (Nov 21)

Guest Lecturer Sean Welleck

Reading Material

• Highly Recommended Reading: Inference Algorithms Survey (Welleck et al. 2024)

Guest Lecture - Beidi Chen (Nov 26)

Guest Lecturer Beidi Chen