Machine Translation and Sequence to Sequence Models

CS 11-731
Language Technologies Institute, School of Computer Science
Carnegie Mellon University
Tuesday/Thursday 1:30-2:50PM, GHC4102

Instructors/TAs:

Instructor: Graham Neubig (gneubig@cs.cmu.edu)
Office hours: Monday 4:00-5:00PM (GHC5409)
TAs: Qinlan Shen and Dongyeop Kang (cs11-731-sp2017-tas@cs.cmu.edu)
Office hours: Tuesday 3:00-4:00PM (Dongyeop@GHC5713), Wednesday 11:00-12:00AM (Qinlan@GHC6405)
Questions and Discussion: Ideally in class or through piazza so we can share information with the class, but email and office hours are also OK.

Course Description

Machine Translation and Sequence-to-Sequence Models is an introductory graduate-level course surveying the primary approaches and methods for developing systems to translate between human languages, or other sequential data. The main objective of the course is to obtain basic understanding and implementation skills for modern methods for MT and sequence transduction, including how to design models, how to learn the model parameters, how to search for the best output, and how to create training data. The course will focus on machine translation, but also briefly cover tasks such as dialog response generation, image caption generation, and others.

Pre-requisites: This course has no official pre-requisites, although 11-711 "Algorithms for NLP" or 10-701 "Machine Learning" would be helpful.

Course format: Classes will take the following format:

Reading: Before the class, you will be given a reading assignment that you should read before coming to class that day.
Questions: At the beginning of the class, the instructor/TAs will field any questions about the reading assignment.
Quiz: After questions have been answered there will be a short quiz that tests the knowledge in the reading assignment.
Pseudo-code Walk/Discussion: The instructor will walk through some pseudo-code that implements one of the main concepts introduced in the reading assignment and answer any questions.
Implementation: The remainder of the class period will be allocated to actually implementing the method described in the pseudo-code walk. Instructors/TAs will field questions during this time.

Discussion: In the second half of the class, we will be having a paper discussion period on Thursday of every week. Before the class, everyone will be expected to read one of three papers, and turn in a some answers to brief open-ended questions in place of the quiz. During the discussion, everyone will be expected to make at least one critical comment about the paper: Was there anything that you think should have been done better, or examined more? Do you have any ideas for ways to expand the method? So please think of this while reading the paper.

Grading: The course will be graded based on the grades of the small quizzes, assignments, and a course project. The in-class implementation exercises during the first half will be turned in as two small-ish assignments. During the second half, your implementation assignments will focus on completing a larger class final project, which will be expected to be novel and interesting (and potentially useful or publishable).

Things to Prepare: Please bring a computer to every class, and make sure you have an account on GitHub. If either of these pose a problem, please consult with the instructors on the first day of class.

Assignments

Assignment Policies: Assignments and the final project may be done in groups of 1-3. If you work in a group of more tha one person, please use a shared git repository and commit the code that you write, and in reports note who did what part of the project. Assignments done in groups of 2-3 will be expected to 2-3 times as significant as assignments done by one person.

Send to the TAs your names, andrew IDs, and the link to a github repository containing code, output for the "test" and "blind" sets, and a report of 2-4 pages. The report should be named "report.pdf". The output should be tokenized and lowercased, and stored in the "output/" directory. The names of your primary results should be "output/test.primary.en" and "output/blind.primary.en", and should use only the provided IWSLT data for training the system. You can also submit additional outputs that use other methods, or use additional resources other than the IWSLT data listed below. In this case, replace primary by an arbitrary string, and we will calculate results for these as well.

All assignments are expected to be conducted under the CMU policy for academic integrity. All rules here apply and violations will be subject to zero credit on the assignment, or other disciplinary measures. In particular, while you may base your implementation on the pseudo-code provided by the TAs or instructor, copying code of other students in the class who are not part of your assignment group is (obviously) not allowed.

Assignment 1: Create an attentional German-English neural machine translation model using the provided IWSLT data. Add at least one or two modifications over the baseline for full credit. Due date: ~~2/20~~ 2/24, 11:59PM Eastern time.
Assignment 2: Create a phrase-based German-English neural machine translation model using the provided IWSLT data. The assignment includes: implementation of the IBM Model 1 alignment algorithm (uni-directional), the phrase-extract algorithm, and code to turn phrases into a WFST. Make some improvement over this baseline for full credit. Due date: 3/20, 11:59PM Eastern time.
- Template Code: Contains some, but not all of the scripts required to train the system.
- Example Output: An example of the output when the full baseline code is run using the validation set as training data.
Project: The class project is free-topic, as long as it concerns machine translation or sequence-to-sequence models. We expect a substantial amount of creativity. We will have an oral presentation for the projects on 5/2 and 5/4, then the a research-paper style report in TACL, WMT, or NIPS format (the target length is 6-8 pages). Due date: 5/15, 11:59PM Eastern time.

Training Data

IWSLT German-English: Data from the IWSLT2016 workshop for German-English translation, separated into a training set, validation set (the dev2010 set from IWSLT), and test set (the tst2010 set). There is an additional blind test set for which translations are not provided. Tokenization has been done with the tokenizer from NLTK (.tok), and this tokenized data has further been lowercased (.low). In addition, a filtered set with sentences with at least one and no more than 50 words has been provided (.tok.filt and .low.filt). This is version 4 of the data, updated 2/8/2017, 3:20PM.

Course Schedule

Date	Topic	Notes
1/17	Introduction (Updated 1/18, 1:50PM), Statistical Machine Translation Overview
1/19	Language Models 1: n-gram Language Models (Updated 1/18, 1:50PM)
1/24	Language Models 2: Log-linear Language Models (Updated slightly 1/23, 11:00AM)
1/26	Language Models 3: Feed-forward Neural LMs (Updated 1/26, 9:45AM)
1/31	Guest Lecture by Robert Frederking: Rule-based Machine Translation
2/2	Language Models 4: Recurrent Neural Network LMs
2/7	Neural MT 1: Encoder-Decoder Models
2/9	Neural MT 2: Attentional Models	Assignment 1
2/14	Evaluation of Generated Output	Example of Assesments and Automatic Scores, Data
2/16	Project Discussion Day	Prepare to answer two questions: (Required) What kind of things are you interested in pursuing for the project? (Optional) If you have any specific ideas, please share them.
2/21	Symbolic MT 1: IBM Models
2/23	Symbolic MT 2: Weighted Finite State Transducers (Updated 2/21, 8:30PM)
2/28	Symbolic MT 3: Phrase-based Machine Translation (Updated algorithm slightly 2/28, 6:00AM)
3/2	Symbolic MT 4: Tree-based Machine Translation	Assignment 2: Create a Symbolic Model (planned due date: 3/20)
3/7,3/9	Other Sequence-to-Sequence Tasks	Paper Reading Candidates: Persona-based Conversation Model (Dialog) Connectionist Temporal Classification (Speech Recognition) Recurrent Neural Network Grammars (Parsing)
3/14,3/16	Spring Break, No Class
3/21,3/23	Advanced Topics 1: Parameter Optimization	Paper Reading Candidates: Reinforcement learning: Sequence-level training for recurrent neural networks Evaluation measures: Best lexical metric + Optional SeqGAN
3/28,3/30	Advanced Topics 2: Hybrid Neural/Symbolic Models	Paper Reading Candidates: Neural features in symbolic models: Weighting transducers with neural context Symbolic features in neural models: Discrete translation lexicons Extracting knowledge: Neural hidden Markov model
4/4,4/6	Advanced Topics 3: Subword Models	Paper Reading Candidates: A Character-Level Decoder without Explicit Segmentation for Neural Machine Translation Neural Machine Translation of Rare Words with Subword Units Factored Translation Models
4/11,4/13	Advanced Topics 4: Multilingual and Multi-task Learning	Paper Reading Candidates: Ask me anything: Dynamic memory networks for NLP Google's multi-lingual neural machine translation system
4/18	Guest Lecture by LP Morency
4/20	School Break, No Class
4/25,4/27	Advanced Topics 5: Ensembling/System Combination	Paper Reading Candidates: Consensus Network Decoding Sequence-level Knowledge Distillation
5/2,5/4	Final Project Discussion