CS 11-711: Advanced NLP

Assignments

The aim of the assignment and project is to build basic understanding and advanced implementation skills needed to do build cutting-edge systems or do cutting-edge research using neural networks fro NLP, culminating with a project that demonstrates these abilities through a project.

Read all the instructions on this page carefully
You are responsible for reading these instructions and following them carefully. If you do not, you may be marked down as a result.

Assignment Policies

Working in Teams: There are 4 assignments in the class. Assignment 1 must be done individually, while Assignments 2, 3, and 4 must be done in teams of 2-3 (individual submissions will not be accepted for these assignments). If you are having trouble finding a group, the instructor and TAs will help you find one after the first initial survey.

Submission Information: To submit your assignment you must submit via canvas a zip file containing:

your code: This should be in a directory "code" in the top directory unless specified otherwise.
system outputs (assignments 1 and 2): The format will be specified separately for each assignment.
a report (assignments 2, 3 and 4, optional for assignment 1): This should be named "report.pdf" in the top directory. This is for assignments 2, 3 and 4, and can be up to 7 pages for assignments 2 and 3 and 9 pages for assignment 4. References are not included in the page count, and it is OK to submit appendices that include supplementary information such as hyperparameter settings or additional output examples, although there is no guarantee that the TAs will read them. Submissions that exceed the page count will be penalized one third grade for each page over (e.g. A to A- or A- to B+). You may also submit report.pdf for assignment 1 if you have any interesting infromation to convey to the TAs, for example if you did anything interesting above and beyond the minimal requirements.
a link to a github repository containing your code (assignments 2, 3 and 4): This should be a single line file "github.txt" in the top directory. Your github repository must be viewable to the TAs in charge of the assignment by the submission deadline. If your repository is private make it accessible to the TAs by the submission deadline. If your repository is not visible to the TAs, your assignment will not be considered complete, so if you are worried please submit well in advance of the deadline so we can confirm the submission is visible. We use this repository to check contributions of all team members.

Late Day Policy: In case there are unforeseen circumstances that don't let you turn in your assignment on time, 5 late days total for assignments 2, 3, and 4 will be allowed. Note that other than these late days, we will not be making exceptions and extending deadlines except for health reasons, so please try to be frugal with your late days and use them only if necessary. Assignments that are late beyond the allowed late days will be graded down one third-grade per day late (e.g. A to A- for one day, and A to B+ for two days).

Plagiarism/Code Reuse Policy: All assignments are expected to be conducted under the CMU policy for academic integrity. All rules here apply and violations will be subject to penalty including zero credit on the assignment, failing the course, or other disciplinary measures. In particular, in your implementation:

Code or pseudo-code provided by the TAs or instructor may be used freely without restriction.
You may not just re-use an existing implementation written by someone else. The implementation should basically be your own.
Code written by other students in the class cannot be used (except, obviously, you can share code within your group for assignments 2, 3, and 4).

If you are doing a similar project for a graded class at CMU (including independent studies or directed research), you must declare so on your report, and note which parts of the project are for 11-711, and which parts are for the other class. Consult with the TA mailing list if you are unsure.

Consulting w/ Instructors/TAs: For assignments and projects, you are free to consult as much as you want, any time you want with the instructors and TAs. That is what we're here for, and in no way is this considered cheating. In fact, if you don't have much experience with NLP previously, it will be helpful to liberally consult with the instructors and TAs to learn about how to do the implementation and finish the assignments. So please do so.

Because this is a project-based course we assume that many of the students taking the course will be interested in turning their assignments or project into research papers. In this case, if you have received useful advice from the instructor or TAs that made the project significantly better, consider inviting them to be co-authors on the paper. Of course, you do not need to do so just because the paper is a result of the class, only if you feel that their advice or help made a contribution.

Details of Each Assignment

Assignment 1: Build Your Own BERT (Due date 10/2, or 9/28 for 3 points extra credit)

Individual assignment. Build Your Own BERT

Assignment 2: End-to-end NLP System Building (submission 10/26, report 10/31)

Group assignment. NLP From Scratch. This will be done as a group.

Assignment 3: Project Proposal State-of-the-art Reimplementation (Due 11/14)

Group assignment. Assignment 3 will involve two parts. (1) You will perform a literature survey on a topic of interest, and propose a project topic based on this literature survey. (2) You will reproduce the evaluation numbers of a competitive baseline model for a task related to this project topic (not necessarily the same). In other words, you must get the same numbers as the previous paper on the same dataset.

In your report, perform an analysis of what remaining errors this model makes (ideally with concrete examples of failure cases), and describe how what you plan to do in the final project will improve on this. If you are interested in tackling a task that does not have a neural baseline in the final project, you may also describe how you plan to adapt this model to the new task and, based on your error analysis, what difficulties you predict in doing so.

The grading rubric for the project proposal component is as follows:

A+: Exceptional or surprising. Goes far beyond most other submissions.
A: A survey that covers all the major relevant papers in the field and a well-grounded project proposal based on this survey.
A-: The survey has a good analysis but is missing a few pieces of relevant related work, or is quite complete but is lacking in critical analysis or forward directions.
B+: The survey is either quite lacking in coverage or analysis, or is decent but not complete in both aspects.
B or B-: The survey is lacking in both coverage and analysis, but does make an attempt to cover some related research.
C+ or below: Clear lack of effort or incompleteness.

The grading rubric for the reproduction component is as follows:

A+: Exceptional or surprising. Goes far beyond most other submissions.
A: Numbers that meet or exceeds the previously reported results. A comprehensive analysis of the results, and forward-looking plans for further development.
A-: Similarly, a complete re-implementation with competitive result numbers, but less analysis or forward-looking plans for development than assignments rewarded an A.
B+: An implementation and evaluation numbers exist, but they do not match previous work in the field. Or the analysis or forward-looking plans may be seriously lacking.
B or B-: Two or more of the above three elements are lacking.
C+ or below: Clear lack of effort or incompleteness.

Assignment 4: Final Project (Due 12/9)

Group assignment. The final project work will be expected to be a novel contribution that either (1) introduces new techniques for one of the existing tasks in the assignment using a significant amount of technical sophistication utilizing one of the more advanced techniques introduced in the class, or (2) tackles a new NLP task with an NLP model that is motivated by the unique problems posed by the application domain, (3) applies an existing NLP method to a new language or domain with improvements specifically tailored to the unique challenges posed by that language or domain. The grading rubric is as follows:

A+: Exceptional or surprising. Goes far beyond most other submissions.
A: A respectable research contribution that is novel and effective, and could be submitted largely as-is as a paper to an academic conference.
A-: A respectable research contribution that has some small incomplete parts, but is largely complete and promising.
B+: An idea that is novel, but the results may not be there yet, or the analysis is short.
B or B-: Results, analysis, or novelty are lacking.
C+ or below: Clear lack of effort or incompleteness.

Negative Results: Sometimes experiments don’t work as planned. If you try hard to get positive results but are not successful, you may still get a good grade by clearly describing why you thought your methods would work, and then performing an analysis of why your initial assumptions were incorrect, leading to results that did not match your initial expectations. The bar for paper writing, experimentation, and analysis will be a bit higher in these cases, as we want to make sure that you really made a serious effort.

Example Tasks

Below is a list of suggested NLP tasks that you may use for your assignments and projects. It is completely fine, and highly encouraged, to tackle other tasks, but you must confirm with the instructor/TAs in the initial questionnaire (or if you decide to change after the questionnaire, please send us an email).