Assignments

The aim of the assignment and project is to build the skills needed to do build cutting-edge systems or do cutting-edge research, culminating with a project that demonstrates these abilities through a project.

Read all the instructions on this page carefully
You are responsible for reading these instructions and following them carefully. If you do not, you may be marked down as a result.

Assignment Policies

Working in Teams: All projects in this class must be done in teams of 2-3. Individual submissions will not be accepted. If you are having trouble finding a group, the instructor and TAs will help you find one after the first initial survey.

Submission Information: To submit your assignment, send the following to the TA mailing list:

  • your names and andrew IDs
  • a report: This should be up to 5 pages for the assignments, 9 pages for the projects. References are not included in the page count, and it is OK to submit appendices that include supplementary information such as hyperparameter settings or additional output examples, although there is no guarantee that the TAs will read them. Submissions that exceed the page count will be penalized one half grade. Submit this as an attachment to your email. The moment this PDF is delivered to the TA mailing list is the time your assignment is treated as being turned in.
  • a link to a github repository containing your code: Your github repository must be viewable to the TAs and instructor by the submission deadline. If your repository is private make it accessible to us (github IDs neubig, vgtomahawk, rooa, zihangdai, pmichel31415, junjiehu, qizhex). If your repository is not visible to us, your assignment will not be considered complete, so if you are worried please submit well in advance of the deadline so we can confirm the submission is visible.

Late Day Policy: In case there are unforeseen circumstances that don't let you turn in your assignment on time, 5 late days total over the two assignments will be allowed (late days may not be applied to the project). Note that other than these late days, we will not be making exceptions and extending deadlines, so please try to be responsible frugal with your late days and use them only if necessary. Note that the second assignment is harder than the first one, so it'd be a good idea to try to save your late days for the second assignment if possible. Assignments that are late beyond the allowed late days will be graded down one half-grade per day late.

Plagiarism/Code Reuse Policy: All assignments are expected to be conducted under the CMU policy for academic integrity. All rules here apply and violations will be subject to penalty including zero credit on the assignment, failing the course, or other disciplinary measures. In particular, in your implementation:

  • Pseudo-code provided by the TAs or instructor may be used freely without restriction.
  • You may not just re-use an existing implementation written by someone else. The implementation should basically be your own.
  • Fragments of code found online can be used (assuming the license so permits), but if they are significant, please cite these in your report. Failure to do so will be treated as being in violation of the assignment rules.
  • Code written by other students in the class not in your assignment group cannot be used.

Consulting w/ Instructors/TAs: For assignments and projects, you are free to consult as much as you want, any time you want with the instructors and TAs. That is what we're here for, and in no way is this considered cheating. In fact, if you don't have much experience with neural networks previously, it may be necessary to liberally consult with the instructors and TAs to learn about how to do the implementation and finish the assignments. So please do so.

Because this is a project-based course I assume that many of the students taking the course will be interested in turning their assignments or project into research papers. In this case, if you have received useful advice from the instructor or TAs that made the project significantly better, consider inviting them to be co-authors on the paper. Of course, you do not need to do so just because the paper is a result of the class, only if you feel that their advice or help made a contribution.

Details of Each Assignment

Initial Questionnaire and Environment Check (Due 2/5)

At the beginning of the class, you will be asked to turn in two things: an (1) initial questionnaire regarding what task you are interested in tackling and who will be in your group and (2) a log file from a simple benchmark program demonstrating that you have set up an environment where you can run experiments in an expeditious manner. Details of both of these are forthcoming.

Checkpoint Assignment 1 (2/26)

Checkpoint 1 will involves two parts: a survey and a simple baseline implementation. Each part of the assignment will be graded separately, and the final grade for the assignment will be the average of the two grades.

Survey Component: In the survey, explain the task that you would like to tackle in concrete terms, and also cover all of the relevant recent research on the topic. The grading rubric for the survey component is as follows:

  • A+: Exceptional or surprising. Goes far beyond most other submissions.
  • A: A complete survey that covers all the major relevant papers in the field and has a critical analysis of their strengths, weaknesses, or applicability.
  • A-: The survey has a good analysis but is missing a few pieces of relevant related work, or is quite complete but is lacking on critical analysis.
  • B+: The survey is either quite lacking in coverage or analysis, or is decent but not complete in both aspects.
  • B or B-: The survey is lacking in both coverage and analysis, but does make an attempt to cover some related research.
  • C+ or below: Clear lack of effort or incompleteness.

Implementation Component: For your given task, choose a related research paper and find the data set they are using, obtain their dataset, and code a simple neural network that is able to get some accuracy number on this dataset. This number does not have to match previous work and can just be done with a simple and naive model. Analyze the results, and see where your simple baseline is making mistakes. The grading rubric for the implementation component is as follows:

  • A+: Exceptional or surprising. Goes far beyond most other submissions.
  • A: A complete implementation that calculates results on a dataset (for existing tasks, a standard one). An analysis of the results, and description of how this analysis will guide future development.
  • A-: A complete implementation with result numbers, less analysis or forward-looking plans for development than assignments rewarded an A.
  • B+: An implementation and evaluation numbers exist, but either the implementation, the analysis, or the forward-looking plans for development still have rough edges.
  • B or B-: Two or more of the above three elements are lacking.
  • C+ or below: Clear lack of effort or incompleteness.

Checkpoint Assignment 2 (3/26)

Checkpoint 2 will involve reproducing the evaluation numbers of a state-of-the-art baseline model for the task of interest with code that you have implemented from scratch (e.g., you are not allowed to simply run existing code, nor copy large chunks from an existing implementation of the particular model of interest). In other words, you must get the same numbers as the previous paper on the same dataset.

In your report, also perform an analysis of what remaining errors this model makes (ideally with concrete examples of failure cases), and describe how you plan to create a new model for the final project that will address these error cases. If you are interested in tackling a different task in the final project, you may also describe how you adopted the existing model to this new task and perform your error analysis on the new task (although you must report results on the task that the state-of-the-art model was originally designed for.

The grading rubric for this checkpoint is as follows:

  • A+: Exceptional or surprising. Goes far beyond most other submissions.
  • A: A complete re-implementation that meets or exceeds the state of the art. An analysis of the results, and forward-looking plans for further development.
  • A-: Similarly, a complete re-implementation with competitive result numbers, but less analysis or forward-looking plans for development than assignments rewarded an A.
  • B+: An implementation and evaluation numbers exist, but they do not match previous work in the field. Or the analysis or forward-looking plans may be seriously lacking.
  • B or B-: Two or more of the above three elements are lacking.
  • C+ or below: Clear lack of effort or incompleteness.

Final Project (Due 5/7)

The final project work will be expected to be a novel research contribution that either (1) introduces new techniques for one of the existing tasks in the assignment using a significant amount of technical sophistication utilizing one of the more advanced techniques introduced in the class, or (2) tackles a new NLP task with a neural network model that is motivated by the unique problems posed by the application domain. The grading rubric is as follows:

  • A+: Exceptional or surprising. Goes far beyond most other submissions.
  • A: A respectable research contribution that is novel and effective, and could be submitted largely as-is as a paper to an academic conference.
  • A-: A respectable research contribution that has some small incomplete parts, but is largely complete and promising.
  • B+: An idea that is novel, but the results may not be there yet, or the analysis is short.
  • B or B-: Results, analysis, or novelty are lacking.
  • C+ or below: Clear lack of effort or incompleteness.

Suggested Tasks

Below is a list of suggested NLP tasks that you may use for your assignments and projects. It is completely fine, and highly encouraged, to tackle other tasks, but you must confirm with the instructor/TAs in the initial questionnaire (or if you decide to change after the questionnaire, please re-file your questionnaire with the new task details, then send us an email).