Intro 4 - Neural Network Basics and Toolkit Construction (9/9/2021)
Content:
- Overview of a Neural Networks and Neural Network Toolkits
- Defining Differentiable Functions
- The Forward Algorithm
- The Backward Algorithm
- Parameter Updates
Reading Material
- Reference: An Overview of Gradient Descent Algorithms. (Ruder 2016)
- Reference: The Marginal Value of Adaptive Gradient Methods. (Wilson et al. 2017)
- Reference: Stronger Baselines for Neural MT. (Denkowski Neubig 2017)
- Reference: Dropout. (Srivastava et al. 2014)
- Reference: Dropconnect. (Wan et al. 2013)
- Reference: Marginal Value of Adaptive Gradient Methods (Wilson et al. 2017)
- Reference: A Tour of PyTorch Internals
Slides: NN Toolkit Slides
Video: NN Toolkit Video
Sample Code: minnn-assignment