Deep Neural Networks

This course will aim to introduce students to the core fundamentals of modern deep multi-layered neural networks, while still remaining grounded in practice. The underpinning assumption in its design is that while students may have experience (especially hands-on experience) in machine learning, data science or general software engineering — they have not worked with deep learning or taken prior courses in the area.

Logistics

  • Lecturer: Nicholas D. Lane
  • Compute: Access to GPU resources is kindly provided by Google Compute Cloud credits offered to students for free.

Sessions

  • November 19th - 23rd, 2018
  • March 11th - 15th, 2019
  • November TBA, 2019
  • March TBA, 2020

Objectives

At the conclusion of this course students should understand:

  1. The principles and approaches for learning with deep neural networks.
  2. The main variants of deep learning (such convolutional and recurrent architectures), and their typical applications.
  3. The key concepts, issues and practices when training and modeling with deep architectures; as well as have hands-on experience in using deep learning frameworks for this purpose.
  4. How to implement basic versions of some of the core deep network algorithms (such as back-propagation)
  5. How deep learning fits within the context of other machine learning approaches, and what tasks it is considered to be suited and not well suited to perform.

Contents

  1. Introduction to Neural Networks and Deep Learning: Background, History and Intuition
  2. Supervised Training Methods, and Basic Architectures: Multi-layer Perceptrons, Backpropagation, Stochastic Gradient Descent
  3. Convolutional Networks and Image Applications
  4. Unsupervised Methods and Related Architectures: Autoencoders and Generative Adversarial Networks
  5. Recurrent Networks and Time-series/Sequential Applications
  6. Modeling Audio, and Case studies in Speech and Sounds: Data Augmentation and Transfer Learning
  7. Practical Implications due to Hardware, Systems and the Cloud: Scalability and Efficiency of Training and Inference

Requirements

Students registering for this course must satisfy three area of background knowledge.

  1. Mathematical Foundations — Undergraduate courses in the following topics: calculus, probability and linear algebra
  2. Programming Skills — Undergraduate level programming courses or experience that indicates programming proficiency
  3. Machine Learning Experience — Students are required to have taken at least one prior course in machine learning at an introductory level during earlier study. Such a course should have provided basic machine learning concepts and ideas, as well as described some popular modelling approaches (at the time the course was taken). There is no expectation this course would have covered any content related to deep learning or types of neural networks.

Professional Masters Program

We offer this course as part of the Profession Masters Program at the Computer Science Department (formerly known as the Software Engineering Program – SEP). Further details of this program can be found here, with the other courses on this program listed here.

Practicals