Designing, Visualizing and Understanding Deep Neural Networks

UC Berkeley

Course Description

From the UC Berkeley course catalog:

Deep Networks have revolutionized computer vision, language technology, robotics and control. They have growing impact in many other areas of science and engineering. They do not, however, follow a closed or compact set of theoretical principles. In Yann Lecun’s words they require “an interplay between intuitive insights, theoretical modeling, practical implementations, empirical studies, and scientific analyses.” This course attempts to cover that ground.

Important: In this course, we will use Edstem to post announcements and important information. It is the student’s responsibility to actively monitor the Ed for any important announcements.

Useful course links:

• bCourse: link
• Ed: link
• Gradescope: link

Textbooks (Optional)

• The text "Deep Learning: Foundations & Concepts" by Christopher Bishop & Hugh Bishop is recommended but not required. The free-to-use online version is at Bishop Book
• Dive into Deep Learning D2LAI is an excellent interactive online textbook and set of resources for Deep Learning ! (a PDF version of the entire book is also available online)

Lectures

Lectures are Tuesdays and Thursdays, 6:30PM - 8PM, in 10 Evans or online via Zoom. Lecture slides are provided via this website, and lecture videos are provided via the bCourses “Media Gallery”. Students are responsible for all lecture content.

This Ed post “Lecture Schedule” contains more info about the lecture schedule, including: location (eg 10 Evans vs online Zoom) and lecture recording links.

Here is an optional weekly reading list of supplemental material: link. While this is not required for the course, we believe that the material here can enhance understanding of the course and, more broadly, gain further exposure to the DNN field.

Lecture Slides

• Lecture 01 [Week 1, 2024/08/29] Introduction
  • Bishop Book: Chapter 1
• Lecture 02 [Week 2, 2024/09/03] ML Review Part 1
  • Bishop Book: Chapter 2
  • Reading: Binary classification & logistic regression
• Lecture 03 [Week 2, 2024/09/05] ML Review Part 2
• Lecture 04 [Week 3, 2024/09/10] Neural Networks
• Lecture 05 [Week 3, 2024/09/12] Optimization
  • Bishop Book: Chapter 7
• Lecture 06 [Week 4, 2024/09/17] Building Blocks
  • Python demo code: [tensor_demo.py] [two_layer_linear_nn_demo.py] [normalization_motivation.py] [batch_norm_demo.py] [layer_norm_demo.py] [two_layer_nn_bells_and_whistles.py]
• Lecture 07 [Week 4, 2024/09/19] Convolutional Neural Networks ("ConvNets")
  • Bishop Book: Chapter 10
• Lecture 08 [Week 5, 2024/09/24] Backprop Part 2
• Lecture 09 [Week 5, 2024/09/26] Recurrent Neural Networks ("RNNs")
• Lecture 10 [Week 06, 2024/10/01] Computer Vision
• Lecture 11 [Week 06, 2024/10/03] Transformers - Part 1
• Lecture 12 [Week 07, 2024/10/08] Transformers - Part 2
• Lecture 13 [Week 07, 2024/10/10] Transformers - Part 3
  • Transformers Notebook (Attention, Decoder-Encoder)
• Lecture 14 [Week 08, 2024/10/15] Transformers - Part 4
• Lecture 15 [Week 08, 2024/10/17] MidTerm Review
• Lecture N/A [Week 09, 2024/10/22] MidTerm
• Lecture N/A [Week 09, 2024/10/24] No lecture
• Lecture N/A [Week 10, 2024/10/29] No lecture
• Lecture 16 [Week 10, 2024/10/31] NLP: Pretraining
  • Pretraining word2vec
• Lecture 17 [Week 11, 2024/11/05] Visual Transformer, Masked Autoencoder
• Lecture 18 [Week 10, 2024/10/31] NLP Pretraining: Deeply Encoded Representations
  • GPT (Radford et al 2018)
  • GPT (Devlin et al 2019)
  • InstructGPT (Ouyang et al 2022)
• Lecture 19 [Week 12, 2024/11/12] Accelerating and scaling DNN training (GPU)
  • Guest Talk. Speaker: William Chen (TA), Robotics and DL
• Lecture 20 [Week 12, 2024/11/14] Final Project: Overview and Context
  • Assessing Software Fault Risk with Machine Learning
• Lecture 21 [Week 13, 2024/11/19] Recommendation Systems
• Lecture 22 [Week 13, 2024/11/21] Guest Talk. Speaker: Yuxi Liu (TA), Neural Networks: 1900 - 1990
• Lecture 23 [Week 14, 2024/11/26] Guest Talk. Speaker: Jinjian Liu (Tutor), Repository to Environment
• Lecture N/A [Week 14, 2024/11/28] No lecture (Thanksgiving holiday)
• Lecture 24 [Week 15, 2024/12/03] Speaker: Eric Kim, Closing Lecture (1/2)
• Lecture 25 [Week 15, 2024/12/05] Principled structures in deep learning-based autoregressive modeling of high-dimensional multi-scale chaotic dynamical systems. Guest Lecture on Deep Learning for Climate Modeling by Prof Ashesh Chattopadhyay of UCSC

Discussion Sections

The discussion sections will not cover new material, but rather will give you additional practice solving problems. You can attend any discussion section you like. However, if there are less crowded sections that fit your schedule, those offer more opportunities for you to interact with your TA.

Section Notes

• Discussion 01 (Week 3): Section Notes, Solution

• Discussion 02 (Week 4): Section Notes, Solution

• Discussion 03 (Week 5): Section Notes, Solution. Notebook. Open notebook on Google Colab

• Discussion 04 (Week 6): Section Notes, Solution

• Discussion 05 (Week 7): Section Notes, Solution

• Discussion 06 (Week 8): Section Notes, Solution

• Discussion 07 (Week 9): Section Notes, Solution

• Discussion 08 (Week 11): Section Notes, Solution

• Discussion 09 (Week 12): Section Notes, Solution

• Discussion 10 (Week 13): Section Notes, Solution

Homeworks

All homeworks are graded for accuracy. See the course syllabus for info about collaboration, slip day, late policy.

Homeworks

• Homework 1, due Tues Oct 8th 2024 11:59 PM PST

• Homework 2, due Sun Oct 27th 2024 11:59 PM PST

• Homework 3, due Fri Nov 22nd 2024 11:59 PM PST

• Homework 4, due Sun Dec 15th 2024 11:59 PM PST

• Final Project, due Fri Dec 20th 2024 11:59 PM PST