\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

DL10 · Deep Learning Theory and Frontiers

A free, illustrated L4 course of 10 modules in the Deep Learning track. Modules include: Expressivity and depth; Loss landscapes; Implicit bias of gradient descent.

DL10 L4 · Expert 10 modules · about 40 lessons Planned · Ring 4

Part of the Deep Learning track. Ships in Ring 4 (Expert) of the roadmap.

Modules

1. Expressivity and depth
2. Loss landscapes
3. Implicit bias of gradient descent
4. Neural tangent kernel and infinite width
5. Double descent and generalization
6. Grokking and training dynamics
7. Lottery tickets and pruning
8. Continual learning and loss of plasticity
9. Mechanistic interpretability
10. Scaling laws

Signature illustrations

The pictures a learner should still see in their head a year later:

Two-dimensional slices of loss landscapes, double-descent curves, circuit diagrams of learned features, plasticity across a task sequence, log-log scaling plots.

Status

This course is planned, not written yet. It ships with Ring 4 (Expert).

Every lesson will follow the same twelve-part blueprint with at least three purpose-made figures. Watch the repository to hear when modules ship, or help write and review it.