\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

M03.10 · Tensors and index notation for deep learning

A free, illustrated module of 4 lessons and one interactive in M03 Linear Algebra II: Decompositions: Tensors as arrays with many axes; Index notation and the summation convention; einsum: any contraction from scratch; Tensors inside a network layer.

M03.10 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of M03 Linear Algebra II: Decompositions in the Mathematics track.

Deep learning moves numbers around in arrays with many axes: batches of vectors, of sequences, of images, of attention heads. These four lessons give those arrays a clear meaning (axes, shape and order, and how to move axes without scrambling the numbers), a compact language for every sum over them (index notation and the summation convention), one function that computes any such sum (einsum, built from scratch first), and then use all three to read a dense layer, multi-head attention scores and a low-rank weight matrix shape by shape.

What you need first. Matrix multiplication (M02.5), rank (M02.7), transposes, traces and quadratic forms (M02 and M03.4), NumPy arrays, reshape and broadcasting (P02.1 and P02.2). The last lesson ties back to low-rank approximation (M03.6).

Lessons

1. Tensors as arrays with many axes M03.10.1 25 min

   In deep learning a tensor is an array with several named axes: a batch of vectors, of sequences, of images. Learn shape and order, move axes with transpose and moveaxis, and see why reshape cannot do that job.

2. Index notation and the summation convention M03.10.2 25 min

   Write any product of matrices and vectors as letters with indices, drop the summation sign, and prove identities by moving letters: transposes of products, traces, quadratic forms and the gradient of a quadratic form.

3. einsum: any contraction from scratch M03.10.3 25 min

   Turn an index expression into a subscript string such as ij,jk->ik, build einsum from scratch with one loop over every index, then use numpy.einsum, batched matmul and einsum_path, and see why the order of a chain of products can change its cost a hundredfold.

4. Tensors inside a network layer M03.10.4 25 min

   Read a dense layer, multi-head attention scores and a low-rank weight matrix as contractions with named axes: every reshape, transpose, einsum and softmax with its shape, worked by hand on two tokens and two heads, then in NumPy.

The interactive

M03.10-X1 · einsum explorer. Pick or type a subscript string such as ij,jk->ik and set the size of every letter. Each operand is drawn as a block whose axes carry their letters, the summed letters are marked, and the read-out gives the formula, the output shape and the number of multiply-adds, or says why the string is invalid. It lives in einsum: any contraction from scratch.