\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

DL01.9 · Datasets, loaders and batching

A free, illustrated module of 4 lessons and one interactive in DL01 Neural Network Foundations: A dataset is something you can index; Batching from scratch, then DataLoader; Batch size and the noise of a gradient; Splits, stratification and samplers.

DL01.9 L2 · Intermediate 4 lessons · about 110 min Live · in review

Part of DL01 Neural Network Foundations in the Deep Learning track.

DL01.8 used a DataLoader as a black box. These four lessons open it up on the course’s image dataset, the 1,797 digits of scikit-learn: a dataset is anything that can say how many items it holds and hand over item i; a loader shuffles the indices, cuts them into batches and stacks the items; the batch size sets how far each batch gradient strays from the full gradient, a spread that falls as 1/|B|; and the split of the digits into 1,077 training, 360 validation and 360 test images is fixed once, stratified, with samplers and class weights for a rare class.

What you need first. The training loop of DL01.8 and PyTorch tensors, losses and optimizers (DL01.7). Mini-batch gradient descent and why its gradient is unbiased (ML01.5.4), and splits without leakage (ML01.2.4).

Lessons

1. A dataset is something you can index DL01.9.1 25 min

   PyTorch’s map-style dataset asks only two things of your data: how many items there are, and what item i is. Write one from scratch over the 1,797 digit images of scikit-learn, with the right shapes and dtypes, then meet TensorDataset and decide where a transform belongs.

2. Batching from scratch, then DataLoader DL01.9.2 25 min

   Shuffle the indices with a seeded generator, cut them into batches, keep or drop the short last one, and stack the items: a loader in fifteen lines. Then DataLoader does the same job, batch for batch, and its default collate function is opened up.

3. Batch size and the noise of a gradient DL01.9.3 30 min

   A batch gradient is right on average but wrong in every single draw. Derive how its spread around the full gradient falls as 1/|B|, with the correction for sampling without replacement, measure it on the digits network, and weigh the cost of an epoch against the updates it buys.

4. Splits, stratification and samplers DL01.9.4 30 min

   Fix the course’s split of the digits once: stratified twice into 1,077 training, 360 validation and 360 test images. See why random_split does not stratify, then let samplers and class weights rescue a class cut to a tenth of its images.

The interactive

DL01.9-X1 · Epoch shuffler. Twenty-four examples as tiles. Set the batch size, switch shuffle and drop_last, and step through an epoch to see which tiles form each batch, while each batch’s step is drawn beside the full-batch step on a two-parameter loss and the read-out compares their spread with sigma^2/|B| times (n - |B|)/(n - 1). It lives in Batch size and the noise of a gradient.