\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

P02.4 · DataFrames and Series

A free, illustrated module of 4 lessons and one interactive in P02 Scientific Python: NumPy, pandas and Plotting: A Series: values with labels; A DataFrame: columns that share an index; Selecting rows and columns: loc and iloc; Alignment: labels are matched, not positions.

P02.4 L1 · Core 4 lessons · about 100 min Live · in review

Part of P02 Scientific Python: NumPy, pandas and Plotting in the Programming and Data track.

NumPy arrays hold numbers; pandas gives every number a label. These four lessons build a Series by hand as an array plus a dictionary of positions, put three Series side by side as a DataFrame that shares one index, select rows and columns by label with loc and by position with iloc, and show what labels do in arithmetic: pandas matches them before it adds, so values meet the right partner and NaN marks a label with none.

What you need first. Dictionaries and classes (P01.6, P01.9), arrays, shapes and dtypes (P02.1), reductions along an axis and broadcasting (P02.2), and indexing, boolean masks and views (P02.3).

Lessons

1. A Series: values with labels P02.4.1 25 min

   A pandas Series is a NumPy array with a label on every value. Build one by hand from an array and a dictionary of positions, then use pd.Series, loc and iloc, and see why Series.std gives 3.464 where np.std gives 3.0.

2. A DataFrame: columns that share an index P02.4.2 25 min

   A DataFrame is a dictionary of Series that share one index. Build the weather table by hand and with pd.DataFrame, take a first look with shape, dtypes, head, info and describe, add a column, and reduce along each axis.

3. Selecting rows and columns: loc and iloc P02.4.3 25 min

   loc picks rows and columns by label, iloc by position. See why a label slice keeps its end, filter rows with a boolean Series, order them with sort_values, and write with one .loc so that copy-on-write never loses the change.

4. Alignment: labels are matched, not positions P02.4.4 25 min

   When two pandas objects meet in arithmetic, their labels are matched first. Build alignment by hand over the union of labels, see where NaN comes from, centre a table by columns and by rows with sub(axis=0), and use fill_value and reindex.

The interactive

P02.4-X1 · Label aligner. Two short Series whose labels can be taken out, put back or turned round with buttons. Their sum is drawn twice, added by position as NumPy would and matched by label as pandas does, with lines joining the paired rows and NaN where a label has no partner. It lives in Alignment: labels are matched, not positions.