In deep learning a tensor is an array with several named axes: a batch of vectors, of sequences, of images. Learn shape and order, move axes with transpose and moveaxis, and see why reshape cannot do that job.
M03.10.1L2 Intermediate25 minIn review (what this means)
The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.
Read the shape of a deep-learning tensor, such as (B, T, d) or (B, C, H, W), naming what each axis indexes, and count its entries.
Tell the order of an array (its number of axes) from the rank of a matrix, with an example where they differ.
Choose the axes tuple for numpy.transpose, or the call to numpy.moveaxis, that turns one axis order into another.
Explain, by the reading order, why reshape cannot move an axis, and predict what it produces instead.
The question
A library hands you a batch of colour images as an array of shape (32, 3, 28, 28): 32 images, 3 colour channels, 28 by 28 pixels. The next step wants shape (32, 28, 28, 3). The call x.reshape(32, 28, 28, 3) runs without an error and gives exactly that shape, and every picture in it is scrambled. What went wrong, and what is the right call?
Intuition
A matrix has two axes: rows and columns. The arrays that flow through a neural network have more. A batch of sentences has one axis for the sentence, one for the position of a word in it and one for the numbers that describe each word. A batch of images has four.
M03.10.1-F1
The shapes of deep learning. Each new axis adds one more index you need to pick out a single number. The letters are the index names used in this module: \(b\) for the example in a batch, \(t\) for the position in a sequence, \(e\) for the embedding (feature) entry, \(c\), \(h\), \(w\) for channel, row and column of an image.
In deep learning such an array is called a tensor. Each axis has a size, and the list of sizes, in order, is the shape. The number of axes is the order. To pick out one number you give one index per axis.
The axes are not interchangeable. In shape (32, 3, 28, 28) the second axis means “colour channel” and the third means “pixel row”. Changing the order of the axes means moving every number to a new address, so that the same number is found under the new names. That is a job for a transpose, not for a reshape.
Build it up
Step 1. An entry needs one index per axis. Take the order-3 array of shape (2, 3, 4) that holds the numbers 0 to 23 in order. In code, indices start at 0. The entry x[1, 2, 0] sits in layer 1, row 2, column 0.
M03.10.1-F2
An order-3 array of shape (2, 3, 4), drawn as two layers. The entry x[1, 2, 0] is 20. In the reading order, layer 1 starts after 12 entries and row 2 after 8 more, so it sits at position 20.
Step 2. The size is the product of the sizes. Shape (2, 3, 4) holds \({2 \cdot 3 \cdot 4} = 24\) numbers. A batch of shape (32, 3, 28, 28) holds 75,264.
Step 3. The reading order. NumPy stores the entries in one line, with the last index changing fastest: x[0, 0, 0], x[0, 0, 1], and so on. This is the reading order of P02.1.3. For shape (2, 3, 4) one step in the first index skips \({3 \cdot 4} = 12\) places and one step in the second skips 4.
Step 4. Two kinds of change. A reshape keeps the line of numbers and cuts it into a new shape. A transpose keeps every number’s meaning and gives the axes a new order. The first changes the shape without moving a single number in the line. The second moves numbers in the line, because their addresses change.
A tensor of order\(k\) is an array of real numbers \(x_{i_1 i_2 \dots i_k}\) with one index per axis; index \(i_m\) runs from 1 to \(n_m\) (from 0 to \(n_m - 1\) in code). Its shape is \((n_1, \dots, n_k)\) and its size is the product \(n_1 n_2 \cdots n_k\).
A permutation of the axes is an ordering \((p_1, \dots, p_k)\) of the numbers \(1, \dots, k\). It turns x into the array y of shape \((n_{p_1}, \dots, n_{p_k})\) given by
In words: output axis \(m\)is input axis \(p_m\), and every entry is copied to the address that names the same position along every axis. NumPy writes this np.transpose(x, axes), with the tuple counted from 0. With no tuple it reverses all the axes, which for a matrix is the ordinary transpose. np.moveaxis(x, source, destination) is the special case that moves one axis and keeps the others in their old order.
Order is not rank. The order of an array counts its axes. The rank of M02.7.2 counts the independent columns of a matrix. Two matrices of the same order can have different ranks, and an order-3 array has no matrix rank at all. Older texts and some libraries say “rank” for the number of axes; this course says order, to keep the two apart.
M03.10.1-F5
Same order, different rank; and an order-3 array, for which the matrix rank is not defined.
Why reshape cannot move an axis. Take the image of the worked example below, shape (3, 2, 2) in the order (channel, row, column). In the reading order, entry \(x[c, h, w]\) sits at position
\[
\begin{aligned}
p &= (c \cdot 2 + h) \cdot 2 + w \\
&= 4c + 2h + w .
\end{aligned}
\]
The three values of pixel \((0, 0)\) are at positions 0, 4 and 8. A reshape to (2, 2, 3) cuts the line into groups of three, so its “pixel \((0, 0)\)” is positions 0, 1 and 2. Those are the first three red values. A reshape can only cut the line; it can never bring positions 0, 4 and 8 together. A transpose does exactly that.
Go deeper: the mathematician's tensor, and why the two meanings share a name
In mathematics a matrix \(\mat{A}\) also describes a bilinear form, \(\beta(\vect{x}, \vect{y}) = \vect{x}\T \mat{A} \vect{y} = \sum_{i}\sum_{j} a_{ij} x_i y_j\): a function of two vectors that is linear in each when the other is held fixed. The matrix is the table of its values on basis vectors, \(a_{ij} = \beta(\vect{e}_i, \vect{e}_j)\) (Axler 9.4).
An order-3 array does the same for three vectors: \(\tau(\vect{x}, \vect{y}, \vect{z}) = \sum_{i,j,k} t_{ijk}\, x_i y_j z_k\) is linear in each slot separately, a 3-linear form (Axler 9.25), and \(t_{ijk} = \tau(\vect{e}_i, \vect{e}_j,
\vect{e}_k)\). To a mathematician the tensor is the multilinear map, and the array is its description in one basis; change the basis and the array changes by a fixed rule, while the map stays the same. Deep learning keeps the array and mostly forgets the map. The name travelled with the array.
A third meaning is worth knowing so that it does not surprise you: the tensor rank of an order-3 array is the smallest number of outer products \({\vect{x} \otimes \vect{y} \otimes \vect{z}}\) that add up to it (Kolda and Bader, section 3.1). That is yet another number, and it is not the order either.
Worked example
Move the channel axis of one small image to the end. The image has 3 channels (R, G, B) of 2 by 2 pixels, stored channels first as shape (3, 2, 2). The red values are 1 to 4, the green 5 to 8 and the blue 9 to 12.
Find the axes tuple. We want the order (row, column, channel). Count the input axes from 0: channel is 0, row is 1, column is 2.
Output axis 0 is the row, input axis 1.
Output axis 1 is the column, input axis 2.
Output axis 2 is the channel, input axis 0.
So the tuple is \((1, 2, 0)\), and the rule of the definition reads \(y[h, w, c] = x[c, h, w]\).
M03.10.1-F3
Channels first to channels last, entry by entry. Pixel \((0, 0)\) of the output gathers the top-left value of each channel: 1, 5 and 9.
Fill the output. Pixel \((0, 0)\) is \((x[0,0,0],\, x[1,0,0],\, x[2,0,0]) = (1, 5, 9)\). In the same way pixel \((0, 1)\) is \((2, 6, 10)\), pixel \((1, 0)\) is \((3, 7, 11)\) and pixel \((1, 1)\) is \((4, 8, 12)\). Each pixel holds one red, one green and one blue value, as it should.
Undo it. To go back, ask the same question of y: channel is its axis 2, row its axis 0, column its axis 1, so y.transpose(2, 0, 1) restores x.
With a batch. For shape (B, C, H, W) the batch axis stays first and the tuple becomes \((0, 2, 3, 1)\). That is np.moveaxis(x, 1, -1): move axis 1 to the end, keep the rest in order.
Code
First the permutation from scratch: build the output’s shape, then visit every output address and copy the entry it names.
from itertools import productimport numpy as npdef permute(x, axes):"""Output axis m is input axis axes[m]: y[j_0, ..., j_k-1] = x[i] with i[axes[m]] = j_m.""" shape =tuple(x.shape[a] for a in axes) y = np.empty(shape, dtype=x.dtype)for index in product(*(range(n) for n in shape)): source = [0] * x.ndimfor m, a inenumerate(axes): source[a] = index[m] y[index] = x[tuple(source)]return yimage = np.arange(1, 13).reshape(3, 2, 2) # channels first: R = 1..4, G = 5..8, B = 9..12y = permute(image, (1, 2, 0))print("shape", y.shape, " pixel (0, 0):", y[0, 0], " pixel (1, 1):", y[1, 1])print("reshape instead, pixel (0, 0):", image.reshape(2, 2, 3)[0, 0])
transpose: (2, 2, 2, 3) moveaxis agrees: True
matches the scratch version: True
reshape keeps the reading order: True
transpose keeps the reading order: False
default transpose reverses every axis: (2, 2, 3, 2)
x[1, 2, 0] = 20 size 24 order 3 images: 75264
rank of I: 3 rank of ones: 1
np.transpose and np.moveaxis usually return a view: the same numbers in memory, read through a new order of axes. How that works, with strides, is the subject of P08.5.
Explore
This module’s interactive, the einsum explorer, lives in M03.10.3. Try the preset ij->ji there: it is the transpose of this lesson, written as a string of index letters.
Pitfalls
M03.10.1-F4
Wrong beside right. Both calls give shape (2, 2, 3). Only the transpose keeps each pixel’s three colours together.
“The shape is right, so the data is right.”
A reshape to the target shape always succeeds when the sizes multiply to the same total, and it keeps the reading order, so it mixes values from different channels into one pixel. To change the order of axes, use np.transpose with an axes tuple or np.moveaxis. A reshape is right only when it merges or splits neighbouring axes, as in splitting a width of 8 into 2 heads of 4 (lesson M03.10.4).
“.T swaps the last two axes.”
For a matrix it does. For an array of order 3 or more, x.T reverses all the axes: shape (2, 3, 2, 2) becomes (2, 2, 3, 2). To swap only the last two, write np.swapaxes(x, -1, -2) or x.transpose(0, 1, 3, 2).
“A 3 by 3 matrix of ones has rank 2, because it has two axes.”
That is its order. Its rank is 1: every column is the same. A network’s weight matrix has order 2 whatever its rank, and lesson M03.10.4 builds one of small rank on purpose.
Check yourself
Recall
1. What is the order of an array of shape (B, T, d), and how many numbers does it hold?
Show answer
Order 3: it has three axes. It holds \({B \cdot T \cdot d}\) numbers.
2. In the reading order of NumPy, which index changes fastest?
Show answer
The last one. For shape (2, 3, 4), entry x[i, j, l] sits at position \({(i \cdot 3 + j) \cdot 4 + l}\).
3. What does np.transpose(x, (1, 2, 0)) do to an array of shape (3, 2, 2)?
Show answer
Output axis 0 is input axis 1, output axis 1 is input axis 2 and output axis 2 is input axis 0. The shape becomes (2, 2, 3) and \(y[h, w, c] = x[c, h, w]\).
4. Why can a reshape not turn channels-first into channels-last?
Show answer
A reshape keeps the reading order and only cuts the line into a new shape. The three values of one pixel sit at positions 0, 4 and 8, and no cut brings them together.
5. Give a matrix whose order and rank differ.
Show answer
The 3 by 3 matrix of ones: order 2, rank 1. (Any matrix of rank other than 2 will do.)
Exercises
1.★ A batch of 64 colour images of 32 by 32 pixels arrives in PyTorch’s order (N, C, H, W). Give its shape, its size, and the shape after np.moveaxis(x, 1, -1).
Show solution
Shape (64, 3, 32, 32). Size \({64 \cdot 3 \cdot 32 \cdot 32} = 196{,}608\). Moving axis 1 (the channel) to the end gives (64, 32, 32, 3), the order TensorFlow uses by default.
2.★★ A tensor of attention queries has shape (B, T, h, dh) = (2, 5, 4, 8). Find the axes tuple that gives shape (B, h, T, dh), the resulting shape, and the tuple that undoes it.
Show solution
Output axis 0 is the batch (input axis 0), axis 1 the head (input axis 2), axis 2 the position (input axis 1) and axis 3 the head entry (input axis 3): the tuple is \((0, 2, 1, 3)\) and the shape is (2, 4, 5, 8). Applying \((0, 2, 1, 3)\) again swaps axes 1 and 2 back, so it is its own inverse.
3.★★★ Let x be the array of shape (2, 3) holding 0 to 5. Write out x.reshape(3, 2) and x.T, and show they differ. Then prove that for any array of shape (1, n) the transpose and the reshape to (n, 1) are equal.
Show solution
x is \([[0, 1, 2], [3, 4, 5]]\). x.reshape(3, 2) cuts the line 0 to 5 into pairs: \([[0, 1], [2, 3], [4, 5]]\). x.T has \(y[j, i] = x[i, j]\): \([[0, 3], [1, 4], [2, 5]]\). They differ already in entry \([0, 1]\).
Now let x have shape (1, n), and let y = x.T, of shape (n, 1), with \(y[j, 0] = x[0, j]\). The reading order of y runs over \(j = 0, 1, \dots, n - 1\) and lists \(x[0, 0], x[0, 1], \dots, x[0, n - 1]\), which is the reading order of x. An array is fixed by its shape and its reading order, and x.reshape(n, 1) has the same shape and keeps that reading order, so the two are equal. Moving an axis of size 1 never changes the reading order.
Summary card
A tensor is an array with named axes; moving axes is a transpose, never a reshape.
Order is not rank: ones(3, 3) has order 2 and rank 1.
Sources and next steps
Sources. Each was opened and checked against this lesson on 15 November 2026, and the survey of Kolda and Bader on 15 November 2026.
Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, section 2.1: a tensor as “an array of numbers arranged on a regular grid with a variable number of axes”, with an element written \(A_{i,j,k}\).
NumPy reference, numpy.transpose (the \(i\)-th axis of the result is input axis axes[i]; with no tuple the axes are reversed), numpy.moveaxis (other axes remain in their original order) and numpy.reshape (the default C order reads with the last index changing fastest).
PyTorch reference, torch.nn.Conv2d: input of shape (N, Cin, H, W), with N the batch size; TensorFlow reference, tf.nn.conv2d: data_format defaults to “NHWC”, with “NCHW” as the alternative.
Sheldon Axler, Linear Algebra Done Right, 4th edition (open access), section 9A (bilinear forms, 9.1, and the matrix of a bilinear form, 9.4) and section 9B (\(m\)-linear forms, 9.25), for the block on the mathematician’s tensor.
Tamara G. Kolda and Brett W. Bader, Tensor Decompositions and Applications, SIAM Review 51(3), 2009: the order of a tensor is its number of dimensions, also called ways or modes (section 2), and the rank of a tensor is the smallest number of rank-one tensors that sum to it (section 3.1).
![Two blue grids of three rows and four columns, titled layer b equals 0 and layer b equals 1. The first holds 0 to 11 row by row, the second 12 to 23. Row indices 0, 1, 2 are written down the left side and labelled t, and column indices 0 to 3 along the top, labelled e. In the second grid, the cell in row 2, column 0 holds 20 and is shaded yellow, with a pointer reading x[1, 2, 0] = 20. Below: shape (2, 3, 4): 2 times 3 times 4 equals 24 entries; reading order: position 1 times 12 plus 2 times 4 plus 0 equals 20.](figures/M03.10.1-F2.svg)
![Left, in blue, x of shape (3, 2, 2), channels first: three 2 by 2 grids titled c equals 0 (R), c equals 1 (G) and c equals 2 (B), holding 1 2 3 4, 5 6 7 8 and 9 10 11 12, with the top-left cell of each shaded yellow. Right, in vermillion, y of shape (2, 2, 3), channels last: four pixels of three cells each, arranged in rows h equals 0 and 1 and columns w equals 0 and 1, holding 1 5 9, 2 6 10, 3 7 11 and 4 8 12; the pixel 1 5 9 is shaded yellow. A green arrow labelled x.transpose(1, 2, 0) runs from left to right. Below: y[h, w, c] equals x[c, h, w], so pixel (0, 0) collects 1, 5 and 9; every pixel keeps its own three values.](figures/M03.10.1-F3.svg)
