\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

Linear independence

A set of vectors is linearly dependent when one of them is spare: it can be built from the others. Learn the two ways to say this, prove that they are the same statement, test small sets by hand and in NumPy, and see why spare columns make a model’s weights ambiguous.

M02.2.3L1 Core25 minIn review

Before this lesson

M02.2.1 Linear combinations M02.2.2 Span: everything you can reach

After this lesson you can

• State the definition of linear independence in terms of the combinations that give the zero vector.
• Prove that a set is dependent exactly when one of its vectors is a linear combination of the others.
• Test small sets for independence: two vectors, three vectors in the plane, and three vectors in space.
• Explain why dependent columns give a linear model many sets of weights with identical predictions.

In review. The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

The question

Record a temperature in Celsius and again in Fahrenheit, and give a linear model both columns, together with the column of ones that lets a model add a constant to every prediction. The model now has three weights to tune, but the three columns do not do three different jobs: one of them can be built from the other two. How do we say precisely that a set of vectors contains a spare, and how do we test for one?

From the last lesson

1. What is the span of a set of vectors?

Show answer

The set of all their linear combinations: everything you can reach by scaling them and adding.

2. When does adding a vector \(\vect{w}\) to a set leave its span unchanged?

Show answer

Exactly when \(\vect{w}\) is already in the span of the set.

Intuition

The last lesson ended with vectors that add nothing: a second vector on the line of the first, a third vector in the sheet of the other two. A set that contains such a spare vector is called linearly dependent. A set with no spare, in which every vector contributes a direction of its own, is linearly independent.

M02.2.3-F1

Two views of one dependent set. Left: \(\vect{w} = (3, 0)\) is spare, because \({2\,\vect{u}} + {1\,\vect{v}}\) already reaches its tip. Right: the same fact as a walk that returns to its start without standing still.

Both panels say the same thing about the vectors \(\vect{u} = (1, -1)\), \(\vect{v} = (1, 2)\) and \(\vect{w} = (3, 0)\). On the left, one vector is a combination of the others. On the right, a combination with weights that are not all zero comes back to the origin. This lesson shows that the two views always go together.

Build it up

Step 1. Two vectors. Two vectors are dependent exactly when one is a multiple of the other, so that both lie on one line through the origin. If \(\vect{v} = {k\,\vect{u}}\), then \(\vect{v}\) adds nothing to the span of \(\vect{u}\).

Step 2. The zero vector spoils any set. The zero vector is a combination of any other vectors: give them all the weight zero. So a set that contains \(\vect{0}\) always has a spare.

M02.2.3-F2

Three small sets, tested by eye. Two vectors on different lines are independent. Two vectors on one line are dependent. Any set that contains the zero vector is dependent.

Step 3. Three vectors in the plane. These are always dependent. If the first two are not parallel, they already span the whole plane (lesson M02.2.2), so the third is a combination of them. If the first two are parallel, the set is dependent already.

Step 4. The round trip. Take the spare vector of the figure, \(\vect{w} = {2\,\vect{u}} + {1\,\vect{v}}\), and move everything to one side:

\[ 2\,\vect{u} + 1\,\vect{v} - 1\,\vect{w} = \vect{0}. \]

This is a linear combination that gives the zero vector although its weights, \(2\), \(1\) and \(-1\), are not all zero. Such a combination is called non-trivial. The trivial combination, with every weight zero, gives the zero vector for any set of vectors and tells us nothing.

Step 5. Why the round trip is the better test. “One vector is a combination of the others” makes you guess which vector is the spare. “Some non-trivial combination gives zero” treats all the vectors alike, and it turns into equations that can simply be solved. That is why it is used as the definition.

The formal statement

NoteDefinition: linear independence

The vectors \(\vect{v}_1, \dots, \vect{v}_k\) are linearly independent if the only weights with

\[ c_1\vect{v}_1 + c_2\vect{v}_2 + \cdots + c_k\vect{v}_k = \vect{0} \]

are \(c_1 = c_2 = \cdots = c_k = 0\). Otherwise they are linearly dependent: some weights, not all zero, give the zero vector.

NoteTheorem: the two statements are the same

Let \(k \ge 2\). The vectors \(\vect{v}_1, \dots, \vect{v}_k\) are linearly dependent if and only if at least one of them is a linear combination of the others.

Proof, first direction. Suppose the vectors are dependent, so \(c_1\vect{v}_1 + \cdots + c_k\vect{v}_k = \vect{0}\) with some weight not zero. Call that weight \(c_j\). Move every other term to the right-hand side:

\[ c_j\,\vect{v}_j = -\sum_{i \neq j} c_i\,\vect{v}_i . \]

Because \(c_j \neq 0\) we may divide by it:

\[ \vect{v}_j = \sum_{i \neq j} \left(-\frac{c_i}{c_j}\right) \vect{v}_i . \]

So \(\vect{v}_j\) is a linear combination of the others.

Proof, second direction. Suppose one vector is a combination of the others, say \(\vect{v}_j = \sum_{i \neq j} {d_i\,\vect{v}_i}\). Subtract \(\vect{v}_j\) from both sides:

\[ \sum_{i \neq j} d_i\,\vect{v}_i \;-\; 1\,\vect{v}_j = \vect{0}. \]

This is a combination of all \(k\) vectors that gives zero, and the weight on \(\vect{v}_j\) is \(-1\), which is not zero. So the vectors are dependent. \(\blacksquare\)

The theorem says “at least one”, not “every”. The first direction also shows which vectors can play the spare: any vector that has a non-zero weight in some round trip.

A single vector is a special case with no “others”: the set \(\{\vect{v}\}\) is dependent exactly when \(\vect{v} = \vect{0}\). (If \({c\,\vect{v}} = \vect{0}\) with \(c \neq 0\), multiplying by \(1/c\) gives \(\vect{v} = \vect{0}\).)

Tests for small sets. Each follows from results already proved.

• Any set containing the zero vector is dependent. Give \(\vect{0}\) the weight 1 and every other vector the weight 0.
• Two vectors in the plane are independent exactly when \(D = u_1 v_2 - u_2 v_1 \neq 0\). If \(D \neq 0\), each target is reached by exactly one pair of weights (lesson M02.2.1). The target \(\vect{0}\) is reached by the weights \(0, 0\), so by no others. If \(D = 0\), then \(\vect{u} = \vect{0}\) or \(\vect{v} = {k\,\vect{u}}\) (lesson M02.2.2), and \({1\,\vect{u}} + {0\,\vect{v}} = \vect{0}\) or \({k\,\vect{u}} - {1\,\vect{v}} = \vect{0}\) is a non-trivial combination.
• Three vectors in the plane are always dependent. If \(\vect{u}, \vect{v}\) are dependent, a non-trivial \({a\,\vect{u}} + {b\,\vect{v}} = \vect{0}\) extends to \({a\,\vect{u}} + {b\,\vect{v}} + {0\,\vect{w}} = \vect{0}\). If they are independent, then \(D \neq 0\), so \(\vect{w} = {a\,\vect{u}} + {b\,\vect{v}}\) for some weights, and \({a\,\vect{u}} + {b\,\vect{v}} - {1\,\vect{w}} = \vect{0}\).
• In general, solve. Write \(c_1\vect{v}_1 + \cdots + c_k\vect{v}_k = \vect{0}\) entry by entry and solve the equations. If only the zero weights work, the set is independent.

Why a spare vector matters: recipes stop being unique. Suppose the round trip \(c_1\vect{v}_1 + \cdots + c_k\vect{v}_k = \vect{0}\) is non-trivial, and take any vector \(\vect{x} = a_1\vect{v}_1 + \cdots + a_k\vect{v}_k\) of the span. Adding \(s\) times the round trip changes the weights but not the vector, for every number \(s\):

\[ \begin{aligned} &(a_1 + s c_1)\vect{v}_1 + \cdots + (a_k + s c_k)\vect{v}_k \\ &= \vect{x} + s\,\vect{0} = \vect{x} . \end{aligned} \]

So every vector of the span has infinitely many recipes. For a model, that means infinitely many sets of weights that make identical predictions.

Worked example

Test two sets of three vectors in space. Both use \(\vect{u} = (1, 0, 1)\) and \(\vect{v} = (0, 1, 1)\). The third vector is \(\vect{w} = (1, 1, 0)\) in the first set and \(\vect{w} = (1, 1, 2)\) in the second.

M02.2.3-F3

The same test on two sets. On the left the equations force every weight to be zero. On the right one weight stays free, and choosing it to be 1 produces a round trip.

First set, \(\vect{w} = (1, 1, 0)\). Write out \({a\,\vect{u}} + {b\,\vect{v}} + {c\,\vect{w}} = \vect{0}\):

\[ \begin{aligned} &a\,(1, 0, 1) + b\,(0, 1, 1) + c\,(1, 1, 0) \\ &\;= (a + c,\;\; b + c,\;\; a + b) = (0, 0, 0). \end{aligned} \]

The first entry gives \(a = -c\) and the second gives \(b = -c\). The third entry then reads \(a + b = -2c = 0\), so \(c = 0\), and with it \(a = 0\) and \(b = 0\). Only the zero weights work. The set is independent.

Second set, \(\vect{w} = (1, 1, 2)\). The same steps:

\[ \begin{aligned} &a\,(1, 0, 1) + b\,(0, 1, 1) + c\,(1, 1, 2) \\ &\;= (a + c,\;\; b + c,\;\; a + b + 2c) \\ &\;= (0, 0, 0). \end{aligned} \]

Again \(a = -c\) and \(b = -c\). But now the third entry reads \(-c - c + 2c = 0\), which is true for every \(c\). The weight \(c\) is free. Take \(c = 1\): then \(a = -1\), \(b = -1\), and

\[ \begin{aligned} &-\vect{u} - \vect{v} + \vect{w} \\ &\;= (-1 + 1,\;\; -1 + 1,\;\; -1 - 1 + 2) \\ &\;= (0, 0, 0). \end{aligned} \]

A non-trivial combination gives zero, so the set is dependent. Rearranged, \(\vect{w} = \vect{u} + \vect{v}\): the third vector lies in the sheet of the first two.

The temperature columns. Three readings in Celsius are \(\vect{c} = (0, 10, 20)\). In Fahrenheit they are \(\vect{f} = (32, 50, 68)\), and the column of ones is \(\vect{1} = (1, 1, 1)\). Each Fahrenheit reading is 32 plus 1.8 times the Celsius reading, so the combination \({32\,\vect{1}} + {1.8\,\vect{c}} - {1\,\vect{f}}\) is zero in every entry:

\[ \begin{aligned} 32 \cdot 1 + 1.8 \cdot 0 - 32 &= 0, \\ 32 \cdot 1 + 1.8 \cdot 10 - 50 &= 0, \\ 32 \cdot 1 + 1.8 \cdot 20 - 68 &= 0 . \end{aligned} \]

The round trip has weights \((32,\, 1.8,\, -1)\). So the model \({w_0\,\vect{1}} + {w_1\,\vect{c}} + {w_2\,\vect{f}}\) cannot have unique weights. The weights \((5, 2, 0)\) predict \({5\,\vect{1}} + {2\,\vect{c}} = (5, 25, 45)\). Subtract the round trip once to get the weights \((-27,\, 0.2,\, 1)\), which predict

\[ \begin{aligned} -27 + 0.2 \cdot 0 + 32 &= 5, \\ -27 + 0.2 \cdot 10 + 50 &= 25, \\ -27 + 0.2 \cdot 20 + 68 &= 45 . \end{aligned} \]

Different weights, the same predictions. No amount of data can tell them apart.

Code

From scratch: a round trip is a certificate of dependence, and checking one takes a single combination. For two vectors in the plane the test is the number \(D\).

def combine(weights, vectors):
    """Scale each vector by its weight, then add the results."""
    total = [0] * len(vectors[0])
    for c, v in zip(weights, vectors):
        total = [t + c * x for t, x in zip(total, v)]
    return tuple(total)

def independent_pair(u, v):
    """Two vectors in the plane are independent exactly when D is not zero."""
    return u[0] * v[1] - u[1] * v[0] != 0

u, v, w = (1, -1), (1, 2), (3, 0)
print("2u + 1v - 1w =", combine([2, 1, -1], [u, v, w]))
print("u, v independent:", independent_pair(u, v))
print("(1, 2), (2, 4) independent:", independent_pair((1, 2), (2, 4)))

u3, v3 = (1, 0, 1), (0, 1, 1)
print("-u - v + (1, 1, 2) =", combine([-1, -1, 1], [u3, v3, (1, 1, 2)]))

2u + 1v - 1w = (0, 0)
u, v independent: True
(1, 2), (2, 4) independent: False
-u - v + (1, 1, 2) = (0, 0, 0)

For larger sets, NumPy counts the independent directions among the columns of an array. The count is called the rank (module M02.7 defines it properly). The vectors are independent exactly when the rank equals the number of vectors.

import numpy as np

def independent(vectors):
    columns = np.column_stack(vectors)
    rank = np.linalg.matrix_rank(columns)
    return bool(rank == len(vectors))

print("u, v, (1, 1, 0):", independent([u3, v3, (1, 1, 0)]))
print("u, v, (1, 1, 2):", independent([u3, v3, (1, 1, 2)]))

ones, celsius = np.ones(3), np.array([0.0, 10.0, 20.0])
fahrenheit = 32 + 1.8 * celsius
X = np.column_stack([ones, celsius, fahrenheit])
print("temperature columns independent:", independent([ones, celsius, fahrenheit]))
print("weights (5, 2, 0) predict    ", X @ [5, 2, 0])
print("weights (-27, 0.2, 1) predict", X @ [-27, 0.2, 1])

u, v, (1, 1, 0): True
u, v, (1, 1, 2): False
temperature columns independent: False
weights (5, 2, 0) predict     [ 5. 25. 45.]
weights (-27, 0.2, 1) predict [ 5. 25. 45.]

Columns that are nearly dependent cause trouble too. Here two vectors are almost parallel, and reaching a modest target takes enormous weights of opposite sign.

nearly = np.column_stack([[1.0, 1.0], [1.0, 1.01]])
print("D =", round(1.0 * 1.01 - 1.0 * 1.0, 4))
print("weights for the target (0, 1):", np.linalg.solve(nearly, [0.0, 1.0]).round(6))

D = 0.01
weights for the target (0, 1): [-100.  100.]

Explore

The span explorer in lesson M02.2.2 shows independence directly. While its two vectors are independent the whole plane is shaded and the target has exactly one pair of weights. Drag one vector onto the line of the other: the shading collapses to a line, and the read-out reports either no weights or infinitely many. The Nearly parallel button shows the effect of the last code cell: large weights of opposite sign.

Pitfalls

M02.2.3-F4

Wrong beside right. On the left no two of the three vectors are parallel, yet the set is dependent, because all three lie in one sheet. On the right the third vector leaves the sheet of the other two.

“No two of the vectors are parallel, so the set is independent.”

Checking pairs is not enough. The vectors \((1, 0, 1)\), \((0, 1, 1)\) and \((1, 1, 2)\) point along three different lines, but the third is the sum of the first two. Independence is a property of the whole set.

“Dependent means every vector is a combination of the others.”

Only at least one must be. In the set \((1, 0)\), \((2, 0)\), \((0, 1)\) the second vector is twice the first, so the set is dependent. But \((0, 1)\) is not a combination of the other two, which only reach the horizontal axis.

“The weights \(0, 0, 0\) give the zero vector, so the vectors are dependent.”

All-zero weights give the zero vector for every set. The question is whether any other weights do.

“My columns are not exactly dependent, so I am safe.”

Nearly dependent columns are a real hazard in machine learning, known as multicollinearity. As the code showed, weights can become huge and nearly cancel. A tiny change in the target then swings them wildly, so the fitted weights mean little even when the predictions are fine.

Check yourself

Recall

1. What does it mean for vectors \(\vect{v}_1, \dots, \vect{v}_k\) to be linearly independent?

Show answer

The only weights with \(c_1\vect{v}_1 + \cdots + c_k\vect{v}_k = \vect{0}\) are \(c_1 = \cdots = c_k = 0\).

2. What is the other way to say that a set of two or more vectors is dependent?

Show answer

At least one of the vectors is a linear combination of the others.

3. When are two vectors dependent?

Show answer

When one is a multiple of the other, so that both lie on one line through the origin. For vectors in the plane this is the same as \(D = u_1 v_2 - u_2 v_1 = 0\).

4. Why is any set that contains the zero vector dependent?

Show answer

Give the zero vector the weight 1 and every other vector the weight 0. The combination is the zero vector, and its weights are not all zero.

5. How many vectors in the plane can be linearly independent at most?

Show answer

Two. Any three vectors in the plane are dependent.

Exercises

1. ★ Decide whether each set is independent. For a dependent set, give a round trip. (a) \((1, 2), (3, 6)\) (b) \((1, 2), (2, 1)\) (c) \((1, 2), (2, 1), (5, 7)\).

Show solution

(a) Dependent: \(D = {1 \cdot 6} - {2 \cdot 3} = 0\), and \({3\,(1, 2)} - {1\,(3, 6)} = (0, 0)\).

(b) Independent: \(D = {1 \cdot 1} - {2 \cdot 2} = -3 \neq 0\).

(c) Dependent, as any three vectors in the plane are. To find the round trip, write \((5, 7)\) as \({a\,(1, 2)} + {b\,(2, 1)}\): the equations \(a + 2b = 5\) and \(2a + b = 7\) give \(a = 3\), \(b = 1\). So

\[ \begin{aligned} &3\,(1, 2) + 1\,(2, 1) - 1\,(5, 7) \\ &\;= (3 + 2 - 5,\; 6 + 1 - 7) \\ &\;= (0, 0). \end{aligned} \]

2. ★★ Are \((1, 1, 0)\), \((0, 1, 1)\), \((1, 0, 1)\) independent? What about \((1, 1, 0)\), \((0, 1, 1)\), \((1, 2, 1)\)?

Show solution

First set:

\[ \begin{aligned} &a\,(1, 1, 0) + b\,(0, 1, 1) \\ &\qquad + c\,(1, 0, 1) \\ &\;= (a + c,\; a + b,\; b + c) \\ &\;= (0, 0, 0). \end{aligned} \]

The first entry gives \(a = -c\). The second gives \(b = -a = c\). The third then reads \(b + c = 2c = 0\), so \(c = 0\) and \(a = b = 0\). Independent.

Second set: \((1, 2, 1) = (1, 1, 0) + (0, 1, 1)\), so \({1\,(1, 1, 0)} + {1\,(0, 1, 1)} - {1\,(1, 2, 1)} = (0, 0, 0)\). Dependent.

3. ★★★ Suppose \(\vect{v}_1, \dots, \vect{v}_k\) are independent and \(\vect{w}\) is not in their span. Prove that \(\vect{v}_1, \dots, \vect{v}_k, \vect{w}\) are independent.

Show solution

Suppose \(c_1\vect{v}_1 + \cdots + c_k\vect{v}_k + {c\,\vect{w}} = \vect{0}\). We must show that every weight is zero.

If \(c \neq 0\), we could solve for \(\vect{w}\): \(\vect{w} = -\tfrac{c_1}{c}\vect{v}_1 - \cdots - \tfrac{c_k}{c}\vect{v}_k\). That would put \(\vect{w}\) in the span of the others, which it is not. So \(c = 0\).

The equation is now \(c_1\vect{v}_1 + \cdots + c_k\vect{v}_k = \vect{0}\). Because \(\vect{v}_1, \dots, \vect{v}_k\) are independent, \(c_1 = \cdots = c_k = 0\).

All the weights are zero, so the larger set is independent. This is the precise form of “a vector outside the span adds a new direction”.

Summary card

Dependent means one vector is spare. Independent means none is.

M02.2.3-F1

A spare vector is a round trip.

• Independent. \(c_1\vect{v}_1 + \cdots + c_k\vect{v}_k = \vect{0}\) only when every \(c_i = 0\).
• Dependent. Some weights, not all zero, give \(\vect{0}\). Equivalently, at least one vector is a combination of the others.
• Two vectors in the plane. Independent exactly when \(D = u_1 v_2 - u_2 v_1 \neq 0\).
• Always dependent. Any set containing \(\vect{0}\), and any three vectors in the plane.
• Consequence. In a dependent set, every vector of the span has infinitely many recipes.

Sources and next steps

Sources. Each was opened and checked against this lesson on 12 November 2026.

• Sheldon Axler, Linear Algebra Done Right, 4th edition (open access). Section 2A: definition 2.15 (linearly independent), definition 2.17 (linearly dependent), example 2.16 (a list of two vectors is independent exactly when neither is a multiple of the other), example 2.18 (a list containing the zero vector is dependent) and the linear dependence lemma 2.19.
• Dan Margalit and Joseph Rabinoff, Interactive Linear Algebra, Section 2.5, Linear Independence. The theorem that a set is linearly dependent if and only if one of the vectors is in the span of the other ones, and the facts about two collinear vectors and sets containing the zero vector.
• Marc Deisenroth, Aldo Faisal and Cheng Soon Ong, Mathematics for Machine Learning (free PDF). Section 2.5, definition 2.12 (linear dependence and independence, through non-trivial combinations) and the remark that lists the properties used here.
• Grant Sanderson, Linear combinations, span, and basis vectors, 3Blue1Brown, Linear Algebra, chapter 2. Vectors are linearly dependent when one can be removed without reducing the span.
• scikit-learn user guide, Section 1.1.1, Ordinary Least Squares. When columns of the design matrix are approximately linearly dependent, the least-squares estimate becomes highly sensitive to random errors in the target: multicollinearity.
• NumPy reference, numpy.linalg.matrix_rank: the rank is the number of singular values greater than a tolerance.

Next. M02.2.4 Basis and dimension asks for the two good properties at once: a set that spans everything and has no spare.

This lesson unlocks

M02.2.4 Basis and dimension

Your progress is saved in this browser only. No account, no tracking.