Skip to main content

Understanding Linear Algebra: Data Science Edition

Randall Pruim

Section 3.1 The span of a set of vectors

Matrix multiplication allows us to rewrite a linear system in the form \(A\xvec = \bvec\text{.}\) Besides being a more compact way of expressing a linear system, this form allows us to think about linear systems geometrically since matrix multiplication is defined in terms of linear combinations of vectors.
We now return to our two fundamental questions, rephrased here in terms of matrix multiplication.
  • Existence: Is there a solution to the equation \(A\xvec=\bvec\text{?}\)
  • Uniqueness: If there is a solution to the equation \(A\xvec=\bvec\text{,}\) is it unique?
In this section, we focus on the existence question and see how it leads to the concept of the span of a set of vectors.
Before we get going, let’s load the NumPy and SymPy packages and set some print options to reduce the number of digits displayed. Here we choose to display 4 digits; suppress = True suppresses scientific notation for small values.

Preview Activity 3.1.1. The existence of solutions.

  1. If the equation \(A\xvec = \bvec\) is inconsistent, what can we say about the pivot positions of the augmented matrix \(\left[\begin{array}{r|r} A \amp \bvec \end{array}\right]\text{?}\)
  2. Consider the matrix \(A\)
    \begin{equation*} A = \left[ \begin{array}{rrr} 1 \amp 0 \amp -2 \\ -2 \amp 2 \amp 2 \\ 1 \amp 1 \amp -3 \end{array}\right]\text{.} \end{equation*}
    If \(\bvec=\threevec{2}{2}{5}\text{,}\) is the equation \(A\xvec = \bvec\) consistent? If so, find a solution.
  3. If \(\bvec=\threevec{2}{2}{6}\text{,}\) is the equation \(A\xvec = \bvec\) consistent? If so, find a solution.
  4. Identify the pivot positions of \(A\text{.}\)
  5. For our two choices of the vector \(\bvec\text{,}\) one equation \(A\xvec = \bvec\) has a solution and the other does not. What feature of the pivot positions of the matrix \(A\) tells us to expect this?
Solution.
  1. We know there must be a pivot position in the rightmost column of the augmented matrix.
  2. We construct the augmented matrix
    \begin{equation*} \left[\begin{array}{rrr|r} 1 \amp 0 \amp -2 \amp 2 \\ -2 \amp 2 \amp 2 \amp 2 \\ 1 \amp 1 \amp -3 \amp 5 \end{array}\right] \sim \left[\begin{array}{rrr|r} 1 \amp 0 \amp -2 \amp 2 \\ 0 \amp 1 \amp -1 \amp 3 \\ 0 \amp 0 \amp 0 \amp 0 \end{array}\right]\text{,} \end{equation*}
    which shows that the system is consistent. The solution space is described parametrically as
    \begin{equation*} \xvec=\threevec{2}{3}{0}+x_3\threevec{2}{1}{1}\text{.} \end{equation*}
  3. Now the augmented matrix is
    \begin{equation*} \left[\begin{array}{rrr|r} 1 \amp 0 \amp -2 \amp 2 \\ -2 \amp 2 \amp 2 \amp 2 \\ 1 \amp 1 \amp -3 \amp 6 \end{array}\right] \sim \left[\begin{array}{rrr|r} 1 \amp 0 \amp -2 \amp 0 \\ 0 \amp 1 \amp -1 \amp 0 \\ 0 \amp 0 \amp 0 \amp 1 \end{array}\right] \end{equation*}
    showing that the equation \(A\xvec=\bvec\) is inconsistent.
  4. There are two pivot positions in \(A\text{,}\) as shown.
    \begin{equation*} \left[\begin{array}{rrr} {\mathbf 1} \amp 0 \amp -2 \\ -2 \amp {\mathbf 2} \amp 2 \\ 1 \amp 1 \amp -3 \\ \end{array}\right] \sim \left[\begin{array}{rrr} {\mathbf 1} \amp 0 \amp -2 \\ 0 \amp {\mathbf 1} \amp -1 \\ 0 \amp 0 \amp 0 \\ \end{array}\right]\text{.} \end{equation*}
  5. Since there is a row of \(A\) that does not have a pivot position, it is possible to augment \(A\) by a vector \(\bvec\) so that we obtain a pivot position in the rightmost column of the augmented matrix. In this case, we have an inconsistent system.

Subsection 3.1.1 The span of a set of vectors

In the preview activity, we considered a \(3\by3\) matrix \(A\) and found that the equation \(A\xvec = \bvec\) has a solution for some vectors \(\bvec\) in \(\real^3\) and has no solution for others. We will introduce a concept called span that describes the vectors \(\bvec\) for which there is a solution.
We can write an \(m\by n\) matrix \(A\) in terms of its columns
\begin{equation*} A = \left[\begin{array}{rrrr} \vvec_1\amp\vvec_2\amp\cdots\amp\vvec_n \end{array}\right]\text{.} \end{equation*}
Remember that Proposition 1.4.4 says that the equation \(A\xvec = \bvec\) is consistent if and only if we can express \(\bvec\) as a linear combination of \(\vvec_1,\vvec_2,\ldots,\vvec_n\text{.}\)

Definition 3.1.1.

The span of a set of vectors \(\vvec_1,\vvec_2,\ldots,\vvec_n\) is the set of all linear combinations that can be formed from the vectors.
Alternatively, if \(A = \begin{bmatrix} \vvec_1 \amp \vvec_2 \amp \cdots \amp \vvec_n \end{bmatrix}\text{,}\) then the span of the vectors consists of all vectors \(\bvec\) for which the equation \(A\xvec = \bvec\) is consistent.

Example 3.1.2.

Considering the set of vectors \(\vvec=\twovec{-2}{1}\) and \(\wvec = \twovec{8}{-4}\text{,}\) we see that the vector
\begin{equation*} \bvec = 3\vvec + \wvec = \twovec2{-1} \end{equation*}
is one vector in the span of the vectors \(\vvec\) and \(\wvec\) because it is a linear combination of \(\vvec\) and \(\wvec\text{.}\)
Equivalently, since
\begin{equation*} \begin{bmatrix} -2 \amp 8 \\ 1 \amp -4 \\ \end{bmatrix} \twovec{3}{1} = \twovec{2}{-1} \end{equation*}
we know that
\begin{equation*} \bvec = \twovec2{-1} \end{equation*}
is in the span of \(\vvec\) and \(\wvec\text{:}\)
To determine whether the vector \(\bvec=\twovec{5}{2}\) is in the span of \(\vvec\) and \(\wvec\text{,}\) we form the matrix
\begin{equation*} A = \begin{bmatrix} \vvec \amp \wvec \end{bmatrix} = \begin{bmatrix} -2 \amp 8 \\ 1 \amp -4 \\ \end{bmatrix} \end{equation*}
and consider the equation \(A\xvec=\bvec\text{.}\)
We have
\begin{equation*} \left[ \begin{array}{rr|r} -2 \amp 8 \amp 5 \\ 1 \amp -4 \amp 2 \\ \end{array} \right] \sim \left[ \begin{array}{rr|r} 1 \amp -4 \amp 0 \\ 0 \amp 0 \amp 1 \\ \end{array} \right], \end{equation*}
which shows that the equation \(A\xvec = \bvec\) is inconsistent. Therefore, \(\bvec=\twovec52\) is one vector that is not in the span of \(\vvec\) and \(\wvec\text{.}\)

Activity 3.1.2.

Let’s look at two examples to develop some intuition for the concept of span.
  1. First, we will consider the set of vectors
    \begin{equation*} \vvec = \twovec{1}{2}, ~~~\wvec = \twovec{-2}{-4}\text{.} \end{equation*}
    Instructions.
    The diagram below can be used to construct linear combinations whose weights \(c\) and \(d\) may be varied using the sliders at the top. The vectors \(\vvec\) and \(\wvec\) are outlined while the linear combination
    \begin{equation*} c\vvec + d\wvec \end{equation*}
    is shaded in red.
    Figure 3.1.3. An interactive diagram for constructing linear combinations of the vectors \(\vvec\) and \(\wvec\text{.}\)
    1. What vector is the linear combination of \(\vvec\) and \(\wvec\) with weights:
      • \(c = 2\) and \(d=0\text{?}\)
      • \(c = 1\) and \(d=1\text{?}\)
      • \(c = 0\) and \(d=-1\text{?}\)
    2. Can the vector \(\twovec{2}{4}\) be expressed as a linear combination of \(\vvec\) and \(\wvec\text{?}\) Is the vector \(\twovec{2}{4}\) in the span of \(\vvec\) and \(\wvec\text{?}\)
    3. Can the vector \(\twovec{3}{0}\) be expressed as a linear combination of \(\vvec\) and \(\wvec\text{?}\) Is the vector \(\twovec{3}{0}\) in the span of \(\vvec\) and \(\wvec\text{?}\)
    4. Describe the set of vectors in the span of \(\vvec\) and \(\wvec\text{.}\)
    5. For what vectors \(\bvec\) does the equation
      \begin{equation*} \left[\begin{array}{rr} 1 \amp -2 \\ 2 \amp -4 \end{array}\right] \xvec = \bvec \end{equation*}
      have a solution?
  2. We will now look at an example where
    \begin{equation*} \vvec = \twovec{2}{1}, ~~~\wvec = \twovec{1}{2}\text{.} \end{equation*}
    Instructions.
    In a similar way, the diagram below can be used to construct linear combinations
    \begin{equation*} c\vvec + d\wvec. \end{equation*}
    Figure 3.1.4. An interactive diagram for constructing linear combinations of the vectors \(\vvec\) and \(\wvec\text{.}\)
    1. What vector is the linear combination of \(\vvec\) and \(\wvec\) with weights:
      • \(c = 2\) and \(d=0\text{?}\)
      • \(c = 1\) and \(d=1\text{?}\)
      • \(c = 0\) and \(d=-1\text{?}\)
    2. Can the vector \(\twovec{-2}{2}\) be expressed as a linear combination of \(\vvec\) and \(\wvec\text{?}\) Is the vector \(\twovec{-2}{2}\) in the span of \(\vvec\) and \(\wvec\text{?}\)
    3. Can the vector \(\twovec{3}{0}\) be expressed as a linear combination of \(\vvec\) and \(\wvec\text{?}\) Is the vector \(\twovec{3}{0}\) in the span of \(\vvec\) and \(\wvec\text{?}\)
    4. Describe the set of vectors in the span of \(\vvec\) and \(\wvec\text{.}\)
    5. For what vectors \(\bvec\) does the equation
      \begin{equation*} \left[\begin{array}{rr} 2 \amp 1 \\ 1 \amp 2 \end{array}\right] \xvec = \bvec \end{equation*}
      have a solution?
Solution.
  1. For the first set of vectors, we find:
    1. We can form the linear combinations:
      • When \(c = 2\) and \(d=0\text{,}\) the linear combination is \(\twovec{2}{4}\text{.}\)
      • When \(c = 1\) and \(d=1\text{,}\) the linear combination is \(\twovec{-1}{-2}\text{.}\)
      • When \(c = 0\) and \(d=-1\text{,}\) the linear combination is \(\twovec{2}{4}\text{.}\)
    2. Yes, we saw that there are at least two ways. For instance, when the weights are \(c=2\) and \(d=0\text{.}\) This means that \(\twovec24\) is in the span of \(\vvec\) and \(\wvec\text{.}\)
    3. No. No matter how we change the weights, the linear combination lies on the line through \(\vvec\) and \(\wvec\text{.}\) This means that \(\twovec{3}{0}\) is not in the span of \(\vvec\) and \(\wvec\text{.}\)
    4. The span of \(\vvec\) and \(\wvec\) is the set of all vectors on the line through \(\vvec\text{.}\)
    5. If the equation has a solution, \(\bvec\) must lie on the line defined by \(\vvec\text{.}\)
  2. For the second set of vectors, we have:
    1. We can form the linear combinations:
      • When \(c = 2\) and \(d=0\text{,}\) the linear combination is \(\twovec42\text{.}\)
      • When \(c = 1\) and \(d=1\text{,}\) the linear combination is \(\twovec{3}{3}\text{.}\)
      • When \(c = 0\) and \(d=-1\text{,}\) the linear combination is \(\twovec{-1}{-2}\text{.}\)
    2. Yes. Using the diagram, we see that \(\twovec{-2}{2}=-2\vvec + 2\wvec\text{.}\) This means that \(\twovec{-2}{2}\) is in the span of \(\vvec\) and \(\wvec\text{.}\)
    3. Yes. Using the diagram, we see that \(\twovec{3}{0}=2\vvec-1\wvec\text{.}\) This means that \(\twovec{3}{0}\) is in the span of \(\vvec\) and \(\wvec\text{.}\)
    4. Every two-dimensional vector is in the span of \(\vvec\) and \(\wvec\text{.}\)
    5. The equation has a solution for every \(\bvec\text{.}\)
This activity aims to convey the geometric meaning of span. Remember that we can think of a linear combination of the two vectors \(\vvec\) and \(\wvec\) as a recipe for walking in the plane \(\real^2\text{.}\) We first move a prescribed amount in the direction of \(\vvec\) and then a prescribed amount in the direction of \(\wvec\text{.}\) The span consists of all the places we can walk to.

Example 3.1.5.

Let’s consider the vectors \(\vvec=\twovec20\) and \(\wvec=\twovec{-1}1\) as shown in Figure 3.1.6.
Figure 3.1.6. The vectors \(\vvec\) and \(\wvec\) and some linear combinations they create.
The figure shows us that \(\bvec = \vvec + 2\wvec = \twovec02\) is a linear combination of \(\vvec\) and \(\wvec\text{.}\) Indeed, we can verify this algebraically by solving the matrix equation
\begin{equation*} \begin{bmatrix}\vvec \amp \wvec \end{bmatrix} ~ \xvec = \twovec02, \end{equation*}
whose corresponding augmented matrix has the reduced row echelon form
\begin{equation*} \left[ \begin{array}{rr|r} 2 \amp -1 \amp 0 \\ 0 \amp 1 \amp 2 \\ \end{array} \right] \sim \left[ \begin{array}{rr|r} 1 \amp 0 \amp 1 \\ 0 \amp 1 \amp 2 \\ \end{array} \right]. \end{equation*}
Because this system is consistent, we know that \(\bvec=\twovec02\) is in the span of \(\vvec\) and \(\wvec\text{.}\) The solution \(\xvec = \twovec12\) provides the weights.
In fact, we can say more. Notice that the coefficient matrix
\begin{equation*} \begin{bmatrix} 2 \amp -1 \\ 0 \amp 1 \\ \end{bmatrix} \sim \begin{bmatrix} 1 \amp 0 \\ 0 \amp 1 \\ \end{bmatrix} \end{equation*}
has a pivot position in every row. This means that for any other vector \(\bvec\text{,}\) the augmented matrix corresponding to the equation \(\begin{bmatrix}\vvec \amp \wvec \end{bmatrix} ~\xvec = \bvec\) cannot have a pivot position in its rightmost column:
\begin{equation*} \left[ \begin{array}{rr|r} 2 \amp -1 \amp * \\ 0 \amp 1 \amp * \\ \end{array} \right] \sim \left[ \begin{array}{rr|r} 1 \amp 0 \amp * \\ 0 \amp 1 \amp * \\ \end{array} \right]. \end{equation*}
Therefore, the equation \(\begin{bmatrix}\vvec \amp \wvec \end{bmatrix} ~\xvec = \bvec\) is consistent for every two-dimensional vector \(\bvec\text{,}\) which tells us that every two-dimensional vector is in the span of \(\vvec\) and \(\wvec\text{.}\) In this case, we say that the span of \(\vvec\) and \(\wvec\) is \(\real^2\text{.}\)
The intuitive meaning is that we can walk to any point in the plane by moving an appropriate distance in the \(\vvec\) and \(\wvec\) directions.

Example 3.1.7.

Now let’s consider the vectors \(\vvec=\twovec{-1}1\) and \(\wvec=\twovec2{-2}\) as shown in Figure 3.1.8.
Figure 3.1.8. The vectors \(\vvec\) and \(\wvec\) and some linear combinations they create.
From the figure, we expect that \(\bvec = \twovec02\) is not a linear combination of \(\vvec\) and \(\wvec\text{.}\) Once again, we can verify this algebraically by constructing the linear system
\begin{equation*} \begin{bmatrix}\vvec \amp \wvec \end{bmatrix} ~ \xvec = \twovec02. \end{equation*}
The augmented matrix has the reduced row echelon form
\begin{equation*} \left[ \begin{array}{rr|r} -1 \amp 2 \amp 0 \\ 1 \amp -2 \amp 2 \\ \end{array} \right] \sim \left[ \begin{array}{rr|r} 1 \amp -2 \amp 0 \\ 0 \amp 0 \amp 1 \\ \end{array} \right], \end{equation*}
from which we see that the system is inconsistent. Therefore, \(\bvec=\twovec02\) is not in the span of \(\vvec\) and \(\wvec\text{.}\)
We should expect this behavior from the coefficient matrix
\begin{equation*} \begin{bmatrix} -1 \amp 2 \\ 1 \amp -2 \\ \end{bmatrix} \sim \begin{bmatrix} 1 \amp -2 \\ 0 \amp 0 \\ \end{bmatrix}. \end{equation*}
Because the second row of the coefficient matrix does not have a pivot position, it is possible for a linear system \(\begin{bmatrix}\vvec \amp \wvec \end{bmatrix} ~\xvec = \bvec\) to have a pivot position in its rightmost column:
\begin{equation*} \left[ \begin{array}{rr|r} -1 \amp 2 \amp * \\ 1 \amp -2 \amp * \\ \end{array} \right] \sim \left[ \begin{array}{rr|r} 1 \amp -2 \amp 0 \\ 0 \amp 0 \amp 1 \\ \end{array} \right]. \end{equation*}
If we notice that \(\wvec = -2\vvec\text{,}\) we see that any linear combination of \(\vvec\) and \(\wvec\text{,}\)
\begin{equation*} c\vvec + d\wvec = c\vvec -2d\vvec = (c-2d)\vvec, \end{equation*}
is actually a scalar multiple of \(\vvec\text{.}\) Therefore, the span of \(\vvec\) and \(\wvec\) is the line defined by the vector \(\vvec\text{.}\) Intuitively, this means that we can only walk to points on this line using these two vectors.

Notation 3.1.9.

We will denote the span of the set of vectors \(\vvec_1,\vvec_2,\ldots,\vvec_n\) by \(\laspan{\vvec_1,\vvec_2,\ldots,\vvec_n}\text{.}\)
In Example 3.1.5, we saw that \(\laspan{\vvec,\wvec} = \real^2\text{.}\) However, for the vectors in Example 3.1.7, we saw that \(\laspan{\vvec,\wvec}\) is simply a line.

Subsection 3.1.2 Pivot positions and span

A set of vectors \(\vvec_1,\vvec_2,\ldots,\vvec_n\) naturally defines a matrix \(A = \begin{bmatrix}\vvec_1\amp\vvec_2\amp \cdots \amp \vvec_n\end{bmatrix}\) whose columns are the given vectors. As we’ve seen, a vector \(\bvec\) is in \(\laspan{\vvec_1,\vvec_2,\ldots,\vvec_n}\) precisely when the linear system \(A\xvec=\bvec\) is consistent.
The previous examples point to the fact that the span is related to the pivot positions of \(A\text{.}\) While Section 3.2 and Section 4.6 develop this idea more fully, we will now examine the possibilities in \(\real^3\text{.}\)

Activity 3.1.3.

In this activity, we will look at the span of sets of vectors in \(\real^3\text{.}\)
  1. Suppose \(\vvec=\threevec{1}{2}{1}\text{.}\) Give a geometric description of \(\laspan{\vvec}\) and a rough sketch of \(\vvec\) and its span in Figure 3.1.10.
    Figure 3.1.10. A three-dimensional coordinate system for sketching \(\vvec\) and its span.
  2. Now consider the two vectors
    \begin{equation*} \evec_1 = \threevec{1}{0}{0},~~~ \evec_2 = \threevec{0}{1}{0}\text{.} \end{equation*}
    Sketch the vectors below. Then give a geometric description of \(\laspan{\evec_1,\evec_2}\) and a rough sketch of the span in Figure 3.1.11.
    Figure 3.1.11. A coordinate system for sketching \(\evec_1\text{,}\) \(\evec_2\text{,}\) and \(\laspan{\evec_1,\evec_2}\text{.}\)
  3. Let’s now look at this situation algebraically by writing write \(\bvec = \threevec{b_1}{b_2}{b_3}\text{.}\) Determine the conditions on \(b_1\text{,}\) \(b_2\text{,}\) and \(b_3\) so that \(\bvec\) is in \(\laspan{\evec_1,\evec_2}\) by considering the linear system
    \begin{equation*} \left[\begin{array}{rr} \evec_1 \amp \evec_2 \\ \end{array}\right] ~\xvec = \bvec \end{equation*}
    or
    \begin{equation*} \left[\begin{array}{rr} 1 \amp 0 \\ 0 \amp 1 \\ 0 \amp 0 \\ \end{array}\right] \xvec = \threevec{b_1}{b_2}{b_3}\text{.} \end{equation*}
    Explain how this relates to your sketch of \(\laspan{\evec_1,\evec_2}\text{.}\)
  4. Consider the vectors
    \begin{equation*} \vvec_1 = \threevec{1}{1}{-1},~~ \vvec_2 = \threevec{0}{2}{1}. \end{equation*}
    1. Is the vector \(\bvec=\threevec{1}{-2}{4}\) in \(\laspan{\vvec_1,\vvec_2}\text{?}\)
    2. Is the vector \(\bvec=\threevec{-2}{0}{3}\) in \(\laspan{\vvec_1,\vvec_2}\text{?}\)
    3. Give a geometric description of \(\laspan{\vvec_1,\vvec_2}\text{.}\)
  5. Consider the vectors
    \begin{equation*} \vvec_1 = \threevec{1}{1}{-1}, \vvec_2 = \threevec{0}{2}{1}, \vvec_3 = \threevec{1}{-2}{4}\text{.} \end{equation*}
    Form the matrix \(\left[\begin{array}{rrrr} \vvec_1 \amp \vvec_2 \amp \vvec_3 \end{array}\right]\) and find its reduced row echelon form.
    What does this tell you about \(\laspan{\vvec_1,\vvec_2,\vvec_3}\text{?}\)
  6. If the span of a set of vectors \(\vvec_1,\vvec_2,\ldots,\vvec_n\) is \(\real^3\text{,}\) what can you say about the pivot positions of the matrix \(\left[\begin{array}{rrrr} \vvec_1\amp\vvec_2\amp\ldots\amp\vvec_n \end{array}\right]\text{?}\)
  7. What is the smallest number of vectors such that \(\laspan{\vvec_1,\vvec_2,\ldots,\vvec_n} = \real^3\text{?}\)
Solution.
  1. \(\laspan{\vvec}\) is the line defined by \(\vvec\text{.}\)
  2. \(\laspan{\evec_1,\evec_2}\) is the \(xy\)-plane.
  3. For the linear system to be consistent, we need \(b_3=0\text{,}\) which means that the third coordinate of the vector \(\bvec\) must be 0 for \(\bvec\) to be in \(\laspan{\evec_1,\evec_2}\text{.}\) In other words, \(\bvec\) must lie in the \(xy\)-plane.
  4. We consider the two cases.
    1. We have the augmented matrix
      \begin{equation*} \left[\begin{array}{rr|r} 1 \amp 0 \amp 1 \\ 1 \amp 2 \amp -2 \\ -1 \amp 1 \amp 4 \end{array}\right] \sim \left[\begin{array}{rr|r} 1 \amp 0 \amp 0 \\ 0 \amp 1 \amp 0 \\ 0 \amp 0 \amp 1 \end{array}\right]\text{,} \end{equation*}
      which shows that the system is inconsistent. Therefore, \(\bvec\) is not in \(\laspan{\vvec_1, \vvec_2}\text{.}\)
    2. We have the augmented matrix
      \begin{equation*} \left[\begin{array}{rr|r} 1 \amp 0 \amp -2 \\ 1 \amp 2 \amp 0 \\ -1 \amp 1 \amp 3 \end{array}\right] \sim \left[\begin{array}{rr|r} 1 \amp 0 \amp -2 \\ 0 \amp 1 \amp 1 \\ 0 \amp 0 \amp 0 \end{array}\right]\text{,} \end{equation*}
      which shows that the system is consistent. Therefore, \(\bvec\) is in \(\laspan{\vvec_1, \vvec_2}\text{.}\)
    3. The span is the plane in \(\real^3\) defined by \(\vvec_1\) and \(\vvec_2\text{.}\)
  5. We have the reduced row echelon form
    \begin{equation*} \left[\begin{array}{rrr} 1 \amp 0 \amp 1 \\ 1 \amp 2 \amp -2 \\ -1 \amp 1 \amp 4 \end{array}\right] \sim \left[\begin{array}{rrr} 1 \amp 0 \amp 0 \\ 0 \amp 1 \amp 0 \\ 0 \amp 0 \amp 1 \end{array}\right]\text{.} \end{equation*}
    Since there is a pivot position in every row, this says that every equation \(A\xvec=\bvec\) is consistent. The \(\laspan{\vvec_1, \vvec_2, \vvec_3}\) is therefore \(\real^3\text{.}\)
  6. There must be a pivot position in every row.
  7. If a set of vectors spans \(\real^3\text{,}\) its corresponding matrix must have a pivot position in every row. Because there can be at most one pivot position in a column, there must be at least three columns. Therefore, the smallest number of vectors that span \(\real^3\) is three.
The types of sets that appear as the span of a set of vectors in \(\real^3\) are relatively simple.
  • First, with a single nonzero vector, all linear combinations are simply scalar multiples of that vector so that the span of this vector is a line, as shown in Figure 3.1.12.
    Figure 3.1.12. The span of a single nonzero vector is a line.
    Notice that the matrix formed by this vector has one pivot position. For example,
    \begin{equation*} \threevec{-2}{3}{1} \sim \threevec{1}{0}{0}\text{.} \end{equation*}
  • The span of two vectors in \(\real^3\) that do not lie on the same line will be a plane, as seen in Figure 3.1.13.
    Figure 3.1.13. The span of these two vectors in \(\real^3\) is a plane.
    For example, the vectors
    \begin{equation*} \vvec_1=\threevec{-2}{3}{1},~~~ \vvec_2=\threevec{1}{-1}{3} \end{equation*}
    lead to the matrix
    \begin{equation*} \left[\begin{array}{rr} -2 \amp 1 \\ 3 \amp -1 \\ 1 \amp 3 \\ \end{array}\right] \sim \left[\begin{array}{rr} 1 \amp 0 \\ 0 \amp 1 \\ 0 \amp 0 \\ \end{array}\right] \end{equation*}
    with two pivot positions.
  • Finally, a set of three vectors, such as
    \begin{equation*} \vvec_1=\threevec12{-1},~~~ \vvec_2=\threevec201,~~~ \vvec_3=\threevec{-2}20 \end{equation*}
    may form a matrix having three pivot positions
    \begin{equation*} \left[\begin{array}{rrr} \vvec_1 \amp \vvec_2 \amp \vvec_3 \end{array}\right] = \left[\begin{array}{rrr} 1 \amp 2 \amp -2 \\ 2 \amp 0 \amp 2 \\ -1 \amp 1 \amp 0 \\ \end{array}\right] \sim \left[\begin{array}{rrr} 1 \amp 0 \amp 0 \\ 0 \amp 1 \amp 0 \\ 0 \amp 0 \amp 1 \\ \end{array}\right], \end{equation*}
    one in every row. When this happens, no matter how we augment this matrix, it is impossible to obtain a pivot position in the rightmost column:
    \begin{equation*} \left[\begin{array}{rrr|r} 1 \amp 2 \amp -2 \amp *\\ 2 \amp 0 \amp 2 \amp * \\ -1 \amp 1 \amp 0 \amp * \\ \end{array}\right] \sim \left[\begin{array}{rrr|r} 1 \amp 0 \amp 0 \amp *\\ 0 \amp 1 \amp 0 \amp * \\ 0 \amp 0 \amp 1 \amp * \\ \end{array}\right]. \end{equation*}
    Therefore, any linear system \(\begin{bmatrix}\vvec_1\amp\vvec_2\amp\vvec_3\end{bmatrix} ~\xvec = \bvec\) is consistent, which tells us that \(\laspan{\vvec_1,\vvec_2,\vvec_3} = \real^3\text{.}\)
To summarize, we looked at the pivot positions in a matrix whose columns are the three-dimensional vectors \(\vvec_1,\vvec_2,\ldots,\vvec_n\text{.}\) We found that with
  • one pivot position, the span was a line.
  • two pivot positions, the span was a plane.
  • three pivot positions, the span was \(\real^3\text{.}\)
Though we will return to these ideas later, for now take note of the fact that the span of a set of vectors in \(\real^3\) is a relatively simple, familiar geometric object.
The reasoning we led us to conclude that the span of a set of vectors is \(\real^3\) when the associated matrix has a pivot position in every row applies more generally.
This tells us something important about the number of vectors needed to span \(\real^m\text{.}\) Suppose we have \(n\) vectors \(\vvec_1,\vvec_2,\ldots,\vvec_n\) that span \(\real^m\text{.}\) The proposition tells us that the matrix \(A = \left[\begin{array}{rrrr} \vvec_1\amp\vvec_2\amp\ldots\amp\vvec_n \end{array}\right]\) has a pivot position in every row, such as in this reduced row echelon matrix.
\begin{equation*} \left[\begin{array}{rrrrrr} 1 \amp 0 \amp * \amp 0 \amp * \amp 0 \\ 0 \amp 1 \amp * \amp 0 \amp * \amp 0 \\ 0 \amp 0 \amp 0 \amp 1 \amp * \amp 0 \\ 0 \amp 0 \amp 0 \amp 0 \amp 0 \amp 1 \\ \end{array}\right]. \end{equation*}
Since a matrix can have at most one pivot position in a column, there must be at least as many columns as there are rows, which implies that \(n\geq m\text{.}\) For instance, if we have a set of vectors that span \(\real^{632}\text{,}\) there must be at least 632 vectors in the set.
We have thought about a linear combination of a set of vectors \(\vvec_1,\vvec_2,\ldots,\vvec_n\) as the result of walking a certain distance in the direction of \(\vvec_1\text{,}\) followed by walking a certain distance in the direction of \(\vvec_2\text{,}\) and so on. If \(\laspan{\vvec_1,\vvec_2,\ldots,\vvec_n} = \real^m\text{,}\) this means that we can walk to every point in \(\real^m\) using the directions \(\vvec_1,\vvec_2,\ldots,\vvec_n\text{.}\) Intuitively, this proposition is telling us that we need at least \(m\) directions to have the flexibility needed to reach every point in \(\real^m\text{.}\)

Terminology.

Because span is a concept that is connected to a set of vectors, we say, “The span of the set of vectors \(\vvec_1, \vvec_2, \ldots, \vvec_n\) is ....” While it may be tempting to say, “The span of the matrix \(A\) is ...,” we should instead say “The span of the columns of the matrix \(A\) is ....”

Subsection 3.1.3 Span and linear models

Consider a linear model with an intercept and two precitors. The fitted vector for such a model has the form
\begin{equation*} \yhat = \hat\beta_0 \onevec + \hat\beta_1 \xvec_1 + \hat\beta_2 \xvec_2\text{.} \end{equation*}
From a linear algebra perspective, this means that \(\yhat\) is in \(\laspan{\onevec, \xvec_1, \xvec_2}\text{.}\) If we have \(n\) observations in our data, then the data vector \(\yvec\) lives in \(\real^n\text{,}\) but the fitted vector \(\yhat\) lives in \(\laspan{\onevec, \xvec_1, \xvec_2}\text{,}\) a subspace of dimension 3 called the model space.
In Chapter 6 we will learn how to use linear algebra to determine which vector in the model space is closest to \(\yvec\text{.}\) That will be the vector \(\yhat\text{.}\) For now, the important thing is that we can use some geometric intuition to understand linear models if we think of the model space as the span of the predictor vectors.
For example, adding predictors to a model increases the dimension of the model space and leads to fitted vectors \(\yhat\) that are closer to the data vector \(\yvec\text{.}\) We say that models with more predictors are more flexible or have more model degrees of freedom because we have more "room" to search for a best \(\yhat\text{.}\)
Somewhat counterintuitively, it is not always good to fit the data better. In an extreme case, if we have \(n\) linearly independent 1  predictors (including \(\onevec\)), then the model space is all of \(\real^n\text{,}\) so \(\yhat = \yvec\) and the model fits the data perfectly. But a model that fits the data this well is almost surely overfitting -- it is learning both the generalizable features of the data and the idiosyncrasies of the data. Because of this, a "larger" model (one with a higher-dimensional model space) may be less useful at predicting new data than a "smaller" model that fits the data less well but generalizes better.

Subsection 3.1.4 Summary

We defined the span of a set of vectors and developed some intuition for this concept through a series of examples.
  • The span of a set of vectors \(\vvec_1,\vvec_2,\ldots,\vvec_n\) is the set of linear combinations of the vectors. We denote the span by \(\laspan{\vvec_1,\vvec_2,\ldots,\vvec_n}\text{.}\)
  • A vector \(\bvec\) is in \(\laspan{\vvec_1,\vvec_2,\ldots,\vvec_n}\) if an only if the linear system
    \begin{equation*} \left[\begin{array}{rrrr} \vvec_1\amp\vvec_2\amp\ldots\vvec_n \end{array}\right] ~\xvec = \bvec \end{equation*}
    is consistent.
  • If the \(m\by n\) matrix
    \begin{equation*} \left[\begin{array}{rrrr} \vvec_1\amp\vvec_2\amp\ldots\vvec_n \end{array}\right] \end{equation*}
    has a pivot position in every row, then the span of these vectors is \(\real^m\text{;}\) that is, \(\laspan{\vvec_1,\vvec_2,\ldots,\vvec_n} = \real^m\text{.}\)
  • Any set of vectors that spans \(\real^m\) must have at least \(m\) vectors.
  • The model space of a linear model is the span of the predictor vectors in the data. The fitted response \(\yhat\) is always in this span.

Exercises 3.1.5 Exercises

1.

In this exercise, we will consider the span of some sets of two- and three-dimensional vectors.
  1. Consider the vectors
    \begin{equation*} \vvec_1 = \twovec{1}{-2}, \vvec_2 = \twovec{4}{3}\text{.} \end{equation*}
    1. Is \(\bvec = \twovec{2}{1}\) in \(\laspan{\vvec_1,\vvec_2}\text{?}\)
    2. Give a geometric description of \(\laspan{\vvec_1,\vvec_2}\text{.}\)
  2. Consider the vectors
    \begin{equation*} \vvec_1=\threevec{2}{1}{3}, \vvec_2=\threevec{-2}{0}{2}, \vvec_3=\threevec{6}{1}{-1}\text{.} \end{equation*}
    1. Is the vector \(\bvec=\threevec{-10}{-1}{5}\) in \(\laspan{\vvec_1,\vvec_2,\vvec_3}\text{?}\)
    2. Is the vector \(\vvec_3\) in \(\laspan{\vvec_1,\vvec_2,\vvec_3}\text{?}\)
    3. Is the vector \(\bvec=\threevec{3}{3}{-1}\) in \(\laspan{\vvec_1,\vvec_2,\vvec_3}\text{?}\)
    4. Give a geometric description of \(\laspan{\vvec_1,\vvec_2,\vvec_3}\text{.}\)

2.

Provide a justification for your response to the following questions.
  1. Suppose you have a set of vectors \(\vvec_1,\vvec_2,\ldots,\vvec_n\text{.}\) Can you guarantee that \(\zerovec\) is in \(\laspan{\vvec_1\,\vvec_2,\ldots,\vvec_n}\text{?}\)
  2. Suppose that \(A\) is an \(m \by n\) matrix. Can you guarantee that the equation \(A\xvec = \zerovec\) is consistent?
  3. What is \(\laspan{\zerovec,\zerovec,\ldots,\zerovec}\text{?}\)

3.

For both parts of this exercise, give a geometric description of sets of the vectors \(\bvec\) and include a sketch.
  1. For which vectors \(\bvec\) in \(\real^2\) is the equation
    \begin{equation*} \left[\begin{array}{rr} 3 \amp -6 \\ -2 \amp 4 \\ \end{array}\right] \xvec = \bvec \end{equation*}
    consistent?
  2. For which vectors \(\bvec\) in \(\real^2\) is the equation
    \begin{equation*} \left[\begin{array}{rr} 3 \amp -6 \\ -2 \amp 2 \\ \end{array}\right] \xvec = \bvec \end{equation*}
    consistent?

4.

Consider the following matrices:
\begin{equation*} A = \left[\begin{array}{rrrr} 3 \amp 0 \amp -1 \amp 1 \\ 1 \amp -1 \amp 3 \amp 7 \\ 3 \amp -2 \amp 1 \amp 5 \\ -1 \amp 2 \amp 2 \amp 3 \\ \end{array}\right],~~~ B = \left[\begin{array}{rrrr} 3 \amp 0 \amp -1 \amp 4 \\ 1 \amp -1 \amp 3 \amp -1 \\ 3 \amp -2 \amp 1 \amp 3 \\ -1 \amp 2 \amp 2 \amp 1 \\ \end{array}\right]\text{.} \end{equation*}
Do the columns of \(A\) span \(\real^4\text{?}\) Do the columns of \(B\) span \(\real^4\text{?}\)

5.

Determine whether the following statements are true or false and provide a justification for your response. Throughout, we will assume that the matrix \(A\) has columns \(\vvec_1,\vvec_2,\ldots,\vvec_n\text{;}\) that is,
\begin{equation*} A = \left[\begin{array}{rrrr} \vvec_1\amp\vvec_2\amp\ldots\amp\vvec_n \end{array}\right]\text{.} \end{equation*}
  1. If the equation \(A\xvec = \bvec\) is consistent, then \(\bvec\) is in \(\laspan{\vvec_1,\vvec_2,\ldots,\vvec_n}\text{.}\)
  2. The equation \(A\xvec = \vvec_1\) is consistent.
  3. If \(\vvec_1\text{,}\) \(\vvec_2\text{,}\) \(\vvec_3\text{,}\) and \(\vvec_4\) are vectors in \(\real^3\text{,}\) then their span is \(\real^3\text{.}\)
  4. If \(\bvec\) can be expressed as a linear combination of \(\vvec_1, \vvec_2,\ldots,\vvec_n\text{,}\) then \(\bvec\) is in \(\laspan{\vvec_1,\vvec_2,\ldots,\vvec_n}\text{.}\)
  5. If \(A\) is a \(8032\by 427\) matrix, then the span of the columns of \(A\) is a set of vectors in \(\real^{427}\text{.}\)

6.

This exercise asks you to construct some matrices whose columns span a given set.
  1. Construct a \(3\by3\) matrix whose columns span \(\real^3\text{.}\)
  2. Construct a \(3\by3\) matrix whose columns span a plane in \(\real^3\text{.}\)
  3. Construct a \(3\by3\) matrix whose columns span a line in \(\real^3\text{.}\)

7.

Provide a justification for your response to the following questions.
  1. Suppose that we have vectors in \(\real^8\text{,}\) \(\vvec_1,\vvec_2,\ldots,\vvec_{10}\text{,}\) whose span is \(\real^8\text{.}\) Can every vector \(\bvec\) in \(\real^8\) be written as a linear combination of \(\vvec_1,\vvec_2,\ldots,\vvec_{10}\text{?}\)
  2. Suppose that we have vectors in \(\real^8\text{,}\) \(\vvec_1,\vvec_2,\ldots,\vvec_{10}\text{,}\) whose span is \(\real^8\text{.}\) Can every vector \(\bvec\) in \(\real^8\) be written uniquely as a linear combination of \(\vvec_1,\vvec_2,\ldots,\vvec_{10}\text{?}\)
  3. Do the vectors
    \begin{equation*} \evec_1=\threevec{1}{0}{0}, \evec_2=\threevec{0}{1}{0}, \evec_3=\threevec{0}{0}{1} \end{equation*}
    span \(\real^3\text{?}\)
  4. Suppose that \(\vvec_1,\vvec_2,\ldots,\vvec_n\) span \(\real^{438}\text{.}\) What can you guarantee about the value of \(n\text{?}\)
  5. Can 17 vectors in \(\real^{20}\) span \(\real^{20}\text{?}\)

8.

The following observation will be helpful in this exercise. If we want to find a solution to the equation \(AB\xvec = \bvec\text{,}\) we could first find a solution to the equation \(A\yvec = \bvec\) and then find a solution to the equation \(B\xvec = \yvec\text{.}\)
Suppose that \(A\) is a \(3\by 4\) matrix whose columns span \(\real^3\) and \(B\) is a \(4\by 5\) matrix. In this case, we can form the product \(AB\text{.}\)
  1. What is the shape of the product \(AB\text{?}\)
  2. Can you guarantee that the columns of \(AB\) span \(\real^3\text{?}\)
  3. If you know additionally that the span of the columns of \(B\) is \(\real^4\text{,}\) can you guarantee that the columns of \(AB\) span \(\real^3\text{?}\)

9.

Suppose that \(A\) is a \(12\by12\) matrix and that, for some vector \(\bvec\text{,}\) the equation \(A\xvec=\bvec\) has a unique solution.
  1. What can you say about the pivot positions of \(A\text{?}\)
  2. What can you say about the span of the columns of \(A\text{?}\)
  3. If \(\cvec\) is some other vector in \(\real^{12}\text{,}\) what can you conclude about the equation \(A\xvec = \cvec\text{?}\)
  4. What can you about the solution space to the equation \(A\xvec =\zerovec\text{?}\)

10.

Suppose that
\begin{equation*} \vvec_1 = \fourvec{3}{1}{3}{-1}, \vvec_2 = \fourvec{0}{-1}{-2}{2}, \vvec_3 = \fourvec{-3}{-3}{-7}{5}\text{.} \end{equation*}
  1. Is \(\vvec_3\) a linear combination of \(\vvec_1\) and \(\vvec_2\text{?}\) If so, find weights such that \(\vvec_3 = a\vvec_1+b\vvec_2\text{.}\)
  2. Show that a linear combination
    \begin{equation*} a\vvec_1 + b\vvec_2 + c\vvec_3 \end{equation*}
    can be rewritten as a linear combination of \(\vvec_1\) and \(\vvec_2\text{.}\)
  3. Explain why \(\laspan{\vvec_1,\vvec_2,\vvec_3} = \laspan{\vvec_1,\vvec_2}\text{.}\)

11.

As defined in this section, the span of a set of vectors is generated by taking all possible linear combinations of those vectors. This exercise will demonstrate the fact that the span can also be realized as the solution space to a linear system.
We will consider the vectors
\begin{equation*} \vvec_1=\threevec{1}{0}{-2}, \vvec_2=\threevec{2}{1}{0}, \vvec_3=\threevec{1}{1}{2} \end{equation*}
  1. Is every vector in \(\real^3\) in \(\laspan{\vvec_1,\vvec_2,\vvec_3}\text{?}\) If not, describe the span.
  2. To describe \(\laspan{\vvec_1,\vvec_2,\vvec_3}\) as the solution space of a linear system, we will write
    \begin{equation*} \bvec=\threevec{a}{b}{c}\text{.} \end{equation*}
    If \(\bvec\) is in \(\laspan{\vvec_1,\vvec_2,\vvec_3}\text{,}\) then the linear system corresponding to the augmented matrix
    \begin{equation*} \left[\begin{array}{rrr|r} 1 \amp 2 \amp 1 \amp a \\ 0 \amp 1 \amp 1 \amp b \\ -2\amp 0 \amp 2 \amp c \\ \end{array}\right] \end{equation*}
    must be consistent. This means that a pivot cannot occur in the rightmost column. Perform row operations to put this augmented matrix into a triangular form. Now identify an equation in \(a\text{,}\) \(b\text{,}\) and \(c\) that tells us when there is no pivot in the rightmost column. The solution space to this equation describes \(\laspan{\vvec_1,\vvec_2,\vvec_3}\text{.}\)
  3. In this example, the matrix formed by the vectors \(\left[\begin{array}{rrr} \vvec_1\amp\vvec_2\amp\vvec_3 \\ \end{array}\right]\) has two pivot positions. Suppose we were to consider another example in which this matrix had had only one pivot position. How would this have changed the linear system describing \(\laspan{\vvec_1,\vvec_2,\vvec_3}\text{?}\)
See Section 3.2 for more about linear independence.
<Prev^TopNext>