Skip to main content

Understanding Linear Algebra: Data Science Edition

Randall Pruim

Section 2.1 What can we expect

At its heart, the subject of linear algebra is about linear equations and, more specifically, sets of two or more linear equations. Proposition 1.4.4 shows that there multiple ways to express this important concept notationally. And we will want to become comfortable with all of these ways, and with moving back and forth among them.
Google routinely deals with a set of trillions of equations each of which has trillions of unknowns. We will eventually understand how to deal with that kind of complexity. To begin, however, we will look at situations in which there are a small number of equations and a small number of unknowns. In spite of their relative simplicity, these smaller examples are rich enough to demonstrate some fundamental concepts that will motivate much of our exploration.

Subsection 2.1.1 Some simple examples

Activity 2.1.1.

In this activity, we consider sets of linear equations having just two unknowns. In this case, we can graph the solutions sets for the equations, which allows us to visualize different types of behavior.
  1. On the grid below, graph the lines
    \begin{equation*} \begin{aligned} y \amp = x+1 \\ y \amp = 2x-1 \\ \end{aligned}\text{.} \end{equation*}
    At what point or points \((x,y)\text{,}\) do the lines intersect? How many points \((x,y)\) satisfy both equations?
  2. On the grid below, graph the lines
    \begin{equation*} \begin{aligned} y \amp = x+1 \\ y \amp = x-1 \\ \end{aligned}\text{.} \end{equation*}
    At what point or points \((x,y)\text{,}\) do the lines intersect? How many points \((x,y)\) satisfy both equations?
  3. On the grid below, graph the line
    \begin{equation*} y = x+1\text{.} \end{equation*}
    How many points \((x,y)\) satisfy this equation?
  4. On the grid below, graph the lines
    \begin{equation*} \begin{aligned} y \amp = x+1 \\ y \amp = 2x-1 \\ y \amp = -x. \\ \end{aligned} \end{equation*}
    At what point or points \((x,y)\text{,}\) do the lines intersect? How many points \((x,y)\) satisfy all three equations?
Answer.
  1. This is exactly one point, the point \((2,3)\text{,}\) that satisfies both equations.
  2. There are no points that satisfy both equations.
  3. There are infinitely many points.
  4. There are no points that satisfy all three equations.
Solution.
  1. The graph of the two lines is as shown on the right. There is a single point, the point \((2,3)\text{,}\) at which the lines intersect. Therefore, there is a single point that satisfies both equations.
  2. These two lines are parallel, which means there is no point at which the lines intersect. Therefore, there is no point that satisfies both equations.
  3. There are infinitely many points that lie on this line and that, therefore, satisfy this single equation.
  4. These three lines do not have a common intersection point. Consequently, there is no point satisfying all three equations.
The examples in this introductory activity demonstrate several possible outcomes for the solutions to a set of linear equations. Notice that we are interested in points that satisfy each equation in the set and that these are seen as intersection points of the lines. Similar to the examples considered in the activity, three types of outcomes are seen in Figure 2.1.1.
Figure 2.1.1. Three possible graphs for sets of linear equations in two unknowns.
In this figure, we see that
  • With a single equation, there are infinitely many points \((x,y)\) satisfying that equation.
  • Adding a second equation adds another condition we place on the points \((x,y)\) resulting in a single point that satisfies both equations.
  • Adding a third equation adds a third condition on the points \((x,y)\text{,}\) and there is no point that satisfies all three equations.
Generally speaking, a single equation will have many solutions, in fact, infinitely many. As we add equations, we add conditions which lead, in a sense we will make precise later, to a smaller number of solutions. Eventually, we have too many equations and find there are no points that satisfy all of them.
This example illustrates a general principle to which we will frequently return.

Solutions to sets of linear equations.

Given a set of linear equations, there are either:
  • infinitely many points,
  • exactly one point, or
  • no points
that satisfy every equation in the set.
Notice that we can see a bit more. In Figure 2.1.1, we are looking at equations in two unknowns. Here we see that
  • One equation has infinitely solutions.
  • Two equations have exactly one solution.
  • Three equations have no solutions.
It seems reasonable to wonder if the number of solutions depends on whether the number of equations is less than, equal to, or greater than the number of unknowns. Of course, one of the examples in the activity shows that there are exceptions to this simple rule, as seen in Figure 2.1.2. For instance, two equations in two unknowns may correspond to parallel lines so that the set of equations has no solutions. It may also happen that a set of three equations in two unknowns has a single solution. However, it seems safe to think that the more equations we have, the smaller the set of solutions will usually be.
Figure 2.1.2. A set of two equations in two unknowns can have no solutions, and a set of three equations can have one solution.
Let’s also consider some examples of equations having three unknowns, which we call \(x\text{,}\) \(y\text{,}\) and \(z\text{.}\) Just as solutions to linear equations in two unknowns formed straight lines, solutions to linear equations in three unknowns form planes.
When we consider an equation in three unknowns graphically, we need to add a third coordinate axis, as shown in Figure 2.1.3.
Figure 2.1.3. Coordinate systems in two and three dimensions.
As shown in Figure 2.1.4, a linear equation in two unknowns, such as \(y=0\text{,}\) is a line while a linear equation in three unknowns, such as \(z=0\text{,}\) is a plane.
Figure 2.1.4. The solutions to the equation \(y=0\) in two dimensions and \(z=0\) in three.
In three unknowns, the set of solutions to one linear equation forms a plane. The set of solutions to a pair of linear equations is seen graphically as the intersection of the two planes. As in Figure 2.1.5, we typically expect this intersection to be a line.
Figure 2.1.5. A single plane and the intersection of two planes.
When we add a third equation, we are looking for the intersection of three planes, which we expect to form a point, as in the left of Figure 2.1.6. However, in certain special cases, it may happen that there are no solutions, as seen on the right.
Figure 2.1.6. Two examples showing the intersections of three planes.

Activity 2.1.2.

This activity considers sets of equations having three unknowns. In this case, we know that the solutions of a single equation form a plane. If it helps with visualization, consider using \(3\by5\)-inch index cards to represent planes.
  1. Is it possible that there are no solutions to two linear equations in three unknowns? Either sketch an example or state a reason why it can’t happen.
  2. Is it possible that there is exactly one solution to two linear equations in three unknowns? Either sketch an example or state a reason why it can’t happen.
  3. Is it possible that the solutions to four equations in three unknowns form a line? Either sketch an example or state a reason why it can’t happen.
  4. What would you usually expect for the set of solutions to four equations in three unknowns?
  5. Suppose we have a set of 500 linear equations in 10 unknowns. Which of the three possibilities would you expect to hold?
  6. Suppose we have a set of 10 linear equations in 500 unknowns. Which of the three possibilities would you expect to hold?
Answer.
  1. Yes.
  2. No.
  3. Yes.
  4. We would expect there to be no solutions.
  5. We would expect there to be no solutions.
  6. We would expect there to be infinitely many solutions.
Solution.
  1. Yes, it is possible if the two planes are parallel to one another.
  2. No, this is not possible. Two planes will either intersect in a line, if they are not parallel, or not intersect at all, if they are parallel.
  3. Yes, it is possible that four planes intersect in a line. One may sketch four planes that intersect in, say, the \(z\)-axis.
  4. In general, we would expect there to be no solutions to four equations in three unknowns because there are more equations than unknowns.
  5. Since there are more equations than unknowns, we would expect there to be no solutions. We cannot guarantee this, however.
  6. Since there are fewer equations than unknowns, we would expect there to be infinitely many solutions. We cannot guarantee this, however.

Subsection 2.1.2 Systems of linear equations

Now that we have seen some simple examples, let’s agree on some terminology to help us think more precisely about sets of equations.
First, we considered a linear equation having the form
\begin{equation*} y = 2x - 1\text{.} \end{equation*}
It will be convenient for us to rewrite this so that all the unknowns are on one side of the equation:
\begin{equation*} -2x + y = -1\text{.} \end{equation*}
More generally, the equation of a line can always be expressed in the form
\begin{equation*} ax + by = c, \end{equation*}
which gives us the flexibility to describe all lines. For instance, vertical lines, such as \(x=3\text{,}\) may be represented in this form.
Notice that each term on the left is the product of a constant and the first power of an unknown. In the future, we will want to consider equations having many more unknowns, which we will sometimes denote as \(x_1, x_2, \ldots, x_n\text{.}\) This leads to the following definition:

Definition 2.1.7.

A linear equation in the unknowns \(x_1,x_2,\ldots,x_n\) may be written in the form
\begin{equation*} a_1x_1 + a_2x_2 + \ldots + a_nx_n = b\text{,} \end{equation*}
where \(a_1,a_2,\ldots,a_n\) are real numbers known as coefficients. We also say that \(x_1,x_2,\ldots,x_n\) are the variables in the equation.
By a linear system of equations or a linear system, we mean a set of linear equations written in a common set of unknowns.
For instance,
\begin{equation*} \begin{alignedat}{4} 2x_1 \amp {} + \amp {} 1.2x_2 \amp {}-{} \amp 4x_3 \amp {}={} \amp 3.7 \\ -0.1x_1 \amp {} \amp {} \amp {} + {} \amp x_3 \amp {}={} \amp 2 \\ x_1 \amp {}+{} \amp x_2 \amp {}-{} \amp x_3 \amp {}={} \amp 1.4 \\ \end{alignedat} \end{equation*}
is an example of a linear system.

Definition 2.1.8.

A solution to a linear system is simply a set of numbers \(x_1 = s_1, x_2 = s_2, \ldots, x_n=s_n\) that satisfy all the equations in the system.
For instance, we earlier considered the linear system
\begin{equation*} \begin{alignedat}{3} -x \amp {}+{} \amp y \amp {} = {} \amp 1 \\ -2x \amp {}+{} \amp y \amp {} = {} \amp -1. \\ \end{alignedat} \end{equation*}
To check that \((x,y) = (2,3)\) is a solution, we verify that the following equations are true.
\begin{equation*} \begin{alignedat}{3} -2 \amp {}+{} \amp 3 \amp {} = {} \amp 1 \\ -2(2) \amp {}+{} \amp 3 \amp {} = {} \amp -1. \\ \end{alignedat} \end{equation*}

Definition 2.1.9.

We call the set of all solutions the solution space of the linear system.

Activity 2.1.3. Linear equations and their solutions.

  1. Which of the following equations are linear? Please provide a justification for your response.
    1. \begin{equation*} 2x + xy -3y^2 = 2 \text{.} \end{equation*}
    2. \begin{equation*} -2x_1 + 3x_2 +4x_3 - x_5 = 0 \text{.} \end{equation*}
    3. \begin{equation*} x = 3z - 4y \text{.} \end{equation*}
  2. Consider the system of linear equations:
    \begin{equation*} \begin{alignedat}{4} x \amp {}+{} \amp y \amp \amp \amp {}={} \amp 3 \\ \amp \amp y \amp {}-{} \amp z \amp {}={} \amp 2 \\ 2x \amp {}+{} \amp y \amp {}+{} \amp z \amp {}={} \amp 4. \\ \end{alignedat} \end{equation*}
    1. Is \((x,y,z) = (1,2,0)\) a solution?
    2. Is \((x,y,z) = (-2,1,0)\) a solution?
    3. Is \((x,y,z) = (0,-3,1)\) a solution?
    4. Can you find a solution in which \(y = 0\text{?}\)
    5. Do you think there are other solutions? Please explain your response.
Answer.
  1. There are two linear equations in this list.
    1. This is not a linear equation.
    2. This is a linear equation.
    3. This is a linear equation.
  2. We will see that the system of linear equations has infinitely many solutions.
    1. This is a solution.
    2. This is a not solution.
    3. This is a not solution.
    4. Yes, \((x,y,z) = (3,0,-2)\) is a solution.
    5. Yes, because there should be infinitely many solutions.
Solution.
  1. There are two linear equations in this list.
    1. This is not a linear equation due to the presence of \(y^2\) in the third term on the left.
    2. This is a linear equation.
    3. This is a linear equation since it can be rewritten in the form \(x + 4y - 3z = 0\text{.}\)
  2. We will see that the system of linear equations has infinitely many solutions.
    1. Yes, this is a solution since all three equations are satisfied if we set \(x=1\text{,}\) \(y=2\text{,}\) and \(z=0\text{.}\)
    2. No, this is not a solution since the first equation is not satisfied if \(x=-2\) and \(y=1\text{.}\)
    3. This is also not a solution.
    4. If \(y=0\text{,}\) then we arrive at the system of three linear equations:
      \begin{equation*} \begin{alignedat}{4} x \amp \amp \amp {}={} \amp 3 \\ \amp {}-{} \amp z \amp {}={} \amp 2 \\ 2x \amp {}+{} \amp z \amp {}={} \amp 4. \\ \end{alignedat} \end{equation*}
      We have a solution when \(x=3\) and \(z=-2\text{.}\) Therefore, \((x,y,z)=(3,0,-2)\) is a solution to the original system of equations.
    5. Since we have found two solutions to the system of equations, we should expect that there are infinitely many. Therefore, there should be many other solutions.

Subsection 2.1.3 Summary

The point of this section is to build some intuition about the behavior of solutions to linear systems through consideration of some simple examples. We will develop a deeper and more precise understanding of these phenomena in our future explorations.
  • A linear equation is one that may be written in the form
    \begin{equation*} a_1x_1 + a_2x_2 + \ldots + a_nx_n = b\text{.} \end{equation*}
  • A linear system is a set of linear equations and a solution is a set of values assigned to the unknowns that make each equation true.
  • We came to expect that a linear system has either infinitely many solutions, exactly one solution, or no solutions.
  • When we add more equations to a system, the solution space usually seems to become smaller.

Exercises 2.1.4 Exercises

1.

Give a brief explanation of your response to the following questions.
  1. Suppose we have a system of 500 linear equations in 10 unknowns. What would be a reasonable guess about which of the three possibilities for the set of solutions holds?
  2. Suppose we have a system of 10 linear equations in 500 unknowns. What would be a reasonable guess about which of the three possibilities for the set of solutions holds?

2.

Which of the following equations are linear? Please provide a justification for your response.
  1. \begin{equation*} 2x + xy -3y^2 = 2 \text{.} \end{equation*}
  2. \begin{equation*} -2x_1 + 3x_2 +4x_3 - x_5 = 0 \text{.} \end{equation*}
  3. \begin{equation*} x = 3z - 4y \text{.} \end{equation*}

3.

Consider the system of linear equations:
\begin{equation*} \begin{alignedat}{4} x \amp {}+{} \amp y \amp \amp \amp {}={} \amp 3 \\ \amp \amp y \amp {}-{} \amp z \amp {}={} \amp 2 \\ 2x \amp {}+{} \amp y \amp {}+{} \amp z \amp {}={} \amp 4. \\ \end{alignedat} \end{equation*}
  1. Is \((x,y,z) = (1,2,0)\) a solution?
  2. Is \((x,y,z) = (-2,1,0)\) a solution?
  3. Is \((x,y,z) = (0,-3,1)\) a solution?
  4. Can you find a solution in which \(y = 0\text{?}\)
  5. Do you think there are other solutions? Please explain your response.

4.

This exercise concerns systems of equations having three unknowns. In this case, we know that the solution space of a single equation is a plane. If it helps with visualization, consider using \(3\by5\) index cards to represent planes.
  1. Is it possible that the solution space of a system of four equations in three unknowns is a line? Either sketch an example or give a reason why it can’t happen.
  2. What would you usually expect for the solution space to a system of four equations in three unknowns?
  3. Is it possible that there are no solutions to a system of two equations in three unknowns? Either sketch an example or give a reason why it can’t happen.
  4. Is it possible that there is exactly one solution to a system of two equations in three unknowns? Either sketch an example or give a reason why it can’t happen.
<Prev^TopNext>