The Spectral Theorem and singular value decompositions

Skip to main content

\(\newcommand{\avec}{{\boldsymbol a}} \newcommand{\bvec}{{\boldsymbol b}} \newcommand{\cvec}{{\boldsymbol c}} \newcommand{\dvec}{{\boldsymbol d}} \newcommand{\dtil}{\widetilde{\boldsymbol d}} \newcommand{\evec}{{\boldsymbol e}} \newcommand{\fvec}{{\boldsymbol f}} \newcommand{\mvec}{{\boldsymbol m}} \newcommand{\nvec}{{\boldsymbol n}} \newcommand{\pvec}{{\boldsymbol p}} \newcommand{\qvec}{{\boldsymbol q}} \newcommand{\rvec}{{\boldsymbol r}} \newcommand{\svec}{{\boldsymbol s}} \newcommand{\tvec}{{\boldsymbol t}} \newcommand{\uvec}{{\boldsymbol u}} \newcommand{\vvec}{{\boldsymbol v}} \newcommand{\wvec}{{\boldsymbol w}} \newcommand{\xvec}{{\boldsymbol x}} \newcommand{\yvec}{{\boldsymbol y}} \newcommand{\zvec}{{\boldsymbol z}} \newcommand{\betavec}{{\boldsymbol \beta}} \newcommand{\E}{\operatorname{E}} \newcommand{\zerovec}{{\boldsymbol 0}} \newcommand{\onevec}{{\boldsymbol 1}} \newcommand{\real}{{\mathbb R}} \newcommand{\twovec}[2]{\left[\begin{array}{r}#1 \\ #2 \end{array}\right]} \newcommand{\ctwovec}[2]{\left[\begin{array}{c}#1 \\ #2 \end{array}\right]} \newcommand{\threevec}[3]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \end{array}\right]} \newcommand{\cthreevec}[3]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \end{array}\right]} \newcommand{\fourvec}[4]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]} \newcommand{\cfourvec}[4]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]} \newcommand{\fivevec}[5]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]} \newcommand{\cfivevec}[5]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]} \newcommand{\mattwo}[4]{\left[\begin{array}{rr}#1 \amp #2 \\ #3 \amp #4 \\ \end{array}\right]} \newcommand{\laspan}[1]{\text{Span}\{#1\}} \newcommand{\bcal}{{\cal B}} \newcommand{\ccal}{{\cal C}} \newcommand{\scal}{{\cal S}} \newcommand{\wcal}{{\cal W}} \newcommand{\ecal}{{\cal E}} \newcommand{\coords}[2]{\left\{#1\right\}_{#2}} \newcommand{\gray}[1]{\color{gray}{#1}} \newcommand{\lgray}[1]{\color{lightgray}{#1}} \newcommand{\rank}{\operatorname{rank}} \newcommand{\row}{\text{Row}} \newcommand{\col}{\text{Col}} \renewcommand{\row}{\text{Row}} \newcommand{\nul}{\text{Nul}} \newcommand{\var}{\text{Var}} \newcommand{\cov}{\text{Cov}} \newcommand{\corr}{\text{corr}} \newcommand{\len}[1]{\left|#1\right|} \newcommand{\bbar}{\overline{\bvec}} \newcommand{\bhat}{\widehat{\bvec}} \newcommand{\bperp}{\bvec^\perp} \newcommand{\atilde}{\tilde{\avec}} \newcommand{\btilde}{\tilde{\bvec}} \newcommand{\vhat}{\widehat{\vvec}} \newcommand{\uhat}{\widehat{\uvec}} \newcommand{\what}{\widehat{\wvec}} \newcommand{\xhat}{\widehat{\xvec}} \newcommand{\xmean}{\overline{\xvec}} \newcommand{\xbar}{\overline{\xvec}} \newcommand{\xtilde}{\tilde{\xvec}} \newcommand{\Xtilde}{\tilde{X}} \newcommand{\yhat}{\widehat{\yvec}} \newcommand{\ymean}{\overline{\yvec}} \newcommand{\ybar}{\overline{\yvec}} \newcommand{\ytilde}{\tilde{\yvec}} \newcommand{\yperp}{\yvec^\perp} \newcommand{\betahat}{\widehat{\betavec}} \newcommand{\Sighat}{\widehat{\Sigma}} \newcommand{\by}{\times} \newcommand{\transpose}{\top} \newcommand{\proj}[2]{\operatorname{proj}\left(#1 \to #2\right)} \newcommand{\projsub}[2]{\operatorname{proj}_{#2}(#1)} \newcommand{\lt}{<} \newcommand{\gt}{>} \newcommand{\amp}{&} \definecolor{fillinmathshade}{gray}{0.9} \newcommand{\fillinmath}[1]{\mathchoice{\colorbox{fillinmathshade}{$\displaystyle \phantom{\,#1\,}$}}{\colorbox{fillinmathshade}{$\textstyle \phantom{\,#1\,}$}}{\colorbox{fillinmathshade}{$\scriptstyle \phantom{\,#1\,}$}}{\colorbox{fillinmathshade}{$\scriptscriptstyle\phantom{\,#1\,}$}}} \)

Chapter 7 The Spectral Theorem and singular value decompositions

Chapter 5 demonstrated several important uses for the theory of eigenvalues and eigenvectors. For example, knowing the eigenvalues and eigenvectors of a matrix \(A\) enabled us to make predictions about the long-term behavior of dynamical systems in which some initial state \(\xvec_0\) evolves according to the rule \(\xvec_{k+1} = A\xvec_k\text{.}\)

We can’t, however, apply this theory to every problem we might meet. First, eigenvectors only exist when the matrix \(A\) is square, and we have seen situations, such as the least squares problems in Section 6.5, where the matrices we’re interested in are not square. Second, even when \(A\) is square, there may not be a basis for \(\real^m\) consisting of eigenvectors of \(A\text{,}\) an important condition we required for some of our work.

This chapter introduces singular value decompositions, whose singular values and singular vectors may be viewed as a generalization of eigenvalues and eigenvectors. In fact, we will see that every matrix, whether square or not, has a singular value decomposition and that knowing it gives us a great deal of insight into the matrix. It’s been said that having a singular value decomposition is like looking at a matrix with X-ray vision as the decomposition reveals essential features of the matrix.

<Prev ^Top Next>