Objectives: Topics
-
Orthogonal projections and their properties
-
Best approximations
Print preview
Section 6.3 Orthogonal Projections
Handout 6.3 Orthogonal Projections
Objectives: Goals
-
Apply orthogonality and projections to:
-
compute projections and distances,
-
express vectors using orthogonal decompositions,
-
construct best approximations.
-
Motivating Question
For the matrix \(A\) and vector \(\vec b\text{,}\) which vector \(\widehat b \in \Col(A)\) is closest to \(\vec b\text{?}\)
\begin{equation*}
A=\begin{bmatrix}
1 \amp 2\\
3 \amp 0\\
-4 \amp -2
\end{bmatrix},
\qquad
\vec b=\begin{bmatrix}1\\1\\1\end{bmatrix}.
\end{equation*}
Theorem 6.36 Orthogonal Decomposition.
If \(W\) is a subspace of \(\mathbb{R}^n\text{,}\) then each \(\vec y\in\mathbb{R}^n\) has a unique decomposition:
\begin{equation*}
\vec y = \widehat y + w^\perp
\end{equation*}
where \(\widehat y \in W\) and \(w^\perp \in W^\perp\text{.}\) In fact, if \(\{\vec u_1,\dots,\vec u_p\}\) is any orthogonal basis for \(W\text{,}\) then
\begin{align*}
\widehat y \amp = \proj_W \vec y \\
\amp = \proj_{\vec u_1}\vec y + \cdots + \proj_{\vec u_p}\vec y\\
\amp = \frac{\vec y\cdot\vec u_1}{\vec u_1\cdot\vec u_1}\vec u_1 + \cdots + \frac{\vec y\cdot\vec u_p}{\vec u_p\cdot\vec u_p}\vec u_p
\end{align*}
The vector \(\widehat y\) is called the orthogonal projection of \(\vec y\) onto \(W\) and can be written \(\widehat y = \proj_{W}\vec y\text{.}\)
We have that \(w^\perp = \vec y - \widehat y\text{.}\)
Example 6.37.
Let \(\{\vec u_1,\dots,\vec u_5\}\) be an orthonormal basis of \(\mathbb{R}^5\) and let \(W=\operatorname{Span}\{\vec u_1,\vec u_2\}\text{.}\)
For any \(\vec y\in\mathbb{R}^5\text{,}\) write the orthogonal decomposition: \(\vec y=\widehat y + w^\perp\text{,}\) with \(\widehat y\in W\) and \(w^\perp\in W^\perp\text{.}\)
Solution.
Since \(left\{\vec u_1, \dots \vec u_5\}\) is an orthonormal basis for \(\mathbb{R}^5\text{,}\) we have
\begin{equation*}
\vec y = c_1 \vec u_1 + c_2\vec u_2 + c_3\vec u_3 + c_4\vec u_4 + c_5 \vec u_5\text{,}
\end{equation*}
where \(c_i = \vec y \cdot \vec u_i\) by Theorem 6.30. Note that \(\{\vec u_1, \vec u_2\}\) is an orthonormal basis for \(W\text{,}\) so by Theorem 6.36, we have
\begin{equation*}
\widehat y = \proj_{\vec u_1}\vec y + \proj_{\vec u_2}\vec y = (\vec y \cdot \vec u_1)\vec u_1 + (\vec y \cdot \vec u_2)\vec u_2 \in W\text{.}
\end{equation*}
It follows that
\begin{equation*}
\vec w^\perp = \vec y - \widehat y = (\vec y \cdot \vec u_3) \vec u_3 + (\vec y \cdot \vec u_4)\vec u_4 + (\vec y \cdot \vec u_5)\vec u_5 \in W^\perp\text{.}
\end{equation*}
Example 6.38.
Given
\begin{equation*}
\vec y=\begin{bmatrix}-1\\2\\6\end{bmatrix},\quad
\vec u_1=\begin{bmatrix}3\\-1\\2\end{bmatrix},\quad
\vec u_2=\begin{bmatrix}1\\-1\\-2\end{bmatrix},
\end{equation*}
construct the decomposition \(\vec y = \widehat y + w^\perp\text{,}\) where \(\widehat y\) is the orthogonal projection of \(\vec y\) onto \(W=\operatorname{Span}\{\vec u_1,\vec u_2\}\text{.}\)
Solution.
Note that \(\mathcal{B} = \left\{\vec u_1, \vec u_2\right\}\) is an orthogonal basis for \(W\) since \(\vec u_1 \cdot \vec u_2 = 0\text{.}\) Therefore,
\begin{align*}
\widehat y \amp = \proj_W\vec y \\
\amp = \proj_{\vec u_1}\vec y + \proj_{\vec u_2}\vec y\\
\amp = \dfrac{\vec y \cdot \vec u_1}{\vec u_1 \cdot \vec u_1} \vec u_1 + \dfrac{\vec y \cdot \vec u_2}{\vec u_2 \cdot \vec u_2}\vec u_2\\
\amp = \dfrac{7}{14}\begin{bmatrix} 3 \\ -1 \\ 2 \end{bmatrix} - \dfrac{15}{6}\begin{bmatrix} 1 \\ -1 \\ -2 \end{bmatrix}\\
\amp = \begin{bmatrix} -1 \\ 2 \\ 6 \end{bmatrix}\\
\amp = \vec y
\end{align*}
Note that \(\widehat y = \vec y\text{.}\) What does this mean? Since \(\widehat y\) must be a vector in \(W\text{,}\) it must be the case that \(\vec y \in W\text{.}\) Hopefully, it makes sense that for any vector already in \(W\text{,}\) say \(\vec w \in W\text{,}\) we will have \(\widehat w = \proj_W \vec w = \vec w\text{.}\) In this example, we have
\begin{equation*}
w^\perp = \vec y - \widehat y = \vec 0\text{.}
\end{equation*}
Recall that in general we should have that \(\widehat y \cdot w^\perp = 0\text{.}\) This is definitely true here.
Theorem 6.39 Best Approximation Theorem.
Let \(W\subseteq\mathbb{R}^n\) and \(\vec y\in\mathbb{R}^n\text{.}\) If \(\widehat y\) is the orthogonal projection of \(\vec y\) onto \(W\text{;}\) that is \(\widehat y = \proj_W\vec y\text{,}\) then for every \(\vec w\in W\) with \(\vec w\ne \widehat y\text{,}\)
\begin{equation*}
\lVert\vec y-\widehat y\rVert < \lVert\vec y-\vec w\rVert.
\end{equation*}
Thus, \(\widehat y\) is the unique closest vector in \(W\) to \(\vec y\text{.}\)
Definition 6.40 Distance from a Subspace.
Let \(W \subset \mathbb{R}^n\) and \(\vec y \in \mathbb{R}^n\text{.}\) The distance of a vector \(\vec y\) from a subspace \(W\), denoted \(\dist(\vec y, W)\text{,}\) is the minimum distance between \(\vec y\) and a vector \(\vec w \in W\text{;}\) that is,
\begin{equation*}
\dist(\vec y, W) = \operatorname{min}\left\{\dist(\vec y, \vec w) : w \in W\right\}\text{.}
\end{equation*}
By Theorem 6.39, we see that the vector in \(W\) which is closest to \(\vec y\) is \(\widehat y = \proj_W\vec y\text{.}\) Therefore,
\begin{equation*}
\dist(\vec y, W) = \dist(\vec y, \widehat y)\text{.}
\end{equation*}
Example 6.41.
Using the same vectors as Example 6.38:
\begin{equation*}
\vec y=\begin{bmatrix}-1\\2\\6\end{bmatrix},\quad
\vec u_1=\begin{bmatrix}3\\-1\\2\end{bmatrix},\quad
\vec u_2=\begin{bmatrix}1\\-1\\-2\end{bmatrix},
\end{equation*}
compute the distance from \(\vec y\) to \(W=\operatorname{Span}\{\vec u_1,\vec u_2\}\text{.}\)
Solution.
Recall form Example 6.38, we have that \(\widehat y = \vec y\text{.}\) Therefore, we have that
\begin{equation*}
\dist(\vec y, W) = \dist(\vec y, \widehat y) = \dist(\vec y, \vec y) = 0\text{,}
\end{equation*}
which makes sense since \(\vec y \in W\text{.}\)
Example 6.42.
Find the distance from \(\vec y\) to the subspace \(W = \Span\{\vec v_1, \vec v_2, \vec v_3\}\text{,}\) where
\begin{equation*}
\vec y = \begin{bmatrix} 11 \\ 1 \\ -2 \\ 6 \end{bmatrix}, \qquad \vec v_1 = \begin{bmatrix} 1 \\ 1 \\ 0 \\ -1 \end{bmatrix}, \qquad \vec v_2 = \begin{bmatrix} 1 \\ 0 \\ 1 \\ 1 \end{bmatrix}, \qquad \vec v_3 = \begin{bmatrix} 0 \\ -1 \\ 1 \\ -1\end{bmatrix}\text{.}
\end{equation*}
Solution.
First, we find \(\widehat y\text{.}\) Note that \(\left\{\vec v_1, \vec v_2, \vec v_3\right\}\) is an orthogonal basis for \(W\text{.}\) Check this. It follows that
\begin{align*}
\widehat y \amp = \proj_W\vec y \\
\amp = \proj_{\vec v_1}\vec y + \proj_{\vec v_2}\vec y + \proj_{\vec v_3}\vec y \\
\amp = \dfrac{\vec y \cdot \vec v_1}{\vec v_1 \cdot \vec v_1}\vec v_1 + \dfrac{\vec y \cdot \vec v_2}{\vec v_2 \cdot \vec v_2}\vec v_2 + \dfrac{\vec y \cdot \vec v_3}{\vec v_3 \cdot \vec v_3}\vec v_3 \\
\amp = 2\begin{bmatrix} 1 \\ 1 \\ 0 \\ -1 \end{bmatrix} + 5\begin{bmatrix} 1 \\ 0 \\ 1 \\ 1 \end{bmatrix} - 3\begin{bmatrix} 0 \\ -1 \\ 1 \\ -1 \end{bmatrix} \\
\amp = \begin{bmatrix} 7 \\ 5 \\ 2 \\ 6 \end{bmatrix}
\end{align*}
Note that in this example \(\widehat y \neq \vec y\text{.}\) Therefore, \(\vec y \not \in W\text{,}\) so there is a nonzero distance between \(\vec y\) and \(W\text{.}\) It follows that
\begin{equation*}
\dist(\vec y, W) = \dist(\vec y, \widehat y) = \|\vec y - \widehat y\| = \left\|\begin{bmatrix} 4 \\ -4 \\ -4 \\ 0 \end{bmatrix}\right\| = \sqrt{48}\text{.}
\end{equation*}
Example 6.43.
Indicate whether each statement is true or false. If true, explain why. If false, give a counterexample.
(a)
If \(\vec x\) is orthogonal to \(\vec v\) and \(\vec w\text{,}\) then \(\vec x\) is orthogonal to \(\vec v - \vec w\text{.}\)
Solution.
This is true. One way to prove this is by direct calculation. Since \(\vec x\) is orthogonal to \(\vec v\) and \(\vec w\text{,}\) we have that \(\vec x \cdot \vec v = 0\) and \(\vec x \cdot \vec w = 0\text{.}\) It follows that
\begin{equation*}
\vec x \cdot (\vec v - \vec w) = \vec x \cdot \vec v - \vec x \cdot \vec w = 0 - 0 = 0\text{.}
\end{equation*}
Thus, \(\vec x\) is orthogonal to \(\vec v - \vec w\text{.}\)
Another way to prove this is to strictly use theory. Note that if \(\vec v\) and \(\vec w\) are orthogonal to \(\vec x\text{,}\) then \(\vec v, \vec w \in \Span\{\vec x\}^\perp\text{.}\) Since \(\Span\{\vec x \}^\perp\) is a subspace, it is closed under linear combinations, so \(\vec v - \vec w \in \Span\{\vec x\}^\perp\text{.}\) Thus, \(\vec x\) is also orthogonal to \(\vec v - \vec w\text{.}\)
(b)
If \(\proj_{W}{\vec y} = \vec y\text{,}\) then \(\vec y \in W\text{.}\)
Solution.
This is true. By very definition, \(\proj_W\vec y \in W\text{,}\) so if \(\vec y = \proj_W\vec y\text{,}\) then \(\vec y \in W\text{.}\)
(c)
If \(\vec y = \vec u_1 + \vec v_1\text{,}\) where \(\vec u_1 \in W\) and \(\vec v_1 \in W^\perp\text{,}\) then \(\vec u_1\) is the orthogonal projection of \(\vec y\) onto \(W\text{.}\)
Solution.
This is true. By Theorem 6.36, we have \(\vec y = \widehat y + w^\perp\text{,}\) where \(\widehat y \in W\) and \(w^\perp \in W^\perp\) are uniquely chosen. Thus, we must have that
\begin{equation*}
\vec u_1 = \widehat y = \proj_W \vec y\text{.}
\end{equation*}
We can see this more explicitly using the transitive property and properties of subspace and orthogonal complements. Note that
\begin{equation*}
\vec y = \vec u_1 + \vec u_2 \quad \text{and} \quad \vec y = \widehat y + w^\perp\text{.}
\end{equation*}
Therefore, we have that
\begin{equation*}
\vec u_1 + \vec v_1 = \widehat y + w^\perp \quad \implies \quad \vec u_1 - \widehat y = w^\perp - \vec v_1\text{.}
\end{equation*}
Since \(w^\perp, \vec v_1 \in W^\perp\) and since \(W^\perp\) is closed under linear combinations (it is a subspace), we have that \(\vec u_1 - \widehat y = w^\perp - \vec v_1\in W^\perp\text{.}\) We also know that \(\vec u_1 - \widehat y \in W\) since each vector is in \(W\text{.}\) It follows that the vector \(\vec u_1 - \widehat y\) is a vector that is in both \(W\) and \(W^\perp\text{,}\) which means that it is a vector in \(W\) that is also orthogonal to every vector in \(W\text{,}\) including itself. The only vector orthogonal to itself is \(\vec 0\text{.}\) Thus,
\begin{equation*}
\vec u_1 - \widehat y = \vec 0 \quad \implies \quad \vec u_1 = \vec y = \proj_W\vec y\text{.}
\end{equation*}
