Objectives: Topics
-
Orthogonal sets of vectors
-
Orthogonal bases and projections
Print preview
Section 6.2 Orthogonal Sets, Orthogonal Bases, and Projections
Handout 6.2 Orthogonal Sets, Orthogonal Bases, and Projections
Objectives: Goals
-
Apply orthogonality concepts to:
-
compute orthogonal projections and distances,
-
express a vector as a linear combination of orthogonal vectors,
-
characterize bases for subspaces of \(\mathbb{R}^n\text{,}\)
-
construct orthonormal bases.
-
Motivating Question
What are the special properties of the following basis for \(\mathbb{R}^3\text{?}\)
\begin{equation*}
\frac{1}{\sqrt{11}}\begin{bmatrix}3\\1\\1\end{bmatrix},\quad
\frac{1}{\sqrt{6}}\begin{bmatrix}-1\\2\\1\end{bmatrix},\quad
\frac{1}{\sqrt{66}}\begin{bmatrix}-1\\-4\\7\end{bmatrix}
\end{equation*}
Definition 6.20 Orthogonal Set.
A set of vectors \(\{\vec u_1,\dots,\vec u_p\}\) is an orthogonal set if \(\vec u_j\cdot\vec u_k=0\) whenever \(j\ne k\text{.}\)
Example 6.21.
Fill in missing entries so that \(\{\vec u_1,\vec u_2,\vec u_3\}\) is an orthogonal set:
\begin{equation*}
\vec u_1 = \begin{bmatrix}4\\0\\1\end{bmatrix},\qquad
\vec u_2 = \begin{bmatrix}-2\\0\\ \phantom{1} \end{bmatrix},\qquad
\vec u_3 = \begin{bmatrix} \phantom{-1} \\ \phantom{1} \\ \phantom{1} \end{bmatrix}
\end{equation*}
Solution.
Let \(a\) represent the missing element in \(\vec u_2\text{.}\) It follows that
\begin{equation*}
0 = \vec u_1 \cdot \vec u_2 = \begin{bmatrix} 4 \\ 0 \\ 1 \end{bmatrix}\cdot \begin{bmatrix} -2 \\ 0 \\ a \end{bmatrix} = -8 + a \quad \implies \quad a = 8\text{.}
\end{equation*}
To find \(\vec u_3\text{,}\) we need to find a vector that is orthogonal to both \(\vec u_1\) and \(\vec u_2\text{.}\) Note that \(\vec v_3 = \vec 0\) works since the zero vector is orthogonal to every vector. Usually, we are interested in finding vectors that are linearly independent so that they will form a basis for our space, so let’s find a vector \(\vec u_3\) other than the zero vector. Let \(\vec u_3 = \begin{bmatrix}x_1 \\ x_2 \\ x_3 \end{bmatrix}\) and note that
\begin{equation*}
\vec u_1 \cdot \vec u_3 = 0 \quad \implies \quad 4x_1 + 0x_2 + 1x_3 = 0
\end{equation*}
and
\begin{equation*}
\vec u_2 \cdot \vec u_3 = 0 \quad \implies \quad -2x_1 + 0x_2 + 8x_3 = 0\text{.}
\end{equation*}
It follows that \(\vec u_3\) can be any vector in \(\Nul(A)\text{,}\) where \(A = \begin{bmatrix} 4 \amp 0 \amp 1 \\ -2 \amp 0 \amp 8 \end{bmatrix}\text{.}\) Feel free to find a non-zero vector in \(\Nul(A)\) by row reducing \(A\) and following the normal process. Or, you might be able to observe that
\begin{equation*}
\vec u_3 = \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix} \in \Nul(A)\text{,}
\end{equation*}
which makes it a fine choice for the third vector in the orthogonal set.
Theorem 6.22 Linear Independence of Orthogonal Sets.
If \(\{\vec u_1,\dots,\vec u_p\}\) is an orthogonal set of vectors, then:
\begin{equation*}
\left\lVert c_1\vec u_1 + \cdots + c_p\vec u_p\right\rVert^2
= c_1^2\lVert\vec u_1\rVert^2 + \cdots + c_p^2\lVert\vec u_p\rVert^2.
\end{equation*}
Furthermore, if all the vectors \(\vec u_1\) are nonzero, then the set is linearly independent.
Theorem 6.23 Expansion in an Orthogonal Basis.
If \(\{\vec u_1,\dots,\vec u_p\}\) is an orthogonal basis for a subspace \(W\text{,}\) then any \(\vec w\in W\) can be written:
\begin{equation*}
\vec w = c_1\vec u_1 + \cdots + c_p\vec u_p,
\end{equation*}
where
\begin{equation*}
c_k = \frac{\vec w\cdot\vec u_k}{\vec u_k\cdot\vec u_k}.
\end{equation*}
Solution.
Proof: Suppose \(\{\vec u_1,\dots,\vec u_p\}\) is an orthogonal basis for a subspace \(W\text{.}\) Then, we know that \(\vec u_i \cdot \vec u_j = 0\) for each \(i \neq j\) (since this is an orthogonal set) and \(\vec u_i \cdot \vec u_i \neq 0\) (since \(\vec u_i \neq \vec 0\)). Let \(\vec w \in W\text{.}\) Then, \(\vec w\) can be expressed uniquely as a linear combination of the basis vectors; that is
\begin{equation*}
\vec w = c_1\vec u_1 + c_2\vec u_2 + \cdots + c_p\vec u_p
\end{equation*}
for a unique choice of \(c_1, c_2 ,\dots, c_p\text{.}\) Note what happens if we dot product both sides of the above equation with \(\vec u_k\) for some \(k \in \{1,2,\dots, p\}\text{:}\)
\begin{align*}
\vec w \cdot \vec u_k \amp = (c_1\vec u_1 + \cdots + c_k\vec u_k + \cdots + c_p\vec u_p) \cdot \vec u_k \\
\vec u_k \cdot \vec w \amp = c_1(\vec u_1 \cdot \vec u_k) + \cdots + c_k(\vec u_k \cdot \vec u_k) + \cdots + c_p(\vec u_p \cdot u_k) \\
\vec w \cdot \vec u_k \amp = c_1(0) + \cdots + c_{k-1}(0) + c_k(\vec u_k \cdot \vec u_k) + c_{k+1}(0) + \cdots + c_p(0)
\end{align*}
It follows that
\begin{equation*}
\vec w \cdot \vec u_k = c_k(\vec u_k \cdot \vec u_k)\text{.}
\end{equation*}
Since \(\vec u_k \cdot \vec u_k \neq 0\text{,}\) we divide both sides by that quantity to get
\begin{equation*}
c_k = \dfrac{\vec w \cdot \vec u_k}{\vec u_k \cdot \vec u_k}\text{.}
\end{equation*}
Example 6.24.
Let
\begin{equation*}
\vec x=\begin{bmatrix}1\\1\\1\end{bmatrix},\qquad
\vec u=\begin{bmatrix}1\\-2\\1\end{bmatrix},\qquad
\vec v=\begin{bmatrix}-1\\0\\1\end{bmatrix},\qquad
\vec s=\begin{bmatrix}3\\-4\\1\end{bmatrix}.
\end{equation*}
and let \(W\) be the subspace of \(\mathbb{R}^3\) that is orthogonal to \(\vec x\text{.}\)
(a)
Check that \(\mathcal{B} = \left\{\vec u,\vec v\right\}\) forms an orthogonal basis for \(W\text{.}\)
Solution.
First, we show that \(\mathcal{B}\) is an orthogonal set. Note that
\begin{equation*}
\vec u \cdot \vec v = \begin{bmatrix} 1 \\ -2 \\ 1 \end{bmatrix} \cdot \begin{bmatrix} -1 \\ 0 \\ 1 \end{bmatrix} = -1 + 0 + 1 = 0\text{.}
\end{equation*}
Therefore, \(\vec u\) and \(\vec v\) are orthogonal, so \(\mathcal{B}\) is an orthogonal set.
Next, we show that \(\mathcal{B}\) is a basis for \(W\text{.}\) From the description of \(W\text{,}\) we see that \(W = \Span\{\vec x\}^\perp\text{.}\) Using the two results given in Definition 6.12, we see that
\begin{align*}
\dim(W) + \dim(W^\perp) \amp = \dim(\mathbb{R}^3)\\
\dim(W) + \dim(\Span\{\vec x\}) \amp = 3\\
\dim(W) \amp = 3-\dim(\Span\{\vec x\})\\
\dim(W) \amp = 2\text{.}
\end{align*}
Therefore, a basis for \(W\) must contain two linearly independent vectors that live in \(W\text{.}\) We can see that \(\vec u\) and \(\vec v\) since they are not multiples of one another. Another way to see this is that non-zero orthogonal vectors are always linearly independent. Lastly, we need to show that \(\vec u\) and \(\vec v\) actually live in \(W\text{.}\) Note that
\begin{equation*}
\vec u \cdot \vec x = 1 -2 + 1 = 0 \quad \text{and} \quad \vec v \cdot \vec x = -1 + 0 + 1 = 0
\end{equation*}
Thus, \(\vec u\) is orthogonal to \(\vec x\) and \(\vec v\) is orthogonal to \(\vec x\text{.}\) Hence, \(\vec u, \vec v \in W\text{,}\) so \(\mathcal{B}\) is a basis for \(W\text{.}\) Since \(\mathcal{B}\) was also an orthogonal set, it is an orthogonal basis for \(W\text{.}\)
(b)
Verify that \(\vec s \in W\text{.}\)
Solution.
Note that \(\vec s \cdot \vec x = 3 - 4 + 1 = 0\text{,}\) so \(\vec s\) is orthogonal to \(\vec x\text{.}\) Thus, \(\vec s \in W\text{.}\)
(c)
Compute the expansion of \(\vec s\) in the basis \(\mathcal{B}\text{.}\)
Solution.
Since \(\vec s \in W\) and since \(\mathcal{B} = \{\vec u, \vec v\}\) is a basis for \(W\text{,}\) we know there exists a unique choice of \(c_1,c_2 \in \mathbb{R}\) so that
\begin{equation*}
\vec s = c_1\vec u + c_2 \vec v\text{.}
\end{equation*}
\begin{equation*}
c_1 = \dfrac{\vec s \cdot \vec u}{\vec u \cdot \vec u} = \dfrac{3 + 8 + 1}{1 + 4 + 1} = \dfrac{12}{6} = 2
\end{equation*}
and
\begin{equation*}
c_2 = \dfrac{\vec s \cdot \vec v}{\vec v \cdot \vec v} = \dfrac{-3 + 0 + 1}{1 + 0 + 1} = \dfrac{-2}{2} = -1\text{.}
\end{equation*}
Therefore, we have that
\begin{equation*}
\vec s = 2\vec u - \vec v\text{,}
\end{equation*}
which can be easily checked.
Definition 6.25 Orthogonal Projection.
For a nonzero vector \(\vec u\) and vector \(\vec v\text{,}\) the orthogonal projection of \(\vec v\) onto \(\vec u\) is:
\begin{equation*}
\proj_{\vec u}(\vec v) = \frac{\vec v\cdot\vec u}{\vec u\cdot\vec u}\vec u.
\end{equation*}
Visualization of \(\operatorname{proj}_{\vec u}(\vec v)\) as the closest point in the span of \(\vec u\) to \(\vec v\text{.}\)
Theorem 6.26 Properties of Orthogonal Projections.
Let \(\vec u\) be a nonzero vector. Then, the vector \(\proj_{\vec u}\vec v\) is the vector in \(\Span\{\vec u\}\) which is closest to \(\vec v\text{.}\) Also, the vector \(\vec w=\vec v - \operatorname{proj}_{\vec u}(\vec v)\) is orthogonal to \(\vec u\text{.}\) Furthermore,
\begin{equation*}
\|\vec v\|^2 = \|\proj_{\vec u}{\vec v}\|^2 + \|\vec w\|^2\text{.}
\end{equation*}
Example 6.27.
Let \(\vec y = \begin{bmatrix} 7 \\ 6 \end{bmatrix}\) and \(\vec u = \begin{bmatrix} 4 \\ 2 \end{bmatrix}\text{.}\) Write \(\vec y\) as the sum of two orthogonal vectors, one in \(\Span\{\vec u\}\) and one orthogonal to \(\vec u\text{.}\)
Solution.
We want to find vectors \(\vec v_1 \in \Span\{\vec u\}\) and \(\vec v_2 \in \Span\{\vec u\}^\perp\) so that \(\vec y = \vec v_1 + \vec v_2\text{.}\) Note that this is exactly what projection will give me. We have that
\begin{equation*}
\vec v_1 = \proj_{\vec u}(\vec y) = \dfrac{\vec y \cdot \vec u}{\vec u \cdot \vec u}\vec u = \dfrac{28 + 12}{16 + 4}\begin{bmatrix} 4 \\ 2 \end{bmatrix} = \begin{bmatrix} 8 \\ 4 \end{bmatrix} \in \Span\{\vec u\}\text{.}
\end{equation*}
From Theorem 6.26, we see that \(\vec v_1\) is the vector in \(\Span\{\vec u\}\) which is closest to \(\vec y\text{.}\) To find \(\vec v_2 \in \Span\{\vec u\}^\perp\text{,}\) we use Theorem 6.26 and find that
\begin{equation*}
\vec v_2 = \vec y - \vec v_1 = \begin{bmatrix} 7 \\ 6 \end{bmatrix} - \begin{bmatrix} 8 \\ 4 \end{bmatrix} = \begin{bmatrix} -1 \\ 2 \end{bmatrix}\text{.}
\end{equation*}
We see that \(\vec v_2 \cdot \vec u = 0\text{.}\) Therefore,
\begin{equation*}
\vec y = \vec v_1 + \vec v_2 = \begin{bmatrix} 8 \\ 4 \end{bmatrix} + \begin{bmatrix} -1 \\ 2 \end{bmatrix}
\end{equation*}
is a decomposition of \(\vec y\) into two vectors, one living in \(\Span\{\vec u\}\) and the other living in \(\Span\{\vec u\}^\perp\text{.}\) We will see in Theorem 6.36 that this decomposition is unique.
Example 6.28.
Let \(L\) be the line spanned by \(\vec u=\begin{bmatrix}1\\1\\1\\1\end{bmatrix}\text{.}\)
(a)
Find the projection of \(\vec v = \begin{bmatrix} -3 \\ 5 \\ 6 \\ -4 \end{bmatrix}\) onto the line \(L\text{.}\)
Solution.
Note that
\begin{equation*}
\proj_L\vec v = \proj_{\vec u}\vec v = \dfrac{-3 + 5 + 6 - 4}{1 + 1 + 1 + 1}\begin{bmatrix} 1 \\ 1 \\ 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 1 \\ 1 \\ 1 \\ 1 \end{bmatrix} = \vec u\text{.}
\end{equation*}
There isn’t anything too special about \(\proj_{\vec u}\vec v = \vec u\text{.}\) Now, if we had instead gotten \(\proj_{\vec u}\vec v = \vec v\text{,}\) then we would learn that \(\vec v \in \Span\{\vec u\}\text{,}\) so \(\vec v\) is the vector in \(\Span\{\vec u\}\) which is closest to \(\vec v\text{.}\)
(b)
How close is \(\vec v\) to the line \(L\text{?}\)
Solution.
Note that the vector \(\vec w = \vec v - \proj_{\vec u}\vec v\) keeps track of how far away \(\vec v\) is from the line \(\Span\{\vec u\}\text{.}\) It follows that the distance \(\vec v\) is from \(L\) is given by
\begin{equation*}
\|\vec v - \proj_{\vec u}\vec v\| = \left\|\begin{bmatrix} -4 \\ 4 \\ 5 \\ -5 \end{bmatrix}\right\| = \sqrt{16 + 16 + 25 + 25} = \sqrt{82}\text{.}
\end{equation*}
Definition 6.29 Orthonormal Basis.
An orthonormal basis for a subspace \(W\) is an orthogonal basis in which each vector has length 1.
Theorem 6.30 Expansion in an Orthonormal Basis.
If \(\{\vec u_1,\dots,\vec u_p\}\) is orthonormal basis for \(W\) and \(\vec w\in W\text{,}\) then
\begin{equation*}
\vec w = (\vec w\cdot\vec u_1)\vec u_1 + \cdots + (\vec w\cdot\vec u_p)\vec u_p,
\end{equation*}
and
\begin{equation*}
\|\vec w\| =
\sqrt{
(\vec w\cdot\vec u_1)^2 + \cdots + (\vec w\cdot\vec u_p)^2
}.
\end{equation*}
Solution.
Proof: The first result is a direct corollary of Theorem 6.23 since for each \(k \in \{1,2, \dots, p\}\text{,}\) we have that \(\vec u_k \cdot \vec u_k = 1\) because \(\vec u_k\) is a unit vector.
Example 6.31.
Let \(W = \Span\left\{\begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}\right\}^\perp\text{.}\) Find the missing entries in an orthonormal basis for \(W\text{:}\)
\begin{equation*}
u=\frac{1}{\sqrt{\phantom{-1}}}
\begin{bmatrix}1\\0\\ \phantom{-1} \end{bmatrix},
\qquad
v=\frac{1}{\sqrt{\phantom{-1}}}
\begin{bmatrix} \phantom{-1} \\ \phantom{-1} \\ \phantom{-1} \end{bmatrix}
\end{equation*}
Then, express \(\vec w = \begin{bmatrix} 3 \\ 6 \\ -9 \end{bmatrix} \in W\) as a linear combination of \(\vec u\) and \(\vec v\text{.}\)
Solution.
Let \(\vec x = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}\text{.}\) In order for \(\vec u \cdot \vec x = 0\text{,}\) we see that the missing entry must be \(-1\text{.}\) Since the vector \(\begin{bmatrix} 1 \\ 0 \\ -1 \end{bmatrix}\) has magnitude \(\sqrt{2}\text{,}\) we see that
\begin{equation*}
\vec u = \dfrac{1}{\sqrt{2}}\begin{bmatrix}1 \\ 0 \\ -1 \end{bmatrix}\text{.}
\end{equation*}
We need to choose \(\vec v\) so that it is orthogonal to \(\vec x\) (which verifies that \(\vec v \in W\)) and orthogonal to \(\vec u\) (since we want an orthonormal basis). By observation, we see that the vector \(\begin{bmatrix} 1 \\ -2 \\ 1 \end{bmatrix}\) satisfies the orthogonality conditions. Since that vector has length \(\sqrt{6}\text{,}\) we choose
\begin{equation*}
\vec v = \dfrac{1}{\sqrt{6}}\begin{bmatrix} 1 \\ -2 \\ 1\end{bmatrix}\text{.}
\end{equation*}
By Theorem 6.30, we have that
\begin{equation*}
\vec w = (\vec w \cdot \vec u) \vec u + (\vec w \cdot \vec v) \vec v = \dfrac{12}{\sqrt{2}}\vec u - \dfrac{18}{\sqrt{6}}\vec v\text{.}
\end{equation*}
Checking, we see that
\begin{equation*}
\begin{bmatrix} 3 \\ 6 \\ -9 \end{bmatrix} = \dfrac{12}{\sqrt{2}}\left(\dfrac{1}{\sqrt{2}} \begin{bmatrix} 1 \\ 0 \\ -1 \end{bmatrix}\right) - \dfrac{18}{\sqrt{6}}\left(\dfrac{1}{\sqrt{6}}\begin{bmatrix} 1 \\ -2 \\ 1 \end{bmatrix}\right) = 6\begin{bmatrix} 1 \\ 0 \\ -1 \end{bmatrix} - 3 \begin{bmatrix} 1 \\ -2 \\ 1\end{bmatrix}
\end{equation*}
Theorem 6.32 Orthonormal Columns.
An \(m \times n\) matrix \(U\) has orthonormal columns if and only if \(U^TU = I_n\text{.}\)
Can \(U\) have orthonormal columns if \(n>m\text{?}\)
Solution.
Suppose \(U\) has more columns than rows. The, we know that \(U\) cannot have a pivot in each column. Therefore, the columns of \(U\) are linearly dependent. The only way they can form an orthogonal set is if the columns corresponding to free variables are the zero vector. However, this makes it impossible for each column to have length 1. Therefore, a matrix with more columns than rows can never have orthonormal columns.
Definition 6.33 Orthogonal Matrix.
An orthogonal matrix is a square matrix whose columns are orthonormal.
Theorem 6.34 Mapping Properties of Matrices with Orthonormal Columns.
If \(U\) is a matrix with orthonormal columns, then:
-
Preserves angles: \((U\vec x)\cdot(U\vec y)=\vec x\cdot\vec y\text{.}\)
Solution.
Using the defintion of dot product (Definition 6.1), we see that\begin{equation*} (U\vec x)\cdot (U\vec y) = (U\vec x)^T(U\vec y) = \vec x^T U^TU\vec y\text{.} \end{equation*}By Theorem 6.32, we see that \(U^TU = I_n\text{.}\) Therefore,\begin{equation*} (U\vec x)\cdot(U\vec y) = \vec x^TU^TU\vec y = \vec x^T\vec y = \vec x \cdot \vec y\text{.} \end{equation*} -
Preserves length: \(\| U\vec x\| = \|\vec x\|\text{.}\)
Solution.
This is a direct consequence of using Definition 6.4 and the above result with \((U\vec x)\) and itself. -
Preserves orthogonality.
Solution.
This is direct consequence of Theorem 6.18 and (a) above.
Example 6.35 Example.
Compute the length of
\begin{equation*}
\begin{bmatrix}
\frac12 \amp \frac{2}{\sqrt{14}}\\
\frac12 \amp \frac{1}{\sqrt{14}}\\
\frac12 \amp \frac{-3}{\sqrt{14}}\\
\frac12 \amp 0
\end{bmatrix}
\begin{bmatrix}
\sqrt{5}\\
-2
\end{bmatrix}.
\end{equation*}
Solution.
Note that the matrix \(U = \begin{bmatrix}
\frac12 \amp \frac{2}{\sqrt{14}}\\
\frac12 \amp \frac{1}{\sqrt{14}}\\
\frac12 \amp \frac{-3}{\sqrt{14}}\\
\frac12 \amp 0
\end{bmatrix}\) is a matrix with orthonormal columns. Therefore, we can use Theorem 6.34(b) to find that
\begin{equation*}
\left\|\begin{bmatrix}
\frac12 \amp \frac{2}{\sqrt{14}}\\
\frac12 \amp \frac{1}{\sqrt{14}}\\
\frac12 \amp \frac{-3}{\sqrt{14}}\\
\frac12 \amp 0
\end{bmatrix} \begin{bmatrix} \sqrt{5} \\ -2 \end{bmatrix}\right\| = \left\|\begin{bmatrix}\sqrt{5} \\ -2 \end{bmatrix}\right\| = \sqrt{5 + 4} = 3.
\end{equation*}
