Orthogonal sets of vectors

Section 3.1 Orthogonal sets of vectors

You may recall from elementary linear algebra, or a calculus class, that vectors in

R^{2}

R^{3}

are considered to be quantities with both magnitude and direction. Interestingly enough, neither of these properties is inherent to a general vector space. The vector space axioms specify only algebra; they say nothing about geometry. (What, for example, should be the “angle” between two polynomials?)

🔗

Because vector algebra is often introduced as a consequence of geometry (like the “tip-to-tail” rule), you may not have thought all that carefully about what, exactly, is responsible for making the connection between algebra and geometry. It turns out that the missing link is the humble dot product.

🔗

You probably encountered the following result, perhaps as a consequence of the law of cosines: for any two vectors

u, v \in R^{2},

u \cdot v = ‖ u ‖ ‖ v ‖ \cos θ,

🔗

where

θ

is the angle between

u

and

v .

Here we see both magnitude and direction (encoded by the angle) defined in terms of the dot product.

🔗

While it is possible to generalize the idea of the dot product to something called an inner product, we will first focus on the basic dot product in

R^{n} .

Once we have a good understanding of things in that setting, we can move on to consider the abstract counterpart.

🔗

Subsection 3.1.1 Basic definitions and properties

🔗

For most of this chapter (primarily for typographical reasons) we will denote elements of

R^{n}

as ordered

n

-tuples

(x_{1}, \dots, x_{n})

rather than as column vectors.

🔗

Definition 3.1.1.

🔗

Let

x = (x_{1}, x_{2}, \dots, x_{n})

and

y = (y_{1}, y_{2}, \dots, y_{n})

be vectors in

R^{n} .

The dot product of

x

and

y,

denoted by

x \cdot y

is the scalar defined by

x \cdot y = x_{1} y_{1} + x_{2} y_{2} + \dots + x_{n} y_{n} .

🔗

The norm of a vector

x

is denoted

‖ x ‖

and defined by

‖ x ‖ = \sqrt{x_{1}^{2} + x_{2}^{2} + \dots + x_{n}^{2}} .

🔗

Note that both the dot product and the norm produce scalars. Through the Pythagorean Theorem, we recognize the norm as the length of

x .

The dot product can still be thought of as measuring the angle between vectors, although the simple geometric proof used in two dimensions is not that easily translated to

n

dimensions. At the very least, the dot product lets us extend the notion of right angles to higher dimensions.

🔗

Definition 3.1.2.

🔗

We say that two vectors

x, y \in R^{n}

are orthogonal if

x \cdot y = 0 .

🔗

It should be no surprise that all the familiar properties of the dot product work just as well in any dimension. The folowing properties can be confirmed by direct computation, so the proof is left as an exercise.

🔗

Theorem 3.1.3.

🔗

For any vectors

x, y, z \in R^{n},

$x \cdot y = y \cdot x$
$x \cdot (y + z) = x \cdot y + x \cdot z$
For any scalar $c,$ $x \cdot (c y) = (c x) \cdot y = c (x \cdot y)$
$x \cdot x \geq 0,$ and $x \cdot x = 0$ if and only if $x = 0$

🔗

Remark 3.1.4.

🔗

The above properties, when properly abstracted, become the defining properties of a (real) inner product. (A complex inner product also involves complex conjugates.) For a general inner product, the requirement

x \cdot x \geq 0

is referred to as being positive-definite, and the property that only the zero vector produces zero when dotted with itself is called nondegenerate. Note that we have the following connection between norm and dot product:

‖ x ‖^{2} = x \cdot x .

🔗

For a general inner product, this can be used as a definition of the norm associated to an inner product.

🔗

Exercise 3.1.5.

🔗

Show that for any vectors

x, y \in R^{n},

we have

‖ x + y ‖^{2} = ‖ x ‖^{2} + 2 x \cdot y + ‖ y ‖^{2} .

Hint.

Use properties of the dot product to expand and simplify.

🔗

Exercise 3.1.6.

🔗

Suppose

R^{n} = span {v_{1}, v_{2}, \dots, v_{k}} .

Prove that

x = 0

if and only if

x \cdot v_{i} = 0

for each

i = 1, 2, \dots, k .

Hint.

Don’t forget to prove both directions! Note that the hypothesis allows you to write

x

as a linear combination of the

v_{i} .

🔗

There are two important inequalities associated to the dot product and norm. We state them both in the following theorem, without proof.

🔗

Theorem 3.1.7.

🔗

Let

x, y

be any vectors in

R^{n} .

Then

$| x \cdot y | \leq ‖ x ‖ ‖ y ‖$
$‖ x + y ‖ \leq ‖ x ‖ + ‖ y ‖$

🔗

The first of the above inequalities is called the Cauchy-Schwarz inequality, which be viewed as a manifestation of the formula

x \cdot y = ‖ x ‖ ‖ y ‖ \cos θ,

🔗

since after all,

| \cos θ | \leq 1

for any angle

θ .

🔗

The usual proof involves some algebraic trickery; the interested reader is invited to search online for the Cauchy-Schwarz inequality, where they will find no shortage of websites offering proofs.

🔗

The second result, called the triangle inequality, follows immediately from the Cauchy-Scwarz inequality and Exercise 3.1.5:

‖ x + y ‖^{2} = ‖ x ‖^{2} + 2 x \cdot y + ‖ y^{2} ‖ \leq ‖ x ‖^{2} + 2 ‖ x ‖ ‖ y ‖ + ‖ y ‖^{2} = (‖ x ‖ + ‖ y ‖)^{2} .

🔗

The triangle inequality gets its name from the “tip-to-tail” picture for vector addition. Essentially, it tells us that the length of any side of a triangle must be less than the sum of the lengths of the other two sides. The importance of the triangle inequality is that it tells us that the norm can be used to define distance.

🔗

Definition 3.1.8.

🔗

For any vectors

x, y \in R^{n},

the distance from

x

y

is denoted

d (x, y),

and defined as

d (x, y) = ‖ x - y ‖ .

🔗

Remark 3.1.9.

🔗

Using properties of the norm, we can show that this distance function meets the criteria of what’s called a metric. A metric is any function that takes a pair of vectors (or points) as input, and returns a number as output, with the following properties:

$d (x, y) = d (y, x)$ for any $x, y$
$d (x, y) \geq 0,$ and $d (x, y) = 0$ if and only if $x = y$
$d (x, y) \leq d (x, z) + d (z, y)$ for any $x, y, z$

🔗

We leave it as an exercise to confirm that the distance function defined above is a metric.

🔗

In more advanced courses (e.g. topology or analysis) you might go into detailed study of these structures. There are three interrelated structures: inner products, norms, and metrics. You might consider questions like: does every norm come from an inner product? Does every metric come from a norm? (No.) Things get even more interesting for infinite-dimensional spaces. Of special interest are spaces such as Hilbert spaces (a special type of infinite-dimensional inner product space) and Banach spaces (a special type of infinite-dimensional normed space).

🔗

Exercise 3.1.10.

🔗

Select all vectors that are orthogonal to the vector

(2, 1, - 3)

$(1, 1, 1)$
Yes! $2 (1) + 1 (1) - 3 (1) = 0 .$
$(3, 1, 2)$
You should find that the dot product is $1,$ not $0,$ so these vectors are not orthogonal.
$(0, 0)$
You might be tempted to say that the zero vector is orthogonal to everything, but we can’t compare vectors from different vector spaces!
$(0, - 3, - 1)$
Yes! We have to be careful of signs here: $2 (0) + 1 (- 3) + (- 3) (- 1) = 0 - 3 + 3 = 0 .$

🔗

Exercise 3.1.11.

🔗

u

is orthogonal to

v

and

v

is orthogonal to

w,

then

u

is orthogonal to

w .

True.
Consider $u = (1, 0, 0),$ $v = (0, 1, 0),$ and $w = (1, 0, 1) .$
False.
Consider $u = (1, 0, 0),$ $v = (0, 1, 0),$ and $w = (1, 0, 1) .$

🔗

Subsection 3.1.2 Orthogonal sets of vectors

🔗

In Chapter 1, we learned that linear independence and span are important concepts associated to a set of vectors. In this chapter, we learn what it means for a set of vectors to be orthogonal, and try to understand why this concept is just as important as independence and span.

🔗

Definition 3.1.12.

🔗

A set of vectors

{v_{1}, v_{2}, \dots, v_{k}}

R^{n}

is called orthogonal if:

$v_{i} \neq 0$ for each $i = 1, 2 \dots, k$
$v_{i} \cdot v_{j} = 0$ for all $i \neq j$

🔗

Exercise 3.1.13.

🔗

Show that the following is an orthogonal subset of

R^{4} .

{(1, 0, 1, 0), (- 1, 0, 1, 1), (1, 1, - 1, 2)}

🔗

Can you find a fourth vector that is orthogonal to each vector in this set?

Hint.

The dot product of the fourth vector with each vector above must be zero. Can you turn this requirement into a system of equations?

🔗

Exercise 3.1.14.

🔗

{v, w}

and

{x, y}

are orthogonal sets of vectors in

R^{n},

then

{v, w, x, y}

is an orthogonal set of vectors.

True.
Try to construct an example. The vector $x$ has to be orthogonal to $y,$ but is there any reason it has to be orthogonal to $v$ or $w ?$
False.
Try to construct an example. The vector $x$ has to be orthogonal to $y,$ but is there any reason it has to be orthogonal to $v$ or $w ?$

🔗

The requirement that the vectors in an orthogonal set be nonzero is partly because the alternative would be boring, and partly because it lets us state the following theorem.

🔗

Theorem 3.1.15.

🔗

Any orthogonal set of vectors is linearly independent.

Strategy.

Any proof of linear independence should start by defining our set of vectors, and assuming that a linear combination of these vectors is equal to the zero vector, with the goal of showing that the scalars have to be zero.

Set up the equation (say,

c_{1} v_{1} + \dots c_{n} v_{n} = 0

), with the assumption that your set of vectors is orthogonal. What happens if you take the dot product of both sides with one of these vectors?

Proof.

Suppose

S = {v_{1}, v_{2}, \dots, v_{k}}

is orthogonal, and suppose

c_{1} v_{1} + c_{2} v_{2} + \dots + c_{k} v_{k} = 0

for scalars

c_{1}, c_{2}, \dots, c_{k} .

Taking the dot product of both sides of the above equation with

v_{1}

gives

\begin{aligned} c_{1} (v_{1} \cdot v_{1}) + c_{2} (v_{1} \cdot v_{2}) + \dots + c_{k} (v_{1} \cdot v_{k}) & = v_{1} \cdot 0 \\ c_{1} ‖ v_{1} ‖^{2} + 0 + \dots + 0 & = 0 . \end{aligned}

Since

‖ v_{1} ‖^{2} \neq 0,

we must have

c_{1} = 0 .

We similarly find that all the remaining scalars are zero by taking the dot product with

v_{2}, \dots, v_{k} .

🔗

Another useful consequence of orthogonality: in two dimensions, we have the Pythagorean Theorem for right-angled triangles. If the “legs” of the triangle are identified with vectors

x

and

y,

and the hypotenuse with

z,

then

‖ x ‖^{2} + ‖ y ‖^{2} = ‖ z ‖^{2},

since

x \cdot y = 0 .

🔗

n

dimensions, we have the following, which follows from the fact that all “cross terms” (dot products of different vectors) will vanish.

🔗

Theorem 3.1.16. Pythagorean Theorem.

🔗

For any orthogonal set of vectors

{x_{1}, \dots, x_{k}}

we have

‖ x_{1} + \dots + x_{k} ‖^{2} = ‖ x_{1} ‖^{2} + \dots + ‖ x_{k} ‖^{2} .

Strategy.

Remember that

‖ x_{1} + \dots + x_{k} ‖^{2} = (x_{1} + \dots + x_{k}) \cdot (x_{1} + \dots + x_{k}),

and use the distributive property of the dot product, along with the fact that each pair of different vectors is orthogonal.

🔗

Our final initial result about orthogonal sets of vectors relates to span. In general, we know that if

y \in span {x_{1}, \dots, x_{k}},

then it is possible to solve for scalars

c_{1}, \dots, c_{k}

such that

y = c_{1} x_{1} + \dots + c_{k} x_{k} .

The trouble is that finding these scalars generally involves setting up, and then solving, a system of linear equations. The great thing about orthogonal sets of vectors is that we can provide explicit formulas for the scalars.

🔗

Theorem 3.1.17. Fourier expansion theorem.

🔗

Let

S = {v_{1}, v_{2}, \dots, v_{k}}

be an orthogonal set of vectors. For any

y \in span S,

we have

y = (\frac{y \cdot v_{1}}{v_{1} \cdot v_{1}}) v_{1} + (\frac{y \cdot v_{2}}{v_{2} \cdot v_{2}}) v_{2} + \dots + (\frac{y \cdot v_{k}}{v_{k} \cdot v_{k}}) v_{k} .

Strategy.

Take the same approach you used in the proof of Theorem 3.1.15, but this time, with a nonzero vector on the right-hand side.

Proof.

Let

y = c_{1} v_{1} + \dots + c_{k} v_{k} .

Taking the dot product of both sides of this equation with

v_{i}

gives

v_{i} \cdot y = c_{i} (v_{i} \cdot v_{i}),

since the dot product of

v_{i}

with

v_{j}

for

i \neq j

is zero.

🔗

One use of Theorem 3.1.17 is determining whether or not a given vector is in the span of an orthogonal set. If it is in the span, then its coefficients must satisfy the Fourier expansion formula. Therefore, if we compute the right hand side of the above formula and do not get our original vector, then that vector must not be in the span.

🔗

Exercise 3.1.18.

🔗

Determine whether or not the vectors

v = (1, - 4, 3, - 11), w = (3, 1, - 4, 2)

belong to the span of the vectors

x_{1} = (1, 0, 1, 0), x_{2} = (- 1, 0, 1, 1), x_{3} = (1, 1, - 1, 2) .

🔗

(We confirmed that

{x_{1}, x_{2}, x_{3}}

is an orthogonal set in Exercise 3.1.13.)

🔗

The Fourier expansion is especially simple if our basis vectors have norm one, since the denominators in each coefficient disappear. Recall that a unit vector in

R^{n}

is any vector

x

with

‖ x ‖ = 1 .

For any nonzero vector

v,

a unit vector (that is, a vector of norm one) in the direction of

v

is given by

\hat{u} = \frac{1}{‖ v ‖} v .

🔗

We often say that the vector

u

is normalized. (The convention of using a “hat” for unit vectors is common but not universal.)

🔗

Exercise 3.1.19.

🔗

Match each vector on the left with a parallel unit vector on the right.

🔗

Remember that a parallel vector is a scalar multiple of the one you started with.

$⟨ 2, - 1, 2 ⟩$
$⟨ \frac{2}{3}, - \frac{1}{3}, \frac{2}{3} ⟩$
$⟨ 3, 0, - 4 ⟩$
$⟨ \frac{3}{5}, 0, - \frac{4}{5} ⟩$
$⟨ 1, 2, 1 ⟩$
$⟨ \frac{1}{\sqrt{6}}, \frac{2}{\sqrt{6}}, \frac{1}{\sqrt{6}} ⟩$
$⟨ 2, 0, 1 ⟩$
$⟨ \frac{2}{\sqrt{5}}, 0, \frac{1}{\sqrt{5}} ⟩$

🔗

Definition 3.1.20.

🔗

A basis

B

R^{n}

is called an orthonormal basis if

B

is orthogonal, and all the vectors in

B

are unit vectors.

🔗

Example 3.1.21.

🔗

In Exercise 3.1.13 we saw that the set

{(1, 0, 1, 0), (- 1, 0, 1, 1), (1, 1, - 1, 2), (1, - 6, - 1, 2)}

🔗

is orthogonal. Since it’s orthogonal, it must be independent, and since it’s a set of four independent vectors in

R^{4},

it must be a basis. To get an orthonormal basis, we normalize each vector:

\begin{aligned} {\hat{u}}_{1} & = \frac{1}{\sqrt{1^{2} + 0^{2} + 1^{2} + 0^{2}}} (1, 0, 1, 0) = \frac{1}{\sqrt{2}} (1, 0, 1, 0) \\ {\hat{u}}_{2} & = \frac{1}{\sqrt{(- 1)^{2} + 0^{2} + 1^{2} + 1^{2}}} (- 1, 0, 1, 1,) = \frac{1}{\sqrt{3}} (- 1, 0, 1, 1) \\ {\hat{u}}_{3} & = \frac{1}{\sqrt{1^{2} + 1^{2} + (- 1)^{2} + 2^{2}}} (1, 1, - 1, 2) = \frac{1}{\sqrt{7}} (1, 1, - 1, 2) \\ {\hat{u}}_{4} & = \frac{1}{\sqrt{1^{2} + (- 6)^{2} + (- 1)^{2} + 2^{2}}} (1, - 6, - 1, 2) = \frac{1}{\sqrt{42}} (1, - 6, - 1, 2) . \end{aligned}

🔗

The set

{{\hat{u}}_{1}, {\hat{u}}_{2}, {\hat{u}}_{3}, {\hat{u}}_{4}}

is then an orthonormal basis of

R^{4} .

🔗

The process of creating unit vectors does typically introduce square root coefficients in our vectors. This can seem undesirable, but there remains value in having an orthonormal basis. For example, suppose we wanted to write the vector