Linear Methods of Applied Mathematics

Evans M. Harrell II and James V. Herod*

version of 5 August 1996, text with Maple. A Mathematica version is also available.

If you wish to print a nicely formatted version of this chapter, you may download the rtf file, which will be interpreted and opened by Microsoft Word.

(Some remarks for the instructor).

This course is concerned with some of the most important methods of applied mathematics, without which many technological developments of the nineteenth and twentieth centuries would have been impossible. When you have finished this course you will be able to solve most of the partial differential equations and integral equations that you are likely to encounter in engineering, and, just as importantly, you will understand what to expect from the solutions, which might emerge in a mysterious fashion from a computer at your workplace. You will also learn techniques that are useful in other contexts - Fourier series, special functions, and iteration.

The leitmotifs are linearity and orthogonality. By this stage inyour education, you are familiar with vectors. You have added them by thehead-to-tail construction, or by adding their components. You also know how to take their scalar (dot) product, producing a number equal to the product of the two lengths and the cosine of the angle between them:

v.w = |v| |w| cos(angle(v,w))

One of the most powerful and astonishing ideas in modern mathematics is that in many regards functions can be treated just like vectors. There is a very close analogy between sets of functions and spaces of vectors, which will guide us to striking solutions to several of the most important differential equations you will use in science and engineering, especially the wave equation, the heat equation, and the potential equation.

Mathematicians have a way of making analogies without the mushiness that so
often afflicts analogies in so many other contexts. It is called *abstraction*.
While abstraction may sometimes seem like theory divorced from reality, when
used properly it is a wonderful, and very practical, tool for solving
problems.

This is why we will begin by making vectors abstract. Let us recall some facts
about everyday vectors. If I have a bag full of vectors, I can add scale them
and add them together. We call the result a
*linear combination*:

I'll normally use Greek letters for scalars (= ordinary real or complex
numbers). It doesn't matter how many are in the combination, but unless I
explicitly state otherwise, I will assume that it is only a finite sum, a
*finite linear combination*. Of course, we can make linear combinations of
functions, too:

and that is another function. In this way, the set of all functions is a vector space.

**Definition I.1**. More formally, a *vector space* over the complex numbers is a
set of entities, abstractly called vectors, for which

1. Any finite linear combination of vectors is a member of the same set

2. The usual commutative rule holds for addition: v + w = w + v,

3. Just for consistency, the usual commutative, associative, and distributive laws hold for vector addition and multiplication by scalars. In other words

,

,

.

Also, 1 v = v.

In practice the rules in 3 are obvious and not very interesting. From these rules, you can show some other properties, such as:

There is a special element, which will be called the zero vector, equal to the scalar 0 times any vector whatsoever. It has the property that for any vector v, 0 + v = v.

For any vector v, there is a negative vector, called -v, such that v + (-v) = 0 (the zero vector).

Certain pretty reasonable conventions will be made, such as writing v - w
instead of v + (-
w).
Great - all the usual stuff works, right? Well, not quite. We *don't* assume
abstractly that we can multiply vectors by one another. Things like the dot
product and the cross product work for some vector spaces but not for others.
The other deep issue lurking in the shadows is infinite
linear combinations,
i.e., infinite series. Fourier liked them, and we'll see lots of them later.
But if the vectors are functions, perhaps the infinite series converges for
some x and not for others. For instance,

when x is a multiple of pi, but it is certainly an improper sum when x = pi/2, and it is not immediately clear what happens for other values of x. What does the series mean then?

1. The usual two-dimensional or three-dimensional vectors, which can be thought of as column vectors such as

[2] [3]

or as row vectors (2,3). In Maple's notation, vectors are represented as lists, that is, a string of items within brackets: [2,3]. The set of all such vectors will be denoted C^{2} or C^{3} (assuming complex entries are allowed - otherwise R^{2} or respectively R^{3}). Maple manipulates them in a straightforward way. First we need to let Maple know we are doing linear algebra (Note the colon).

with(linalg):evalm([2,3]+[4+I,2]); [6+I, 5]

(I is the square root of -1 in Maple, and *evalm* stands for "evaluate matrix," since Maple regards a vector as a matrix with only one row or column.) For the dot product,

dotprod([2,3],[1,2]); 8

2. The set of complex numbers. Here there is no difference between a vector and a scalar, and you can check all the properties pretty easily. We call this a one-dimensional vector space.

3. The set C^{n} of n numbers, manipulated just like 2- or 3- vectors, except that the number of components is some other fixed number, n. For instance,with C^{4}, we might have elements such as [1,2,3,4] and [1,0,-1,-2]

evalm([1,2,3,4]+[1,0,-1,-2]); [ 2, 2, 2, 2 ]dotprod([1,2,3,4],[1,0,-1,-2]); -10evalb("=1*1+2*0+3*(-1)+4*(-2)); true

(The ditto mark " refers to the result of the previous calculation in Maple, and
*evalb*, short for "evaluate Boolean," treats the input as a logical variable.
This is a convenient way to check two calculations which do not look identical.

4. The set of continuous functions of a variable x, 0 < x < 1. The rather stupid function f_{0}(x) = 0 for all x plays the role of the zero element.

5. Instead of listing n numbers, let us multiply them by n different functions, to define another vector space of functions. For example, for some fixed n, consider

a_{1} sin(x) + a_{2} sin(2x) + ... + a_{n} sin(nx),

where the numbers a_{k} can take on any value. Notice that this vector space is
a part of the previous one, in other words, a *subspace* .

6. The set of, say, 2 by 3 matrices. Addition means addition component-wise:

[1 0 i ] [2 -i 0 ] [3 -i i] [-1 pi 2+i] + [-1 -pi 2-i] = [-2 0 4]

and multiplication by scalars affects all components:

[1 0 i ] [5 0 5i ] 5 [-1 pi 2+i] = [-5 5 pi 10+5i]

In Maple a matrix is a list of lists - after all it consists of a string of row vectors:

m1:=[[1,0,I],[-1,Pi,2+I]]; m1:=[[1,0,I],[-1,\pi,2+I]];m2:=[[2,-I,0],[-1,-Pi,2-I]]; m1:=[[2,-I,0],[-1, -\pi,2-I]];evalm(m1+m2); m2 := [[2, - I, 0], [-1, - Pi, 2 - I]]evalm(5*m1); [ 5 0 5 I ] [ ] [ -5 5 Pi 10 + 5 I ]

7. The set of 3-component vectors (x,y,z) such that x - 2y + z = 0. This is a plane through the origin.

**Definitions I.3**. A set of vectors {v_{1}, ..., v_{n}} is
*linearly independent* if it
is impossible to write any of the vectors in the set as a
linear combination of
the rest.
*Linearly dependent* is the opposite notion. The *dimension* of a
vector space V is the largest number n of linearly independent vectors {v_{1},
..., v_{n}} which are contained in V. The *span* of a set {v_{1}, ..., v_{n}} is the
vector space V obtained by considering all possible linear combinations from
the set. We also say that {v_{1}, ..., v_{n}} *spans* V. A set {v_{1}, ..., v_{n}} is a
*basis* for a finite-dimensional vector space V if {v_{1}, ..., v_{n}} is linearly
independent and spans V.

Notice that the only way that two vectors can be linearly dependent is for them to be proportional (parallel) or for one of them to be 0. If the set has the same number of vectors as each vector has components, which frequently is the case, then there is a calculation to test for linear dependence. Array the vectors in a square matrix and calculate its determinant. If the determinant is 0, they are dependent, and otherwise they are independent. For example, consider the vectors {(1,2,3), (4,5,6),(7,8,9)}, which are not obviously linearly dependent. A calculation with Mathematica shows:

In:= Det[{{1,2,3},{4,5,6},{7,8,9}}]

Out= 0

Indeed, we can solve for one of these vectors as a linear combination of the others:

In:= Solve[{7,8,9} == alpha {1,2,3} + beta {4,5,6}, {alpha, beta}]

Out= {{alpha -> -1, beta -> 2}}

Many of our vector spaces will be infinite-dimensional. For example, {sin(n x)} for n = 1, 2, ..., is an infinite, linearly independent set of continuous functions - there is no way to write sin(400 x), for example, as a linear combination of sine functions with lower frequencies, so you can see that each time we introduce a sine function of a higher frequency into the list, it is independent of the sines that were already there. An infinite linearly independent set is a basis for V is every element of V is a limit of finite linear combinations of the set - but we shall have to say more later about such limits. A vector space has lots of different bases, but all bases contain the same number of items.

For practical purposes, the dimension of a set is the number of degrees of
freedom, i.e., the number of parameters it takes to describe the set. For
example, C^{n} has dimension n, and the set of 2 by 3 matrices has six
elements, so its dimension is 6.

**Model problem I.4**. Show that the plane x + y + z = 0 is a two-dimensional
vector space.

The verification of the vector space properties will be left to the reader. Here is how to show that the dimension is 2:

Solution. The solution can't be 3, since there are vectors in R^{3} which are
not in the plane, such as (1,1,1). On the other hand, here are two independent
vectors in the plane: (1,-1,0) and (1,1,-2).

Further observations. A general vector in the plane can be written with 2 parameters multiplying these two vectors, and it is not hard to find the formula to express the general vector this way. First write the general vector in the plane as (x,y,-x-y) (by substituting for z - notice

solve({x=alpha*1+beta*1,y=alpha*(-1)+beta*(1)}, {alpha, beta}); {alpha = - 1/2 y + 1/2 x, beta = 1/2 x + 1/2 y}

Hence the two vectors we found in the solution are a basis for the plane. Any two independent vectors in the plane form a basis.

**Definition I.5**. A *linear transformation* is a function on vectors, with the property that it
doesn't matter whether linear combinations are made before or after the
transformation. Formally,

Linear transformations are also called *linear operators*, or just operators for
short. You know plenty of examples:

**Examples I.6**.

1. Matrices. If M is a matrix and v, w etc. are column vectors, then

Think of rotation and reflection matrices here. If you put a bunch of vectors head-to-tail and rotate the assemblage, or look at it in a mirror, you get the same effect as if you first rotate or reflect the vectors, and then put them head-to-tail. It may be less geometrically obvious when the matrix distorts vectors in a trickier way, or there are more than three dimensions, but it is still true.

2. Derivatives and integrals. As we know,

As Maple puts it:

evalb(diff(alpha*f(x)+beta*g(x),x)=alpha*diff(f(x),x)+beta*diff(g(x),x)); True

3. This may seem silly at first, but the *identity operator* Id, which just
leaves functions alone is a linear transformation:

,

since both sides are just round-about ways of writing
alpha_{1} f_{1} + alpha_{2}f_{2}.
The identity operator is a useful bit of notation for much the same reason as the identity matrix,

[1 0 0] Id = [0 1 0] [0 0 1]

the effect of which on any vector is to leave it unchanged.

A linear transformation is a function defined on vectors, and the output is
always a vector, but *the output need not be the same kind of vector as the
input*. You should be familiar with this from matrices, since a 2 by 3
matrix acts on 3-vectors and produces 2-vectors. Similarly, the operator D
acts on a vector space of functions assumed to be differentiable and produces
functions which are not necessarily differentiable.

**Example I.7**. More exotic possibilities are possible, such as the operator
which acts on 2 by 2 matrices by the rule:

The whole theory we are going to develop begins with the analogy between
Example I.3.1 and Example I.3.2. We can think of the
*linear operator* of
differentiation by x as a kind of abstract matrix, denoted D (as inMathematica,
where f'[x] can be written D[f[x],x]; in Maple, this correspondsto diff) . If we
also think of the functions f and g as entities unto themselves, without focusing
on the variable x, then the expression in thefirst part of Example I.3.2 looks a
whole lot like Example I.3.1:

,

The custom with linear transformations, as with matrices, is to do without the
parentheses when the input variable is clear. It is tempting to manipulate
linear operators in many additional ways as if they were matrices, for instance
to multiply them together. This often works. For instance D^{2} = D D can be
thought of as the second derivative operator, and expressions such as D^{2} + D +
2 Id make sense. In passing, notice that if S is the integration operator of
the second part of Example 2, then D S f = f, so D is the inverse to S in the
sense that D S = Id. (But is S D = Id?)

**Linear ODE's.** In your course on ordinary differential equations, you studied
linear differential equations, a good example of which would be

D^{2}y + Dy + 2y = 0

More specifically, this is an example of a "linear, homogeneous differential
equation of second order, with constant coefficients". We can picture this
equation as one for the *null space* of a *differential operator* A:

A y := (D^{2} + D + 2 Id) y = 0.
(1.1)

(Maple renders this as

A:=u->diff(diff(u,x),x)+diff(u,x)+2*u; / 2 \ | d | / d \ A := u -> |----- u| + |---- u| + 2 u | 2 | \ dx / \ dx /

By definition, the null space N(A) is the set of all vectors solving this
equation; some texts refer to it as the *kernel* of A. There is no difference.
You may remember that the null space of a matrix is always a linear subspace,
and the same is true for the null space of any linear operator. E. g., the
plane of Example 1.2.6 is the null