Linear algebra

Linear algebra is the study of the algebraic properties of linear transformations.

Transformations

A transformation is any operation that transforms an input to an output.

A transformation is linear if every amplification of the input causes a corresponding amplification of the output, and if the sum of inputs equals the sum of their respective outputs.

Every linear transformation corresponds to a matrix.

Scalars

A scalar is any quantity that can be described by a single number. The set of all scalars is called the field of scalars, and is usually \bb{R}.

Vectors

A vector is any member of a vector space; a vector space is any class of objects that can be added together, or multiplied with scalars. A vector space must be closed under vector addition and scalar multiplication.

Vectors can be summed by summing the corresponding components separately, and can be multiplied by a scalar by multiplying each component separately. The sum of vectors of different dimension is undefined.

The field of scalars can be seen as a vector field of dimension one.

There is exactly one vector in a vector space of dimension zero, the zero vector ().

The space \bb{C} of complex numbers can be viewed as a two-dimensional vector space over \bb{R}.

A unit vector is any vector with unit length.

n-tuples as vectors

For any integer n \ge 1, the vector space \bb{R}^n is defined to be the space of all n-tuples of reals.

\bb{R}^n = \{ (x_1,x_2,\cdots,x_n) : x_1,x_2,\cdots,x_n \in \bb{R} \}

Sequences as vectors

Let \bb{R}^\infty denote the vector space of all infinite sequences.

Matrices as vectors

For any integers m,n \ge 1, let M_{m \times n}(\bb{R}) denote the vector space of all m \times n matrices (m rows and n columns) with real entries.

Subspaces

A subspace V of a vector space that contains a specified vector v must also contain the vectors av for all scalars a (a vector space must be closed under scalar multiplication). The set

\{ av : a \in \bb{R} \}

is called the span of v.

If we construct a vector space V that contains the two specified vectors v and w, then V must contain not only v and w and every scalar multiple av and bw of v and w (where a and b are scalars), but also every linear combination av+bw (so that V is closed under vector addition). This will include all scalar multiples anyway.

Spans

A vector of the form av+bw, where a,b are scalars and v,w are vectors, is called a linear combination of v and w.

The set of all linear combinations of v and w is called the span of v and w:

\text{span}(\{ v,w \}) = \{ av + bw : a,b \in \bb{R} \}

The span of two vectors v and w of \bb{R}^3 is the plane through the origin that contains both v and w.

Determing membership in a span

We can determine whether a particular vector is in the span of some other vectors. For example, to determine whether the vector (0,1,2) is in the span of (1,1,1), (3,2,1), (1,0,1), we can find the scalars a_1, a_2, and a_3 such that

(0,1,2) = a_1 (1,1,1) + a_2 (3,2,1) + a_3 (1,0,1)

Multiplying the coefficients and summing the vectors gives us the vector

(a_1 + 3a_2 + a_3, a_1 + 2a_2, a_1 + a_2 + a_3)

This is the same as finding a_1, a_2, a_3 that solve the equations

\begin{aligned} a_1 & +3a_2 & +a_3 & =0 \\ a_1 & +2a_2 & & =1 \\ a_1 & +a_2 & +a_3 & =2 \\ \end{aligned}

This is a linear system of equations: “system” because it consists of more than one equation, and “linear” because the variables a_1, a_2, a_3 only appear as linear factors (as opposed to quadratic factors such as a_1^2 or a_2 a_3, or non-linear factors such as \sin(a_1)). This system can also be written in matrix form:

\begin{pmatrix} 1 & 3 & 1 \\ 1 & 2 & 0 \\ 1 & 1 & 1 \\ \end{pmatrix} \begin{pmatrix} a_1 \\ a_2 \\ a_3 \\ \end{pmatrix} = \begin{pmatrix} 0 \\ 1 \\ 2 \\ \end{pmatrix}

or schematically as

\left( \begin{array}{ccc|c} 1 & 3 & 1 & 0 \\ 1 & 2 & 0 & 1 \\ 1 & 1 & 1 & 2 \\ \end{array} \right)

The best method to solve this system of equations is Gaussian elimination. The idea is to try to make as many components of the matrix zero, using three basic moves:

  • Swap two rows
    The order of the equations in the system is not significant. This move is purely cosmetic.
  • Multiply a row by a constant
    We can multiply both sides of an equation by any non-zero constant. This is mostly cosmetic.
  • Subtract a multiple of one row from another
    This is the main move. Take any row, multiply it by any scalar, and subtract (or add) the result from another row; the original row remains unchanged. Use this to set one or more entries of a row to zero.

The process can be demonstrated for our example with the following moves:

  • Subtract the first row from the second and third rows
\left( \begin{array}{ccc|c} 1 & 3 & 1 & 0 \\ 0 & -1 & -1 & 1 \\ 0 & -2 & 0 & 2 \\ \end{array} \right)
  • Divide the third row by -2
\left( \begin{array}{ccc|c} 1 & 3 & 1 & 0 \\ 0 & -1 & -1 & 1 \\ 0 & 1 & 0 & -1 \\ \end{array} \right)
  • Subtract three copies of the third row from the first, add one copy of the third row to the second
\left( \begin{array}{ccc|c} 1 & 0 & 1 & 3 \\ 0 & 0 & -1 & 0 \\ 0 & 1 & 0 & -1 \\ \end{array} \right)
  • Finally, multiply the second row by -1, subract the second row from the first, and swap the second and third rows
\left( \begin{array}{ccc|c} 1 & 0 & 0 & 3 \\ 0 & 1 & 0 & -1 \\ 0 & 0 & 1 & 0 \\ \end{array} \right)

This gives us the solution, namely that a_1 = 3, a_2 = -1, a_3 = 0, and confirms that the vector (0,1,2) is a member of the span.

For this example there was exactly one solution for a_1, a_2, a_3 — they were exactly determined by the linear system. If there is more than one system, we say the system is under-determined — there are not enough equations to pin down each variable exactly. This often happens if there are more unknowns than equations.

Spanning sets

A set S is said to span a vector space V if \text{span}(S) = V (that is, if every vector of V can be generated as a linear combination of elements of S). We call S a spanning set for V.

An example of a spanning set is the set \{ (1,0,0), (0,1,0), (0,0,1) \} in \bb{R}^3; every vector in \bb{R}^3 can clearly be written as a linear combination of these three vectors.

Another example is the set \{ 1, x, x^2, x^3 \} of P_3(\bb{R}).

Adding a vector to a spanning set will always yield another spanning set, but the additional vector will always be superfluous (the set is linearly dependant).

A spanning set which contains no superfluous vectors is linearly independant, and is called a basis.

Matrices

Diagonal matrix

A matrix is diagonal if all the entries not in the main diagonal (top left to bottom right) are zero, such as:

\begin{pmatrix} 1 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & 3 \\ \end{pmatrix}

Trace

For any integer n \ge 1 and any matrix A \in M_{n \times n}, the trace tr(A) of the matrix is the sum of all entries on the diagonal.

For example, if

\begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \\ \end{pmatrix}

then

tr(A) = 1 + 5 + 9 = 15