Linear algebra

Vectors

A vector is a geometric object with magnitude (length) and direction, but no position. A vector with its tail at the origin O is called a position vector, and does have a position. Vectors are notated either in bold (\bm{v}), with an arrow above (\vec{v}), or with an underline (\underline{v}).

Vectors can be added and scaled to form a vector space.

\vec{a} + \vec{b} = \mat{a_1\\a_2\\\vdots\\a_n} + \mat{b_1\\b_2\\\vdots\\b_n} = \mat{a_1+b_1\\a_2+b_2\\\vdots\\a_n+b_n}

Matrices

x_1 \mat{1 \\ 2} + x_2 \mat{4 \\ 11} = \mat{2 \\ 1}

Pure mathematics

Linear algebra is the study of the algebraic properties of linear transformations.

Transformations

A transformation is any operation that transforms an input to an output. A transformation is linear if the principles of superposition and proportionality hold. Every linear transformation corresponds to a matrix.

The principle of superposition requires that the sum of inputs equals the sum of their respective outputs. For a linear system, if [e_1(t), r_1(t)] are excitation-response pairs, then if the excitation were e(t) = e_1(t) + e_2(t), the response would be r(t) = r_1(t) + r_2(t).

The principle of proportionality requires that every amplification of the input causes a corresponding amplification of the output. For a linear system, if the excitation were C_1e_1(t), where C_1 is constant, then the response would be C_1r_1(t).

Scalars

A scalar is any quantity that can be described by a single number. The set of all scalars is called the field of scalars, and is usually \bb{R}.

Vectors

A vector is any member of a vector space (tautological). A vector space is any class of objects that can be added together or multiplied by scalars. A vector space must be closed under vector addition and scalar multiplication.

The field of scalars can be seen as a vector field of dimension one. There is exactly one vector in a vector space of dimension zero, the zero vector (). The space \bb{C} of complex numbers can be seen as a two-dimensional vector space over \bb{R}.

Vector addition is performed by summing pairs of corresponding components. The sum of vectors of different dimension is undefined. Scalar multiplication is performed by multiplying each component by a scalar value.

Notable vector spaces

A unit vector is any vector with unit length.

n-tuples as vectors

For any integer n \ge 1, the vector space \bb{R}^n is defined to be the space of all n-tuples of reals.

\bb{R}^n = \{ (x_1,x_2,\cdots,x_n) : x_1,x_2,\cdots,x_n \in \bb{R} \}

Sequences as vectors

Let \bb{R}^\infty denote the vector space of all infinite sequences.

Matrices as vectors

For any integers m,n \ge 1, let M_{m \times n}(\bb{R}) denote the vector space of all m \times n matrices (m rows and n columns) with real entries.

Subspaces

A subspace V of a vector space that contains a specified vector v must also contain the vectors av for all scalars a (a vector space must be closed under scalar multiplication). The set

\{ av : a \in \bb{R} \}

is called the span of v.

If we construct a vector space V that contains the two specified vectors v and w, then V must contain not only v and w and every scalar multiple av and bw of v and w (where a and b are scalars), but also every linear combination av+bw (so that V is closed under vector addition). This will include all scalar multiples anyway.

Spans

A vector of the form av+bw, where a,b are scalars and v,w are vectors, is called a linear combination of v and w.

The set of all linear combinations of v and w is called the span of v and w:

\text{span}(\{ v,w \}) = \{ av + bw : a,b \in \bb{R} \}

The span of two vectors v and w of \bb{R}^3 is the plane through the origin that contains both v and w.

Determining membership in a span

We can determine whether a particular vector is in the span of some other vectors. For example, to determine whether the vector (0,1,2) is in the span of (1,1,1), (3,2,1), (1,0,1), we can find the scalars a_1, a_2, and a_3 such that

(0,1,2) = a_1 (1,1,1) + a_2 (3,2,1) + a_3 (1,0,1)

Multiplying the coefficients and summing the vectors gives us the vector

(a_1 + 3a_2 + a_3, a_1 + 2a_2, a_1 + a_2 + a_3)

This is the same as finding a_1, a_2, a_3 that solve the equations

\begin{aligned} a_1 & +3a_2 & +a_3 & =0 \\ a_1 & +2a_2 & & =1 \\ a_1 & +a_2 & +a_3 & =2 \\ \end{aligned}

This is a linear system of equations: “system” because it consists of more than one equation, and “linear” because the variables a_1, a_2, a_3 only appear as linear factors (as opposed to quadratic factors such as a_1^2 or a_2 a_3, or non-linear factors such as \sin(a_1)). This system can also be written in matrix form:

\begin{pmatrix} 1 & 3 & 1 \\ 1 & 2 & 0 \\ 1 & 1 & 1 \\ \end{pmatrix} \begin{pmatrix} a_1 \\ a_2 \\ a_3 \\ \end{pmatrix} = \begin{pmatrix} 0 \\ 1 \\ 2 \\ \end{pmatrix}

or schematically as

\left( \begin{array}{ccc|c} 1 & 3 & 1 & 0 \\ 1 & 2 & 0 & 1 \\ 1 & 1 & 1 & 2 \\ \end{array} \right)

The best method to solve this system of equations is Gaussian elimination. The idea is to try to make as many components of the matrix zero, using three basic moves:

  • Swap two rows
    The order of the equations in the system is not significant. This move is purely cosmetic.
  • Multiply a row by a constant
    We can multiply both sides of an equation by any non-zero constant. This is mostly cosmetic.
  • Subtract a multiple of one row from another
    This is the main move. Take any row, multiply it by any scalar, and subtract (or add) the result from another row; the original row remains unchanged. Use this to set one or more entries of a row to zero.

The process can be demonstrated for our example with the following moves:

  • Subtract the first row from the second and third rows
\left( \begin{array}{ccc|c} 1 & 3 & 1 & 0 \\ 0 & -1 & -1 & 1 \\ 0 & -2 & 0 & 2 \\ \end{array} \right)
  • Divide the third row by -2
\left( \begin{array}{ccc|c} 1 & 3 & 1 & 0 \\ 0 & -1 & -1 & 1 \\ 0 & 1 & 0 & -1 \\ \end{array} \right)
  • Subtract three copies of the third row from the first, add one copy of the third row to the second
\left( \begin{array}{ccc|c} 1 & 0 & 1 & 3 \\ 0 & 0 & -1 & 0 \\ 0 & 1 & 0 & -1 \\ \end{array} \right)
  • Finally, multiply the second row by -1, subract the second row from the first, and swap the second and third rows
\left( \begin{array}{ccc|c} 1 & 0 & 0 & 3 \\ 0 & 1 & 0 & -1 \\ 0 & 0 & 1 & 0 \\ \end{array} \right)

This gives us the solution, namely that a_1 = 3, a_2 = -1, a_3 = 0, and confirms that the vector (0,1,2) is a member of the span.

For this example there was exactly one solution for a_1, a_2, a_3 — they were exactly determined by the linear system. If there is more than one system, we say the system is under-determined — there are not enough equations to pin down each variable exactly. This often happens if there are more unknowns than equations.

Spanning sets

A set S is said to span a vector space V if \text{span}(S) = V (that is, if every vector of V can be generated as a linear combination of elements of S). We call S a spanning set for V.

An example of a spanning set is the set \{ (1,0,0), (0,1,0), (0,0,1) \} in \bb{R}^3; every vector in \bb{R}^3 can clearly be written as a linear combination of these three vectors.

Another example is the set \{ 1, x, x^2, x^3 \} of P_3(\bb{R}).

Adding a vector to a spanning set will always yield another spanning set, but the additional vector will always be superfluous (the set is linearly dependant).

A spanning set which contains no superfluous vectors is linearly independant, and is called a basis.

Matrices

Diagonal matrix

A matrix is diagonal if all the entries not in the main diagonal (top left to bottom right) are zero, such as:

\begin{pmatrix} 1 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & 3 \\ \end{pmatrix}

Trace

For any integer n \ge 1 and any matrix A \in M_{n \times n}, the trace tr(A) of the matrix is the sum of all entries on the diagonal.

For example, if

A = \begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \\ \end{pmatrix}

then

tr(A) = 1 + 5 + 9 = 15