Linear Algebra Review Part 1

References

• Linear Algebra Done Right, 4th Edition, Sheldon Axler

Reference

Helpful YouTube Videos

• Below are some YouTube videos that I watched to better understand some concepts:
• Dimension of Subspace
• • For a subspace ,
  • Examples:
  • Standard Basis: spans and is linearly independent (all of the columns have 1s).

Preface

You are probably about to begin your second exposure to linear algebra. Unlike your first brush with the subject, which probably emphasized Euclidean spaces and matrices, this encounter will focus on abstract vector spaces and linear maps.
[...]
You cannot read mathematics the way you read a novel. If you zip through a page is in less than an hour, you are probably going to fast.

• I am going to ignore this part (I don't have that much time to learn the subject deeply).
• Note for The Notes Below:
• • denotes or
  • typically denote vector spaces over

Vector Spaces

• Linear algebra is the study of linear maps on finite-dimensional vector spaces.
• A vector space is a set with operations of addition and scalar multiplication that satisfy natural algebraic properties.
• = set of real numbers
• Complex numbers were invented so that we can take the square root of negative numbers,
• Definition: Complex Numbers
• • A complex number is an ordered pair , where , but we will write this as .
  • The set of all complex numbers is denoted by :
  • Addition and Multiplication on C are defined by:

• We usually think of the set of all real numbers, , as the subset of C.

• Note that when is used, it is used as a substitute for C or
• Definition: list, length
• • Suppose that n is a nonnegative integer. A list of length n is an ordered collection of n elements (which might be numbers, other lists, or more abstract objects)
  • Two lists are equal if and only if they have the same length and the same elements in the same order
• Lists are often written as elements separated by commas and surrounded by parentheses. Thus a list of length two may be written as: . A list of length n may be written as .
• lists differ from finite sets in two ways: in lists, order matters and repetitions have meaning; in sets, order and repetitions are irrelevant
• Definition: Fⁿ
• • is the set of all lists of length n of elements of F
  • For , we say that x_k is the k^th coordinate of .

• Elements of can be thought of as points or as vectors.
• is an element of , and we may casually refer to it as a point in or a vector in without worrying about whether the geometry of has any physical meaning.
• The additive inverse of a vector in is the vector with the same length but pointing in the opposite direction.

• Definition: Scalar Multiplication in Fⁿ
• • The product of a number and a vector in is computed by multiplying each coordinate in the vector by
  • Here, .
• Scalar multiplications has a nice interpretation in : the constant lambda shrinks or stretches the vector by its own magnitude.
• A field is a set containing at least two distinct elements called 0 and 1, along with the operations of addition and subtraction satisfying all properties in 1.3.

• Definition: Addition, Scalar Multiplication
• • An addition on a set V is a function that assigns an element to each pair of elements
  • A scalar multiplication on a set is a function that assigns an element for each and each .
• Definition: Vector Space
• • A vector space is a set along with an addition on and a scalar multiplication on such that the following properties hold:

• Elements of a vector space are called vectors or points.
• is a vector space over , and is a vector space over .

• Definition: Subspace
• • A subset of is called a subspace of if is also a vector space with the same additive identity, addition, and scalar multiplication as on .
• Definition: Sum of Subspaces
• • Suppose are subspaces of . The sum of , denoted by , is the set of all possible sums of elements of . More precisely:

Finite Dimensional Vector Spaces

• Lists of vectors are usually written without surrounding parentheses.
• Definition: Linear Combination
• • A linear combination of a list of vectors in is a vector of the form where .
• Definition: Span
• • The set of all linear combinations of a list of vectors in is called the span of , denoted by span . In other words:

• • The span of an empty list is defined to be .
• Definition: Spans
• • If equals , we say that the list spans .
• Definition: Finite-Dimensional Vector Space
• • A vector space is called finite-dimensional if some list of vectors in it spans the space.
• Definition: Polynomial, P(F)
• • A function is called a polynomial with coefficients in id there exists such that:

• • for all .
  • is the set of all polynomials with coefficients in F.
• A vector space is called infinite-dimensional if it is not finite-dimensional.
• A list of vectors in is called linearly dependent if it is not linearly independent.
• A basis is a list of vectors in that is linearly independent and spans .
• Any two bases of a finite-dimensional vector space have the same length.
• The dimension of a finite-dimensional vector space is the length of any basis of the vector space.
• The dimension of a finite-dimensional vector space is denoted by .
• If finite-dimensional and is a subspace of
• Suppose is finite-dimensional. Then every linearly independent list of vectors in of length is a basis of .
• Suppose that is finite-dimensional and is a subspace of such that . Then .
• Suppose that is finite-dimensional. Then every spanning list of vectors in of length is a basis of .
• If and are subspaces of a finite-dimensional vector space, then .

Comparison between Finite Sets and Finite-Dimensional vector Spaces:

Linear Maps

• A linear map from to is a function with the following properties:
• • additivity
  • homogeneity
  • • .
• The set of linear maps form to is denoted by
• The set of linear maps form to is denoted by . In other words,
• Suppose is a basis of and . Then there exists a unique linear map such that for each .
• Suppose that and . The sum and the product are the linear maps from to defined by:

• for all .
• With the operations of addition and scalar multiplication as defined above, is a vector space.
• If and , then the product is defined by:

Page 56 - I'm stopping here and moving onto a different method of learning which you should be able to see in a later note Linear Algebra Part 2. This book was too theoretical for me right now.

Linear Algebra Cheat Sheet

• Reference

Matrix Basics

• A matrix is an array of numbers. means that:

• Two matrices can be multiplied if inner dimensions agree:

• Transpose: The transpose operator swaps rows and columns. If then and .
• Vector products: If are column vectors.
• • The inner product is (a.k.a. dot product)
  • The outer product is .
  • They are juts ordinary matrix multiplications.
• Inverse: Let (square). If there exists with or (if one holds, then the other holds with the same ) then is called the inverse of , denoted .
• Some properties of the matrix inverse:
• • is unique if it exists

Vector Norms

• A norm is a function satisfying these properties:
• • is and only if (definiteness)
  • for all (homogeneity)
  • (triangle inequality)

Linear Independence

• A set of vectors is linear independent if:

• If we define the matrix then the columns of A are linearly independent if .
• If the vectors are not linearly independent, then they are linearly dependent. In this case, at least one of the vectors is redundant (can be expressed as a linear combination of the others) i.e. there exists a and real numbers such that:

The Rank of a Matrix

• • = maximum number of linearly independent columns
  • = maximum number of linearly independent rows
• if and , then
• • If then
  • If then
• So, multiplying by an invertible matrix does not alter the rank.
• General properties of matrix rank:
• • is invertible if and only if

Linear Equations

• Given and , linear equations take the form .
• Where we must solve for . Three possibilities:
• • No solutions: Example: and
  • Exactly one solution: and
  • Infinitely many solutions:
• Two common cases:
• • Overdetermined: . Typically no solutions. One approach is least-squares: find to minimize .
  • Underdetermined: Typically infinitely many solutions. One approach is regularization; find the solution to such that is as small as possible.

Least Squares

• When the linear equations are overdetermined and there is no solution, one approach is to find an that almost works by minimizing the 2-norm of the residual:

• This problem always has a solution (not necessarily unique). minimizes the equation above if and only if satisfies the normal equations:

• The normal equations (and therefore (1)) have a unique solution if and only if the columns of are linearly independent, Then,

Range and Nullspace

• Given , we have the definitions:
• Range: . Note:
• Nullspace: . Note:
• The following statements are equivalent:
• • There exists a solution to the equation
• The following statements are equivalent:
• • Solutions to the equation are unique
  • • Remember:

Theorem:

Orthogonal Matrices

Projections

The Single Value Decomposition

• Every can be factored as:

• • is orthogonal, its columns are the left singular vectors
  • is orthogonal, its columns are the right singular vectors
  • is diagonal. are the singular values
• Complete the orthogonal matrices so they become square:

• The singular values are an intrinsic property of A. (The SVD is not unique, but every SVD of A has the same ).

• Singular vectors and singular vales satisfy:

• Suppose (full SVD) as above
• • rank:
  • transpose:
  • pseudoinverse:
• Fundamental subspaces:

• Matrix Norms: