Vector spaces

👋 About the exercise session

Let me first say that from my personal experience, this is the moment in the course where things get confusing. And that is alright. This is because in this part of the course we abstract a lot of things away. Abstract thinking is certainly type of thinking that needs some practice, but is equally important as the thinking about some concrete problems. Therefore, apart from learning about vector spaces, you also learn how to think in an abstract manner. And trust me when I say that this is a useful skill for a data scientist:

• Programming: You might have already heard of object oriented programming, maybe even about classes and instances of classes. Then, class is an abstraction of some object that you want to represent.

• Statistics: Next semester, you will learn about notion of random variables. This is yet another abstraction that allows you to reason about the given dataset at hand.

Bottom line: Abstract thinking is hard, but it is an important skill. Let this be an opportunity to learn it.

Now, to the vector spaces - example of yet another abstraction. Very simply put, vector space is a set of vectors, two operations (addition, scalar multiplication) and scalars (usually real numbers). The reason vector spaces is an abstraction is that it is up to us to define what the vectors actually represent. Here is a couple of examples of vectors:

• A set of ordered tuples of size 3: \((x_1, x_2, x_3)\)

• Set of \(2 \times 3\) matrices

In this session, we will focus on being able to identify vector spaces, work with their properties as well as proving that some subset of vectors is a subspace of a given vector space. Last but not the least, what should be your motivation to learn this apart from practicing your abstract thinking? Here is a couple of reasons:

• School perspective: You need to know this for another mandatory assignment as well as it might appear in the exam

• Least squares: In a couple of weeks, we will talk about least squares, i.e., given set of points, find a best linear model that fits these points. The parameters for this linear model are estimated via a method that is based on the notion of vector (sub)spaces

This was quite an exhaustive intro, sorry. 😞 Let’s get to work!

📓 Notes

Vector space definition and examples

I have already mentioned in the intro what is a vector space, but let me say it again because it is important:

vector space is a set of vectors, two operations (addition, scalar multiplication) and scalars (usually real numbers)

However, it is important to mention that this definition is not entirely complete. In addition, for the above defined set to be considered as vector space, it has to also satisfy 10 axioms defined on the page 161 in the book. Example of one of the axioms is for instance that \(u + v \in V\) where \(V\) is the given set. So why is this useful? Well, if someone gives you a set and tells you that you can assume it is a vector space, you know exactly what algebraic operations you can do and what to expect as a result. Yes, I admit this quite abstract and you will most likely not get into this situation unless you become a professional mathematician whose job is to actually prove certain theorems. Perhaps more concrete example from programming would be that if I hand you given type (class), you can assume that it has certain API (methods and attributes).

Back to vector spaces. In the intro, I have already given you some examples. So what are some non-examples? The books lists a couple:

• Set of all integers: if you multiply given integer by a fraction you get a number that is no longer part of the set of integers. Therefore, formally we say that it is not closed under scalar multiplication and since of the axioms has not been met, we can not say that the set of all integers is a valid vector space.

• The set of all second degree polynomials: if you have two second degree polynomials \(f_1(x) = x^2\) and \(f_2(x) = -x_2 + x\) and you add them you get first degree polynomial. Again, violation of one of the 10 axioms means we can not say that the given set is a vector space

So now we know what is a vector space, some concrete examples and non-examples. One last thing to add, how do we actually prove that given set is a vector space? Well, you simply have to prove that all 10 axioms are satisfied. But within the scope of this course, you do not need to worry since we will only be proving that certain subset of a vector space is its subspace - here the proving procedure is simpler.

Subspace of a vector space

Subspace might sound even more abstract than vector space, but it is not. You are a given a set of vectors which is said to be a subset of a particular vector space. The goal is to determine whether the given subset can be considered as a subspace of the given vector space. In other words, you want find out whether when working with this subset, you can use the exact same assumptions as you could use in the case of the vector space. In a pseudo code, you could write something like:

def is_subspace(set): 
  are_conditions_met = check_conditions(set)
  return are_conditions_met

Now, how does the check_conditions function looks like? Well, as I have promised, it is simpler than proving all 10 axioms. More specifically, to prove that given subset \(W\) is a subspace of a given vector space, it has to meet the following three conditions:

(1) \(W\) has to be non-empty: simply give a concrete example of a vector from this subset

(2) Let \(u, v \in W\), then \(u + v\) must also be part of the \(W\), this is called closure under addition

(3) Let \(u \in W\) and \(c\) be any scalar, then \(cu\) must again be part of the \(W\), this is called closure under scalar multiplication

If all these three are satisfied, you can safely assume that the given subset is a subspace of a given vector space. If any of the conditions is not satisfied, this is no longer possible. See the exercises where I show the proof procedure in detail.

Linear combination of vectors

We are given a vector \(v\) which is part of vector space \(V\). Then we can write it as a linear combination of other vectors \(u_1, \dots, u_k\) from the \(V\) as follows:

where \(c_1, \dots c_k\) are scalars. In practice, you are usually asked to show how to write the given vector \(v\) as a linear combination of given vectors \(u_1, \dots, u_k\). See for instance exercise 4.4.5.

Spanning sets

You are given some set of vectors \(S\) and the question is whether this set is a spanning set for given vector space \(V\). According to the definition, if \(S\) spans \(V\) then any vector \(v\) from \(V\) can be written as a linear combination of the vectors in \(S\). Let me give you a simple example based on the book. Let’s show that:

spans \(R^2\). Let \(v\) be an arbitrary vector from \(V\):

Then it must be true that

where \(u_1, u_2 \in S\). If we rewrite this as a system of linear equations:

All we care about is whether this system has a unique solution. Recall from the previous lecture that we can compute determinant of the coefficient matrix to find this out. If it is non-zero, then there must be a unique solution. And in fact this system does have non-zero determinant, so we proved that \(S\) spans \(V\).

Speaking of spanning sets, there are two more important things to note:

(1) Span of a set: this is a set of vectors which you can get by linearly combining vectors in the given set \(S\). If the set of these vectors that you can get is equal to the all vectors in the given vector space \(V\), you can say that \(S\) spans \(V\)

(2) Span(S) is a subspace of V: if \(S\) is a subset of \(V\) then its span is a subspace of \(V\).

Linear (in)dependence

Given set of vectors \(S\) is linearly independent if the following equation has only trivial solution:

Meaning all scalars are equal to zero. Otherwise, the given set \(S\) is said to be linearly dependent. In practice, you translate the above equation into the system of linear equations, use Gauss-Jordan elimination to solve it and then based on the solution you decide accordingly about the dependence. Sometimes, you can avoid this long procedure by checking if one of vectors in the set \(S\) is the scalar multiple of the other. And in general, if any of the vectors in \(S\) can be written as a linear combination of the other vectors, then \(S\) is linearly dependent set.

In the following lecture about basis and dimension, we will talk more about the use case of span(S) and linear in(dependence). So stay tuned!

⛳️ Learning goals checklist

What a week, a lot of new stuff for sure. Main takeaways are:

• Be able to determine if a given subset of a vector space is a subspace
• Be able to write a vector as a linear combination of other vectors
• Be able to determine if a given set of vectors span a vector space

See you next week!💥