Eigenvectors and Eigenvalues
The eigenvectors and eigenvalues of a matrix play an important role in engineering and science. A few of the areas in which it can be applied include Spectral Clustering, PCA, The Google Page Rank algorithm, facial recognition and quantum mechanics. Nevertheless, what are eigenvectors and eigenvalues, when to use them, and how can they be used? Certainly, for anyone who wants to know machine learning algorithms in a deeper way, knowing eigenvectors and eigenvalues will be a helpful topic and that is precisely what we will discuss in this article.
What are eigenvalues and eigenvectors?
Eigen means characteristic in German. In other words, when we talk about Eigenproblems, we’re referring to identifying what something’s characteristic properties are. When a linear transformation is applied to a vector, its characteristic vector is a nonzero vector that changes by only a scalar factor. By scaling the Eigenvector by lambda, one obtains the corresponding Eigenvalue. For the purpose of understanding eigenvectors, we must understand linear transformations. Scaling, rotation, and shearing are some of the matrix transformation operations. Most often, we think about how these transformations might affect a specific vector when applying them. If they are applied to every vector in this space, it will also be useful to consider what that will look like. Let us look at this linear transformation example, take a look at these three vectors and see what happens if we do a vertical scaling on them:
Using the example above, you can see that the orange vector stayed the same, pointing in the same direction and the same length as before. The blue vector, on the other hand, points in the same direction but has a different length. Lastly, the purple vector used to be 45 degrees, but has now increased in angle and length. Consequently, you can say that with the exception of the horizontal (orange) and vertical (blue) vectors, all the other vector directions changed in some sense, which shows why we referred to the orange and blue vectors as eigenvectors as they represent the characteristics of this particular transformation. Furthermore, we can say that eigenvalue 1 corresponding to the horizontal vector, while eigenvalue 2 corresponding to the vertical vector.
The principle underlying eigenvectors can now simply be defined as those which lie along the same span of a space in both cases, before and after linear transformation. On the other hand, the principle underlying eigenvalues can be defined as the amount of stretch that each of these vectors has undergone.
The linear transformation is shown in the example below as well. As we are all aware, eigenvectors cannot be obtained by rotating these vectors to any degree of rotation. However, we do obtain eigenvectors by rotating to 180 degrees. It is important to note that even though these vectors remain in the same span as they did before, the direction in which they point is the opposite. Thus, all vectors that are eigenvectors in this linear transformation have an eigenvalue of -1, so all vectors in this linear transformation are eigenvectors.
How to calculate eigenvectors?
Our previous reading indicates that the vectors that remain in the same span following a transformation are eigenvectors, even if their lengths change and they point in different directions. However, if they remain in the same span, they are considered eigenvectors. Consider the case of a transformation A. Assuming we want to compute the eigenvalues and eigenvectors of the A matrix, let’s follow these steps:
In the first step, lambda will be multiplied by the identity matrix I as follows:
As a second step, we will take matrix A and subtract lambda times I from it:
As a third step, determine the determinate of this matrix:
The fourth step is to set the determinate to 0 and solve it to determine the eigenvalues of the matrix:
After identifying the eigenvalues, we will use them to calculate the eigenvectors by substituting them into the matrix A (for each eigenvalue separately):
It is therefore the case that for an eigenvalue lambda = 3, the eigenvector x = [1, 2] applies to a vector with a scalar c, in which case the vector [c, 2c] is applicable. As a result, any vector whose first element is twice the second is considered to be an eigenvector of matrix A.
In the case of lambda = -1, let’s do the same thing as before:
Hence, for lambda = -1, we obtain the eigenvector x = [1, -2], which corresponds to the vector [c, -2c], where c is a scalar value.
Hopefully this article has been helpful despite the fact that we don’t need to calculate the Eigenvalues and Eigenvectors by hand every time we use the algorithm but it is still important to understand how the model works on the inside in order to be able to use it confidently.
