{"id":1871,"date":"2018-12-06T12:00:00","date_gmt":"2018-12-06T11:00:00","guid":{"rendered":"https:\/\/kindsonthegenius.com\/blog\/singular-value-decompositionsvd-a-dimensionality-reduction-technique\/"},"modified":"2026-07-05T03:21:26","modified_gmt":"2026-07-05T01:21:26","slug":"singular-value-decompositionsvd-a-dimensionality-reduction-technique","status":"publish","type":"post","link":"https:\/\/kindsonthegenius.com\/blog\/singular-value-decompositionsvd-a-dimensionality-reduction-technique\/","title":{"rendered":"Singular Value Decomposition(SVD) \u2013 A Dimensionality Reduction Technique"},"content":{"rendered":"

Singular Value Decomposition (SVD)<\/strong> is a dimensionality reduction technique similar to PCA but more effective than PCA. It is considered as factorization of a data matrix into three matrices.<\/p>\n

\"\"<\/p>\n

Given a rectangular matrix A which is an n x p matrix, the SVD theorem shows that this matrix can\u00a0 be represented as:<\/p>\n

A = U\u2211V*<\/strong><\/p>\n

where<\/p>\n

A is the original data matrix (n x p)<\/p>\n

U is\u00a0 the left singular vectors<\/p>\n

\u2211\u00a0 contains the singular values on its diagonal\u00a0 as shown below<\/p>\n

V* is right singular vectors<\/p>\n

\"\"<\/p>\n

This\u00a0 singular values matrix is can be taught of as the eigen vectors\u00a0 in the Principal Components Analysis. SVD can actually be interpreted in a similar way as PCA.<\/p>\n

The first column of V corresponds to the largest singular value and the next column of V corresponds to the next largest singular value and so on.<\/p>\n

 <\/p>\n

This being the case, then if we multiply the singular values matrix with the original matrix, we would have:<\/p>\n

XV = U\u2211V*V\u00a0<\/strong><\/p>\n

Since V*V\u00a0 = I, then we would have<\/p>\n

XV = U\u2211<\/strong><\/p>\n

This means that we would have the scores(projection of the original data into the first principal component) after we have computed the SVD since we have U and\u00a0\u2211<\/p>\n

T = U\u2211<\/strong><\/p>\n

This is really easy because since\u00a0\u03c3\u00a0 is filled with zeroes,\u00a0 all we need to do is to multiply every column of U by a single number (you can look at a Review of Matrix Operations).<\/p>\n

So if we are interested in only two principal components, we take two columns of V<\/p>\n

Tr = Ur<\/sub>\u2211r<\/sub><\/p>\n

 <\/p>\n

How Many Principal Components do we Pick<\/strong><\/p>\n

How then do we determine the number of principal components to pick? Whether 2 , 3, 4 or more.<\/p>\n

Let’s first observe that the singular values (eigen values in PCA) are ordered. This means that they get smaller as we move down the diagonal. So we need to create a plot of the cumulative sum of the singular values from 1 to k against k.<\/p>\n

The shape of this curve helps us to understand the data. From the plot we can see how many components needed to explain the variance in the data.<\/p>\n

This would give us a plot like shown in Figure 1.0.<\/p>\n

\"\"
Figure 1.0: Choosing number of principal components<\/figcaption><\/figure>\n

Since the singular values get smaller and smaller, the plot flattens out into a plateau as it gets to the maximum. Then we examine the shape of the curve. There are three options:<\/p>\n

    \n
  • It could have a sharp edge: In this case, the first few components gives us all the variance<\/li>\n
  • It could be a smooth curve: Here the variance depends on so many components<\/li>\n
  • It could be in between<\/li>\n<\/ul>\n

    When the curve reaches a plateau, then adding more components does not contribute any additional explanation to the data.<\/p>\n

    Another approach is to set some threshold on the percentage of the data that needs to be explained. In this case, when enough number of components have been added that explain the variance in the data up to this threshold, then we stop. For instance, how many singular values is needed to explain 90% of the data or 95% of the data and so on.<\/p>\n","protected":false},"excerpt":{"rendered":"

    Singular Value Decomposition (SVD) is a dimensionality reduction technique similar to PCA but more effective than PCA. It is considered as factorization of a data … <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"pagelayer_contact_templates":[],"_pagelayer_content":"","footnotes":""},"categories":[414],"tags":[],"class_list":["post-1871","post","type-post","status-publish","format-standard","hentry","category-programming"],"_links":{"self":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1871","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/comments?post=1871"}],"version-history":[{"count":1,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1871\/revisions"}],"predecessor-version":[{"id":2039,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1871\/revisions\/2039"}],"wp:attachment":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/media?parent=1871"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/categories?post=1871"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/tags?post=1871"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}