{"id":1872,"date":"2018-12-13T12:00:00","date_gmt":"2018-12-13T11:00:00","guid":{"rendered":"https:\/\/kindsonthegenius.com\/blog\/introduction-to-higher-order-singular-value-decomposition-hosvd\/"},"modified":"2026-07-05T03:21:29","modified_gmt":"2026-07-05T01:21:29","slug":"introduction-to-higher-order-singular-value-decomposition-hosvd","status":"publish","type":"post","link":"https:\/\/kindsonthegenius.com\/blog\/introduction-to-higher-order-singular-value-decomposition-hosvd\/","title":{"rendered":"Introduction to Higher Order Singular Value Decomposition (HOSVD)"},"content":{"rendered":"<p>I would start with a brief review of SVD and then gradually ease into Higher Order SVD.\u00a0 As usual, I would try to make it as easy and clear as possible.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-210 aligncenter\" src=\"https:\/\/www.kindsonthegenius.com\/wp-content\/uploads\/2020\/09\/Higher-Order-Singular-Value-Decomposition-1-300x144.jpg\" alt=\"\" width=\"300\" height=\"144\" \/><\/p>\n<ol>\n<li>Review of Singular Value Decomposition(SVD)<\/li>\n<li>About Tensor Product<\/li>\n<li>What is Higher Order SVD (HOSVD)<\/li>\n<li>Basic Concepts of HOSVD<\/li>\n<li>The HOSVD Algorithm<\/li>\n<\/ol>\n<p><strong>Review of Singular Value Decomposition (SVD)<\/strong><\/p>\n<p>Recall that Singular Value decomposition is a technique to decompose a data matrix into three parts.<\/p>\n<p>Given a rectangular matrix A which is an n x p matrix, the SVD theorem shows that the matrix can\u00a0 be represented as:<\/p>\n<p><strong>A = U\u2211V<sup>T<\/sup><\/strong>(same as\u00a0<strong>\u00a0U\u2211V*<\/strong>)<\/p>\n<p>where<\/p>\n<ul>\n<li>A is the original data matrix\u00a0 of size m x n<\/li>\n<li>U is\u00a0 the left singular vectors of size m x r<\/li>\n<li>\u2211\u00a0 is the\u00a0 singular values on its diagonal of size r x r<\/li>\n<li>V* is right singular vectors of size r x n<\/li>\n<\/ul>\n<p>Now we are going to represent the SVD still in a slightly different way. This time as as summation of different components:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-205 aligncenter\" src=\"https:\/\/www.kindsonthegenius.com\/wp-content\/uploads\/2020\/09\/SVD-2-300x241.jpg\" alt=\"\" width=\"161\" height=\"129\" \/><\/p>\n<p>In this case the, symbol\u00a0\u2297 represents the tensor product (remember that a matrix is also a tensor, but of rank 2). This means that each element of a can be represented in this way:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-206 aligncenter\" src=\"https:\/\/www.kindsonthegenius.com\/wp-content\/uploads\/2020\/09\/SVD22-300x246.jpg\" alt=\"\" width=\"171\" height=\"140\" \/><\/p>\n<p>As like before,<\/p>\n<ul>\n<li style=\"text-align: left;\">\u03a3 is a diagonal r x r matrix with non-zero singular values in its diagonal<\/li>\n<li style=\"text-align: left;\">\u03c3 are the actual singular values while<\/li>\n<li style=\"text-align: left;\">u<sub>k<\/sub> are the columns of the matrix U (m x r)<\/li>\n<li style=\"text-align: left;\">v<sub>k<\/sub> are the columns of the matrix V (r x n)<\/li>\n<\/ul>\n<p>This is illustrated in Figure 1a and Figure 1b shown below. Keep in mind that we are still doing a review in SVD, but trying to generalize as this would make understanding of Higher Order SVD easier to follow.<\/p>\n<figure id=\"attachment_208\" aria-describedby=\"caption-attachment-208\" style=\"width: 533px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-208 \" src=\"https:\/\/www.kindsonthegenius.com\/wp-content\/uploads\/2020\/09\/Singular-Value-Decomposition-General-Representation.jpg\" alt=\"\" width=\"533\" height=\"225\" \/><figcaption id=\"caption-attachment-208\" class=\"wp-caption-text\">Figure 1a:\u00a0 SVD &#8211; General Representation<\/figcaption><\/figure>\n<figure id=\"attachment_209\" aria-describedby=\"caption-attachment-209\" style=\"width: 735px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-209 size-large\" src=\"https:\/\/www.kindsonthegenius.com\/wp-content\/uploads\/2020\/09\/Singular-Value-Decomposition-Summation-of-Components-1024x321.jpg\" alt=\"\" width=\"735\" height=\"230\" \/><figcaption id=\"caption-attachment-209\" class=\"wp-caption-text\">Figure 1b: SVD &#8211; Summation of Products<\/figcaption><\/figure>\n<p>Figure 1a and Figure 1b are approximate to the same. In 1 b, we are taking summation over the the product of the slices of the columns of U with slices of\u00a0 rows of V<sup>T<\/sup>. Also not that we are dotting each term with the corresponding singular value.<\/p>\n<p>Now, we are set to now consider Higher Order SVD.\u00a0 The Singular Value Decomposition(SVD) can be generalized to higher order tensors or multi-way arrays in different ways.<\/p>\n<p>In this article, we are going to consider two main approaches, namely:<\/p>\n<ol>\n<li>Tucker decomposition (HoSVD decomposition)<\/li>\n<li>CP Expansion (Canonical decomposition\/Parallel facotorisation)<\/li>\n<\/ol>\n<p>We would continue with these in the next article. Do check in couple of days.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I would start with a brief review of SVD and then gradually ease into Higher Order SVD.\u00a0 As usual, I would try to make it &hellip; <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"pagelayer_contact_templates":[],"_pagelayer_content":"","footnotes":""},"categories":[414],"tags":[],"class_list":["post-1872","post","type-post","status-publish","format-standard","hentry","category-programming"],"_links":{"self":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1872","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/comments?post=1872"}],"version-history":[{"count":1,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1872\/revisions"}],"predecessor-version":[{"id":2040,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/1872\/revisions\/2040"}],"wp:attachment":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/media?parent=1872"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/categories?post=1872"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/tags?post=1872"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}