Joseph Van Name comments on Joseph Van Name’s Shortform

[Joseph Van Name]

Let’s compute some inner products and gradients.

Set up: Let $K$ denote either the field of real or the field of complex numbers. Suppose that $d_1,\dots ,d_r$ are positive integers. Let $m_0,\dots ,m_n$ be a sequence of positive integers with $m_0=m_n=1$. Suppose that $X_{i,j}$ is an $m_{i-1}\times m_i$-matrix whenever $1\le j\le d_i$. Then from the matrices $X_{i,j}$, we can define a $d_1\times \cdots \times d_r$-tensor $T\left(\right(X_{i,j}{)}_{i,j})=(X_{1,i_1}\dots X_{n,i_n}{)}_{i_1,\dots ,i_n}$. I have been doing computer experiments where I use this tensor to approximate other tensors by minimizing the ${\ell }_2$-distance. I have not seen this tensor approximation algorithm elsewhere, but perhaps someone else has produced this tensor approximation construction before. In previous shortform posts on this site, I have given evidence that the tensor dimensionality reduction behaves well, and in this post, we will focus on ways to compute with the tensors $T\left(\right(X_{i,j}{)}_{i,j})$, namely the inner product of such tensors and the gradient of the inner product with respect to the matrices $(X_{i,j}{)}_{i,j}$.

Notation: If $A_1,\dots ,A_r,B_1,\dots ,B_r$ are matrices, then let $\Gamma (A_1,\dots ,A_r;B_1,\dots ,B_r)$ denote the superoperator defined by letting $\Gamma (A_1,\dots ,A_r;B_1,\dots ,B_r)\left(X\right)=A_{1}X{B}_1^{\ast }+\cdots +A_{r}X{B}_r^{\ast }$. Let $\Phi (A_1,\dots ,A_r)=\Gamma (A_1,\dots ,A_r;A_1,\dots ,A_r)$.

Inner product: Here is the computation of the inner product of our tensors.

$⟨T\left(\right(A_{i,j}{)}_{i,j}),T(\left(B_{i,j}{)}_{i,j}\right)⟩$

$=⟨(A_{1,i_1}\dots A_{n,i_n}{)}_{i_1,\dots ,i_n},(B_{1,i_1}\dots B_{n,i_n}{)}_{i_1,\dots ,i_n}⟩$

$={\sum }_{i_1,\dots ,i_n}{A}_{1,i_1}\dots A_{n,i_n}(B_{1,i_1}\dots B_{n,i_n}{)}^{\ast }$

$={\sum }_{i_1,\dots ,i_n}{A}_{1,i_1}\dots A_{n,i_n}{B}_{n,i_n}^{\ast }\dots B_{1,i_1}^{\ast }$

$=\Gamma (A_{1,1},\dots ,A_{1,d_1};B_{1,1},\dots ,B_{1,d_1})\dots \Gamma (A_{n,1},\dots ,A_{n,d_n};B_{n,1},\dots ,B_{n,d_n})$.

In particular, $\parallel T\left(\right(A_{i,j}{)}_{i,j}){\parallel }^2=\Phi (A_{1,1},\dots ,A_{1,d_1})\dots \Phi (A_{n,1},\dots ,A_{n,d_n})$.

Gradient: Observe that ${\nabla }_X\text{Tr}\left(AX\right)=A^T$. We will see shortly that the cyclicity of the trace is useful for calculating the gradient. And here is my manual calculation of the gradient of the inner product of our tensors.

${\nabla }_{X_{\alpha ,\beta }}⟨T\left(\right(X_{i,j}{)}_{i,j}),T(\left(A_{i,j}{)}_{i,j}\right)⟩$

$={\nabla }_{X_{\alpha ,\beta }}{\sum }_{i_1,\dots ,i_n}{X}_{1,i_1}\dots X_{n,i_n}{A}_{n,i_n}^{\ast }\dots A_{1,i_1}^{\ast }$

$={\nabla }_{X_{\alpha ,\beta }}\text{Tr}({\sum }_{i_1,\dots ,i_n}{X}_{1,i_1}\dots X_{n,i_n}{A}_{n,i_n}^{\ast }\dots A_{1,i_1}^{\ast })$

$={\nabla }_{X_{\alpha ,\beta }}\text{Tr}({\sum }_{i_1,\dots ,i_n}{X}_{\alpha ,i_{\alpha }}\dots X_{n,i_n}{A}_{n,i_n}^{\ast }\dots$

$A_{\alpha +1,i_{\alpha +1}}^{\ast }{A}_{\alpha ,i_{\alpha }}^{\ast }{A}_{\alpha -1,i_{\alpha -1}}^{\ast }\dots A_{1,i_1}^{\ast }{X}_{1,i_1}\dots X_{\alpha -1,i_{\alpha -1}})$

$={\nabla }_{X_{\alpha ,\beta }}\text{Tr}({\sum }_{i_{\alpha +1},\dots ,i_n,i_1,\dots ,i_{\alpha -1}}{X}_{\alpha ,\beta }\dots X_{n,i_n}{A}_{n,i_n}^{\ast }\dots$

$A_{\alpha +1,i_{\alpha +1}}^{\ast }{A}_{\alpha ,\beta }^{\ast }{A}_{\alpha -1,i_{\alpha -1}}^{\ast }\dots A_{1,i_1}^{\ast }{X}_{1,i_1}\dots X_{\alpha -1,i_{\alpha -1}})$

$=({\sum }_{i_{\alpha +1},\dots ,i_n,i_1,\dots ,i_{\alpha -1}}{X}_{\alpha +1,i_{\alpha +1}}\dots X_{n,i_n}{A}_{n,i_n}^{\ast }\dots$

$A_{\alpha +1,i_{\alpha +1}}^{\ast }{A}_{\alpha ,\beta }^{\ast }{A}_{\alpha -1,i_{\alpha -1}}^{\ast }\dots A_{1,i_1}^{\ast }{X}_{1,i_1}\dots X_{\alpha -1,i_{\alpha -1}}{)}^T$

$=({\sum }_{i_{\alpha +1},\dots ,i_n,i_1,\dots ,i_{\alpha -1}}{X}_{\alpha +1,i_{\alpha +1}}\dots X_{n,i_n}$

$A_{n,i_n}^{\ast }\dots A_{\alpha +1,i_{\alpha +1}}^{\ast }{A}_{\alpha ,\beta }^{\ast }{A}_{\alpha -1,i_{\alpha -1}}^{\ast }\dots A_{1,i_1}^{\ast }{X}_{1,i_1}\dots X_{\alpha -1,i_{\alpha -1}}{)}^T$

$=[(\Gamma (X_{\alpha +1,1},\dots ,X_{\alpha +1,d_{\alpha +1}};A_{\alpha +1,1},\dots ,A_{\alpha +1,d_{\alpha +1}})\cdots$

$\Gamma (X_{n,1},\dots ,X_{n,d_n};A_{n,1},\dots ,A_{n,d_n})\left(1\right))$

$A_{\alpha ,\beta }^{\ast }$

$(\left(\Gamma \right(A_{\alpha -1,1}^{\ast },\dots ,A_{\alpha -1,d_{\alpha -1}}^{\ast };X_{\alpha -1,1}^{\ast },\dots ,X_{\alpha -1,d_{\alpha -1}}^{\ast })\dots$

$\Gamma (A_{1,1}^{\ast },\dots ,A_{1,d_1}^{\ast };X_{1,1}^{\ast },\dots ,X_{1,d_1}^{\ast })\left(1\right)){]}^T$.