Joseph Van Name’s Shortform

[Joseph Van Name]

Every entry in a matrix counts for the $L_2$-spectral radius similarity. Suppose that $A_1,\dots ,A_r,B_1,\dots ,B_r$ are real $n\times n$-matrices. Set $A^{\otimes 2}=A\otimes A$. Define the $L_2$-spectral radius similarity between $(A_1,\dots ,A_r)$ and $(B_1,\dots ,B_r)$ to be the number

$\frac{\rho (A_1\otimes B_1+\cdots +A_r\otimes B_r)}{\rho (A_1^{\otimes 2}+\cdots +A_r^{\otimes 2}{)}^{1/2}\rho (B_1^{\otimes 2}+\cdots +B_r^{\otimes 2}{)}^{1/2}}$. Then the $L_2$-spectral radius similarity is always a real number in the interval $[0,1]$, so one can think of the $L_2$-spectral radius similarity as a generalization of the value $\frac{|⟨u,v⟩|}{\parallel u\parallel \cdot \parallel v\parallel }$ where $u,v$ are real or complex vectors. It turns out experimentally that if $A_1,\dots ,A_r$ are random real matrices, and each $B_j$ is obtained from $A_j$ by replacing each entry in $B_j$ with $0$ with probability $1-\alpha$, then the $L_2$-spectral radius similarity between $(A_1,\dots ,A_r)$ and $(B_1,\dots ,B_r)$ will be about $\sqrt{\alpha }$. If $u=(A_1,\dots ,A_r),v=(B_1,\dots ,B_r)$, then observe that $\frac{|⟨u,v⟩|}{\parallel u\parallel \cdot \parallel v\parallel }\approx \sqrt{\alpha }$ as well.

Suppose now that $A_1,\dots ,A_r$ are random real $n\times n$ matrices and $C_1,\dots ,C_r$ are the $m\times m$ submatrices of $A_1,\dots ,A_r$ respectively obtained by only looking at the first $m$ rows and columns of $A_1,\dots ,A_r$. Then the $L_2$-spectral radius similarity between $A_1,\dots ,A_r$ and $C_1,\dots ,C_r$ will be about $\sqrt{m/n}$. We can therefore conclude that in some sense $C_1,\dots ,C_r$ is a simplified version of $A_1,\dots ,A_r$ that more efficiently captures the behavior of $A_1,\dots ,A_r$ than $B_1,\dots ,B_r$ does.

If $A_1,\dots ,A_r,B_1,\dots ,B_r$ are independent random matrices with standard Gaussian entries, then the $L_2$-spectral radius similarity between $(A_1,\dots ,A_r)$ and $(B_1,\dots ,B_r)$ will be about $1/\sqrt{r}$ with small variance. If $u,v$ are random Gaussian vectors of length $r$, then $\frac{|⟨u,v⟩|}{\parallel u\parallel \cdot \parallel v\parallel }$ will on average be about $c/\sqrt{r}$ for some constant $c$, but $\frac{|⟨u,v⟩|}{\parallel u\parallel \cdot \parallel v\parallel }$ will have a high variance.

These are some simple observations that I have made about the spectral radius during my research for evaluating cryptographic functions for cryptocurrency technologies.

[Algon]

Your notation is confusing me. If r is the size of the list of matrices, then how can you have a probability of 1-r for r>=2? Maybe you mean 1-1/​r and sqrt{1/​r} instead of 1-r and sqrt{r} respectively?

[Joseph Van Name]

Thanks for pointing that out. I have corrected the typo. I simply used the symbol $r$ for two different quantities, but now the probability is denoted by the symbol $\alpha$.

[Joseph Van Name]

We can use the $L_2-$spectral radius similarity to measure more complicated similarities between data sets.

Suppose that $A_1,\dots ,A_r$ are $m\times m$-real matrices and $B_1,\dots ,B_r$ are $n\times n$-real matrices. Let $\rho \left(A\right)$ denote the spectral radius of $A$ and let $A\otimes B$ denote the tensor product of $A$ with $B$. Define the $L_2$-spectral radius by setting ${\rho }_2(A_1,\dots ,A_r)=\rho (A_1\otimes A_1+\cdots +A_r\otimes A_r{)}^{1/2}$, Define the $L_2$-spectral radius similarity between $A_1,\dots ,A_r$ and $B_1,\dots ,B_r$ as

$\parallel (A_1,\dots ,A_r)\simeq (B_1,\dots ,B_r){\parallel }_2=\frac{\rho (A_1\otimes B_1+\cdots +A_r\otimes B_r)}{{\rho }_2(A_1,\dots ,A_r){\rho }_2(B_1,\dots ,B_r)}$.

We observe that if $C$ is invertible and $\lambda$ is a constant, then

$\parallel (A_1,\dots ,A_r)\simeq (\lambda C{B}_1{C}^{-1},\dots ,\lambda C{B}_r{C}^{-1}){\parallel }_2=1.$

Therefore, the $L_2$-spectral radius is able to detect and measure symmetry that is normally hidden.

Example: Suppose that $u_1,\dots ,u_r;v_1,\dots ,v_r$ are vectors of possibly different dimensions. Suppose that we would like to determine how close we are to obtaining an affine transformation $T$ with $T\left(u_j\right)=v_j$ for all $j$ (or a slightly different notion of similarity). We first of all should normalize these vectors to obtain vectors $x_1,\dots ,x_r;y_1,\dots ,y_r$ with mean zero and where the covariance matrix is the identity matrix (we may not need to do this depending on our notion of similarity). Then $\parallel (x_1{x}_1^{\ast },\dots ,x_r{x}_r^{\ast })\simeq (y_1{y}_1^{\ast },\dots ,y_r{y}_r^{\ast }){\parallel }_2$ is a measure of low close we are to obtaining such an affine transformation $T$. We may be able to apply this notion to determining the distance between machine learning models. For example, suppose that $M,N$ are both the first few layers in a (typically different) neural network. Suppose that $a_1,\dots ,a_r$ is a set of data points. Then if $u_j=M\left(a_j\right)$ and $v_j=M\left(a_j\right)$, then $\parallel (x_1{x}_1^{\ast },\dots ,x_r{x}_r^{\ast })\simeq (y_1{y}_1^{\ast },\dots ,y_r{y}_r^{\ast }){\parallel }_2$ is a measure of the similarity between $M$ and $N$.

I have actually used this example to see if there is any similarity between two different neural networks trained on the same data set. For my experiment, I chose a random collection of $S\subseteq \{0,1{\}}^{32}\times \{0,1{\}}^{32}$ of ordered pairs and I trained the neural networks $M,N$ to minimize the expected losses $E(\parallel N(a)-b{\parallel }^2:(a,b)\in S),E(\parallel M(a)-b{\parallel }^2:(a,b)\in S)$. In my experiment, each $a_j$ was a random vector of length 32 whose entries were 0′s and 1′s. In my experiment, the similarity $\parallel (x_1{x}_1^{\ast },\dots ,x_r{x}_r^{\ast })\simeq (y_1{y}_1^{\ast },\dots ,y_r{y}_r^{\ast }){\parallel }_2$ was worse than if $x_1,\dots ,x_r,y_1,\dots ,y_r$ were just random vectors.

This simple experiment suggests that trained neural networks retain too much random or pseudorandom data and are way too messy in order for anyone to develop a good understanding or interpretation of these networks. In my personal opinion, neural networks should be avoided in favor of other AI systems, but we need to develop these alternative AI systems so that they eventually outperform neural networks. I have personally used the $L_2$-spectral radius similarity to develop such non-messy AI systems including LSRDRs, but these non-neural non-messy AI systems currently do not perform as well as neural networks for most tasks. For example, I currently cannot train LSRDR-like structures to do any more NLP than just a word embedding, but I can train LSRDRs to do tasks that I have not seen neural networks perform (such as a tensor dimensionality reduction).

[Joseph Van Name]

So in my research into machine learning algorithms that I can use to evaluate small block ciphers for cryptocurrency technologies, I have just stumbled upon a dimensionality reduction for tensors in tensor products of inner product spaces that according to my computer experiments exists, is unique, and which reduces a real tensor to another real tensor even when the underlying field is the field of complex numbers. I would not be too surprised if someone else came up with this tensor dimensionality reduction before since it has a rather simple description and it is in a sense a canonical tensor dimensionality reduction when we consider tensors as homogeneous non-commutative polynomials. But even if this tensor dimensionality reduction is not new, this dimensionality reduction algorithm belongs to a broader class of new algorithms that I have been studying recently such as LSRDRs.

Suppose that $K$ is either the field of real numbers or the field of complex numbers. Let $V_1,\dots ,V_n$ be finite dimensional inner product spaces over $K$ with dimensions $d_1,\dots ,d_n$ respectively. Suppose that $V_i$ has basis $e_{i,1},\dots ,e_{i,d_i}$. Given $v\in V_1\otimes \cdots \otimes V_n$, we would sometimes want to approximate the tensor $v$ with a tensor that has less parameters. Suppose that $(m_0,\dots ,m_n)$ is a sequence of natural numbers with $m_0=m_n=1$. Suppose that $X_{i,j}$ is a $m_{i-1}\times m_i$ matrix over the field $K$ for $1\le i\le n$ and $1\le j\le d_i$. From the system of matrices $(X_{i,j}{)}_{i,j}$, we obtain a tensor $T\left(\right(X_{i,j}{)}_{i,j})={\sum }_{i_1,\dots ,i_n}{e}_{i_1}\otimes \cdots \otimes e_{i_n}\cdot X_{1,i_1}\dots X_{n,i_n}$. If the system of matrices $(X_{i,j}{)}_{i,j}$ locally minimizes the distance $\parallel v-T\left(\right(X_{i,j}{)}_{i,j})\parallel$, then the tensor $T\left(\right(X_{i,j}{)}_{i,j})$ is a dimensionality reduction of $v$ which we shall denote by $u$.

Intuition: One can associate the tensor product $V_1\otimes \cdots \otimes V_n$ with the set of all degree $n$ homogeneous non-commutative polynomials that consist of linear combinations of the monomials of the form $x_{1,i_1}\cdots x_{n,i_n}$. Given, our matrices $X_{i,j}$, we can define a linear functional $\varphi :V_1\otimes \cdots \otimes V_n\to K$ by setting $\varphi \left(p\right)=p\left(\right(X_{i,j}{)}_{i,j})$. But by the Reisz representation theorem, the linear functional $\varphi$ is dual to some tensor in $V_1\otimes \cdots \otimes V_n$. More specifically, $\varphi$ is dual to $T\left(\right(X_{i,j}{)}_{i,j})$. The tensors of the form $T\left(\right(X_{i,j}{)}_{i,j})$ are therefore the

Advantages:

1. In my computer experiments, the reduced dimension tensor $u$ is often (but not always) unique in the sense that if we calculate the tensor $u$ twice, then we will get the same tensor. At least, the distribution of reduced dimension tensors $u$ will have low Renyi entropy. I personally consider the partial uniqueness of the reduced dimension tensor to be advantageous over total uniqueness since this partial uniqueness signals whether one should use this tensor dimensionality reduction in the first place. If the reduced tensor is far from being unique, then one should not use this tensor dimensionality reduction algorithm. If the reduced tensor is unique or at least has low Renyi entropy, then this dimensionality reduction works well for the tensor $v$.

2. This dimensionality reduction does not depend on the choice of orthonormal basis $e_{i,1},\dots ,e_{i,d_i}$. If we chose a different basis for each $V_i$, then the resulting tensor $u$ of reduced dimensionality will remain the same (the proof is given below).

3. If $K$ is the field of complex numbers, but all the entries in the tensor $v$ happen to be real numbers, then all the entries in the tensor $u$ will also be real numbers.

4. This dimensionality reduction algorithm is intuitive when tensors are considered as homogeneous non-commutative polynomials.

Disadvantages:

1. This dimensionality reduction depends on a canonical cyclic ordering the inner product spaces $V_1,\dots ,V_n$.

2. Other notions of dimensionality reduction for tensors such as the CP tensor dimensionality reduction and the Tucker decompositions are more well-established, and they are obviously attempted generalizations of the singular value decomposition to higher dimensions, so they may be more intuitive to some.

3. The tensors of reduced dimensionality $T\left(\right(X_{i,j}{)}_{i,j})$ have a more complicated description than the tensors in the CP tensor dimensionality reduction.

Proposition: The set of tensors of the form ${\sum }_{i_1,\dots ,i_n}{e}_{1,i_1}\otimes \cdots \otimes e_{n,i_n}{X}_{1,i_1}\dots X_{n,i_n}$ does not depend on the choice of bases $(e_{i,1},\dots ,e_{i,d_i}{)}_i$.

Proof: For each $i$, let $f_{i,1},\dots ,f_{i,d_i}$ be an alternative basis for $V_i$. Then suppose that $e_{i,j}={\sum }_k{u}_{i,j,k}{f}_{i,k}$ for each $i,j$. Then

${\sum }_{i_1,\dots ,i_n}{e}_{1,i_1}\otimes \cdots \otimes e_{n,i_n}{X}_{1,i_1}\dots X_{n,i_n}$

$={\sum }_{i_1,\dots ,i_n}{\sum }_{k_1}{u}_{1,i_1,k_1}{f}_{1,i_1}\otimes \cdots \otimes {\sum }_{k_n}{u}_{n,i_n,k_n}{f}_{n,i_n}{X}_{1,i_1}\dots X_{n,i_n}$

$={\sum }_{k_1,\dots ,k_n}{f}_{1,k_1}\otimes \cdots \otimes f_{n,k_n}{\sum }_{i_1,\dots ,i_n}{u}_{1,i_1,k_1}\dots u_{n,i_n,k_n}{X}_{1,i_1}\dots X_{n,i,n}$

$={\sum }_{k_1,\dots ,k_n}{f}_{1,k_1}\otimes \cdots \otimes f_{n,k_n}\left({\sum }_{i_1}{u}_{1,i_1,k_1}{X}_{1,i_1}\right)\dots \left({\sum }_{i_n}{u}_{n,i_n,k_n}{X}_{i_n}\right)$. Q.E.D.

A failed generalization: An astute reader may have observed that if we drop the requirement that $m_n=1$, then we get a linear functional defined by letting

$\varphi \left(p\right)=\text{Tr}\left(p\right(\left(X_{i,j}{)}_{i,j}\right))$. This is indeed a linear functional, and we can try to approximate $v$ using a the dual to $\varphi$, but this approach does not work as well.

[Joseph Van Name]

There are some cases where we have a complete description for the local optima for an optimization problem. This is a case of such an optimization problem.

Such optimization problems are useful for AI safety since a loss/​fitness function where we have a complete description of all local or global optima is a highly interpretable loss/​fitness function, and so one should consider using these loss/​fitness functions to construct AI algorithms.

Theorem: Suppose that $U$ is a real,complex, or quaternionic $n\times n$-matrix that minimizes the quantity $\parallel U{\parallel }_2+\parallel U^{-1}{\parallel }_2$. Then $U$ is unitary.

Proof: The real case is a special case of a complex case, and by representing each $n\times n$-quaternionic matrix as a complex $2n\times 2n$-matrix, we may assume that $U$ is a complex matrix.

By the Schur decomposition, we know that $U=VT{V}^{\ast }$ where $V$ is a unitary matrix and $T$ is upper triangular. But we know that $\parallel U{\parallel }_2=\parallel T{\parallel }_2$. Furthermore, $U^{-1}=V{T}^{-1}{V}^{\ast }$, so $\parallel U^{-1}{\parallel }_2=\parallel T^{-1}{\parallel }_2$. Let $D$ denote the diagonal matrix whose diagonal entries are the same as $T$. Then $\parallel T{\parallel }_2\ge \parallel D{\parallel }_2$ and $\parallel T^{-1}{\parallel }_2\ge \parallel D^{-1}{\parallel }_2$. Furthermore, $\parallel T{\parallel }_2=\parallel D{\parallel }_2$ iff T is diagonal and $\parallel T^{-1}{\parallel }_2=\parallel D^{-1}{\parallel }_2$ iff $D$ is diagonal. Therefore, since $\parallel U{\parallel }_2+\parallel U^{-1}{\parallel }_2=\parallel T{\parallel }_2+\parallel T^{-1}{\parallel }_2$ and $\parallel T{\parallel }_2+\parallel T^{-1}{\parallel }_2$ is minimized, we can conclude that $T=D$, so $T$ is a diagonal matrix. Suppose that $T$ has diagonal entries $(z_1,\dots ,z_n)$. By the arithmetic-geometric mean equality and the Cauchy-Schwarz inequality, we know that $\frac{1}{2}\cdot (\parallel (z_1,\dots ,z_n){\parallel }_2+\parallel (z_1^{-1},\dots ,z_n^{-1}\left){\parallel }_2\right)\ge \parallel (|z_1|,\dots ,|z_n|){\parallel }_2\cdot \parallel (|z_1^{-1}|,\dots ,|z_n^{-1})|{\parallel }_2$

$\ge ⟨(|z_1|,\dots ,|z_n|),(|z_1^{-1}|,\dots ,|z_n^{-1})|⟩=\sqrt{n}.$

Here, the equalities hold if and only if $|z_j|=1$ for all $j$, but this implies that $U$ is unitary. Q.E.D.

[Joseph Van Name]

The $L_2$-spectral radius similarity is not transitive. Suppose that $A_1,\dots ,A_r$ are $m\times m$-matrices and $B_1,\dots ,B_r$ are real $n\times n$-matrices. Then define ${\rho }_2(A_1,\dots ,A_r)=\rho (A_1\otimes A_1+\cdots +A_r\otimes A_r{)}^{1/2}$. Then the generalized Cauchy-Schwarz inequality is satisfied:

$\rho (A_1\otimes B_1+\cdots +A_r\otimes B_r)\le {\rho }_2(A_1,\dots ,A_r){\rho }_2(B_1,\dots ,B_r)$.

We therefore define the $L_{2,d}$-spectral radius similarity between $(A_1,\dots ,A_r)$ and $(B_1,\dots ,B_r)$ as $\parallel (A_1,\dots ,A_r)\simeq (B_1,\dots ,B_r)\parallel =\frac{\rho (A_1\otimes B_1+\cdots +A_r\otimes B_r)}{{\rho }_2(A_1,\dots ,A_r){\rho }_2(B_1,\dots ,B_r)}$. One should think of the $L_2$-spectral radius similarity as a generalization of the cosine similarity $\frac{⟨u,v⟩}{\parallel u\parallel \cdot \parallel v\parallel }$ between vectors $u,v$. I have been using the $L_2$-spectral radius similarity to develop AI systems that seem to be very interpretable. The $L_2$-spectral radius similarity is not transitive.

$\parallel (A_1,\dots ,A_r)\simeq (A_1\oplus B_1,\dots ,A_r\oplus B_r)\parallel =1$ and

$\parallel (B_1,\dots ,B_r)\simeq (A_1\oplus B_1,\dots ,A_r\oplus B_r)\parallel =1$, but $\parallel (A_1,\dots ,A_r)\simeq (B_1,\dots ,B_r)\parallel$ can take any value in the interval $[0,1]$.

We should therefore think of the $L_{2,d}$-spectral radius similarity as a sort of least upper bound of $[0,1]$-valued equivalence relations than a $[0,1]$-valued equivalence relation. We need to consider this as a least upper bound because matrices have multiple dimensions.

Notation: $\rho \left(A\right)={lim}_{n\to \infty }\parallel A^n{\parallel }^{1/n}$ is the spectral radius. The spectral radius $A$ is the largest magnitude of an eigenvalue of the matrix $A$. Here the norm does not matter because we are taking the limit.$A\oplus B$ is the direct sum of matrices while $A\otimes B$ denotes the Kronecker product of matrices.

[Joseph Van Name]

Let’s compute some inner products and gradients.

Set up: Let $K$ denote either the field of real or the field of complex numbers. Suppose that $d_1,\dots ,d_r$ are positive integers. Let $m_0,\dots ,m_n$ be a sequence of positive integers with $m_0=m_n=1$. Suppose that $X_{i,j}$ is an $m_{i-1}\times m_i$-matrix whenever $1\le j\le d_i$. Then from the matrices $X_{i,j}$, we can define a $d_1\times \cdots \times d_r$-tensor $T\left(\right(X_{i,j}{)}_{i,j})=(X_{1,i_1}\dots X_{n,i_n}{)}_{i_1,\dots ,i_n}$. I have been doing computer experiments where I use this tensor to approximate other tensors by minimizing the ${\ell }_2$-distance. I have not seen this tensor approximation algorithm elsewhere, but perhaps someone else has produced this tensor approximation construction before. In previous shortform posts on this site, I have given evidence that the tensor dimensionality reduction behaves well, and in this post, we will focus on ways to compute with the tensors $T\left(\right(X_{i,j}{)}_{i,j})$, namely the inner product of such tensors and the gradient of the inner product with respect to the matrices $(X_{i,j}{)}_{i,j}$.

Notation: If $A_1,\dots ,A_r,B_1,\dots ,B_r$ are matrices, then let $\Gamma (A_1,\dots ,A_r;B_1,\dots ,B_r)$ denote the superoperator defined by letting $\Gamma (A_1,\dots ,A_r;B_1,\dots ,B_r)\left(X\right)=A_{1}X{B}_1^{\ast }+\cdots +A_{r}X{B}_r^{\ast }$. Let $\Phi (A_1,\dots ,A_r)=\Gamma (A_1,\dots ,A_r;A_1,\dots ,A_r)$.

Inner product: Here is the computation of the inner product of our tensors.

$⟨T\left(\right(A_{i,j}{)}_{i,j}),T(\left(B_{i,j}{)}_{i,j}\right)⟩$

$=⟨(A_{1,i_1}\dots A_{n,i_n}{)}_{i_1,\dots ,i_n},(B_{1,i_1}\dots B_{n,i_n}{)}_{i_1,\dots ,i_n}⟩$

$={\sum }_{i_1,\dots ,i_n}{A}_{1,i_1}\dots A_{n,i_n}(B_{1,i_1}\dots B_{n,i_n}{)}^{\ast }$

$={\sum }_{i_1,\dots ,i_n}{A}_{1,i_1}\dots A_{n,i_n}{B}_{n,i_n}^{\ast }\dots B_{1,i_1}^{\ast }$

$=\Gamma (A_{1,1},\dots ,A_{1,d_1};B_{1,1},\dots ,B_{1,d_1})\dots \Gamma (A_{n,1},\dots ,A_{n,d_n};B_{n,1},\dots ,B_{n,d_n})$.

In particular, $\parallel T\left(\right(A_{i,j}{)}_{i,j}){\parallel }^2=\Phi (A_{1,1},\dots ,A_{1,d_1})\dots \Phi (A_{n,1},\dots ,A_{n,d_n})$.

Gradient: Observe that ${\nabla }_X\text{Tr}\left(AX\right)=A^T$. We will see shortly that the cyclicity of the trace is useful for calculating the gradient. And here is my manual calculation of the gradient of the inner product of our tensors.

${\nabla }_{X_{\alpha ,\beta }}⟨T\left(\right(X_{i,j}{)}_{i,j}),T(\left(A_{i,j}{)}_{i,j}\right)⟩$

$={\nabla }_{X_{\alpha ,\beta }}{\sum }_{i_1,\dots ,i_n}{X}_{1,i_1}\dots X_{n,i_n}{A}_{n,i_n}^{\ast }\dots A_{1,i_1}^{\ast }$

$={\nabla }_{X_{\alpha ,\beta }}\text{Tr}({\sum }_{i_1,\dots ,i_n}{X}_{1,i_1}\dots X_{n,i_n}{A}_{n,i_n}^{\ast }\dots A_{1,i_1}^{\ast })$

$={\nabla }_{X_{\alpha ,\beta }}\text{Tr}({\sum }_{i_1,\dots ,i_n}{X}_{\alpha ,i_{\alpha }}\dots X_{n,i_n}{A}_{n,i_n}^{\ast }\dots$

$A_{\alpha +1,i_{\alpha +1}}^{\ast }{A}_{\alpha ,i_{\alpha }}^{\ast }{A}_{\alpha -1,i_{\alpha -1}}^{\ast }\dots A_{1,i_1}^{\ast }{X}_{1,i_1}\dots X_{\alpha -1,i_{\alpha -1}})$

$={\nabla }_{X_{\alpha ,\beta }}\text{Tr}({\sum }_{i_{\alpha +1},\dots ,i_n,i_1,\dots ,i_{\alpha -1}}{X}_{\alpha ,\beta }\dots X_{n,i_n}{A}_{n,i_n}^{\ast }\dots$

$A_{\alpha +1,i_{\alpha +1}}^{\ast }{A}_{\alpha ,\beta }^{\ast }{A}_{\alpha -1,i_{\alpha -1}}^{\ast }\dots A_{1,i_1}^{\ast }{X}_{1,i_1}\dots X_{\alpha -1,i_{\alpha -1}})$

$=({\sum }_{i_{\alpha +1},\dots ,i_n,i_1,\dots ,i_{\alpha -1}}{X}_{\alpha +1,i_{\alpha +1}}\dots X_{n,i_n}{A}_{n,i_n}^{\ast }\dots$

$A_{\alpha +1,i_{\alpha +1}}^{\ast }{A}_{\alpha ,\beta }^{\ast }{A}_{\alpha -1,i_{\alpha -1}}^{\ast }\dots A_{1,i_1}^{\ast }{X}_{1,i_1}\dots X_{\alpha -1,i_{\alpha -1}}{)}^T$

$=({\sum }_{i_{\alpha +1},\dots ,i_n,i_1,\dots ,i_{\alpha -1}}{X}_{\alpha +1,i_{\alpha +1}}\dots X_{n,i_n}$

$A_{n,i_n}^{\ast }\dots A_{\alpha +1,i_{\alpha +1}}^{\ast }{A}_{\alpha ,\beta }^{\ast }{A}_{\alpha -1,i_{\alpha -1}}^{\ast }\dots A_{1,i_1}^{\ast }{X}_{1,i_1}\dots X_{\alpha -1,i_{\alpha -1}}{)}^T$

$=[(\Gamma (X_{\alpha +1,1},\dots ,X_{\alpha +1,d_{\alpha +1}};A_{\alpha +1,1},\dots ,A_{\alpha +1,d_{\alpha +1}})\cdots$

$\Gamma (X_{n,1},\dots ,X_{n,d_n};A_{n,1},\dots ,A_{n,d_n})\left(1\right))$

$A_{\alpha ,\beta }^{\ast }$

$(\left(\Gamma \right(A_{\alpha -1,1}^{\ast },\dots ,A_{\alpha -1,d_{\alpha -1}}^{\ast };X_{\alpha -1,1}^{\ast },\dots ,X_{\alpha -1,d_{\alpha -1}}^{\ast })\dots$

$\Gamma (A_{1,1}^{\ast },\dots ,A_{1,d_1}^{\ast };X_{1,1}^{\ast },\dots ,X_{1,d_1}^{\ast })\left(1\right)){]}^T$.

[Joseph Van Name]

So in my research into machine learning algorithms, I have stumbled upon a dimensionality reduction algorithm for tensors, and my computer experiments have so far yielded interesting results. I am not sure that this dimensionality reduction is new, but I plan on generalizing this dimensionality reduction to more complicated constructions that I am pretty sure are new and am confident would work well.

Suppose that $K$ is either the field of real numbers or the field of complex numbers. Suppose that $d_1,\dots ,d_n$ are positive integers and $(m_0,\dots ,m_n)$ is a sequence of positive integers with $m_0=m_n=1$. Suppose that $X_{i,j}$ is an $m_{i-1}\times m_i$-matrix whenever $1\le j\le d_i$. Then define a tensor $T\left(\right(X_{i,j}\left)\right)=(X_{1,i_1}\dots X_{n,i_n}{)}_{i_1,\dots ,i_n}\in K^{d_1}\otimes \cdots \otimes K^{d_n}$.

If $v\in K^{d_1}\otimes \cdots \otimes K^{d_n}$, and $(X_{i,j}{)}_{i,j}$ is a system of matrices that minimizes the value $\parallel v-T\left(\right(X_{i,j}\left)\right)\parallel$, then $T\left(\right(X_{i,j}{)}_{i,j})$ is a dimensionality reduction of $(X_{i,j}{)}_{i,j}$, and we shall denote let $u$ denote the tensor of reduced dimension $T\left(\right(X_{i,j}{)}_{i,j})$. We shall call $u$ a matrix table to tensor dimensionality reduction of type $(m_0,\dots ,m_n)$.

Observation 1: (Sparsity) If $v$ is sparse in the sense that most entries in the tensor $v$ are zero, then the tensor $u$ will tend to have plenty of zero entries, but as expected, $u$ will be less sparse than $v$.

Observation 2: (Repeated entries) If $v$ is sparse and $v=(x_{i_1,\dots ,i_n}{)}_{i_1,\dots ,i_n}$ and the set $\{x_{i_1,\dots ,i_n}:i_1,\dots ,i_n\}$ has small cardinality, then the tensor $u$ will contain plenty of repeated non-zero entries.

Observation 3: (Tensor decomposition) Let $v$ be a tensor. Then we can often find a matrix table to tensor dimensionality reduction $u$ of type $(m_0,\dots ,m_n)$ so that $v-u$ is its own matrix table to tensor dimensionality reduction.

Observation 4: (Rational reduction) Suppose that $v$ is sparse and the entries in $v$ are all integers. Then the value $\parallel u-v{\parallel }^2$ is often a positive integer in both the case when $u$ has only integer entries and in the case when $u$ has non-integer entries.

Observation 5: (Multiple lines) Let $m$ be a fixed positive even number. Suppose that $v$ is sparse and the entries in $v$ are all of the form $r\cdot e^{2\pi in/m}$ for some integer $n$ and $r\ge 0$. Then the entries in $u$ are often exclusively of the form $r\cdot e^{2\pi in/m}$ as well.

Observation 6: (Rational reductions) I have observed a sparse tensor $v$ all of whose entries are integers along with matrix table to tensor dimensionality reductions $u_1,u_2$ of $v$ where $\parallel v-u_1\parallel =3,\parallel v-u_1\parallel =2,\parallel u_2-u_1\parallel =5$.

This is not an exclusive list of all the observations that I have made about the matrix table to tensor dimensionality reduction.

From these observations, one should conclude that the matrix table to tensor dimensionality reduction is a well-behaved machine learning algorithm. I hope and expect this machine learning algorithm and many similar ones to be used to both interpret the AI models that we have and will have and also to construct more interpretable and safer AI models in the future.

[Joseph Van Name]

Suppose that $q,r,d$ are natural numbers. Let $1<p<\infty$. Let $z_{i,j}$ be a complex number whenever $1\le i\le q,1\le j\le r$. Let $L:M_d(C{)}^r\setminus \{0\}\to [-\infty ,\infty )$ be the fitness function defined by letting $L(X_1,\dots ,X_r)$$=({\sum }_{i=1}^q\log \left(\rho \right({\sum }_{j=1}^r{z}_{i,j}{X}_j\left)\right)/q)-\log (\parallel {\sum }_{j=1}^r{X}_j{X}_j^{\ast }{\parallel }_p)/2$. Here, $\rho \left(X\right)$ denotes the spectral radius of a matrix $X$ while $\parallel X{\parallel }_p$ denotes the Schatten $p$-norm of $X$.

Now suppose that $(A_1,\dots ,A_r)\in M_d(C{)}^r\setminus \{0\}$ is a tuple that maximizes $L(A_1,\dots ,A_r)$. Let $M:C^r\setminus \left\{0\right\}\to [-\infty ,\infty )$ be the fitness function defined by letting $M(w_1,\dots ,w_r)=\log \left(\rho \right(w_1{A}_1+\cdots +w_r{A}_r\left)\right)-\log (\parallel (w_1,\dots ,w_r\left){\parallel }_2\right)$. Then suppose that $(v_1,\dots ,v_r)\in C^r\setminus \left\{0\right\}$ is a tuple that maximizes $M(v_1,\dots ,v_r)$. Then we will likely be able to find an $\ell \in \{1,\dots ,q\}$ and a non-zero complex number $\alpha$ where $(v_1,\dots ,v_r)=\alpha \cdot (x_{\ell ,1},\dots ,x_{\ell ,r})$.

In this case, $(z_{i,j}{)}_{i,j}$ represents the training data while the matrices $A_1,\dots ,A_r$ is our learned machine learning model. In this case, we are able to recover some original data values from the learned machine learning model $A_1,\dots ,A_r$ without any distortion to the data values.

I have just made this observation, so I am still exploring the implications of this observation. But this is an example of how mathematical spectral machine learning algorithms can behave, and more mathematical machine learning models are more likely to be interpretable and they are more likely to have a robust mathematical/​empirical theory behind them.

[Joseph Van Name]

I think that all that happened here was the matrices $A_1,\dots ,A_r$ just ended up being diagonal matrices. This means that this is probably an uninteresting observation in this case, but I need to do more tests before commenting any further.