Joseph Van Name comments on Joseph Van Name’s Shortform

[Joseph Van Name]

This is a post about some of the machine learning algorithms that I have been doing experiments with. These machine learning models behave quite mathematically which seems to be very helpful for AI interpretability and AI safety.

Sequences of matrices generally cannot be approximated by sequences of Hermitian matrices.

Suppose that $A_1,\dots ,A_r$ are $n\times n$-complex matrices and $X_1,\dots ,X_r$ are $d\times d$-complex matrices. Then define a mapping $\Gamma (A_1,\dots ,A_r;X_1,\dots ,X_r):M_{n,d}\left(C\right)\to M_{n,d}\left(C\right)$ by $\Gamma (A_1,\dots ,A_r;X_1,\dots ,X_r)\left(X\right)=A_{1}X{X}_1^{\ast }+\cdots +A_{r}X{X}_r^{\ast }$ for all $X$. Define

$\Phi (A_1,\dots ,A_r)=\Gamma (A_1,\dots ,A_r;A_1,\dots ,A_r)$. Define the $L_2$

-spectral radius by setting ${\rho }_2(A_1,\dots ,A_r)=\rho \left(\Phi \right(A_1,\dots ,A_r){)}^{1/2}$. Define the $L_2$-spectral radius similarity between $(A_1,\dots ,A_r)$ and $(X_1,\dots ,X_r)$ by

$\parallel (A_1,\dots ,A_r)\simeq (X_1,\dots ,X_r){\parallel }_2$

$=\frac{\rho \left(\Gamma \right(A_1,\dots ,A_r;X_1,\dots ,X_r\left)\right)}{{\rho }_2(A_1,\dots ,A_r){\rho }_2(X_1,\dots ,X_r)}$.

The $L_2$-spectral radius similarity is always in the interval $[0,1]$. if $n=d$, $A_1,\dots ,A_r$ generates the algebra of $n\times n$-complex matrices, and $X_1,\dots ,X_r$ also generates the algebra of $n\times n$-complex matrices, then $\parallel (A_1,\dots ,A_r)\simeq (X_1,\dots ,X_r){\parallel }_2=1$ if and only if there are $C,\lambda$ with $A_j=\lambda C{X}_j{C}^{-1}$ for all $j$.

Define ${\rho }_{2,d}^H(A_1,\dots ,A_r)$ to be the supremum of

${\rho }_2(A_1,\dots ,A_r)\cdot \parallel (A_1,\dots ,A_r)\simeq (X_1,\dots ,X_r){\parallel }_2$

where $X_1,\dots ,X_r$ are $d\times d$-Hermitian matrices.

One can get lower bounds for ${\rho }_{2,d}^H(A_1,\dots ,A_r)$ simply by locally maximizing $\parallel (A_1,\dots ,A_r)\simeq (X_1,\dots ,X_r){\parallel }_2$ using gradient ascent, but if one locally maximizes this quantity twice, one typically gets the same fitness level.

Empirical observation/​conjecture: If $(A_1,\dots ,A_r)$ are $n\times n$-complex matrices, then ${\rho }_{2,n}^H(A_1,\dots ,A_r)={\rho }_{2,d}^H(A_1,\dots ,A_r)$ whenever $d\ge n$.

The above observation means that sequences of $n\times n$-matrices $(A_1,\dots ,A_r)$ are fundamentally non-Hermitian. In this case, we cannot get better models of $(A_1,\dots ,A_r)$ using Hermitian matrices larger than the matrices $(A_1,\dots ,A_r)$ themselves; I kind of want the behavior to be more complex instead of doing the same thing whenever $d\ge n$

, but the purpose of modeling $(A_1,\dots ,A_r)$ as Hermitian matrices is generally to use smaller matrices and not larger matrices.

This means that the function ${\rho }_{2,d}^H$ behaves mathematically.

Now, the model $(X_1,\dots ,X_r)$ is a linear model of $(A_1,\dots ,A_r)$ since the mapping $A_j\mapsto X_j$ is the restriction of a linear mapping, so such a linear model should be good for a limited number of tasks, but the mathematical behavior of the model $(X_1,\dots ,X_r)$ generalizes to multi-layered machine learning models.