Joseph Van Name

In­ter­pret­ing a di­men­sion­al­ity re­duc­tion of a col­lec­tion of ma­tri­ces as two pos­i­tive semidefinite block di­ag­o­nal matrices

• Joseph Van Name 29 Jan 2024 7:22 UTC

  12 points

  0

  in reply to: FireStormOOO’s comment on: Apol­o­giz­ing is a Core Ra­tion­al­ist Skill

  I do not care to share much more of my reasoning because I have shared enough and also because there is a reason that I have vowed to no longer discuss except possibly with lots of obfuscation. This discussion that we are having is just convincing me more that the entities here are not the entities I want to have around me at all. It does not do much good to say that the community here is acting well or to question my judgment about this community. It will do good for the people here to act better so that I will naturally have a positive judgment about this community.

• Joseph Van Name 18 Jan 2024 18:40 UTC

  10 points

  0

  in reply to: FireStormOOO’s comment on: Apol­o­giz­ing is a Core Ra­tion­al­ist Skill

  Let’s see whether the notions that I have talked about are sensible mathematical notions for machine learning.

  Tensor product-Sometimes data in a neural network has tensor structure. In this case, the weight matrices should be tensor products or tensor sums. Regarding the structure of the data works well with convolutional neural networks, and it should also work well for data with tensor structure to it.

  Trace-The trace of a matrix measures how much the matrix maps vectors onto themselves since

  $\text{Tr}\left(A\right)=c\cdot E(⟨Av,v⟩)$ where $v$ follows the multivariate normal distribution.

  Spectral radius-Suppose that we are iterating a smooth function $f$. Suppose furthermore that $f\left(v\right)=v$ and $u_0$ is near $v$ and $u_{n+1}=f\left(u_n\right)$. We would like to determine whether ${lim}_{n\to \infty }{u}_n=v$ or not. If the Jacobian of $f$ at $v$ has spectral radius less than $1$, then ${lim}_{n\to \infty }{u}_n=v$,. If the Jacobian of $f$ at $v$ has spectral radius greater than $1$, then this limit does not converge.

  The notions that I have been talking about are sensible and arise in machine learning. And understanding these notions is far easier than trying to interpret very large networks like GPT-4 without using these notions. Many people on this site just act like clowns. Karma is only a good metric when the people on the network value substance over fluff. And the only way to convince me otherwise will be for the people here to value posts that involve basic notions like the trace, eigenvalues, and spectral radius of matrices.

  P.S. I can make the trace, determinant, and spectral radius even simpler. These operations are what you get when you take the sum, product, and the maximum absolute value of the eigenvalues. Yes. Those are just the basic eigenvalue operations.

• Joseph Van Name 11 Dec 2023 22:36 UTC

  10 points

  0

  on: Joseph Van Name’s Shortform

  Every entry in a matrix counts for the $L_2$-spectral radius similarity. Suppose that $A_1,\dots ,A_r,B_1,\dots ,B_r$ are real $n\times n$-matrices. Set $A^{\otimes 2}=A\otimes A$. Define the $L_2$-spectral radius similarity between $(A_1,\dots ,A_r)$ and $(B_1,\dots ,B_r)$ to be the number

  $\frac{\rho (A_1\otimes B_1+\cdots +A_r\otimes B_r)}{\rho (A_1^{\otimes 2}+\cdots +A_r^{\otimes 2}{)}^{1/2}\rho (B_1^{\otimes 2}+\cdots +B_r^{\otimes 2}{)}^{1/2}}$. Then the $L_2$-spectral radius similarity is always a real number in the interval $[0,1]$, so one can think of the $L_2$-spectral radius similarity as a generalization of the value $\frac{|⟨u,v⟩|}{\parallel u\parallel \cdot \parallel v\parallel }$ where $u,v$ are real or complex vectors. It turns out experimentally that if $A_1,\dots ,A_r$ are random real matrices, and each $B_j$ is obtained from $A_j$ by replacing each entry in $B_j$ with $0$ with probability $1-\alpha$, then the $L_2$-spectral radius similarity between $(A_1,\dots ,A_r)$ and $(B_1,\dots ,B_r)$ will be about $\sqrt{\alpha }$. If $u=(A_1,\dots ,A_r),v=(B_1,\dots ,B_r)$, then observe that $\frac{|⟨u,v⟩|}{\parallel u\parallel \cdot \parallel v\parallel }\approx \sqrt{\alpha }$ as well.

  Suppose now that $A_1,\dots ,A_r$ are random real $n\times n$ matrices and $C_1,\dots ,C_r$ are the $m\times m$ submatrices of $A_1,\dots ,A_r$ respectively obtained by only looking at the first $m$ rows and columns of $A_1,\dots ,A_r$. Then the $L_2$-spectral radius similarity between $A_1,\dots ,A_r$ and $C_1,\dots ,C_r$ will be about $\sqrt{m/n}$. We can therefore conclude that in some sense $C_1,\dots ,C_r$ is a simplified version of $A_1,\dots ,A_r$ that more efficiently captures the behavior of $A_1,\dots ,A_r$ than $B_1,\dots ,B_r$ does.

  If $A_1,\dots ,A_r,B_1,\dots ,B_r$ are independent random matrices with standard Gaussian entries, then the $L_2$-spectral radius similarity between $(A_1,\dots ,A_r)$ and $(B_1,\dots ,B_r)$ will be about $1/\sqrt{r}$ with small variance. If $u,v$ are random Gaussian vectors of length $r$, then $\frac{|⟨u,v⟩|}{\parallel u\parallel \cdot \parallel v\parallel }$ will on average be about $c/\sqrt{r}$ for some constant $c$, but $\frac{|⟨u,v⟩|}{\parallel u\parallel \cdot \parallel v\parallel }$ will have a high variance.

  These are some simple observations that I have made about the spectral radius during my research for evaluating cryptographic functions for cryptocurrency technologies.

• Joseph Van Name 14 Jun 2023 13:24 UTC

  10 points

  0

  in reply to: Mitchell_Porter’s comment on: Work dumber not smarter

  The intersection of machine learning and generalizations of Laver tables seems to have a lot of potential, so your question about this is extremely interesting to me.

  Machine learning models from generalized Laver tables?

  I have not been able to find any machine learning models that are based on generalized Laver tables that are used for solving problems unrelated to Laver tables. I currently do not directly see any new kinds of deep learning models that one can produce from generalizations of Laver tables, but I would not be surprised if there were a non-standard machine learning model from Laver like algebras that we could use for something like NLP.

  But I have to be skeptical about applying machine learning to generalized Laver tables. Generalizations of Laver tables have intricate structure and complexity but this complexity is ordered. Furthermore, there is a very large supply of Laver-like algebras (even with 2 generators) for potentially any situation. Laver-like algebras are also very easy to compute. While backtracking may potentially take an exponential amount of time, when I have computed Laver-like algebras, the backtracking algorithm seems to always terminate quickly unless it is producing a large quantity or Laver-like algebras or larger Laver-like algebras. But with all this being said, Laver-like algebras seem to lack the steerability that we need to apply these algebras to solving real-world problems. For example, try interpreting (in a model theoretic sense) modular arithmetic modulo 13 in a Laver-like algebra. That is quite hard to do because Laver-like algebras are not built for that sort of thing.

  Here are a couple of avenues that I see as most promising for producing new machine learning algorithms from Laver-like algebras and other structures.

  Let $X$ be a set, and let $\ast$ be a binary operation on $X$ that satisfies the self-distributivity identity $x\ast (y\ast z)=(x\ast y)\ast (x\ast z)$. Define the right powers $x^{\left[n\right]}$ for $n\ge 1$ inductively by setting $x^{\left[1\right]}=x$ and $x^{[n+1]}=x\ast x^{\left[n\right]}$. We say that $(X,\ast ,1)$ is a reduced nilpotent self-distributive algebra if $1$ satisfies the identities $1\ast x=x,x\ast 1=1$ for $x\in X$ and if for all $x\in X$ there is an $n\ge 1$ with $x^{\left[n\right]}=1$. A reduced Laver-like algebra is a reduced nilpotent self-distributive algebra $(X,\ast )$ where if $x_n\in X$ for each $n\ge 0$, then $x_0\ast \cdots \ast x_N=1$ for some $N$. Here we make the convention to put the implied parentheses on the left so that $a\ast b\ast c\ast d=\left(\right(a\ast b)\ast c)\ast d$.

  Reduced nilpotent self-distributive algebras have most of the things that one would expect. Reduced nilpotent self-distributive algebras are often equipped with a composition operation. Reduced nilpotent self-distributive algebras have a notion of a critical point, and if our reduced nilpotent self-distributive algebra is endowed with a composition operation, the set of all critical points in the algebra forms a Heyting algebra. If $(X,\ast )$ is a self-distributive algebra, then we can define $x{\ast }_{0}y=y,x{\ast }_{n+1}y=x{\ast }_n(x\ast y)=x\ast \left(x{\ast }_{n}y\right)$ and$x{\ast }_{\infty }y={lim}_{n\to \infty }x{\ast }_{n}y$ in the discrete topology (this limit always exists for nilpotent self-distributive algebras). We define $x⪯y$ precisely when $x{\ast }_{\infty }y=1$ and $x\simeq y$ precisely when $x⪯y⪯x$. We can define $\text{crit}\left(x\right)$ to be the equivalence class of $x$ with respect to $\simeq$ and $\text{crit}\left[X\right]=X/\simeq$. The operations on $\text{crit}\left[X\right]$ are$\text{crit}\left(x\right)\to \text{crit}\left(y\right)=\text{crit}\left(x{\ast }_{\infty }y\right)$ and $\text{crit}\left(x\right)\wedge \text{crit}\left(y\right)=\text{crit}(x\circ y)$ whenever we have our composition operation $\circ$. One can use computer calculations to add another critical point to a reduced nilpotent self-distributive algebra and obtain a new reduced nilpotent self-distributive algebra with the same number of generators but with another critical point on top (and this new self-distributive algebra will be sub-directly irreducible). Therefore, by taking subdirect products and adding new critical points, we have an endless supply of reduced nilpotent self-distributive algebras with only 2 generators. I also know how to expand reduced nilpotent self-distributive algebras horizontally in the sense that given a finite reduced nilpotent self-distributive algebra $(X,\ast )$ that is not a Laver-like algebra, we can obtain another finite reduced nilpotent self-distributive algebra $(Y,\ast )$ where $X,Y$ are both generated by the same number of elements and these algebras both have the same implication algebra of critical points but $|Y|>>|X|$ but there is a surjective homomorphism $\varphi :Y\to X$. The point is that we have techniques for producing new nilpotent self-distributive algebras from old ones, and going deep into those new nilpotent self-distributive algebras.

  Since reduced nilpotent self-distributive algebras are closed under finite subdirect products, subalgebras, and quotients, and since we have techniques for producing many nilpotent self-distributive algebras, perhaps one can make a ML model from these algebraic structures. On the other hand, there seems to be less of a connection between reduced nilpotent self-distributive algebras and large cardinals and non-commutative polynomials than with Laver-like algebras, so perhaps it is sufficient to stick to Laver-like algebras as a platform for constructing ML models.

  From Laver-like algebras, we can also produce non-commutative polynomials. If $(X,\ast )$ is a Laver-like algebra, and $(x_a{)}_{a\in A}$ is a generating set for $X$, then for each non-maximal critical point $\alpha \in \text{crit}\left[X\right]$, we can define a non-commutative polynomial $p_{\alpha }\left(\right(x_a{)}_{a\in A})$ to be the sum of all non-commutative monomials of the form $x_{a_1}\dots x_{a_n}$ where $a_1\dots a_n\in A^{\ast }$ and $\text{crit}(a_1\ast \cdots \ast a_n)=\alpha$ but where $\text{crit}(a_1\ast \cdots \ast a_m)<\alpha$ for $1\le m<n$. These non-commutative polynomials capture all the information behind the Laver-like algebras since one can reconstruct the entire Laver-like algebra up to critical equivalence from these non-commutative polynomials. These non-commutative polynomials don’t work as well for nilpotent self-distributive algebras; for nilpotent self-distributive algebras, these non-commutative polynomials will instead be rational expressions and one will not be able to recover the entire nilpotent self-distributive algebra from these rational expressions.

  I have used these non-commutative polynomials to construct the infinite product formula

  $\dots p_r(x_1,\dots ,x_n)\dots p_0(x_1,\dots ,x_n)=\frac{1}{1-(x_1+\cdots +x_n)}$ where the polynomials $p_n(x_1,\dots ,x_n)$ are obtained from $n$ different non-trivial rank-into-rank embeddings $j_1,\dots ,j_n:V_{\lambda }\to V_{\lambda }$. This means that the non-commutative ring operations $+,\cdot$ in the rings of non-commutative polynomials are meaningful for Laver-like algebras.

  If I wanted to interpret and make sense of a Laver-like algebra, I would use these non-commutative polynomials. Since non-commutative polynomials are closely related to strings (for NLP) and since the variables in these non-commutative polynomials can be substituted with matrices, I would not be surprised if one can turn Laver-like algebras into a full ML model using these non-commutative polynomials in order to solve a problem unrelated to Laver-like algebras.

  Perhaps, one can also use strong large cardinal hypotheses and logic to produce ML models from Laver-like algebras. Since the existence of a non-trivial rank-into-rank embedding is among the strongest of all large cardinal hypotheses, and since one can produce useful theorems about natural looking finite algebraic structures from these large cardinal hypotheses, perhaps the interplay between the logic using large cardinal hypotheses and finite algebraic structures may be able to produce ML models? For example, one can automate the process of using elementarity to prove the existence of finite algebraic structures. After one has proven the existence of these structures using large cardinal hypotheses, one can do a search using backtracking in order to actually find candidates for these algebraic structures (or if the backtracking algorithm is exhaustive and turns up nothing, then we have a contradiction in the large cardinal hierarchy or a programming error. Mathematicians need to expend a lot of effort into finding an inconsistency in the large cardinal hierarchy this way because I am confident that they won’t find such an inconsistency they will instead find plenty of near misses.). We can automate the process of producing examples of Laver-like algebras that satisfy conditions that were first established using large cardinal hypotheses, and we can perhaps completely describe these Laver-like algebras (up-to-critical equivalence) using our exhaustive backtracking process. This approach can be used to produce a lot of data about rank-into-rank embeddings, but do not see a clear path that allows us to apply this data outside of set theory and Laver-like algebras.

  Using deep learning to investigate generalized Laver tables

  We can certainly apply existing deep learning and machine learning techniques to classical and multigenic Laver tables.

  The n-th classical Laver table is the unique self-distributive algebraic structure $A_n=\left(\right\{1,\dots ,2^n\},{\ast }_n)$ where$x{\ast }_{n}1=x+1\mathrm{mod}{2}^n$ whenever $x\in \{1,\dots ,2^n\}$. The classical Laver tables are up-to-isomorphism the only nilpotent self-distributive algebras generated by a single element.

  Problem: Compute the $n$-th classical Laver table for as large $n$ as we can achieve.

  Solution: Randall Dougherty in his 1995 paper Critical points in an algebra of elementary embeddings II gave an algorithm for computing the 48-th classical Laver table $A_{48}$ and with machine learning, I have personally gotten up to $A_{768}$ (I think I have a couple of errors, but those are not too significant), and I have some of the computation of $A_{1024}$.

  Suppose now that $2^N\le n\le 3\cdot 2^N$. Then Dougherty has shown that one can easily recover the entire function ${\ast }_n$ from the restriction $L_{N,n}:\{1,\dots ,2^{2^N}\}\times \{1,\dots ,2^n\}\to \{1,\dots ,2^n\}$ where $L_{N,n}(x,y)=x{\ast }_{n}y$ for all $x,y$. Suppose now that $1\le x\le 2^n$. Then let $o_n\left(x\right)$ denote the least natural number $m$ such that $x{\ast }_n{2}^m=2^n$. Then there is a number ${\theta }_{n+1}\left(x\right)$ called the threshold of $x$ at $A_{n+1}$ such that $0\le {\theta }_{n+1}\left(x\right)\le 2^{o_n\left(x\right)}$ and ${\theta }_{n+1}\left(2^n\right)=0$ and such that for every $y$ with $1\le y\le 2^n$, we have $x{\ast }_{n+1}y=x{\ast }_{n}y$ whenever $1\le y\le {\theta }_{n+1}\left(x\right)$ and$x{\ast }_{n+1}y=\left(x{\ast }_{n}y\right)+2^n$ for ${\theta }_{n+1}\left(x\right)<y\le 2^{o_n\left(x\right)}$. The classical Laver table $A_{n+1}$ can be easily recovered from the table $A_n$ and the threshold function ${\theta }_{n+1}$. To go beyond Dougherty’s calculation of $A_{48}$ to $A_{768}$ and beyond, we exploit a couple of observations about the threshold function ${\theta }_{n+1}$:

  i: in most cases, ${\theta }_{n+2}\left(x\right)={\theta }_{n+1}\left(x\right)$, and

  ii: in most cases, ${\theta }_{n+1}\left(x\right)=2^i-2^j$ for some $i,j$.

  Let $E\left(n\right)=\{x\in \{1,\dots ,2^{n-2}\}:{\theta }_{n-1}(x)\ne {\theta }_n(x\left)\right\}$, and let $E^{♯}\left(n\right)=E\left(n\right)\cap \{1,\dots ,2^N-2\}$ where $N$ is the least natural number where $n\le 3\cdot 2^N$ and $T_n=\left\{\right(x,{\theta }_n\left(x\right)):x\in E^{♯}(n\left)\right\}$. Then one can easily compute $A_n$ from $A_{n-1}$ and the data $T_n$ by using Dougherty’s algorithm. Now the only thing that we need to do is compute $T_n$ in the first place. It turns out that computing $T_n$ is quite easy except when $n$ is of the form $2^r+1$ or $3\cdot 2^r+1$ for some $r$. In the case, when $n=2^r+1$ or $n=3\cdot 2^r+1$, we initially set $T_n^{\text{temp}}=\varnothing .$ We then repeatedly modify $T_n^{\text{temp}}$ until we can no longer find any contradiction in the statement $T_n=T_n^{\text{temp}}$. Computing $T_n$ essentially amounts to finding new elements in the set $E^{♯}\left(n\right)$ and we find new elements in $E^{♯}\left(n\right)$ using some form of machine learning.

  In my calculation of $A_{768}$, I used operations like bitwise AND, OR, and the bitwise majority operation to combine old elements in $E^{♯}\left(n\right)$ to find new elements in $E^{♯}\left(n\right)$, and I also employed other techniques similar to this. I did not use any neural networks though, but using neural networks seems to be a reasonable strategy, but we need to use the right kinds of neural networks.

  My idea is to use something similar to a CNN where the linear layers are not as general as possible, but the linear layers are structured in a way that reduces the number of weights and so that the structure of the linear layers is compatible with the structure in the data. The elements in $E^{♯}\left(n\right)$ belong to $\{1,\dots ,2^N\}$ and $\{1,\dots ,2^N\}$ has a natural tensor product structure. One can therefore consider $E^{♯}\left(n\right)$ to be a subset of $(R^2{)}^{\otimes n}$. Furthermore, a tuple of $r$ elements in $E^{♯}\left(n\right)$ can be considered as a subset of $\left(R^2\right){)}^{\otimes n}\otimes R^r$. This means that the linear layers from $\left(R^2\right){)}^{\otimes n}\otimes R^r$ to $\left(R^2\right){)}^{\otimes n}\otimes R^r$ should be Kronecker products $A_1\otimes \cdots \otimes A_{n+1}$ or Kronecker sums $A_1\oplus \cdots \oplus A_{n+1}$ (it seems like Kronecker sums with residual layers would work better than Kronecker products). Recall that the Kronecker product and Kronecker sum of matrices $A,B$ are defined by setting $(A\otimes B)(u\otimes v)=Au\otimes Bv$ and $A\oplus B=(A\otimes 1)+(1\otimes B)$ respectively. Perhaps neural networks can be used to generate new elements in $E^{♯}\left(n\right)$. These generative neural networks do not have to be too large nor do they have to be too accurate. A neural network that is 50 percent accurate will be better than a neural network that is 90 percent accurate but takes 10 times as long to compute and is much harder to retrain once most of the elements in $E^{♯}\left(n\right)$ have already been obtained and when the neural network has trouble finding new elements. I also favor smaller neural networks because one can compute $A_{768}$ without using complicated techniques in the first place. I still need to figure out the rest of the details for the generative neural network that finds new elements of $E^{♯}\left(n\right)$ since I want it to be simple and easy to compute so that it is competitive with things like bitwise operations.

  Problem: Translate Laver-like algebras into data (like sequences of vectors or matrices) into a form that machine learning models can work with.

  Solution: The non-commutative polynomials $p_0,\dots ,p_n$ are already in a form that is easier for machine learning algorithms to use. I was able to apply an algorithm that I originally developed for analyzing block ciphers in order to turn the non-commutative polynomials $p_0,\dots ,p_n$ into unique sequences of real matrices $A_0,\dots ,A_n$ (these real matrices do not seem to depend on the initialization since the gradient ascent always seems to converge to the same local maximum). One can then feed this sequence of real matrices $A_0,\dots ,A_n$ into a transformer to solve whatever problem one wants to solve about Laver-like algebras. One can also represent a Laver-like algebra as a 2-dimensional image and then pass that image through a CNN to learn about the Laver-like algebra. This technique may only be used for Laver-like algebras that are small enough for CNNs to fully see, so I do not recommend CNNs for Laver-like algebras, but this is at least a proof-of-concept.

  Problem: Estimate the number of small enough Laver-like algebras (up-to-critical equivalence) that satisfy certain properties.

  Solution: The set of all multigenic Laver tables over an alphabet $A$ forms a rooted tree. We can therefore apply the technique for estimating the number of nodes in a rooted tree described in the 1975 paper Estimating the Efficiency of Backtrack Programs by Donald Knuth. This estimator is unbiased (the mean of the estimated value is actually the number of nodes in the tree), but it will have a very high variance for estimating the number of multigenic Laver tables. In order to reduce the variance for this estimator, we need to assign better probabilities when finding a random path from the root to the leaves, and we should be able to use transformers to assign these probabilities.

  I am pretty sure that there are plenty of other ways to use machine learning to investigate Laver-like algebras.

  Now, nilpotent self-distributive algebras may be applicable in cryptography (they seem like suitable platforms for the Kalka-Teicher key exchange but nobody has researched this). Perhaps a transformer that learns about Laver-like algebras would do better on some unrelated task than a transformer that was never trained using Laver-like algebras. But I do not see any other practical applications of nilpotent Laver-like algebras at the moment. This means that Laver-like algebras are currently a relatively safe problem for investigating machine learning algorithms, so Laver-like algebras are perhaps applicable to AI safety since I currently do not see any way of misaligning an AI for solving a problem related to Laver-like algebras.

• Joseph Van Name 7 Jun 2023 10:41 UTC

  10 points

  7

  on: Work dumber not smarter

  ReLU activation actually comes from real mathematics. People simply don’t recognize neural networks with ReLU activation as mathematical since it is a different and research level area of mathematics that is usually not taught to people. Another reason that ReLU seems non-mathematical is that people seem to have a bias towards mathematical structures that resemble the field of real numbers, integers, or at least rings of matrices. Mathematicians have much less of such a bias, but they still have a bias; I am a mathematician who has studied Laver tables, and nearly nobody has studied Laver tables recently in part because they are non-commutative, non-associative and arise from very large cardinals yet are studied using computer calculations.

  In mathematics, we want the algebraic structures not to be too simple or too complex, but we want them to work in the right way so that we can prove theorems about them. This means that the mathematical structures often have a moderate amount of complexity, but they are often quite simple.

  If we use the logistic or tanh activation function do not seem to give very much interesting mathematics when we combine these specific functions with linearity and when we compose these functions with themselves (by the universal approximation theorem, neural networks with tanh activation can approximate interesting functions but the neural networks themselves are just approximations for what people truly find interesting). On the other hand, we can compose ReLU with linear functions and more copies of ReLU and we will always have piecewise linear functions, and mathematicians can prove theorems about piecewise linear functions. On the other hand, if you use tanh activation, you get functions such as $\tanh \circ \tanh \circ \tanh$ which are quite unappealing to mathematicians since you cannot prove theorems about them that well.

  Give $R$ (or if you want you can make the domain $[-\infty ,\infty ),(-\infty ,\infty ],[-\infty ,\infty ]$) two binary operations $\oplus ,\otimes$ where $x\oplus y=max(x,y),x\otimes y=x+y$. Then the operations $\oplus ,\otimes$ form a semiring. This means that $(x\oplus y)\oplus z=x\oplus (y\oplus z)$,$(x\otimes y)\otimes z=x\otimes (y\otimes z)$ (associativity), $x\oplus y=y\oplus x$,$x\otimes y=y\otimes x$ (commutativity), $x\otimes (y\oplus z)=(x\otimes y)\oplus (x\otimes z)$ (distributivity), etc. The (max) tropical semiring is the semiring $(R\cup \{\infty \},\oplus ,\otimes )$.

  We can define polynomials over tropical semirings.

  A tropical polynomial function is a function $F:R^n\to R$ of the form

  $F(x_1,\dots ,x_n)$$=(a_1\otimes x_{i_{1,1}}\otimes \cdots \otimes x_{i_{d_1,1}})\oplus \cdots \oplus (a_r\otimes x_{i_{1,r}}\otimes \cdots \otimes x_{i_{d_r,r}})$

  $=max(a_1+x_{i_{1,1}}+\cdots +x_{i_{d_1,1}},\dots ,a_r+x_{i_{r,r}}+\cdots +x_{i_{d_r,r}})$.

  Define the tropical division operation $\oslash$ by $x\oslash y=x-y$.

  A tropical rational function is a function $F:R^n\to R$ of the form$F(x_1,\dots ,x_n)=G(x_1,\dots ,x_n)\oslash H(x_1,\dots ,x_n)$. Every tropical polynomial is a tropical rational function. The collection of all tropical rational functions forms a semi-field. One should think of tropical geometry as a kind of algebraic geometry where we replace the field of complex numbers with the tropical semiring.

  Theorem: Let $F:R^n\to R$ be a function. Then the following are equivalent:

  1. $F:R^n\to R$ is a continuous piecewise linear function with integer coefficients.

  2. $F$ is a tropical rational function.

  3. $F$ is computed by a deep neural network with integer-valued weights, real-valued biases, and where the activation functions ${\sigma }_1,\dots ,{\sigma }_l$ are all of the form ${\sigma }_k\left(x\right)=max(x,a_k)$ where $a_k$ is a real constant known as a threshold constant.

  I personally do not see the fact that the coefficients have to be integers as a significant restriction for neural networks. After all, we can just multiply using traditional multiplication any rational-valued ReLU neural network by an integer and make it integer valued.

  Corollary: Let $F:R^n\to R$ be a function. Then the following are equivalent:

  1. $F:R^n\to R$ is a continuous piecewise linear function with rational coefficients.

  2. $F=v\cdot G$ for some tropical rational function $G:R^n\to R$ and rational number $v$ (i.e.$F$ is a root of a tropical rational function).

  3. $F$ is computed by a deep neural network with rational weights, real-valued biases, and where the activation functions ${\sigma }_1,\dots ,{\sigma }_l$ are all of the form ${\sigma }_k\left(x\right)=max(x,a_k)$ where $a_k$ is a real constant known as a threshold constant.

  One can learn more about how tropical geometry relates to neural networks in the paper Tropical Geometry of Deep Neural Networks by Liwen Zhang, Gregory Naitzat, and Lek-Heng Lim. The paper is not too hard.

  Why don’t we see deep neural networks that are built using tropical matrix multiplication?

• Joseph Van Name 11 Sep 2023 22:36 UTC

  8 points

  5

  on: Erdős Prob­lems in Al­gorith­mic Probability

  You claim that Terry Tao says that Benford’s law lack’s a satisfactory explanation. That is not correct. Terry Tao actually gave an explanation for Benford’s law. And even if he didn’t, if you are moderately familiar with mathematics, an explanation for Benford’s law would be obvious or at least a standard undergraduate exercise.

  Let $X$ be a random variable that is uniformly distributed on the interval $[m\cdot \ln (10),n\cdot \ln (10\left)\right]$ for some $m<n$. Let $Y$ denote the first digit of $e^X$. Then show that $P(Y=k)={\log }_{10}\left(\frac{k+1}{k}\right)$.

  For all $n$, let $X_n$ be a uniformly distributed random variable on the interval $[a_n,b_n]$. Let $Y_n$ denote the first digit of $e^{X_n}$. Suppose that ${lim}_{n\to \infty }{b}_n-a_n=+\infty$. Prove that ${lim}_{n\to \infty }P(Y_n=k)={\log }_{10}\left(\frac{k+1}{k}\right)$.

  Terry Tao in that blog post lists the characteristics that imply Zipf’s law here:

  “These laws govern the asymptotic distribution of many statistics $Q$ which

  • (i) take values as positive numbers;

  • (ii) range over many different orders of magnitude;

  • (iiii) arise from a complicated combination of largely independent factors (with different samples of $Q$ arising from different independent factors); and

  • (iv) have not been artificially rounded, truncated, or otherwise constrained in size.”

  Terry Tao merely stated that Benford’s law cannot be proven the same way mathematical theorems are proven; “Being empirically observed phenomena rather than abstract mathematical facts, Benford’s law, Zipf’s law, and the Pareto distribution cannot be “proved” the same way a mathematical theorem can be proved. However, one can still support these laws mathematically in a number of ways, for instance showing how these laws are compatible with each other, and with other plausible hypotheses on the source of the data.”

When perform­ing a di­men­sion­al­ity re­duc­tion on ten­sors, the trace is of­ten zero.

• Joseph Van Name 20 Sep 2023 20:16 UTC

  7 points

  −1

  on: In­ter­pretabil­ity Ex­ter­nal­ities Case Study—Hun­gry Hun­gry Hippos

  I agree that interpretability research is risky, and that one should carefully consider whether it is worth it to perform this interpretability research. I propose that a safer alternative would be to develop machine learning models that are

  1. quite unrelated to the highest performing machine learning models (this is so that capability gains in these safer models do not translate very well to capability gains for the high performing models),

  2. as inherently interpretable as one can make these models,

  3. more limited in functionality than the highest performing machine learning models, but

  4. still functional enough to be useful in practice.

  These characteristics are correlated with each other, so it is likely that one would get all of these characteristics at once. For example, one will most likely not get interpretability for free but instead trade interpretability for performance. Being unrelated to the highest performing machine learning models will also likely reduce the performance of the machine learning models.

  It seems like spectral methods such as PageRank or the eigenvectors of the Laplacian of a graph satisfy some of these characteristics, but it would be good if there were more research into developing spectral methods further to increase their functionality.

  Perhaps interpretability research has yielded such few results because we need to first develop inherently interpretable models.

• Joseph Van Name 25 Jun 2023 21:09 UTC

  7 points

  1

  on: When is cor­re­la­tion tran­si­tive?

  The transitivity of correlation (or of the inner product on the sphere) is a reformulation of the triangle inequality for spheres with the Riemannian metric (and also for the surface of the unit ball in a real Hilbert space).

  Let $d$ be the Riemannian metric on the sphere $S^{n-1}=\{x\in R^n:\parallel x\parallel =1\}$. The value $d(x,y)$ is just the length of the shortest path $\ell :[0,1]\to S^{n-1}$ with $\ell \left(0\right)=x,\ell \left(1\right)=y$. If $x,y\in S^{n-1}$, then $d(x,y)>\parallel x-y\parallel$ since the path $\ell$ cannot be a straight line segment in $R^n$. The metric $d$ can also be defined by $d(x,y)={\cos }^{-1}(⟨x,y⟩)$. From the triangle inequality, if $x,y,z\in S^{n-1}$, we know that ${\cos }^{-1}(⟨x,z)⟩=d(x,z)\le d(x,y)+d(y,z)={\cos }^{-1}⟨x,y⟩+{\cos }^{-1}⟨y,z⟩$. Therefore, by applying the cosine function (which is order reversing) to this inequality, we have $⟨x,z⟩=\cos \left({\cos }^{-1}\right(⟨x,z⟩\left)\right)\ge \cos ({\cos }^{-1}⟨x,y⟩+{\cos }^{-1}⟨y,z⟩)$

  $=\cos \left({\cos }^{-1}\right(⟨x,y⟩\left)\cos \right({\cos }^{-1}(⟨y,z⟩)+\sin \left({\cos }^{-1}\right(⟨x,y⟩\left)\sin \right({\cos }^{-1}(⟨y,z⟩)$

  $=⟨x,y⟩\cdot ⟨y,z⟩-\sqrt{1-⟨x,y{⟩}^2}\cdot \sqrt{1-⟨y,z{⟩}^2}$.

  For the reverse inequality, we apply cosine to the inequality $d(x,z)\ge d(x,y)-d(y,z)$ to obtain

  $⟨x,z⟩\le ⟨x,y⟩\cdot ⟨y,z⟩+\sqrt{1-⟨x,y{⟩}^2}\cdot \sqrt{1-⟨y,z{⟩}^2}$.

• Joseph Van Name 7 Oct 2023 11:16 UTC

  6 points

  −2

  on: Su­per-Ex­po­nen­tial ver­sus Ex­po­nen­tial Growth in Com­pute Price-Performance

  A double exponential model seems very questionable. Is there any theoretical reason why you chose to fit your model with a double exponential? When fitting your model using a double exponential, did you take into consideration fundamental limits of computation? One cannot engineer transistors to be smaller than atoms, and we are approaching the limit to the size of transistors, so one should not expect very much of an increase in the performance of computational hardware. We can add more transistors to a chip by stacking layers (I don’t know how this would be manufactured, but 3D space has a lot of room), but the important thing is efficiency, and one cannot make the transistors more efficient by stacking more layers. With more layers in 3D chips, most transistors will just be off most of the time, so 3D chips provide a limited improvement.

  Landauer’s principle states that to delete a bit of information in computing, one must spend at least $k\cdot T\cdot \ln \left(2\right)$ energy where $k$ is Boltzmann’s constant, $T$ is the temperature, and $\ln \left(2\right)=0.693\dots$. Here,$k\approx 1.38\cdot {10}^{-23}J/K$ (Joules per Kelvin) which is not a lot of energy at room temperature. As the energy efficiency of computation approaches Landauer’s limit, one runs into problems such as thermal noise. Realistically, one should expect to spend more than $100kT$ energy per bit deletion in order to overcome thermal noise (and this is ). If one tries to avoid Landauer’s limit using reversible computation, then the process of computation becomes more complicated, so with reversible computation, one trades energy efficiency per bit operation with the number of operations computed, and the amount of space one uses in performing that computation. The progress in computational hardware capabilities will slow down as one progresses from classical computation to reversible computation. There are also ways of cutting the energy efficiency of deletion of information from $100kT$ per bit to something much closer to $kT\ln \left(2\right)$, but they seem like a complicated engineering challenge.

  The Margolus-Levitin theorem states that it takes $\frac{h}{4E}$ energy to go from a quantum state to an orthogonal quantum state (by flipping a bit, one transforms a state into an orthogonal state) where $h$ is Planck’s constant ($h\approx 6.626\cdot {10}^{-34}J\cdot S$ (Joules times seconds)) and $E$ is the energy. There are other fundamental limits to the capabilities of computation.

• Joseph Van Name 20 Jul 2023 10:26 UTC

  6 points

  1

  on: Six (and a half) in­tu­itions for KL divergence

  The Radon-Nikodym theorem allows us to define the Kullback-Leibler divergence in a more general setting. Even though this setting is more abstract, in this abstract setting it is clear how the KL-divergence is a way to define relative entropy. The abstract setting limits our options to only the correct option.

  Suppose that $f$ is the probability density function of a random variable supported on $[0,1]$. Then we define the continuous entropy of $f$ as ${\int }_0^1-f\left(x\right)\cdot \log \left(f\right(x\left)\right)dx$.

  Suppose that we are given two probability measures $\mu ,\nu$ on the Borel subsets of $[0,1]$ and we know that $\mu$ is the Lebesgue probability measure on $[0,1]$. Then is there any way to abstractly define the continuous entropy of $\nu$ that only refers to the abstract measures $\mu ,\nu$ without referring to $f$ or by recovering $f$ from the abstract properties of $\mu ,\nu$? Yes there is. The Radon-Nikodym theorem allows us to recover $f$ from the measure theoretic properties of $\mu ,\nu$ and it therefore allows us to recover the continuous entropy.

  Suppose that $(X,M)$ is a $\sigma$-algebra. Let $\mu ,\nu$ be two measures on $X$. We say that $\mu$ is absolutely continuous with respect to $\nu$ if $\nu \left(A\right)=0\Rightarrow \mu \left(A\right)=0$, and we write $\mu \ll \nu$ in this case. Recall that a measure $\mu$ is $\sigma$-finite if there are $A_n\in M$ for each natural number $n$ such that $\mu \left(A_n\right)<\infty$ for each $n$ and $X={\bigcup }_{n=0}^{\infty }\mu \left(A_n\right)$.

  Theorem: (Radon-Nikodym theorem) Suppose that $\mu \ll \nu$ and $\mu ,\nu$ are both $\sigma$-finite measures. Then there exists a positive measurable function $f:X\to [0,\infty )$ where $\mu \left(A\right)={\int }_{A}f\left(x\right)d\nu \left(x\right)$ for each $A\in M$.

  The function $f$ is called the Radon-Nikodym derivative of $\mu$ with respect to $\nu$ and is denoted by $f=\frac{d\mu }{d\nu }$. The Radon-Nikodym derivative is unique up to a set of measure zero.

  We therefore define the Kullback-Kiebler divergence as $D_{KL}\left(\mu ||\nu \right)={\int }_X\frac{d\mu }{d\nu }\cdot \log \left(\frac{d\mu }{d\nu }\right)d\nu ={\int }_X\log \left(\frac{d\mu }{d\nu }\right)d\mu$ which coincides with the continuous entropy without the $-1$ factor.

  Now, the assumption that $\nu$ is isomorphic to the Lebesgue probability measure on $[0,1]$ is a surprisingly general assumption that works for many probability measures $\nu$.

  A Boolean algebra $B$ is said to be $\sigma$-complete if whenever $a_n\in B$ for each natural number $n$, the set$\{a_n:n\in N\}$ has a least upper bound which we denote by ${\bigvee }_{n=0}^{\infty }{a}_n$.

  A measure algebra is a pair $(B,\mu )$ where $B$ is a $\sigma$-complete Boolean algebra and $\mu :B\to [0,\infty )$ is a function with $\mu \left(x\right)=0$ iff $x=0$ and $\mu \left({\bigvee }_{n=0}^{\infty }{a}_n\right)={\sum }_{n=0}^{\infty }\mu \left(a_n\right)$ whenever $a_m\wedge a_n=0$ for $m\ne n$. If $(B,\mu )$ is a measure algebra, then define a metric $d$ (and hence a topology and uniformity and whatnot) on $B$ by setting $d(x,y)=\mu (x\oplus y)$. Recall that a topological space is separable if it has a countable dense subset.

  For example, if $M$ is the collection of Lebesgue measurable subsets of $[0,1]$, and $I$ is the collection of measure zero subsets of $[0,1]$, then the quotient Boolean algebra $M/I$ is complete and if $\mu$ is the Lebesgue measure on $M/I$, then $(M/I,\mu )$ is a separable atomless probability measure algebra, and this measure algebra is unique.

  Theorem: (Caratheodory) Every separable atomless probability measure algebra is isomorphic.

  We have an even stronger uniqueness theorem for this measure algebra.

  Theorem: Every Borel probability measure on a complete separable metric space without any atoms is isomorphic to the Lebesgue measure on the interval $[0,1]$.

  Suppose now that $\mu ,\nu$ are Borel probability measures on a complete separable metric space that possibly have atoms. Then there is some measure $\eta$ where $\nu \times \eta$ is isomorphic to the Lebesgue probability measure. In this case, $\frac{d\mu }{d\nu }\left(x\right)=\frac{d(\mu \times \eta )}{d(\nu \times \eta )}(x,y)$ for almost all $x,y$, so

  $D_{KL}(\mu \times \eta ||\nu \times \eta )=\int \log \left(\frac{d(\mu \times \eta )}{d(\nu \times \eta )}\right(x,y\left)\right)d(\mu \times \eta )(x,y)=\int \log \left(\frac{d\mu }{d\eta }\right(x,y\left)\right)d(\mu \times \eta )(x,y)=\int \log \left(\frac{d\mu }{d\eta }\right(x\left)\right)d\mu \left(x\right)$

  $=D_{KL}\left(\mu ||\nu \right)$. In this case, we can pretend that $\nu \times \eta$ is the Lebesgue measure on $[0,1]$ and that $D_{KL}(\mu \times \eta ||\nu \times \eta )$ just measures the continuous entropy of $\mu \times \eta$ with respect to the Lebesgue probabilty measure on $[0,1]$.

  The KL-divergence also generalizes greatly. Given a function $h:[0,\infty )\to [0,\infty )$ ($h$ is usually convex or concave), one can define another measure of similarity between probability measures such as $D_h\left(\mu ||\nu \right)=\int h\left(\frac{d\mu }{d\nu }\right)d\nu$ known as the $h$-divergence. One can also define a measure on $[0,\infty )$ from the Radon-Nikodym derivative by $(\frac{d\mu }{d\nu }{)}_{\ast }\nu$ where if $g:X\to Y$ and $\nu$ is a measure on $X$, then $g_{\ast }\left(\nu \right)$ is the measure on $Y$ defined by $g_{\ast }\left(\nu \right)\left(R\right)=\nu \left(g^{-1}\right[R\left]\right)$. The random variable $(\frac{d\mu }{d\nu }{)}_{\ast }\nu$ gives information about the similarity between the probability measures $\mu ,\nu$ from which one can always recover the $h$-divergence. In this case,$E\left[h\right(\frac{d\mu }{d\nu }{)}_{\ast }\nu \left)\right]=\int h\left(x\right)d\left(\frac{d\mu }{d\nu }{)}_{\ast }\nu \right(x)=\int h(\frac{d\mu }{d\nu }\left(x\right)\left)d\nu \right(x)=D_h(\mu ||\nu )$.

• Joseph Van Name 18 Jan 2024 11:46 UTC

  5 points

  0

  in reply to: Yitz’s comment on: An In­tro­duc­tion To The Man­delbrot Set That Doesn’t Men­tion Com­plex Numbers

  I usually think of the field of complex numbers algebraically, but one can also think of the real numbers, complex numbers, and quaternions geometrically. The real numbers are good with dealing with 1 dimensional space, and the complex numbers are good for dealing with 2 dimensional space geometrically. While the division ring of quaternions is a 4 dimensional algebra over the field of real numbers, the quaternions are best used for dealing with 3 dimensional space geometrically.

  For example, if $U,V$ are open subsets of some Euclidean space, then a function $f:U\to V$ is said to be a conformal mapping when it preserves angles and the orientation. We can associate the 2-dimensional Euclidean space with the field of complex numbers, and the conformal mappings between open subsets of 2-dimensional spaces are just the complex differentiable mappings. For the Mandelbrot set, we need this conformality because we want the Mandelbrot set to look pretty. If the complex differentiable maps were not conformal, then the functions that we iterate in complex dynamics would stretch subsets of the complex plane in one dimension and expand them in the other dimension and this would result in a fractal that looks quite stretched in one real dimension and squashed in another dimension (the fractals would look like spaghetti; oh wait, I just looked at a 3D fractal and it looks like some vegetable like broccoli). This stretching and squashing is illustrated by 3D fractals that try to mimic the Mandelbrot set but without any conformality. The conformality is why the Julia sets are sensible (mathematicians have proven theorems about these sets) for any complex polynomial of degree 2 or greater.

  For the quaternions, it is well-known that the dot product and the cross product operations on 3 dimensional space can be described in terms of the quaternionic multiplication operation between purely imaginary quaternions.

• Joseph Van Name 22 Oct 2023 17:13 UTC

  5 points

  0

  on: [Cross-post]The the­o­ret­i­cal com­pu­ta­tional limit of the So­lar Sys­tem is 1.47x10^49 bits per sec­ond.

  I forgot to mention another source of difficulty in getting the energy efficiency of the computation down to Landauer’s limit at the CMB temperature.

  Recall that the Stefan Boltzmann equation states that the power being emitted from an object by thermal radiation is equal to $P=A\cdot ϵ\cdot \sigma \cdot T^4$. Here, $P$ stands for power, $A$ is the surface area of the object, $ϵ$ is the emissivity of the object ($ϵ$ is a real number with $0\le ϵ\le 1$),$T$ is the temperature, and $\sigma$ is the Stefan-Boltzmann constant. Here, $\sigma \approx 5.67\cdot {10}^{-8}\cdot W\cdot K^{-4}\cdot m^{-2}$.

  Suppose therefore that we want a Dyson sphere with radius $r$ that maintains a temperature of 4 K which is slightly above the CMB temperature. To simplify the calculations, I am going to ignore the energy that the Dyson sphere receives from the CMB so that I obtain a lower bound for the size of our Dyson sphere. Let us assume that our Dyson sphere is a perfect emitter of thermal radiation so that $ϵ=1$.

  Earth’s surface has a temperature of about $300K$. In order to have a temperature of $4K$, our Dyson sphere needs to receive $(1/75{)}^4$ the energy per unit of area. This means that the Dyson sphere needs to have a radius of about $({75}^4{)}^{1/2}={75}^2=5625$ astronomical units (recall that the distance from Earth to the sun is 1 astronomical unit).

  Let us do more precise calculations to get a more exact radius of our Dyson sphere.

  $r^2=\frac{P}{4\pi \sigma (4K{)}^4}$, so $r=\sqrt{\frac{P}{4\pi \sigma (4K{)}^4}}=\frac{1}{2(4K{)}^2}\cdot \sqrt{\frac{P}{\pi \sigma }}$$\approx 1.45\cdot {10}^{15}m$ which is about 15 percent of a light-year. Since the nearest star is 4 light years away, by the time that we are able to construct a Dyson sphere with a radius that is about 15 percent of a light year, I think that we will be able to harness energy from other stars such as Alpha Centauri.

  The fourth power in the Stefan Boltzmann equation makes it hard for cold objects to radiate heat.

• Joseph Van Name 23 Sep 2023 19:21 UTC

  5 points

  0

  in reply to: Nathan Helm-Burger’s comment on: In­ter­pretabil­ity Ex­ter­nal­ities Case Study—Hun­gry Hun­gry Hippos

  I am not expecting any worldwide regulation on AI that prohibits people from using or training unaligned systems (I am just expecting a usual level of regulation). I am mainly hoping for spectral techniques to develop to the point where AI groups will want to use these spectral techniques (or some other method) more and more until they are competitive with neural networks at general tasks or at least complement the deficiencies of neural networks. I also hope that these spectral techniques will remain interpretable and aligned.

  Right now, there are several kinds of tasks in which I would rather use spectral techniques than neural networks. I have been evaluating the cryptographic security of block ciphers with small message size and very small key size (for cryptocurrency research), and it seems like the spectral techniques that I have developed give consistent measures of security for such block ciphers (I am not done with the training yet) and these spectral techniques are better for cryptanalysis than neural networks. I have been able to use these spectral techniques for other problems such as the problem of finding the largest clique in a graph (this is not something that I would have expected before I did it), and right now these spectral techniques are the only way I know how to transform a non-commutative polynomial into something that other machine learning models can work with better.

  Right now, I do not know how to use spectral techniques to replace deep neural networks. I do not know how to use spectral techniques to approximate a universal function and I do not know how to use spectral techniques to make machine learning models with many layers. I hope to be able to solve these problems of spectral techniques, but I agree that there will be a tradeoff between performance and interpretability. The goal is to make this tradeoff favor interpretable, aligned, and safe machine learning models.

• Joseph Van Name 28 May 2023 12:39 UTC

  5 points

  0

  on: Why and When In­ter­pretabil­ity Work is Dangerous

  “Can we control the thought and behavior patterns of a powerful mind at all?”-I do not see why this would not be the case. For example, in a neural network, if we are able to find a cluster of problematic neurons, then we will be able to remove those neurons. With that being said, I do not know how well this works in practice. After removing the neurons (and normalizing so that the remaining neurons are given higher weights), if we do not retrain the neural network, then it could exhibit more unexpected or poor behavior. If we do retrain the network, then the network could regrow the problematic neurons. Furthermore, if we continually remove problematic neuron clusters, then the neural network could become less interpretable. The process of detecting and removing problematic neuron clusters will be a selective pressure that will cause the neurons to either behave well or behave poorly but evade detection. One solution to this problem may be to employ several different techniques for detecting problematic neuron clusters so that it is harder for these problematic clusters to evade detection. Of course, there may still be problematic neuron clusters that evade detection. But these problematic neuron clusters will probably be much less effective at behaving problematically since these problematic neuron clusters would need to trade performance for the ability to evade detection. For example, the process of detecting problematic neuron clusters could detect large problematic neuron clusters, but small problematic neuron clusters could avoid detection. In this case, the small problematic neuron clusters would be less effective and less worrisome simply because smaller neuron clusters would have a more difficult time causing problems.

• Joseph Van Name 18 Jan 2024 12:03 UTC

  4 points

  1

  in reply to: FireStormOOO’s comment on: Apol­o­giz­ing is a Core Ra­tion­al­ist Skill

  Talking about whining and my loss of status is a good way to get me to dislike the LW community and consider them to be anti-intellectuals who fall for garbage like FTX. Do you honestly think the people here should try to interpret large sections of LLMs while simultaneously being afraid of quaternions?

  It is better to comment on threads where we are interacting in a more positive manner.

  I thought apologizing and recognizing inadequacies was a core rationalist skill. And I thought rationalists were supposed to like mathematics. The lack of mathematical appreciation is one of these inadequacies of the LW community. But instead of acknowledging this deficiency, the community here blasts me as talking about something off topic. How ironic!

• Joseph Van Name 7 Aug 2023 9:50 UTC

  4 points

  −6

  on: When perform­ing a di­men­sion­al­ity re­duc­tion on ten­sors, the trace is of­ten zero.

  Massively downvoting mathematics without commenting at all just shows that the people on this site are very low quality specimens who do not care at all about rationality at all but who just want to pretend to be smart.

• Joseph Van Name 26 Jul 2023 20:04 UTC

  4 points

  0

  on: Open Thread—July 2023

  Hello. I am Joseph Van Name. I have been on this site substantially this year, but I never introduced myself. I have a Ph.D. in Mathematics, and I am a cryptocurrency creator. I am currently interested in using LSRDRs and related constructions to interpret ML models and to produce more interpretable ML models.

  Let $A_1,\dots ,A_r\in M_n\left(R\right)$ be $n\times n$-matrices. Define a fitness function $L_{A_1,\dots ,A_r;d}:M_d(R{)}^r\to [0,\infty )$ by letting $L_{A_1,\dots ,A_r;d}(X_1,\dots ,X_r)=\frac{\rho (A_1\otimes X_1+\cdots +A_r\otimes X_r)}{\rho (X_1\otimes X_1+\cdots +X_r\otimes X_r{)}^{1/2}}$ ($\rho \left(A\right)$ denotes the spectral radius of $A$ while $A\otimes B$ is the tensor product of $A$ and $B$). If $(X_1,\dots ,X_r)$ locally maximizes $L_{A_1,\dots ,A_r;d}(X_1,\dots ,X_r)$, then we say that $(X_1,\dots ,X_r)$ is an $L_{2,d}$-spectral radius dimensionality reduction (LSRDR) of $(A_1,\dots ,A_r)$ (and we can generalize the notion of an LSRDR to a complex and quaternionic setting).

  I originally developed the notion of an LSRDR in order to evaluate the cryptographic security of block ciphers such as the AES block cipher or small block ciphers, but it seems like LSRDRs have much more potential in machine learning and interpretability. I hope that LSRDRs and similar constructions can make our AI safer (LSRDRs do not seem directly dangerous at the moment since they currently cannot replace neural networks).

In­ter­pret­ing a ma­trix-val­ued word em­bed­ding with a math­e­mat­i­cally proven char­ac­ter­i­za­tion of all optima