James Camacho

Karma: 311

James Camacho 10 Jan 2026 19:35 UTC
1 point
0
in reply to: jessicata’s comment on: Understanding complex conjugates in quantum mechanics
Perhaps I’m missing something you’re saying, or you’re missing something I’m saying. In general, the process of finding a “conjugate space” is not an involution. We do not have inner products or Hilbert spaces. There are no pairs. We have to motivate the inner product, and it is motivated by first motivating multilinear forms, and then motivating bilinear forms. But bilinear forms only arise in the alternating representation.

There is a terminology issue here, because “dual” literally means a paired, mirror space, so treating the inner product as a bilinear form in a space $\otimes$ its dual only works in the alternating representation. What you’re actually doing to generate that dual space is to look at the connected components of

${A : ρ^{λ} (A) \in I m (π^{λ})} .$

For the alternating representation, there is the component connected to the origin— $S O (n)$ —as well as the mirrored component $J^{- 1} S O (n) J$ . But there are many more connected components for other representations, exactly $| I m (π^{λ}) |$ of them. Maybe some of them have involutions to each other, but not all of them. Not all of them are “dual” or “conjugate” in the literal sense of the word. This is the terminology issue, and I think the main source of confusion.

Also, my analogy does not break down for odd $n$ . Note that $V_{λ}$ is an irreducible module of $(C^{n})^{\otimes n}$ , not the same space, and $(V_{λ})^{\otimes k}$ is an entirely new tensor product. I was trying to keep my comment from growing longer than it already was, so I may have left out many other little details like this that would help with interpretation.

James Camacho 10 Jan 2026 19:19 UTC
1 point
0
in reply to: Enrique Mark’s comment on: A Loser’s Reflections

You label this a “standard leftist critique,” but you haven’t pointed out why this critique is invalid. You imply it doesn’t hold water without actually providing an argument against it.

Whether or not the critique is valid, the action of criticizing can be invalid here. It is similar to how yelling, “fire!” in a packed theater is rarely the correct way of transmitting information about flaws in the world around you. I think most people would appreciate an explanation for why you are suppressing potentially true free speech, but the explanation does not necessarily need to come paired with the suppression. So, you can have something like this occur:

Alice: I think we ought to overthrow the government.

Bob: Shut up!

Bob may be worried about others overhearing this and turning them in, even if he agrees with Alice. But he can’t say that, at least not in a public forum. I think the issue with “standard leftist critiques” is that these memes are highly virulent, and most people are not innoculated against them. Systems are hard to fix, so people who have been infected with these memes—even if they are true—may take worse actions than if they had never heard the critique.

James Camacho 10 Jan 2026 17:22 UTC
1 point
0
on: Understanding complex conjugates in quantum mechanics
This is to expand on my earlier comment. It really deserves to be part its own post, and in fact I was already in the process of writing that post when I came across Jessica’s work here. Her explanation for the antilinear-linear inner product is interesting—and initially seemed like it was doing something right—but completely different from my own approach. Well actually, I didn’t have an explanation for why the inner product was antilinear-linear, I had only gotten to the point where I knew there was a bilinear form for physical observables, so I was hoping to incorporate some of Jessica’s ideas into my own post. However, there was one detail she was missing, and that was, “why is it a bilinear form in the first place?” I had the explanation, but I could not see it materializing from her approach. Ultimately, I’ve concluded that the reason her approach seems to work is coincidental, and the mirror, conjugate space is not related to the inner product.
So, why a bilinear form? It is because observations are gauge-invariant, and gauge-invariant things are generated by the field strength tensor $F = D \land D$ , where $D = d + A$ is the contravariant derivative along the spacetime manifold. Why does $F = \land^{2} D$ rather than $D$ or $\land^{3} D$ ? Because $d^{2} = 0$ .
The boundary or derivative operator $d$ comes from simplicial complexes:
$d [v_{1} \land v_{2} \land \dots \land v_{n}] = n \sum i = 1 s g n (12 \dots i) [v_{1} \land \dots \land v_{i - 1} \land v_{i + 1} \land \dots \land v_{n}]$
These simplicial compelexes are in turn one piece of the tensor product. For example:
$v_{1} \otimes v_{2} = \frac{1}{2!} [(v_{1} \otimes v_{2} + v_{2} \otimes v_{1}) + (v_{1} \otimes v_{2} - v_{2} \otimes v_{1})]$
We can identify
$v_{1} \land v_{2} = v_{1} \otimes v_{2} - v_{2} \otimes v_{1}$
which matches
$s g n (σ) v_{1} \land v_{2} = v_{σ_{1}} \land v_{σ_{2}} .$
It seems rather strange to do this, until you recall the Schur-Weyl duality. Consider the tensor space $(C^{n})^{\otimes n}$ and group actions
$\begin{matrix} σ (v_{1} \otimes v_{2} \otimes \dots \otimes v_{n}) & = v_{σ_{1}} \otimes v_{σ_{2}} \otimes \dots \otimes v_{σ_{n}} & σ \in S_{n} g (v_{1} \otimes v_{2} \otimes \dots \otimes v_{n}) & = (g v_{1}) \otimes (g v_{2}) \otimes \dots \otimes (g v_{n}) & g \in G L (n, C) \end{matrix}$
These group actions commute, so they are mutual centralizers and decompose into irreducible representations together. We can decompose tensors into a direct sum of these irreducible representations, i.e. break them up into their “symmetric pieces”:
$(C^{n})^{\otimes n} = ⨁ \begin{matrix} λ ⊢ n λ_{1}^{'} \leq n \end{matrix} S_{λ} \otimes V_{λ}$
The corresponding group actions for each piece are represented by $π^{λ} \otimes ρ^{λ}$ . For the antisymmetric, or $s g n$ representation, $S_{λ} \otimes V_{λ}$ is a one-dimensional vector space—it contains one basis vector, $v_{1} \land v_{2} \land \dots \land v_{n}$ , and multiples of that vector. This means $s g n$ consists of $1 \times 1$ matrices (or scalars), and has lots of nice properties due to being abelian. Rotations will be smooth and transitive, as speck1447 explains here. This is why the boundary operator typically uses this irreducible representation.
However, we can define a more general boundary $d$ as a map to a symmetric piece of one degree lower
$d : V_{λ} \otimes S_{λ} \to V_{μ} \otimes S_{λ}$
where $μ \subset λ$ and $| μ | = | λ | - 1$ . It looks exactly the same, but we use $π^{λ} \otimes ρ^{λ}$ rather than $s g n$ to permute an index to the front and drop it. The reason $d^{2} = 0$ for the $s g n$ representation is because there are $2!$ permutations you can drop two indices in, and summing the different permutations gives you
$\sum σ \in S_{2} s g n (σ) = 0.$
In general, you will get $d^{k} = 0$ when
$\sum σ \in S_{k} π^{λ} (σ) = 0.$
The minimal $k$ is not always two, so other representations will ultimately lead to multilinear forms of different degrees. An equivalent way to look at it is from the $G L (n, C)$ side. Rotations are essentially continuous permutations, so the generalized idea of a rotation is an $n \times n$ matrix $A$ where
$ρ^{λ} (A) = I_{dim V_{λ}}$
(the kernel of $ρ^{λ}$ ). Mirroring and other orthogonal matrices are generalized to
$ρ^{λ} (A) \in I m (π^{λ}) .$
We are looking for a multilinear form $B : (V_{λ})^{\otimes k} \to C$ that is invariant to orthogonal transformations, so for any $v \in (V_{λ})^{\otimes k}$ and $g \in I m (π^{λ})$
$B (g v) = B (v)$
(note: $g$ is applied diagonally). It is enough to find a single vector where $g v = v$ for all $g \in I m (π^{λ})$ . Then we can set $B (v) = 1$ and for non-multiples of $v$ , $B (w) = 0$ . The number of antisymmetrizers is the number of rows $λ_{1}^{'}$ , so there will always be some $g$ where $g v = - v \neq v$ until $k > λ_{1}^{'}$ . This gives the minimal $k = λ_{1}^{'} + 1$ .
This equivalent view is closer to what Jessica is doing. You can think of $(V_{λ})^{\otimes k}$ as the original vector space plus several “dual spaces”. However, the “mirror images” come from $I m (π^{λ})$ , and
$| I m (π^{λ}) | = | S_{n} / ker (π^{λ}) | = ⎧ ⎨ ⎩ \begin{matrix} 1 & λ = (n) 2 & λ = (1, 1, \dots, 1) n! & otherwise. \end{matrix}$
So, for any non-trivial and non-alternating representation,
$λ_{1}^{'} + 1 \leq n < n!$
There are way more mirror images than dual spaces! It is almost a lucky coincidence that
$λ_{1}^{'} + 1 = 2 = | I m (π^{λ}) |$
for the alternating representation, and in general, we should not expect to be able to find dual spaces by looking at mirror images.

James Camacho 10 Jan 2026 4:09 UTC
1 point
0
on: Understanding complex conjugates in quantum mechanics
I disagree. In general, you need a multilinear form of degree $k$ where $1 \subset (π_{λ})^{\otimes k}$ , while the number of mirror spaces is the degree of $I m (π_{λ})$ , You are very lucky that the rotations and mirroring you use come from the alternating representation of $S_{n}$ where these happen to match, since $s g n \otimes s g n ≅ 1$ and $I m (s g n) = \pm 1$ .
I am in the process of writing a much longer comment, but I think the primary question your post leaves unanswered is, “why a bilinear form, not any other degree?” and pulling on that thread unravels the understanding this post allegedly gives.

James Camacho 9 Jan 2026 2:32 UTC
3 points
3
in reply to: zoop’s comment on: A Loser’s Reflections
I think your proposal does not lead to branches getting removed. I think
1. The branch is easiest to remove by first moving around it, and then breaking it from the other side.
2. Once you are on the other side, it is individually rational to continue on your way. The branch is somone else’s problem.
I think having 後輩 or children you value could keep it your problem, enough so you still bother to remove the branch. However, your effort here would only marginally help millions of people, only one of which is the person you personally care about. It is individually more rational to focus your efforts on just the people you care about.

Alternatively, you could value removing such obstacles as a terminal goal. I think this exhibits in humans as an intolerance to such obstacles. The mental anguish significantly decreases once they are no longer bumping against the obstacle, so I think it is rather rare to find someone with a strong enough aversion to remove the branch from the other side, but was able push past a much stronger aversion to make it there.

Now, maybe the solution is to help people push past their aversions. Encourage them to still go to university, even if they believe the world ought to be different. But does this actually help fix the problem? It does not matter if they are 10x more able to fix it, if they care 10% as much. Maybe it relieves some of their suffering, but surely the system is producing much more unnecessary suffering than this proposal would relieve? Plus, it isn’t suffering of people you care about. If you are being individually rational, it seems like your best move is to tell your friends and family to work within the system, while letting the deviants do their own thing to fix it.

James Camacho 8 Jan 2026 16:19 UTC
4 points
1
in reply to: zoop’s comment on: A Loser’s Reflections
I agree that the critique is a fairly standard wokist criticism of meritocracy. I think the author’s two main failures are in adopting a self-centered moral position (“how I want the world to be is how it ought to be”), and poor training. The first is a common mistake, but leaves me wondering, “why should society cater to your whims? Why would they be better off, according to their desires?” It is not enough to say the system creates perverse incentives, if you cannot design a better system for people to adopt. That is his second issue. He does not have the mathematical training to design a better stable system, and reading more humanities papers will only reinforce that something is wrong, without helping him suggest improvements. This is why he can come across as whining or indulging in some superiority fantasy.

I think what you miss is people take different attitudes towards unnecessary suffering. I can imagine if you were hiking along a trail and came across a fallen tree, you would calmly duck under it or climb around it. It is the path you are walking on, and if you wish to reach your desired destination with the least effort, you have to pay these small prices. Others would kick the log for getting in their way, stubbing a toe and not solving anything, but then submit to the path laid out for them. The blessed few carry a hatchet, and would clear the way for those that come after them. Then there is the author, who is so indignant about fallen trees unnecessarily being in their way, but carry no hatchets, that they end up storming off the beaten path and try stomping out their own trail. It usually does not work out for them, and even if they succeed, the new trail is longer and worse than the original trail.

I think the first kind of person is defecting in this scenario, unless they put a few dollars towards fixing the trail when they leave. While broader society is not smart or informed enough to punish such defections—and thus defecting really can be your selfishly optimal move—that does mean it is not defection. Telling people to shut up and climb over the log is also defection. It is one thing to say, “the trail is not good, but there is not much you can do about it, so for your own sake, just climb over it.” Jerret Ye (@L.M.Sherlock) has published another of the author’s articles arguing essentally this. But it is entirely another thing to say, “yes, the trail is not good, but stop complaining already. Just shut up and climb over it.” This is what you are doing, and if everyone did it, the trail would never improve.

I was wrong when I said you had completely missed the author’s second narrative. I believe I was frustrated by what I saw as defection, and thought surely you wouldn’t be defecting if you knew it was defection. But that is a modelling error, because there are many other reasons you might think your comment was prosocial. Most likely, if you can shut up complainers, it makes the system more stable and helps others not go through the author’s pain. I would even agree with an argument that the author has not thought this through well enough to propose good solutions, and so it is better they say nothing at all than risk destroying an almost good thing. I do not know exactly why you wrote such a harsh critique, but I really wonder if there was a better way to achieve your goals.

I do think you and the author are playing a cooperative game here. Both of you have a goal that includes others living better lives due to your writing. In such cooperative games, honesty is usually the best policy. Just put a couple sentences at the top explaining your goals, maybe something like, “I believe these memes are harmful to spread, so I am purposefully being harsh with the author in the hopes that they and others like them put more thought into their writing before publishing.” Of course, then all your readers will wonder why you think the memes are harmful to spread, and so you woud have to explain that, but I think this kind of process would significantly improve your comment.

I do want to expand on my point about the different kinds of people. Some people find it intolerable to move around an obstacle. I am much more similar to the author in this way, and really tried to avoid attending university. I applied to many tech jobs, but failed to get interviews except at some of the lowest-paying ones. To quote the author’s experience, which matches mine,

I’m afraid the company’s HR backend probably didn’t even see my resume. A simple “filter by education,” and sorry, all those outside the selection are automatically deleted… How do you prove yourself to someone who can’t even see you? It’s impossible.

I had significantly higher quailfications on paper—a high school diploma, great standardized test scores, and better competition results—but even I could not be seen by a human. So, although I really hated wasting my time on paperwork, I went to MIT. I mostly tried to take classes I was interested in and luckily for me I was interested in a broad education. My largest frustrations were actually in some required science classes: introductory chemistry, probability, algorithms, and whatnot. I did my best to substitute them with more useful classes, but I was told for many of them, “you just have to take it. You cannot test out or substitute them.” The whole process was very frustrating, especially because I am so intolerable to unnecessary obstacles that only exist because someone never bothered to design the system a little better. What made it worse is MIT had a better system (at least for me) fifty years ago, and it had evolved to be worse in terms of free learning, but better in terms of their typical student’s income after graduation. Anyway, I remember thinking over and over, “why am I doing all this,” and I know I would have dropped out if I did not graduate early. Even still, I almost dropped out to join a startup instead of doing my final semester.

I could make this a much longer post, but what I’m trying to highlight is there are some people who see problems that should not exist, and it causes them a ton of mental anguish. You seem generally unbothered, and can go around the obstacles in stride, but it is really painful for people like me. Logically, the additional effort is merely a lot, but mentally it is intolerable. And, the author has much more difficulty in this regard than I do. Part of it is a stricter system, part of it is less natural talent the system would reward them for, and part of it is having an even stronger mental aversion than me to disfunction.

James Camacho 7 Jan 2026 23:16 UTC
5 points
0
on: A Loser’s Reflections
I understand the sentiment. It feels like a waste of effort to study for exams you do not enjoy, do not make you more capable, and do not improve your self-image. I did not experience much of this, as the American standardized exams are significantly easier than the Chinese ones and cannot measure what the elite institutions are looking for, which forces a more holistic approach. It also causes other awful downstream effects, so I do not recommend imitating America here.
I also agree that a lottery system is better than assigning ranks and filtering for the top. There are many goals the university could have in mind when selecting candidates, which they abstract into some process that ranks candidates. If this process is well-known, such as with entrance exams, candidates can spend an inordinate amount of time optimizing for this proxy instead of aligning closer to the university’s ideal student. An opaque process is worse, as part of the optimization process becomes uncovering the black box, which in America comes in the form of $20k “college admissions consultants”. The solution is to make a transparent process for scoring or ranking candidates, but then to randomize the selection.
The author’s proposal for the lottery is wrong. Most admissions, especially at elite institutions, follow a Pareto distribution. Most admissions, especially at elite institutions, are somewhat random, due to application reviewers being human and needing to eat lunch. Thus, almost everyone admitted to elite institutions won a thresholded lottery. These universities like to humble their students during orientation week by saying, “we could have chosen the next 1,000 students instead of you,” which is somewhat false, but true for the bottom 900 of them. It may be different in China (or Taiwan), and this could be a quirk of the American system. However, the American system has a similar educational arms race, so a thresholded lottery is not the solution.
The solution is an exponential lottery. There are diminishing returns to optimizing a score function. To give an example with language learning, it takes about as long to understand 50% of the words on the screen as the next 25%. Scores are logarithmic with effort. If
$Pr [selection] \propto exp (score / temperature)$
candidates would be indifferent between the cost of studying and its expected reward. Only those that have a different, better reward in mind will continue studying, naturally killing the arms race. I believe the right temperature choice should keep the free energy constant across admissions cycles, but I am not sure.

James Camacho 7 Jan 2026 22:01 UTC
3 points
1
in reply to: zoop’s comment on: A Loser’s Reflections
I think there are two things the author is doing here. One of them is whining about not wanting to pay the price the system demands to reap its benefits. The other is arguing that all this pain and suffering is unnecessary for the system to function as it claims it intends to function. This is why they call it a “rat race” (内卷 ^[1] ), bring up the distinction between Pareto and Kaldor-Hicks improvements, talk about about how the system is self-reproducing—which means it survives despite disfunction—and mention how external negative pressure is the natural reaction. You seemed to have completely missed this second narrative. Although they may live in a disfunctional system, the world is a much bigger place. New systems can be created.
1. ↩︎
  I think the term refers to runaway signal inflation.

James Camacho 10 Dec 2025 1:46 UTC
1 point
0
in reply to: abstractapplic’s comment on: D&D Sci Thanksgiving: the Festival Feast Evaluation & Ruleset
Fewer rows might not give interpretable/rules-based solutions an advantage. I tried training on only the first 100 or 20 rows, and I got CDEFMW (15.66) and EMOPSV (15.34) as the predicted best meals. Admittedly CDEFMW shows up in the first 100 rows scoring 18 points, but not EMOPSV. Maybe a human with 20 rows could do better by coming up with a lot of hypothetical rules, but it seems tough to beat the black box.

James Camacho 1 Dec 2025 15:48 UTC
2 points
0
in reply to: Ape in the coat’s comment on: The Joke
I think your main counterpoint to what I said is that people are doing an optimization process where they look at the data while simultaneously doing a search for a better theory. In fact, you cannot even disentangle their brain from the reality that created and runs it, so even a best attempt at theory first, observation second is doomed to fail.

I think the second, stronger sentence is mostly wrong. You do not need a universe similar enough to our universe to produce reasoning similar to ours, just one that can produce similar reasoning and has an incentive to. That incentive can be as little as, “I wonder what physics looks like in 3+1 dimensions?” just like our physicists wonder what it looks like in more or less dimensions, with different fundamental constants, with different laws of motion, with positive spacetime curvature, and so on. Or, we can just shove a bunch of data from our universe into theirs, and reward them for figuring it out (i.e. training LLMs).

As for the first, weaker sentence, yes this is true. Pretty much everyone has tight feedback loops, probably because the search space is too large to first categorize its entirety and then match the single branch you end up observing. I think the role of observation here is closer to moving attention to certain areas of the search space, rather than moving the search tree forward (see Richard Ngo’s shortform on chess). The thing is, this process is unnecessary for simple things. You probably learned to solve TicTacToe by playing a bunch of games, but you could have just solved it. I think the concept of trees are relatively simple, though of course if you want a refined concept like its protein composition or DNA sequencing, yeah that space is too big and you probably have to just go out and observe it.

I don’t really understand your point about unsupervised learning. With unsupervised learning, you can just run a bunch of data through your model until it learns something. That’s the observation → theory pipeline and it’s astoundingly inefficient and bad at generalization. Humans could do the same with 100x fewer examples, which is the gap models need to clear to solve ARC-AGI. Humans are probably doing something closer to theory → observation.

James Camacho 1 Dec 2025 14:53 UTC
1 point
0
in reply to: Richard_Kennaway’s comment on: The Joke
Presumably prepend or append “is wrong.”

James Camacho 29 Nov 2025 19:37 UTC
5 points
−7
on: The Joke
I think you’re missing something here. Science, and more specifically physics, is built on first theorizing or philosophizing, coming up with a lot of potential worlds a priori, and only looking to see which one you probably fall in after the philosophizing is done. How do you know a tree exists? Well, I bet a good philosopher from a different universe could come up with the concept of trees only knowing we live in 3+1 dimensions:
- 3+1 dimensions is enough to figure out there are electric and nuclear forces, and thus fusion and interesting chemical reactions going on.
- Those interesting chemical reactions are going to eventually form self replicators which are going to harness the cheapest energy available—fusion.
- There are only a few phases of matter (percolation theory), one of which is solid, so you’d expect some lifeforms to evolve that harness fusion energy while standing on solid ground.
- Evolution is going to make these lifeforms grow taller to be closer to the energy source than their neighbors.
- So something like trees will exist.
Looking isn’t needed to know trees really exist, just to know that tree over there really exists. That involves a bit of cyclical reasoning, which is basically because you cannot prove a system consistent within the system. The best you can do is check that the current statement isn’t inconsistent, such as if someone says, “how do you know that tree over there really exists,” while pointing at empty air, and you say, “it doesn’t.”

I think my issue with empiricism is that it does not generalize, at all. A better way to compress sense data is to first come up with several theories, then use a few bits to point out which theory is correct and how it is being applied.

James Camacho 27 Nov 2025 18:10 UTC
2 points
−1
on: Postmodernism for STEM Types: A Clear-Language Guide to Conflict Theory
The mental frame I’ve found myself getting into when interacting with conflict practitioners is treating them not as defecting against me, but against everyone. Even if I agree with someone’s position, I will give a pretty scathing reply or employ the dark arts even harder to make them look stupid. Why? Well, in adversarial games, there is nothing to gain from honesty, so the only reason they are using words at all is to manipulate people. The only way to have an honest discussion on the topic is to get them to stop playing adversarial games around it. If I guess they’re just too stupid to realize what they’re doing (as in, they sound like they’re just repeating memes instead of carefully picking their words), I think it’s a better idea to break them out of the adversarial game by forcing them to actually think (by being very precise in pointing out the issue with their rhetoric). But if they know what they’re doing? Better to take them out of the game.

My guess is most people in America’s political climate are conflict practitioners. I do not know if this was the case 50 years ago, but I suspect it was not so, and it is merely an issue with democracy where infecting the populace with adversarial memes is a great way to get elected so after time (and especially the internet) the populace is more and more disease-ridden. If everyone were smarter, they would be more immune to banal manipulation, but I think the memes would also just be more adversarially selected.

James Camacho 27 Nov 2025 17:19 UTC
2 points
0
in reply to: clone of saturn’s comment on: Postmodernism for STEM Types: A Clear-Language Guide to Conflict Theory
Why do some people, who lack the ability to write this well, assume no one else has that ability and thus it must have been written by AI?

James Camacho 26 Nov 2025 0:44 UTC
1 point
0
in reply to: Jesper L.’s comment on: Maybe Insensitive Functions are a Natural Ontology Generator?
I don’t have any more thoughts on this at present, and I probably won’t think too much on it in the future, as it isn’t super interesting to me.

James Camacho 25 Nov 2025 20:39 UTC
1 point
0
in reply to: Jesper L.’s comment on: Maybe Insensitive Functions are a Natural Ontology Generator?
This is mostly correct, though I think there are phase changes making some $β$ more natural than others.

James Camacho 25 Nov 2025 14:42 UTC
2 points
0
in reply to: Jesper L.’s comment on: Maybe Insensitive Functions are a Natural Ontology Generator?
Haha, I did initially start with trying to be more explanatory but that ended after a few sentences. Where I think this could immediately improve a lot of models is by replacing the VAEs everyone is using in diffusion models with information bottleneck autoencoders. In short: VAEs are viruses. In long: VAEs got popular because they work decently well, but they are not theoretically correct. Their paper gestures at a theoretical justification, but it settles for less than is optimal. They do work better than vanilla autoencoders, because they “splat out” encodings which lets you interpolate between datapoints smoothly, and this is why everyone uses them today. If you ask most people using them, they will tell you it’s “industry standard” and “the right way to do things, because it is industry standard.” An information bottleneck autoencoder also ends up “splatting out” encodings, but has the correct theoretical backing. My expectations are you will automatically get things like finer details and better instruction following (“the table is on the apple”), because bottleneck encoders have more pressure to conserve encoding bits for such details.

There are probably a few other places this would be useful—for example, in LLM autoregression, you should try to minimize the mutual information between the embeddings and the previous tokens—but I have yet to do any experiments in other places. This is because estimating the mutual information is hard and makes training more fragile.

In terms of just philosophy, well I don’t particularly care for just the subject of philosophy. Philosophers too often assign muddy meanings to words and wonder why they’re confused ten propositions in. My goal when interacting with such sophistry is usually to define the words and figure out what that entails. I think philosophers just do not have the mathematical training to put into words what they mean, and even with that training it’s hard to do and will often be wrong. For example, I do not think the information bottleneck is a proper definition of “ontology” but is closer to “describing an ontology”. It does not say why something is the way it is, but it helps you figure out what it is. It’s a way to find natural ontologies, but does not say anything about how they came to be.

James Camacho 25 Nov 2025 1:53 UTC
3 points
2
on: Thou art rainbow: Consciousness as a Self-Referential Physical Process
A couple of terms that seem relevant:
1. Holonomy—a closed loop which changes things that move around it. I think consciousness is generally built out of a bunch of holonomies which take sense data, change the information in the sense data while remaining unchanged itself, and shuttle it off for more processing. In a sense, genes and memes are holonomies operating at a bigger scale, reproducing across groups of humans rather than merely groups of neurons.
2. Russell’s vicious circle principle—self-reference invariably leads to logical contradictions. This means an awareness of self-referential consciousness (or phenomenological experience) cannot be a perfect awareness, you must be compressing some of the ‘self’ when you refer to yourself.

James Camacho 25 Nov 2025 0:59 UTC
2 points
0
in reply to: Jesper L.’s comment on: Maybe Insensitive Functions are a Natural Ontology Generator?
The tricky part when parsing John’s post is understanding what he means by “insensitive functions.” He doesn’t define it anywhere, and I think it’s because he was pointing at an idea but didn’t yet have a good definition for it. However, the example he gives—conservation of energy—occurs because the laws of physics are insensitive to some kind of symmetry, in this particular case time-translation. I’ve been thinking a lot about the relationship symmetries + physics + information theory this past year or two, and you can see some of my progress here and here. To me, it felt kind of natural to jump to “insensitive functions” being a sort of stochastic symmetry in the data.
I haven’t fleshed out exactly what that means. For exact symmetries, we can break up the data into a symmetry-invariant piece and the symmetry factor
$x = (^x, g) .$
However, it feels like in real data, there is not such a clean separation. It’s closer to something like this: we could write $x$ “big-edian” form, so that we get finer and finer details about $x$ as we read off more bits. My guess is there is an “elbow” in the importance of the bits, similar to how Chebyshev series have this elbow in coefficient magnitude that chebfun identifies and chops off for quicker calculations:
(Source: Chopping a Chebyshev Series)
In fact, as I’m writing this, I just realized that your autoencoder model could just be a discrete cosine (Chebyshev series) transform. It won’t be the best autoencoder to exist, but it is what JPEG uses. Anyway, I think the “arm”—or the bits to the left of the elbow—seems to form a natural ontology. The bits to the left of it seem to be doing something to help describe $x$ , but not the bits to the right.
How does this relate to symmetries? Well, an exact symmetry is cleanly separable, which means the bits could be added after every other bit—it’s far to the right of the elbow. Chopping off the elbow does satisfy our idea of “ontology” in the exact symmetry case. Then all we need to do is create a model that chops off those uninteresting bits. The parameter $β$ in the information bottleneck pretty much specifies a chopping point. The first term, $I (Y; g (X))$ , says to keep important bits, while the second term, $I (g (x); X)$ says to cut out unimportant bits, and $β$ specifies at what point bits become too unimportant to leave in. You can slowly increase $β$ until things start catastrophically failing (e.g. validation loss goes down), at which point you’ve probably identified the elbow.

James Camacho 24 Nov 2025 19:31 UTC
18 points
0
on: Maybe Insensitive Functions are a Natural Ontology Generator?
I am sure you are already aware of this, but the conserved quantities we see come from symmetries in the function space (Noether’s theorem). The question I think you should be asking is, how do we extend this to random variables in information theory?

I am not sure of The Answer™, but I have an answer and I believe it is The Answer™: with the information bottleneck. Suppose we have a map $f : X \to Y$ from some bits in the world to some properties we care about. In my head, I’m using the example of $X = MNIST digits$ and $Y = their classifications$ . If there is a symmetry in $X$ that $f (X)$ is invariant to, this means we should lose no predictive information by transforming a sample $x$ according to that symmetry

$\begin{matrix} I (Y; X) & = I (Y; g X) \forall g \in G      symmetry ⟹ I (Y; X) & = I (Y; X / G) \end{matrix}$

If we already know the symmetry we can force our predictor to be invariant to it. This is most commonly seen in chemistry models where they choose graph neural networks that are invariant to vertex and edge relabeling. However, usually it is hard to specify the exact symmetries (how do you code a “seven” is symmetric to stretching out its leg a little?), and the symmetries may also not be exact (a “seven” can bleed into a “one” by shortening its arm). The solution is to first run $X$ through an autoencoder model that automatically finds these exact symmetries, and the inexact ones up to whatever bit precision you care about.

If we replace the exact $X / G$ by the autoencoder $g (X)$ , then

$I (Y; X) = I (Y; X / G) \approx I (Y; g (X)) \leq I (Y; X)$

means we want to maximize the mutual information between $Y$ (the property we care about, like “sevenness”) and $g (X)$ . Also,

$H (g (X)) \approx H (X / G) = H (X) - H (G),$

meaning the more symmetries $g (X)$ captures, the smaller its entropy should be. Since $X$ probably has some fixed entropy, this is equivalent to minimizing the mutual information between $X$ and $g (X)$ . Together, we have a tradeoff between maximizing $I (Y; g (X))$ and minimizing $I (g (X); X)$ , which is just the information bottleneck:

$max I (Y; g (X)) - β I (g (X); X) .$

The larger the $β$ the more “stochastic symmetries” are eliminated, which means it gets closer to the essence of “sevenness” or whatever properties are in $Y$ , but further from saying anything else about $X$ . The fun thing you can do is make $X$ and $Y$ the same entity, and now you are getting the essence of $X$ unsupervised (e.g., with MNIST, though it does have reconstruction loss too).

Finally, a little evidence that seems to align with autoencoders being the solution comes from adversarial robustness. For a decade or so, it was believed that generalization and adversarial robustness are counter to one another. This seems a little ridiculous to me now, but then again I was not old enough to be aware of the problem before the myth was dispelled (this myth has been dispelled, right? People today know that generalization and robustness are essentially the same problem, right?). Anyway, the way everyone was training for adversarial robustness is they took the training images, perturbed them as little as possible to get the model to mispredict (adversarial backprop), and then trained on these new images. This ended up making the generalization error worse (“Robustness May Be at Odds with Accuracy”). It turns out if you just first autoencode the images or use a GAN to keep the perturbations on-manifold, then it generalizes better (“Disentangling Adversarial Robustness and Generalization ”). Almost like it captured the ontology better.