programjames

Karma: 515

programjames 9 Sep 2026 10:20 UTC
0 points
0
on: James Camacho’s Shortform
If your public benefit corporation of people ~30 IQ points higher than the population average has solved the alignment problem, why is it misaligned with the public’s benefit?

programjames 31 Aug 2026 0:17 UTC
2 points
0
on: Intelligence Scales as the Logarithm of Compute (& Data)
I take more of a theoretical approach to these kinds of things. What could we say about intelligence scaling, just from the maths of what makes things intelligent?

The definition of intelligence is finding short programs. Those programs can explain phenomena (traditional AI), search an environment (RL), or find short programs (metalearning). The structure for “finding better programs” looks like a loop of technology → empowerment → resources → technology.

Before the world got too complicated, people could empower themselves with all technology, so we had

and

This leads to hyperbolic growth in resources, which looks like an approaching population singularity in late 2026. (see the doomsday equation). However, humans have an upper limit for empowerment due to their weak brains, and by the 1960s polymaths had died out. Instead

and trends—such as Moore’s law and GDP—became exponential. Note that the resources also diverged from human resources, and many technologies became memetic in nature: better software, better training, better mathematical structures. ^[1]

In traditional AI or RL, finding short programs is essentially crypto mining. Humans design a training algorithm and hardware, and throw it at the problem. The search space covered should match the human trends—exponential. So, the shortness of the program is log-exponential in time, a linear increase in IQ and capabilities.

In metalearning, suddenly the loop closes again. Weak human brains are no longer a bottleneck for empowering search with better technology, so the trend becomes hyperbolic once more, at least until the AI runs into the limits of physics.
1. ↩︎
  It is also plausible that the hyperbolic trend continued, just in more immaterial spaces than human population or chip manufacturing.

programjames 30 Aug 2026 3:06 UTC
1 point
0
in reply to: joseph_c’s comment on: Is there only one FairBot?
Can you please fix your comment?
```
;; Play the opponent against a copy of 𝐂𝐨𝐨𝐩𝐞𝐫𝐚𝐭𝐞𝐁𝐨𝐭'𝐬 𝐜𝐨𝐝𝐞, not JustBot's code:
'(λ (my-code opponent-code)
    (prove `(,opponent-code ',opponent-code ',my-code)))
```
The source of my confusion is:
1. You say “CooperateBot’s code” but insert Fairbot’s code.
2. What is “just” about cooperating with FairBot?

programjames 30 Aug 2026 2:31 UTC
1 point
0
in reply to: joseph_c’s comment on: Is there only one FairBot?
You have a typo. JustBot is inserting a copy of FairBot’s code. I think, given the name JustBot, you probably meant to play the opponent against a copy of CooperateBot:
```
(define (just-bot my-code opponent-code)
    (define (cooperate-bot my-code opponent-code)
        #t)
    (prove
        `(,opponent-code ',opponent-code ',cooperate-bot)))
```

programjames 3 Aug 2026 23:32 UTC
1 point
0
in reply to: mruwnik’s comment on: “Bad faith” means intentionally misrepresenting your beliefs
Would you support the neologism, “acting on bad faith,” to clarify the person in question may only be a representative of a bad faith organization?

programjames 2 Aug 2026 2:17 UTC
0 points
−2
in reply to: Eye You’s comment on: Elizabeth’s Shortform
Jargon, rituals, and slogans are absolutely harmful. Cults do so much more damage to people’s minds that is hard for an external observer to pick up on. After all, they wouldn’t be an external observer if they internalized the cult messaging.

programjames 29 Jul 2026 0:27 UTC
1 point
0
in reply to: Alex Gibson’s comment on: leogao’s Shortform
It’s mostly an issue of, “how much effort is communication for each party?”
1. If you are writing an article that will be read by thousands of people, it makes sense to spend several hours to save each of them a couple minutes.
2. If you are teaching students who are new to the subject, and presumably have difficulty communicating where their confusion is, it makes sense to spend a lot of effort modelling them and their understanding, so your lecture builds intuitively on top of that instead of them having to contort their brains until it clicks.
3. If you are writing a comment that only a couple other people will ever read, and you don’t expect it to be controversial or difficult to understand, stream-of-thought writing is probably fine.
4. If you are debating someone on the internet, both of you are acting in good faith, and both of you are about as smart, you should write like the other person will spend the effort to understand, and read like the other person is as smart and acting in good faith as you assume they are. Doing anything different is acting in bad faith. Either you are treating them like they are stupid or they are neglecting to put in the same amount of effort as you. Now I understand there is not the intention of acting in bad faith, and in fact acting like people are lazier or stupider than they actually are may lead to better communication as a general rule of thumb, but anytime you change norms so unintentional bad faith gets rewarded, people start committing a worse sin: they act in bad faith and don’t even know why that was wrong!

programjames 28 Jul 2026 21:42 UTC
6 points
−2
in reply to: leogao’s comment on: leogao’s Shortform
I have some experience being, “the huge tangled ball of yarn where every claim you dig into seems to uncover even more bizarre claims.”

My experience is that you (the digger, not leogao in particular) are making invalid assumptions, usually crude heuristics/intuitions, I point out those invalid assumptions with very precise language, and you completely ignore the precision and come up with a crude heuristic for what I said.

For example, a recent conversation I had on LessWrong:

(you) a big issue with Risk as an analogue to grand strategy is that it presumes that war is correct and zero-sum (rather than negative sum in reality).

(me) War is not zero sum in Risk. There are six players so a large conflict between two of them is usually negative sum. Also, war is usually positive sum in reality and Risk, e.g. evolution, proxy wars.

(you) all competitive board games are zero sum by default and I can’t imagine under what model wars are positive sum in reality

The first comment has two crude intuitions:
1. Winning in board games is zero sum → interactions in board games are zero sum
2. The most prominent wars in my mind are negative sum → war is negative sum
I immediately point out how these heuristics are not true:
1. Zero sum games with more than two players can contain subgames that are nonzero sum.
2. Evolution is akin to a war, and positive sum for the competitors. Proxy wars can be positive sum for the greater powers.
My best guess is they smoothed out my two claims to:
1. zero sum games are not always zero sum
2. wars, which we all know are negative sum, are positive sum
Then they walk away thinking, “man, I tried to dig into their confused epistemology, but it’s such a tangled ball of yarn that every time I follow a thread it just reveals even more bizarre claims. Like come on, he’s saying some obviously tautologically false things.”

My best guess for the miscommunication is I learned to read/write (long form) from math/coding, and pretty much no one else did. So if I say something in a very particular, precise manner, they—unlike me—never learned to pay attention to the precision. The issue is not my sending channel capacity, but their receiving, and when a packet gets dropped they do not slow down and reread. No, they conclude that the bizarre object they reconstruct must be what was originally transmitted.

I think this also mostly explains your explanations: People who don’t converge repeatedly (and from your perspective obstinately) say you are simplifying things in ways that actually matter. And, they usually have to be smart enough, or you would reconstruct their model relatively easily and not have to carefully think through what they say, the implications of what they say, and the implications of what they didn’t say (e.g. can you immediately think of a strong counterargument? then they probably did too, thought a little more, and realized it wasn’t a strong counterargument so they didn’t bother to address it).

programjames 26 Jul 2026 18:24 UTC
2 points
0
in reply to: StefanHex’s comment on: StefanHex’s Shortform
You are looking for low information directions in activation space. That is, what directions have the model already compressed and would take very few bits to move the model to activate more in that direction?

A Gaussian perturbation to the model parameters (not activations) gives an unbiased estimate for the number of bits it takes to update the model under gradient flow to activate in that direction. To be precise, if is an activation (or loss), then

where is the learning rate (how quickly you want to move the model). For the RHS automatically includes information regularization—how much information is there in the activation plus the information to move the model there? This is actually much better than treating the activation (or loss) as free information.

A Gaussian perturbation to the model activations (not parameters) is almost a Gaussian perturbation to the parameters, as neural networks are almost linear. So this is why you still see signal when perturbing the activations instead of the parameters.

All of this gets much more complicated with singular learning theory. In truth, is only the number of bits it takes to move a nonsingular model under gradient flow. But neural networks are very singular. A very expensive way to recover the correct estimate is to do a whole bunch of Monte Carlo simulations, where you actually train the network while adding a tiny bit of noise.

programjames 24 Jul 2026 14:30 UTC
1 point
0
in reply to: Linch’s comment on: War – What is it Good For?
Might as well say everyone in the real world is in a zero-sum competition over control of Tegmark IV. But for the most part, wars take place on much smaller battlefields. E.g. trading spots in Asia to get territory cards, fighting a 2v1 to get another continent. Both of which are positive sum, as they increase the probability that one of the players involved will be the ultimate victor.

programjames 24 Jul 2026 4:58 UTC
1 point
0
in reply to: Linch’s comment on: War – What is it Good For?
Your comment is completely incorrect.
1. War is not zero sum in Risk. There are six players so any large conflict between two of them is usually negative sum.
2. War [edit: by occurrences, not weighted by scale] is usually positive sum in reality and Risk, e.g. evolution, proxy wars.

programjames 20 Jul 2026 15:09 UTC
3 points
0
on: War – What is it Good For?
I think the theory is directionally correct, but there are a few mistakes:
1. Fewer wars between distal countries is only because they cannot project their power that far, and if they try a nearby country would opportunistically subsume them. Closer geographic distance also happens to make the countries similar in other ways, but those other ways have ~no explanatory power for the war.
2. Alliances form to make going to war positive EV, not the other way around. This is why the alliance that instigates a war is usually positive EV—because that alliance was put together specifically for that war.
I would recommend watching some games between the top players in Risk, a simplified simulation of war. They spell out what their plans are, and why. Obviously a game is not perfectly applicable to the real world, but Risk is the equivalent to grand strategy as Go is to positional strategy and Chess battlefield strategy.

programjames 15 Jul 2026 14:10 UTC
1 point
0
on: James Camacho’s Shortform
Positional embeddings are group representations. This is how you find them, and this is also how you parameterize them to make them differentiable/learnable. To give a few examples:
1. Rotary positional embeddings are representations of the translation group. The -dimensional translation group is abelian, so its irreducible representations are all one-dimensional , specified by a momentum vector : . You can get a multi-dimensional representation by choosing several , which are your learnable parameters.
2. If you wanted RoPE embeddings to also preserve rotation (perhaps you have scenes viewed at arbitrary angles), you are looking for representations of the special Euclidean group. The irreducible representations in four dimensions are your familiar particles: electrons, protons, etc., and in general are parameterized by a momentum vector and a discrete spin vector. You would have to do some kind of annealing process to train the spin vector (or you could just make it constant).
3. If you want to do attention on quadtrees, well, what is the group? Suppose every node in the quadtree has local coordinates scaled to . Moving up/down/to a neighbor in the quadtree requires an affine transformation of the local coordinates, which compose, so we can label every node by its affine transformation to the root , where is the node’s translation in the image and is its depth in the tree. The finite-dimensional irreducible representations are parameterized by momentum vectors as well as a modulus and basis vectors and look like where permutes the basis vectors.

programjames 30 Jun 2026 22:46 UTC
4 points
−4
in reply to: Anthony DiGiovanni’s comment on: Functional Decision Theory: Not Even Wrong, Also Wrong
My experience taking philosophy classes at MIT was academic philosophers are trained very poorly at scrutinizing philosophical arguments. People adjacent to philosophy—logicians, mathematicians, linguists, (not physicists)—are generally better able to work through philosophical arguments. The first two because learning the language forces them to learn how to think argumentatively, the third because it deals with similar structure without the baggage introduced by academic philosophers.

For example, if I present the Grandfather Paradox to a professor in these four fields, I expect:
1. The logician would ask me for the axioms, and keep pushing until I admit one of them is, “your grandfather cannot die.”
2. The mathematician would say, “obviously you cannot kill your grandfather, otherwise you have a contradiction.”
3. The linguist would ask me what exactly I mean by grandfather—biological? has he frozen sperm? does he have children yet? a twin brother?—and conclude, “you can kill your grandfather, at least the person that word refers to in the mind of the time traveler.”
4. The philosopher would say, “oh that’s such an interesting thought experiment. I don’t know, can you? It doesn’t seem like you can’t, and yet that seems like it would create a contradiction.” Then they would try putting the paradox in premise-conclusion form, and two hours later conclude, “well many of these axioms and implications seem a little fishy, but I would have to say you can/cannot [equally likely] kill your grandfather”.

programjames 29 Jun 2026 21:34 UTC
7 points
0
in reply to: Bentham's Bulldog’s comment on: Functional Decision Theory: Not Even Wrong, Also Wrong
I acknowledge the snark. I get annoyed when people repeatedly make the mistake of not defining their terms, running rampant with them, being shocked when they get mismatched intuitions, and conclude the undefinitions wrong. It’s no more logical than, “you’re wrong because I feel that way.”

I explain why this isn’t right. Only one algorithm is dependent on yours. The other is just correlated.

Correlated is doing a lot of equivocating in your intuitions. It’s merely correlated not causal, he says! What’s the difference? everyone asks. Oh, there is none, they are extensionally identical, but using the word correlated will trick the functional decision theorist into taking a different action.

One man’s modus tollens is another man’s modus ponens.

You say, “since my intuitions imply the functional decision theorist will take different actions in these extensionally equivalent scenarios, clearly the functional decision theorist can be Dutch booked.”

I say, “since the functional decision theorist is rational and cannot be Dutch booked, clearly your intuitions are wrong about what the functional decision theorist will do. Go back and straighten out your definitions.”

Second, as I explain, measuring similarity is even more difficult.

I literally told you how to measure similarity: KL(p||q).

Re calculator, whether they output the same thing depends on how you interpret their outputs, as explained in the post.

I didn’t think this was your main objection because you told us the isomorphism. However, if the isomoprhism is unknown, or there isn’t an isomorphism but some other transformation, you can use the mutual information to recover it. Re: mutual information neural estimation.

99.9% accurate in the sense that 99.9% of the time, the predictor guesses right.

And it’s a stipulation of the thought experiment that the predictor is reliable—doesn’t matter if this could exist in the real world.

Ah yes, the principle of explosion proves every proposition true and false. Just sneak in contradictory axioms (by not defining your terms) and you can prove anything you want!

programjames 29 Jun 2026 18:38 UTC
1 point
0
in reply to: Gurkenglas’s comment on: Functional Decision Theory: Not Even Wrong, Also Wrong
There is no such thing.

programjames 29 Jun 2026 18:12 UTC
1 point
0
in reply to: Gurkenglas’s comment on: Functional Decision Theory: Not Even Wrong, Also Wrong
The issue with saying, “this agent,” is you do not actually know its policy. The best anyone can do is generate all programs that output the seen distribution of actions, using error-correction codes for nondeterministic policies. Now you have many theories of varying description lengths for the agent, which you weight according to the Solomonoff prior. We can always describe another agent’s policy with a fixed KL(their policy||your policy) extra error-correction bits, so the utils under a given theory are

sum_{policy} exp(-|theory| - KL(policy||your policy)) utils(policy)

and the total utils are

sum_{theory} sum_{policy} … = constant * sum_{policy} exp(-KL(policy||your policy) utils(policy)

programjames 29 Jun 2026 17:31 UTC
−2 points
−1
on: Functional Decision Theory: Not Even Wrong, Also Wrong
To give a meta summary of the problems in this essay: the author does not define their terms, runs rampant with them, and is then shocked when they run into contradictory intuitions.
Here FDT’s answer is that you should two-box in the first case but not in the second case.
No, it says to two-box in both cases for exactly that axiom of extensionality (“equivalent prediction principle”).
No fact about whether two algorithms are the same
This entire objection is a failure to recognize that 0% and 100% are not probabilities. Rather than saying, “are these the exact same?” which is always impossible to verify to 100% confidence, not just with algorithms, you should ask, “how similar are these policies?” The goal of functional decision theory is to maximize the utility of all agents with similar policies to your own, weighted by exp(-KL(their policy||your policy)). So, for example, your twin that presents exactly the same except on Newcomb’s problem has the maximally distant policy to your own when you run into Newcomb.
The calculator objection is made up. You define your terms and then claim they are not defined! You gave us the isomorphism between the calculator’s algorithm and the neg-calculator’s algorithm. Under the axiom of extensionality, these are the same algorithms. Unless you want to define “algorithm” differently, which involves the minus sign. In which case, they’re slightly different algorithms (see the previous paragraph).
Determining how one algorithm being different would affect another algorithm being different, without depending on the epistemic probability of the second being different if the first was.
Is being exponentially accurate in polynomial time good enough? If so, just use annealing to find the trembling-hand equilibria. If not, you’re asking to solve P=NP.
The predictor has a failure rate of only 1 in a trillion trillion.
How? A rational decision-maker chooses mixed policies (for that entropy bonus), and even when there is certain, painful death on the table, they will almost certainly choose it more than 1 in a trillion trillion times. Even an irrational actor will simply mess up and walk in the wrong direction more often than 1 in a trillion trillion. If they claim they can predict my randomness ahead of time, they are lying. As a good functional decision theorist, I use a quantum random number generator (e.g. stare at a lightbulb while thinking) to prevent my randomness being hacked like that.
The simulation is highly correlated with you, so his guesses about whether you’ll cut off your leg for no benefit are 99.9% accurate.
The issue with most of these scenarios is you are unclear on what you mean by 99.9% accurate. Is this epistemic or aleatoric uncertainty? If it is epistemic, Newcomb is not powerful enough to change his decision based on your algorithm. Your algorithm should be: pretend to be a one-boxer (or leg-cutter) up until actually put in that scenario. If he gives you tests before putting you in the scenario, well an EDT or CDT would certainly do their best to pass the tests as well for the future utility.

If it is only the aleatoric uncertainty in your policy, we have set up as an axiom that Newcomb knows your policy. Then you object, “why not just change the policy?” But you literally just made a stipulation that you can’t! It’s exactly the failure most people make with the Grandfather’s Paradox. What seems to be possible is physically impossible when you impose axiomatic restrictions.

programjames 23 Jun 2026 18:03 UTC
9 points
−3
on: Aim to get rejected often
In a world where no one follows this policy, it is a Kaldor-Hicks improvement that gives you the majority of the gains, a very small percent of the people you contact a smaller gain, and the vast majority of the people you bother a small loss. In the actual world where a lot of people follow this policy, it is not even a Kaldor-Hicks improvement. It is just shifting value from people with filters to those without, with transmission costs along the way. The proper response is to make “the worst that can happen” a whole lot worse so people stop saying, “I might as well try pushing negative externalities on others and destroying value in the world.”

programjames 16 Jun 2026 17:22 UTC
3 points
0
in reply to: programjames’s comment on: James Camacho’s Shortform
An amusing story: I tried training a Connect4 bot purely to maximize entropy in a tournament system, so winning more games increased how often it played (the “selective pressure”) ^[1] . Naturally, I thought increasing the selective pressure would increase its game-winning abilities.

Nope! The stronger the selective pressure got, the longer it would drag out games. When it had a win in one move, it would mark it as one of the worst moves to play, only barely above immediately losing. It wouldn’t survive long enough to get entropy in the higher levels of the tournament, so it was trying to eke out more entropy by delaying winning as long as possible.
1. ↩︎
  This is an oversimplification. It was closer to self-play with, “given a win/draw/loss and a tournament structure, how many more games would it get to play?”

programjames

No fact about whether two algorithms are the same