transhumanist_atom_understander

Karma: 480

transhumanist_atom_understander 12 Mar 2026 14:33 UTC
1 point
0
in reply to: James Payor’s comment on: Payorian cooperation is easy with Kripke frames
I don’t think was a mistake.

This gets at the deeper issue: how do we represent an action having a consequence? I don’t think it’s .

In the Prisoner’s Dilemma tournament, you don’t respond to my action, you respond to what you can prove about my action, so it’s really that’s important.

But I think that even in single player games, we should think of the environment as containing an implementation of us which takes an action when it can prove it’s the “right” action, so even then it’s provability that has consequences.

This may seem unnatural, but that’s just because considered decisions are unnatural, and mostly we just do stuff. But I think it’s a good model of thinking with a notebook open and plenty of time. That is, our model is that the consequences in the environment trace back to the existence of a proof in Peano Arithmetic.

My only reservation really is that I think maybe it should be , where means provable with a consistency assumption. Otherwise, how do I know that my implementation in the environment (for a single-player game) or my opponent’s simulation of me (in the Prisoner’s Dilemma) will actually find a proof of before a proof of ? Even if is provable, the implementation or simulation could get to a contradiction first.

transhumanist_atom_understander 10 Mar 2026 17:56 UTC
6 points
−1
in reply to: James Payor’s comment on: Payorian cooperation is easy with Kripke frames
If I’m the Payorian FairBot and you’re my opponent, I simulate you under the assumption that there’s a proof I cooperate; this is .

So, to stay in the Payorian spirit, I could also simulate you under the assumption that there’s a proof I defect.

That makes me think that a Payorian PrudentBot would satisfy

Since there’s no third program involved, I can just define and to clean this up a bit:

The reason this looks right to me is because when we start drawing Kripke frames it does look like I (PrudentBot) am doing two simulations of you (my opponent), to see what you do assuming my provable cooperation and defection. Let’s assume . Then (and yes, is ),

I’ll cooperate if this can’t be satisfied, meaning we need to show contradictions in two different sets of requirements on the Kripke model. The first is the one from my post:

That’s the one where I simulate you assuming there’s a proof that I cooperate. We rule this out if, in that world, you cooperate (contradicting the in ).

The other:

This is the one where I simulate you assuming there’s a proof I defect. We rule this one out if, in that world, you defect (contradicting the in ).

So, if you cooperate in the simulation where there’s a proof I cooperate, and defect in the simulation where there’s a proof I defect, we rule these both out. That means we can’t satisfy the conditions imposed by my defection, meaning I cooperate.

That sounds prudent.

As for the Löbian PrudentBot testing its opponent on DefectBot and how that seems arbitrary. I mentioned in the FairBot vs FairBot section of my post that looks suspiciously like I’m simulating you facing CooperateBot. This was troubling to me because I shouldn’t care whether my opponent exploits CooperateBot. Similarly, in this proposed Payorian PrudentBot, looks suspiciously like simulating my opponent against DefectBot.

However, I think it’s not really? Although that brings to mind something missing from my post. In my post, when I’m simulating you, I never actually give you the full description of myself. I think a general version of this method requires an additional premise, repeating the description of myself but wrapped in a provability modality so it’s accessible in the simulations. Then my simulated opponent in doesn’t see a CooperateBot; rather, it sees whatever bot I am, but we assume there’s a proof I cooperate. Well… I hope this is different than just running my opponent against a CooperateBot, but I’m worried.

I’m not really sure how any of this will turn out, and I don’t have time to try it on paper right now or learn to use any of the automated tools. But in any case I think it’s good to record what seems like the obvious Payorian PrudentBot, at least to me, even if, as before, the solution ends up non-obvious.

Payorian cooperation is easy with Kripke frames

transhumanist_atom_understander9 Mar 2026 0:29 UTC

69 points

7 comments8 min readLW link

transhumanist_atom_understander 16 Dec 2025 2:05 UTC
1 point
0
in reply to: Cole Wyeth’s comment on: Cole Wyeth’s Shortform
Interesting. Have they shared the GPT chatlog? I don’t see it anywhere.

transhumanist_atom_understander 5 Dec 2025 18:21 UTC
2 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform

It may actually be more affordable to build some kinds of high cost-per-kg structures (e.g. datacenters, high-tech factories) in space than on land.

Than in Berkeley, you mean.

transhumanist_atom_understander 27 Nov 2025 4:15 UTC
4 points
0
on: CFAR update, and New CFAR workshops
Interesting to see what you’ve been reading, but I’m also wondering if you’ve been reading about the history of the human potential movement.

Previously you wrote:

My lead guess is that the barriers and tricky spots we ran into are somewhat similar to those that lots of efforts at self-help / human potential movement / etc. things have run into

It was a long time before I placed rationalism in what I now think of as its proper context. Or rather, two contexts: the human potential movement, and the tradition that Drexler calls “exploratory engineering” (≈L5 society culture).

And I don’t think it was just me that missed this; Scott’s Yes, We Have Noticed The Skulls seems to be looking at a completely different reference class (and even says we’re making new mistakes, the opposite of your later conclusion).

I’d be interested in whatever you have to say about this context or how you acquired it (not necessarily recently, I realize). It seems like an important subject, and if you’re restarting CFAR workshops I suppose you must have had some sources to conclude that you’re not making the familiar mistakes this time.

transhumanist_atom_understander 20 Oct 2025 21:16 UTC
3 points
0
in reply to: Nick_Tarleton’s comment on: Why you should eat meat—even if you hate factory farming
That’s interesting—if it’s broken down not into single amino acids, but a mixture of single amino acids, dipeptides, and tripeptides, that still fits with how I understand the system to work; like we’re breaking it down into pieces, but not reliably into single units, sometimes two or three. And then collagen consists of distinctive tripeptide repeats, so the tripeptides you get from collagen are a distinct mixture rather than just random 3-mers, I didn’t think of that. That these tripeptides actually do something is surprising if true, but why not.

I guess what I was thinking was that when you eat collagen, it doesn’t become your collagen. Which seems to be true: your collagen is made at the ribosome from single amino acids, not assembled from the kind of dipeptides and tripeptides discussed in the paper. So it’s not like you get collagen by eating collagen, the way you get vitamin B12 by eating vitamin B12. But if there’s some totally separate biological effect… well, I can’t rule it out.

transhumanist_atom_understander 20 Oct 2025 16:30 UTC
1 point
0
in reply to: Shibetoshi Doggomoto’s comment on: Why you should eat meat—even if you hate factory farming
Right, it depends on the vegan diet. Grains and legume protein are complementary, one deficient in branched chain amino acids, the other deficient in sulfur-containing amino acids, if I recall correctly. I think it’s an easy failure mode of a vegan diet to be all legume protein, and the gluten-free trend has made this even worse, but that’s a rant for the other day. The point here is that when it comes to dietary protein, all that matters is the amino acid composition. Every protein, including collagen, is broken down in the stomach into component amino acids. And that’s why collagen supplements are a scam, and while I am broadly sympathetic to the message of this post I think rationalists should do better.

transhumanist_atom_understander 14 Oct 2025 23:56 UTC
1 point
0
on: Why you should eat meat—even if you hate factory farming
I eat oysters but am otherwise vegan. The reason I didn’t just go with standard veganism is something like the more general arguments in this post. I had my reasons for nitpicking the details of this post; rationalists should learn some science and thereby be less wrong than the rest of the cultic milieu. But I want to comment again to focus on the positive: this post was a great reminder that I’m not a real vegan and why, and I’ve been making more of an effort to get oysters since reading it.
What links here?
- transhumanist_atom_understander's comment on Why you should eat meat—even if you hate factory farming by KatWoods (20 Oct 2025 16:30 UTC; 1 point)

transhumanist_atom_understander 13 Oct 2025 15:50 UTC
8 points
0
in reply to: Jan Betley’s comment on: The Thirteen-Circle Paradox
And if you want a “certified lower bound on their difference” you can use the Lagrange error bound for the Taylor series. The naive reasoning is that the error of the Taylor series is about the size of the first term you leave out. With the Lagrange error bound you get something like that rigorously. With well-behaved functions like sqrt and sin there’s no obstacle to proving that we’ve gotten the third digit correct (that is, that our error is <0.00001 and so can’t change the third digit). So if they differ in the third digit of our bounded numerical computation, they’re different numbers.

I haven’t actually done that carefully in this case that but the bound depends on the maximum of a higher derivative for the function. For sin that should have absolute value at most 1. For sqrt… well, we don’t want to expand around x=0, but if we expand around say x=4, these derivatives I think are not just bounded, but go to zero.

transhumanist_atom_understander 29 Sep 2025 21:33 UTC
1 point
0
on: Why you should eat meat—even if you hate factory farming
Wait, you think people need to eat collagen? Collagen is just a kind of protein, it’ll get broken down into raw amino acids in the stomach. There can be issues with a vegan diet not getting complete protein (that is, low on one or more essential amino acids) but there’s nothing special about collagen specifically.
What links here?
- transhumanist_atom_understander's comment on Why you should eat meat—even if you hate factory farming by KatWoods (14 Oct 2025 23:56 UTC; 1 point)

transhumanist_atom_understander 4 Sep 2025 16:29 UTC
1 point
0
in reply to: Gordon Seidoh Worley’s comment on: All Exponentials are Eventually S-Curves
I’m surprised at how hard it is for me to think of counterexamples.

I thought surely whale populations due to the slow generation time, but it looks like humpback whale populations have already recovered from whaling, and blue whales will get there before long.

Thinking again—in my baseball example, gravity is pulling the ball into the domain of applicability of the constant acceleration model.

Maybe what’s special about the exponential growth model is it implies escape from its own domain of applicability, in time that grows slowly (logarithmically) with the threshold.

transhumanist_atom_understander 4 Sep 2025 2:22 UTC
1 point
0
on: Löb’s Lemma: an easier approach to Löb’s Theorem
I remember this by analogy to Curry’s paradox.

Where the sentence from Curry’s paradox says “If this statement is true, then $p$ ”, $Ψ$ says “if this statement is provable, then $p$ ”, that is, $□ Ψ \to p$ .

In Curry’s paradox, if the sentence is true, that would indeed imply that $p$ is true. And with $Ψ$ , the situation is analogous, but with truth replaced by provability: if $Ψ$ is provable, then $p$ is provable. That is, $□ Ψ \to □ p$ .

But, unlike in Curry’s paradox, this is not what $Ψ$ itself says! Replacing truth with provability has attenuated the sentence, destroyed its ability to cause paradox.

If only $□ p \to p$ , then we would have our paradox back… and that’s Löb’s theorem.

This is all about $□ Ψ \to □ p$ , just about one direction of the biimplication, whereas the post proves not just that but the other direction. It seems that only this forward direction is used in the proof at the end of the post though.

transhumanist_atom_understander 4 Sep 2025 1:49 UTC
4 points
2
on: All Exponentials are Eventually S-Curves
You say “if we are to accurately model the world”...

If I am modelling the path of a baseball, and I write “F = mg”, would you “correct” me that it’s actually inverse square, that the Earth’s gravitation cannot stay at this strength to arbitrary heights? If you did, I would remind you that we are talking about a baseball game, and not shooting it into orbit—or conclude that you had an agenda other than determining where the ball lands.

What if I’m sampling from a population, and you catch me multiplying probabilities together, as if my draws are independent, as if the population is infinite? Yes there is an end to the population, but as long as it’s far away, the dependence induced by sampling without replacement is negligible.

Well, that’s the question, whether to include an effect in the model or whether it’s negligible. An effect like finite population size, diminishing gravity, or the “crowding” effects that turn an exponential growth model logistic.

And the question cannot be escaped just by noting the effect is important eventually.

transhumanist_atom_understander 3 Sep 2025 3:38 UTC
3 points
0
on: Yudkowsky on “Don’t use p(doom)”
Eliezer in 2008, in When (Not) To Use Probabilities, wrote:

To be specific, I would advise, in most cases, against using non-numerical procedures to create what appear to be numerical probabilities. Numbers should come from numbers.

transhumanist_atom_understander 2 Sep 2025 13:50 UTC
1 point
0
in reply to: dr_s’s comment on: A quantum equivalent to Bayes’ rule
Yeah… well, I thought of the $Z$ because it sounds like we’re getting the probabilities of $Y$ from some experiment. So $Z = z$ is the results of the experiment, which in this case is a vector of frequencies. When I put it like that, it sounds like it’s is just a rhetorical device for saying that we have given probabilities of $Y$ .

But I still seem to need $Z$ for my dictionary. I have $γ (x) = P [X = x]$ . What is $γ^{'} (x)$ ? It is some kind of updated probability of $X = x$ , right? Like we went from one probability to the other by doing an experiment. If I didn’t write $γ^{'} (x) = P [X | Z = z]$ , I’d need something like $γ (x) = P_{1} [X = x]$ and $γ^{'} (x) = P_{2} [X = x]$ .

Reading again, it seems like this is exactly Jeffrey conditionalization. So whether you include some extra variable just depends on what you think of Jeffrey conditionalization.

I feel like I’m missing something, though, about what this experiment is and means. For example, I’m not totally clear on whether we have one state $X$ , and a collection of replicates of state $Y$ ; or is it a collection of replicates of $(X, Y)$ pairs?

Looking at the paper, I see the connection to Jeffrey conditionalization is made explicitly. And it mentions Pearl’s “virtual evidence method”; is this what he calls introducing this $Z$ ? But no clarity on exactly what this experiment is. It just says:

But how should the above be generalized to the situation where the new information does not come in the form of a definite value $y_{0}$ for $Y$ , but as “soft evidence,” i.e., a probability distribution $τ (y)$ ?”

By the way, regarding your coin toss example, I can at least say how this is handled in Bayesian statistics. There are separate random variables for each coin toss. $Y_{1}$ is the first, $Y_{2}$ is the second, etc. If you have $n$ coin tosses, then your sample is a vector $\to Y$ containing $Y_{1}$ to $Y_{n}$ . Then the posterior probability is $P [loaded | \to Y = \to y]$ . This will be covered in any Bayesian statistics textbook as “the Bernoulli model”. My class used Hoff’s book, which provides a quick start.

I guess this example suggests a single unknown $X$ (whether the coin is loaded or not) and replicates of $Y$ .

transhumanist_atom_understander 2 Sep 2025 2:05 UTC
8 points
0
on: A quantum equivalent to Bayes’ rule
The “Classical derivation” made more sense to me after translating it to standard probability notation, so I’m commenting to share the “dictionary” I made for it, as well as the unexpected extra assumption I had to make.

The obvious:

$γ (x) = P [X = x]$

$φ (y | x) = P [Y = y | X = x]$

$^φ (x | y) = P [X = x | Y = y]$

It got tricky with $τ$ . Instead of observing $Y = y$ , we observe something else that gives us a probability distribution over $Y$ . I considered this “something else” to be the value of some other unknown: $Z = z$ . The probability distribution over $y$ is a conditional distribution:

$τ (y) = P [Y = y | Z = z]$

Hate to have $z$ on only one side like that… maybe I should have called it $τ_{z}$ … but I’ll leave it as is.

Then,

$γ^{'} (x) = \sum j P [X = x | Y = y_{j}] P [Y = y_{j} | Z = z]$

Not quite the right formula for a simple interpretation of $γ^{'}$ … if only

$P [X = x | Y = y_{j}] = P [X = x | Y = y_{j}, Z = z]$

This is conditional independence, which could be represented with this Bayes net:

$Z \to Y \to X$

Then, we have

$γ^{'} (x) = P [X = x | Z = z]$

That completes the dictionary.

So to do what feels like ordinary probability theory, I had to introduce this extra unknown $Z$ so that we have something to observe, and then to assume that $Z$ only provides information about $Y$ (and indirectly about $X$ , through $Y$ ).

The way you described $τ$ as some probability distribution resulting from an observation, but not a conditional distribution, is in philosophy called Jeffrey conditionalization. The Stanford Encyclopedia of Philosophy gives this example:

A gambler is very confident that a certain racehorse, called Mudrunner, performs exceptionally well on muddy courses. A look at the extremely cloudy sky has an immediate effect on this gambler’s opinion: an increase in her credence in the proposition (muddy) that the course will be muddy—an increase without reaching certainty. Then this gambler raises her credence in the hypothesis (win) that Mudrunner will win the race, but nothing becomes fully certain. (Jeffrey 1965 [1983: sec. 11.3])

The idea is, we go from one probability distribution over ${muddy, \neg muddy}$ to another, without becoming certain of anything. My introduction of $Z$ corresponds to introducing an unknown representing the status of the sky. I would say we are conditioning on $Z = cloudy$ .

I recalled vaguely that Jaynes discussed Jeffrey conditionalization in Probability Theory, and criticized it for holding only in a special case. I took a look, and sure enough, it’s in section 5.6, and he’s pointing out exactly what I did, right down to the arrows, though he calls it a “logic flow diagram” rather than identifying it as a Pearl-style Bayes net.

transhumanist_atom_understander 25 Apr 2025 13:18 UTC
3 points
0
on: Unknown Probabilities
The last formula in this post, the conservation of expected evidence, had a mistake which I’ve only just now fixed. Since I guess it’s not obvious even to me, I’ll put a reminder for myself here, which may not be useful to others. Really I’m just “translating” from the “law of iterated expectations” I learned in my stats theory class, which was:

$E [E [X | Y]] = E [X]$

This is using a notation which is pretty standard for defining conditional expectations. To define it you can first consider the expected value given a particular value of the random variable $Y$ . Think of that as a function of that particular value: $f (y) = E [X | Y = y]$ Then we define conditional expectation as a random variable, obtained from plugging in the random value of $Y$ : $E [X | Y] = f (Y)$ The problem with this notation is it gets confusing which capital letters are random variables and which are propositions, so I’ve bolded random variables. But it makes it very easy to state the law of iterated expectations.

The law of iterated expectations also holds when “relativized”. That is, $E [E [X | Y] | B] = E [X | B]$ where $B$ is an event. If we wanted to stick to just putting random variables behind the conditional bar we could have used the indicator function of that event.

And this translates to the statement in my post. $X$ is an indicator for the event $H$ , which makes a conditional expectation of it a conditional probability of $H$ . So $E [X | Y]$ is $Θ$ . Our proposition $B$ is the background information $B$ , I used the same symbol there. And the right hand side is another expectation of an indicator and therefore also a probability.

I really didn’t want to define this notation in the post itself, but it’s how I’m trained to think of this stuff, so for my own confidence in the final formula I had to write it out this way.

transhumanist_atom_understander 7 Apr 2025 1:52 UTC
1 point
0
on: How Gay is the Vatican?
It would be nice if you had the sexes of the siblings, since it’s supposedly only the older brothers that count, though I don’t really expect that to change anything.

Really the important thing is just to separate birth order from family size. Usually the way I think of this is, we can look at number of older brothers, with a given number of older siblings. I like this setup because it looks like a randomized trial. I have two older siblings, so do you, meiosis randomizes their sexes.

But I guess with the data you have you can look at birth order with a given family size, so we don’t have to worry about the effect of a larger or smaller family. I… don’t think this is what you did? Did I misunderstand something? It seems like if cardinals come from smaller families, that would show up as lower birth orders.

With 9 million people I’d just split it into categories by number of siblings, with your data I’m not sure.

transhumanist_atom_understander 18 Feb 2025 20:53 UTC
3 points
0
on: AI #103: Show Me the Money
After reading this post, I handed over $200 for a month of ChatGPT pro and I don’t think I can go back. o1-pro and Deep Research are next level. o1-pro often understands my code or what I’m asking about without a whole conversation of clarifying, whereas other models it’s more work than it’s worth to get them focused on the real issue rather than something superficially similar. And then I can use “Deep Research” to get links to webpages relevant to what I’m working on. It’s like… smart Google, basically. Google that knows what I’m looking for. I never would have known this existed if I didn’t hand over the $200.

transhumanist_atom_understander

Payo­rian co­op­er­a­tion is easy with Kripke frames

Payorian cooperation is easy with Kripke frames