joseph_c

Karma: 271

joseph_c 9 Mar 2026 17:28 UTC
7 points
−2
on: Prologue to Terrified Comments on Claude’s Constitution
(Under the hood of your chatbot conversation, the context window contains both the “user” and “assistant” turns. We train the model to fill in the assistant’s part and emit a “stop” token. The chat interface stops sampling at the stop token to let you type the next “user” message, rather than continuing to sample the model’s predictions of what the “user” in the dialogue would say next. It’s more like the model being specialized to write the “AI assistant” character in such dialogues, rather than the model speaking “as itself”.)
Moreover, the chatbot is typically not even trained to predict the user dialogue; in training there is usually a mask which zeroes out any gradients coming from those tokens.

joseph_c 9 Mar 2026 6:11 UTC
9 points
0
on: Payorian cooperation is easy with Kripke frames
I think this can also be used to make a smarter version of PrudentBot. If I remember right, PrudentBot is defined as that is, it cooperates if and only if it can prove that (1) its opponent cooperates with it, and (2) there is some world its opponent defects against DefectBot.

To me, the part about choosing DefectBot in particular to compare with seems a little hacked. What you’re really interested in is seeing if it will defect against you, even if you defect against them. Thus, I propose SharkBot, which I label . SharkBot is defined as It’s like your version of FairBot, but adds another condition, that it’s possible that its opponent’s cooperation leads to the possibility of mutual cooperation. Equivalently, it’s possible that provable defection will lead to the opponent defecting. Thus, SharkBot exploits opponents that don’t change their behavior even when they should know better.

I’m pretty sure this cooperates with itself and FairBot, but defects against CooperateBot and a larger share of stupid agents than PrudentBot does.

joseph_c 7 Mar 2026 6:23 UTC
3 points
0
on: D&D.Sci Release Day: Topple the Tower!
I trained a causal transformer model, and then did a minimax on winning every encounter to get the following:
Objective 1
Be a mage and go RLLRLLRR, which takes you along the route:
1. START
2. Tome of Knowledge
3. Cultist
4. Jaw Worm
5. Campfire
6. Sentries
7. Chosen
8. Campfire
9. The Collector
My model was about 91% sure that you would clear the tower along this route.
Objective 2
Again be a mage. Go LLRLLRRR, which takes you along the route:
1. START
2. Acid Slime
3. Cultist
4. Jaw Worm
5. Sentries
6. Gremlin Nob
7. Cloak of Protection
8. Vanishing Powder
9. The Champion
My model was much less sure that this was a tower-clearing route, only giving 59% confidence.

joseph_c 6 Mar 2026 6:03 UTC
2 points
0
in reply to: G Wood’s comment on: Andrew_Critch’s Shortform
I would call that ethics, not morality. I personally distinguish ethics from morality in that ethics is how a society works together despite people wanting conflicting things, while morality is about achieving the most good (however you define “good”). I don’t think this is an official distinction, but I do think it’s useful to distinguish the two concepts.

joseph_c 7 Feb 2026 17:38 UTC
1 point
0
on: Speedrunning a Mech Interp Research Setup (Remote GPU, Torch, TransformerLens, Cuda, SSH, VS Code)
Overall, the setup looks good, except for one part: I wouldn’t give the remote machine complete access to your GitHub account.

I think maybe a better solution would be to set up a bare repository on the Nebius VM. This serves a second remote repository you can push to from your machine. After you push to the bare repository, you can clone it while only on the Nebius VM to another location on the VM. You can make changes in this second location, then push to the bare repository, which you can pull to your machine, and ultimately push to GitHub.

It is a bit more complicated than just directly pushing to GitHub, though. Maybe someone else knows a simpler solution.

joseph_c 15 Jan 2026 22:21 UTC
1 point
0
on: I Made a Judgment Calibration Game for Beginners (Calibrate)
Thanks for making the game! I played it, and was a little frustrated that not all of your questions had the correct answers. For example, I recall one of the questions as reading, “How many typical transport protocols are on the internet?” which is not quite the same as what was answered, “How many internet transport protocols are commonly used?” You see, any protocol is “typical” for its use cases, even if it is uncommon. It reminded me somewhat of taking the SAT, where the test writers would replace a word with a more common word which doesn’t actually quite mean what they intended. As the commenter below me pointed out, what he does with such questions is go meta—and I kind of like that perspective—but it was nevertheless a little frustrating. Overall though, I liked the game. Thanks for making it.

Note: I edited this the day after I wrote it because I found it a little too antagonistic, and I don’t want to be antagonistic. I really did like the game, and appreciate that you made it.

joseph_c 31 Dec 2025 2:36 UTC
2 points
0
on: The origin of rot
You say that “Perhaps for several millennia, [rot didn’t exist]” and then provide a date about 3300 years ago about when rot might potentially have began. You also say that several millennia is a “very, very long time”. A millennium is 1,000 years. Do you think that life has only existed on Earth for a few thousand years?

joseph_c 14 Dec 2025 21:37 UTC
2 points
0
on: The Axiom of Choice is Not Controversial
Now, where did the weirdness come from here. Well, to me it seems clear that really it came from the fact that the reals can be built out of a bunch of shifted rational numbers, right?

I think the weirdness comes from trying to assign a real number measure, instead of allowing infinitesimals. I’ve never understood why infinite sets are readily accepted, but infinitesimal/infinite measures are not.

EDIT: To explain my reasoning more, suppose you were Pythagoras and your student came to you and drew a geometric diagram with lengths not in a ratio of whole numbers. You have two options here:
1. You can declare that not all lengths are commensurate. Not every ratio of lengths results in a number.
2. You can extend your number system.
Finding the right extension is not an easy problem. Should we extend the numbers to allow square roots (including nesting), but nothing else? This suffices for geometry. But it’s actually more useful to use something like a Cauchy sequence completion: Let any sequence of rational numbers that gets closer and closer together “converge” to a real number. Historically, extending your system of numbers has been what has worked.

When we come across an “immeasurable” set, this to me feels like the same kind of problem. Perhaps we don’t yet have a general consensus on what the “right” extension is to infinitesimals/infinities. However, there clearly are some sets with infinitesimal measure, like the set you constructed. We should figure out a way to give that set infinitesimal measure, not just call it immeasurable.

joseph_c 9 Dec 2025 1:32 UTC
2 points
1
in reply to: GenericModel’s comment on: Gödel’s Ontological Proof
To try to clarify why it felt self-referential. I think there’s a self-reference regardless of whether you talk about classes or not, but it’s more obvious if you talk about classes.

I think the correct mathematical term is “Impredicativity”, not “self-referential”, but I’m no logician.

joseph_c 9 Dec 2025 1:26 UTC
1 point
0
in reply to: GenericModel’s comment on: Gödel’s Ontological Proof
If I were to try to translate this into classes instead of properties, it would look like, “The class of perfect properties contains the property of being every perfect property in this class”. That seems self-referential to me.

joseph_c 9 Dec 2025 1:19 UTC
1 point
0
in reply to: GenericModel’s comment on: Gödel’s Ontological Proof
An object is defined to be G if it has every perfect property, and then G is assumed (by axiom) to be a perfect property, hence being G requires being G. Now that I think about it a bit more, though, this seems more like a greatest-lower-bound situation than a Russell’s paradox situation.

joseph_c 9 Dec 2025 1:15 UTC
1 point
0
in reply to: JBlack’s comment on: Gödel’s Ontological Proof
I didn’t know that coherent logic was actually a term logicians used! I’m not a logician myself—I’m a programmer. Thanks for letting me know!

joseph_c 9 Dec 2025 1:11 UTC
1 point
0
in reply to: GenericModel’s comment on: Gödel’s Ontological Proof
The reason I was saying it looked like a type error was because of the self reference. I’m extremely wary of self-referential definitions because you can quickly run into problems like Russell’s paradox. It seems like sometimes it’s okay to have self-referential definitions (like the greatest lower bound), but I’m not confident that Axiom 4 actually avoids those problems.

joseph_c 9 Dec 2025 0:19 UTC
1 point
0
in reply to: GenericModel’s comment on: Gödel’s Ontological Proof
Could you please link me to that formalization?

joseph_c 9 Dec 2025 0:16 UTC
18 points
11
on: Gödel’s Ontological Proof
I think if you replace the word “God” with “top” and “perfect” with “highest”, it would be much more clear what the proof actually implies about the real world: Very little.

Definition 0: Say that ψ is higher than φ if □ ∀x φ(x) → ψ(x).

Axiom 1: A property higher than a highest property is also highest.
Axiom 2: The negation of a highest property is not highest.
Axiom 3: If a property is highest, then in some world there exists an object with that property.
Definition 1: An object is top if it is every highest property.
Axiom 4: The property “top” is highest. Note: This looks like a a type error to me.
Definition 2: A property φ is highest-generating for an object x if it is true of x, and every highest property of x follows from φ in every possible world.
Definition 3: Superior object. (I changed your name for this as well.) An object x is “superior” if for every highest-generating property of x, every world has an object with that property (not necessarily x).
Axiom 5: A highest property is highest in every world.
Axiom 6: The property “superior” is highest. Probably also a type error.

Sheesh, there sure are a lot of axioms. At this point, I’m not even sure we have a ~~coherent logic~~ sane logic anymore! Especially with those potential type errors.

The end result we prove is that in every world, there exists a top object.

When you divorce the “God” and “perfect” language from the axioms, we don’t really get anything that implies much about the real world or Christianity, do we?

joseph_c 9 Dec 2025 0:15 UTC
1 point
0
on: Gödel’s Ontological Proof
Axioms 4 and 6 look like type errors to me. Could you please explain how they are not?

joseph_c 30 Nov 2025 2:57 UTC
3 points
0
in reply to: lsusr’s comment on: Traditional Food
Thank you for the more detailed recipe! I’m not going to switch to only eating this meal (don’t worry about my vitamin intake!). It’s just something I plan to add to my rotation because it’s easy to make and healthy.

joseph_c 30 Nov 2025 0:42 UTC
3 points
0
on: Traditional Food
Is the bean dish supposed to be more like soup or chili? I tried it out, and it was rather soupy. Maybe I added too much broth. What portions did you use for the canned tomatoes, beans, and broth?

joseph_c 23 Nov 2025 23:59 UTC
3 points
0
in reply to: Kaj_Sotala’s comment on: The Enemy Gets The Last Hit
What’s going on is something like adverse selection in an auction. In reality, chess is a solvable game, so that in perfect play win/loss/draw probabilities are all 0 or 1 for every board state. However, you don’t know what these probabilities are, so you use a model. A naive player might just want to play the action which takes them to the board state they model as having the highest probability of winning. However, this fails to take into account the fact that one’s model can be wrong, and so one will tend to pick actions which lead to board states which one mispredicts as being better than they actually are.

If one can accurately model how inaccurate one’s model tends to be at certain board states, then one can do fine without ending on one’s opponent’s move by discounting board states one models as modelling poorly. This is nontrivial for humans to do correctly, however.

Instead, a heuristic one can use is to just let one’s opponent make the last move in one’s search tree. This gives one a lower bound on how good each board state is (an optimistic guess for one’s opponent is a pessimistic guess for one’s self), so one’s choices of board states will not be catastrophically biased. By catastrophically biased, I mean that in chess it’s much easier to wreck your game than to make a stunningly clever move which causes you to win, so that being too optimistic is much worse than being too pessimistic.

joseph_c 15 Nov 2025 7:26 UTC
1 point
0
on: Lambda Calculus Prior
This is not at all my specialty, but might the problem go away if instead of directly passing in the next term into your lambda calculus machine, you first quote it? By “quoting”, I mean converting it to a representation that the lambda calculus machine can inspect, like the QUOTE operator in Lisp.

joseph_c

Objective 1

Objective 2