Bunthut

• Bunthut 3 Sep 2026 23:07 UTC

  10 points

  0

  on: Cat-Bel­ling Problems

  And Irving said it was a good question and he might need to get back to me on that.

  I didn’t see the discussion, but it seems odd to me that a debate-aligner would grant the premise here—isn’t the whole idea of debate that informative responses do exist in very specific parts of the dialogue-tree? I don’t think the empirical case is as damning as you make it sound:

  Mathematics mostly operates by the Scholastic Method. It has parts with easily demonstrable implications, but also other parts which are quite far from that, and it’s mostly fine.

  And people do fail to recognise the right answer, justified with the right reasons, at times—but that is also not always the best argument (to them) for that right answer. You might have needed to present it differently, or attack the reasons people don’t accept the arguments first, or...etc. We don’t demand you do this stuff to get Science Genius Credits afterwards, but that also means you can find examples of people failing to appreciate the Perfect Science Genius, without that necessarily bearing on judging debates of Science-and-Pedagogy Geniuses.

  I still don’t really expect this to work, but more so on priors.

  What links here?

  • StanislavKrym's comment on Cat-Bel­ling Problems by Eliezer Yudkowsky (4 Sep 2026 0:53 UTC; -1 points)

Geo­met­ric Ra­tion­al­ity acts lin­early in ad­di­tive scenarios

• Bunthut 27 Jul 2026 21:43 UTC

  3 points

  0

  on: Informality

  What I do want to claim, perhaps, is that you don’t really understand something unless you can translate it into a transparent-context description.

  Versions of this have appeared in philosophy before. The classical group of objections is over-identification. Naively, you would have to say that inertial mass and gravitational mass mean the same, as do “p is prime” and ” is an integral domain”. (These sorts of problems motivated David Lewis’ modal realism, to extensionalize modal logic—but the mathematical/​neccessary case would need separate handling)

  But this is the tame version of the problem, because it’s already treating individual predicates in an otherwise extensional language. If you want to translate opaque messes into into extensional expressions, you’re operating on the level of sentences. The extension of a sentence is typically its truth value, which would mean there are only two of them: true and false. This leads into problems like Quines indeterminacy of translation. Modality doesn’t help: Expressions become more determined the more their truth-value, or belief in them, is observed to vary.

  And while it doesn’t relate to the formal apparatus here, I’d also link my older and somewhat more galactic answer to similar problems.

• Bunthut 22 Mar 2026 9:45 UTC

  1 point

  0

  on: On The In­de­pen­dence Axiom

  I think you missed this prior discussion of non-independent decision making.

  You might also be interested in my post relating ergodicity and utility. This proposal actually is independent, which works because it has an infinite set of possible outcomes (which also violates vNM).

  Im generally open to nonindependence, but I don’t like resolute choice, because it has a special “time of strategy fixing” as an additional fact. This doesn’t play so well once we leave the world of toy model worlds that begin with their game. Sophisticated choice is even worse, because it assumes a definite end. The specific nonindependent things people want to do often don’t have this problem, so I think it’s worth having some kind of theoretical intermediate here.

• Bunthut 4 Nov 2025 14:29 UTC

  5 points

  1

  on: Lack of So­cial Grace is a Lack of Skill

  I was not, however, socially adroit, so what I said was “why do you care about something boring like horses?”

  She complained to the teachers about me being rude, which in hindsight was entirely fair.

  How normal would you rate the reaction? While you were being rude, I find it strange to complain to the teachers about just that. In my experience, not many people would do that past elementary school.

• Bunthut 3 Nov 2025 17:33 UTC

  4 points

  0

  on: Are ap­ples made of cells?

  I can’t get spoiler tag to work so here it is

  The seeds at least need to contain cells, because new plants with cells grow out of them. It’s not a single cell because it’s solid on the inside—I don’t know any living cell that is, and it would be very difficult for substance transport. It also shows cell damage: a cut apple browns on the air, and the sharper the knife the less it does it. I don’t think a case like bones can even be excluded without a microscope.

• Bunthut 10 Sep 2025 23:32 UTC

  3 points

  1

  in reply to: lsusr’s comment on: En­light­en­ment AMA

  You mean discursive thinking like that voice in your head? Yeah, we do that in shikantaza. After you turn it off what’s left is the sensation of the breath, the sound of wind chimes outside and, if your eyes are open, the image of the wall in front of you. It takes me about 30 minutes of deliberate intention in a peaceful room to get into this state.

  I think getting rid of the voice in my head temporarily is very easy. Trivially, by replacing it with a loud repeating dum dum dum sound in my head, though Im not sure that counts. But I’ve also just done 30s of no auditory imagination while looking around in my non-blank room, and it took maybe 5s to get there. Is this one of those buddhist terms of art where it actually means way more than a layperson would reasonably think it does?

• Bunthut 10 Sep 2025 23:06 UTC

  1 point

  0

  in reply to: Kaj_Sotala’s comment on: En­light­en­ment AMA

  According to predictive coding, believing you’ll take an action just is how you take it, and believing you’ll achieve a goal just is how you intend it. This would mean if you desire more than you can achieve, you experience prediction error, but if you desire less than you can achieve, you just underachieve with no psychological warning.

• Bunthut 25 Jun 2025 9:35 UTC

  1 point

  0

  on: Futarchy’s fun­da­men­tal flaw

  Suppose b is the true bias of the coin (which the supercomputer will compute). Then your expected return in this game is

  𝔼[max(b, 0.50)] = 0.50 + 𝔼[max(b-0.50, 0)]

  No. That formula would imply that, if the coin is 30% for sure and you buy it for 0.3, you make 0.2 in expectation, which you don’t, you make 0 regardless of what price you buy at.

  Note that this kind of problem has also shown up in decision theory more generally. This is a good place to start. In particular, it seems like your problem can be fixed with epsilon exploration (if it doesn’t do so automatically, as per Soares), both the EDT and CDT variant should work.

• Bunthut 20 Jun 2025 20:50 UTC

  1 point

  0

  in reply to: XelaP’s comment on: In­tel­li­gence Is Not Magic, But Your Thresh­old For “Magic” Is Pretty Low

  A simple version of this is done for panoramic photos. If he looked at the city from a consistent direction throughout the flight, that’s all that’s needed. If the direction varied, it can’t have varied a lot—he had to at least see the sides of the building he was drawing, if maybe from a different angle, and not all the buildings would have been parallel. That kind of rotation seems doable with current image transformers (and that’s only neccesary if the drawing has accurate angles even over long distances).

  In any case, I don’t think it matters to my argument if current ML can do it. All the parts that might be difficult for the computer are doable even for normal humans, and therefore not magical. The only thing that’s added to the normal human skill here is perfect memory, which we know is easy for computers and have known for a long time.

• Bunthut 20 Jun 2025 20:34 UTC

  1 point

  0

  in reply to: Steven Byrnes’s comment on: Are su­per­hu­man sa­vants real?

  To clarify the question: I agree that there is variation in talent and that some very talented people can do things most could never. My question is, if you look at the distribution of talent among normal people, and then check how many standard deviations out our savant candidate is, then what’s the chance at least one person with that talent would exist? Basically, is this just the normal right tail that’s expected from additive genetic reshuffling, or an “X-man”.

[Question] Are su­per­hu­man sa­vants real?

• Bunthut 16 Jun 2025 17:33 UTC

  1 point

  −1

  on: In­tel­li­gence Is Not Magic, But Your Thresh­old For “Magic” Is Pretty Low

  Example 3: Stephen Wiltshire. He made a nineteen-foot-long drawing of New York City after flying on a helicopter for 20 minutes, and he got the number of windows and floors of all the buildings correct.

  I think ~everyone understands that computers can do this. The “magical” part is doing it with a human brain, not doing it at all. Similarly, blindfolded chess is not more difficult than normal chess for computers. That may take a little knowledge to see. And “doing it faster” is again clear. So the threshold for magic you describe is not the one even the most naive use for AI.

• Bunthut 28 Apr 2025 14:47 UTC

  1 point

  0

  on: Why Have Sen­tence Lengths De­creased?

  Sentence lengths have declined.

  Data: I looked for similar data on sentence lengths in german, and the first result I found covering a similar timeframe was wikipedia referencing Kurt Möslein: Einige Entwicklungstendenzen in der Syntax der wissenschaftlich-technischen Literatur seit dem Ende des 18. Jahrhunderts. (1974), which does not find the same trend:

  Year	wps
  1770	24,50
  1800	25,54
  1850	32,00
  1900	23,58
  1920	22,72
  1940	19,60
  1960	19,90

  This data on scientific writing starts lower than any of your english examples from that time, and increases initially, but arrives in the same place (insofar as wps are comparably across languages, which I think is fine for english and german).

• Bunthut 1 Apr 2025 15:12 UTC

  1 point

  0

  in reply to: habryka’s comment on: LessWrong has been ac­quired by EA

  6 picolightcones as well, don’t think that changed.

• Bunthut 1 Apr 2025 14:54 UTC

  1 point

  0

  on: LessWrong has been ac­quired by EA

  Before logging in I had 200 LW-Bux, and 3 virtues. Now I have 50 LW and 8 virtues, and I didn’t do anything. Whats that? Is there any explanation of how this stuff works?

• Bunthut 1 Apr 2025 14:45 UTC

  3 points

  0

  in reply to: Kaj_Sotala’s comment on: Ge­netic fit­ness is a mea­sure of se­lec­tion strength, not the se­lec­tion target

  I think your disagreement can be made clear with more formalism. First, the point for your opponents:

  When the animals are in a cold place, they are selected for a long fur coat, and also for IGF, (and other things as well). To some extent, these are just different ways of describing the same process. Now, if they move to a warmer place, they are now selected for a shorter fur instead, and they are still selected for IGF. And there’s also a more concrete correspondence to this: they have also been selected for “IF cold long fur, ELSE short fur” the entire time. Notice especially that there are animals actually implementing this dependent property—it can be evolved just fine, in the same way as the simple properties. And in fact, you could “unroll” the concept of IGF into a humongous environment-dependent strategy, which would then always be selected for, because all the environment-dependence is already baked in.

  Now on the other hand, if you train an AI first on one thing, and then on another, wouldn’t we expect it to get worse at the first again? Indeed, we would also expect a species living in the cold for very long to lose those adaptations relevant to the heat. The reason for this in both cases are, broadly speaking, limits and penalties to complexity. I’m not sure very many people would have bought the argument in the previous paragraph—we all know unused genetic code decays over time. But in the behavioral/​cognitive version with intentionally maximizing IGF that makes it easy to ignore the problems, we’re not used to remembering the physical correlates of thinking. Of course, a dragonfly couldn’t explicitly maximize IGF, because its brain is to small to even understand what that is, and developing that brain has demands for space and energy incompatible with the general dragonfly life strategy. The costs of cognition are also part of the demands of fitness, and the dragonfly is more fit the way it is, and similarly I think a human explicitly maximizing IGF would have done worse for most of our evolution^[1] because the odds you get something wrong are just too high with current expenditure on cognition, better to hardcode some right answers..

  I don’t share your optimistic conclusion however. Because the part about selecting for multiple things simultanuously, that’s true. You are always selecting for everything thats locally extensionally equivalent to the intended selection criteria. There is not a move you could have done in evolutions place, to actually select for IGF instead of [various particular things], this already is what happens when you select for IGF, because it’s the complexity, rather than different intent, that lead to the different result^[2]. Similarly, reinforcement learning for human values will result is whatever is the simplest^[3] way to match human values on the training data.

  1. ^

     and even today, still might if sperm donations et al weren’t possible

  2. ^

     I don’t think you’ve tried to come up with what that different move might look like for evolution, but it’s strongly implied they exist for both it and the AI situation.

  3. ^

     in the sense of that architecture

• Bunthut 13 Mar 2025 8:53 UTC

  14 points

  −1

  in reply to: the gears to ascension’s comment on: Tro­jan Sky

  for AIs, more robust adversarial examples—especially ones that work on AIs trained on different datasets—do seem to look more “reasonable” to humans.

  Then I would expect they are also more objectively similar. In any case that finding is strong evidence against manipulative adversarial examples for humans—your argument is basically “there’s just this huge mess of neurons, surely somewhere in there is a way”, but if the same adversarial examples work on minds with very different architectures, then that’s clearly not why they exist. Instead, they have to be explained by some higher-level cognitive factors shared by ~anyone who gets good at interpreting a wide range of visual data.

  The really obvious adversarial example of this kind in human is like, cults, or so

  Cults use much stronger means than is implied by adversarial examples. For one, they can react to and reinforce your behaviour—is a screen with text promising you things for doing what it wants, with escalating impact and building a track record an adversarial example? No. Its potentially worrying, but not really distinct from generic powerseeking problems. The cult also controls a much larger fraction of your total sensory input over an extended time. Cult members spreading the cult also use tactics that require very little precision—there isn’t information transmitted to them on how to do this, beyond simple verbal instructions. Even if there are more precision-needing ways of manipulating individuals, its another thing entirely to manipulate them into repeating those high precision strategies that they couldn’t themselves execute correctly on purpose.

  if you’re not personally familiar with hypnosis

  I think I am a little bit. I don’t think that means what you think it does. Listening-to-action still requires comprehension of the commands, which is much lower bandwidth than vision, and its a structure thats specifically there to be controllable by others, so it’s not an indication that we are controllable to others in other bizzare ways. And you are deliberately not being so critical—you haven’t, actually, been circumvented, and there isn’t really a path to escalating power—just the fact youre willing to oey someone in a specific context. Hypnosis also ends on its own—the brain naturally tends back towards baseline, implanting a mechanism that keeps itself active indefinitely is high-precision.

• Bunthut 13 Mar 2025 0:25 UTC

  4 points

  −4

  in reply to: the gears to ascension’s comment on: Tro­jan Sky

  Ok, thats mostly what I’ve heard before. I’m skeptical because:

  1. If something like classical adversarial examples existed for humans, it likely wouldn’t have the same effects on different people, or even just viewed from different angles, or maybe even in a different mood.

  2. No known adversarial examples of the kind you describe for humans. We could tell if we had found them because we have metrics of “looking similar” which are not based on our intuitive sense of similarity, like pixelwise differences and convolutions. All examples of “easily confused” images I’ve seen were objectively similar to what theyre confused for.

  3. Somewhat similar to what Grayson Chao said, it seems that the influence of vision on behaviour goes through a layer of “it looks like X”, which is much lower bandwidth than vision in total. Ads have qualitatively similar effects to what seeing their content actually happen in person would.

  4. If adversarial examples exist, that doesn’t mean they exist for making you do anything of the manipulators choosing. Humans are, in principle, at least as programmable as a computer, but that also means there are vastly more courses of action than possible vision inputs. In practice, propably not a lot of high-cognitive-function-processing could be commandeered by adversarial inputs, and behaviours complex enough to glitch others couldn’t be implemented.

• Bunthut 12 Mar 2025 10:53 UTC

  2 points

  0

  in reply to: the gears to ascension’s comment on: Tro­jan Sky

  I just thought through the causal graphs involved, there’s probably enough bandwidth through vision into reliably redundant behavior to do this

  Elaborate.