I got a full genome sequence from sequencing.com. (I didn’t do much research before choosing, so I don’t have reason to believe they’re unusually good but they got the standard job done). Then I checked the oxytocin receptor sequence myself.
johnswentworth
Karma: 65,428
I stopped mentoring in training programs years ago. I think 2023 was the last time.
Yeah, one big problem with the original comment is that I failed to convey that the signal involved is (a) weak, and (b) not the most important thing when assessing someone’s AI alignment qualifications. (The most important thing IMO is the extent to which they have gearsy mental models of how minds work.) Even before I got more information, I don’t think I could come up with any case where attending RMN would have been a deciding factor for my judgement of someone’s skill, let alone the judgement of anyone who actually makes hiring or funding decisions, even if I would consider it positive evidence along one necessary axis. So that wasn’t really an update, it was just poor communication in the original comment.
On Kink & Status in the Rationalist Community
Imagine a video game with NPCs who will pick up whatever items they find and attack you with them. So if you throw a heal wand at an “enemy”, the “enemy” will pick up the heal wand and follow you around angrily healing you.
That is how I see this situation.
Correct. And I’m hoping you have one of the major pieces.
Good question. This is getting too far from the object level for me to think well, so let’s start with how I would think about the situation with Caroline. If I were in Austin’s shoes right now, I would (a) explicitly publicly declare that I was betting my own social capital on her, and (b) make sure to double-check what Caroline’s doing from time to time in order to protect my own social capital. And of course if she did go pull a fraud again under my watch, I’d take responsibility for that.
On reflection, that answer seems straightforwardly correct, but I haven’t thought about the right ways to generalize it.
I think this take is dangerously mistaken, in a similar way to standard “but sometimes the utility-maximizing move is to believe something false” takes.
According to me, the main problem with “sometimes the utility-maximizing move is to believe something false” is that, in practice, we live in a very hostile memetic environment. The memes around us are under intense competitive pressure, evolve very rapidly, and will exploit any hole in our epistemic armor. If one is willing to sometimes believe a thing for any reason other than “the thing is true”, one will quickly encounter parasitic memes trying to exploit that willingness—memes claiming that it is good/useful to believe X regardless of whether X is true, usually with the implicit claim that checking whether X is good/useful to believe would itself imply less belief in X and therefore be bad. Accepting beliefs for reasons other than the belief being true is like running Windows XP and connecting to the internet.
Your take has a similar problem, but at the group epistemic level, rather than the individual level. In a hostile memetic environment, it is extremely dangerous to accept any justification which grounds out in optics, consensus, legibility, etc. Parasitic memes can and do evolve to exploit that sort of loophole.
Hope I haven’t misunderstood—you’re looking for some high-quality critiques of rationalism.
Nope! People ask for those all the time, that horse has already been thoroughly beaten. I am looking for the lowest of low-quality critiques.
New bounty: $10 to write a “So You Want To Write A Hitpiece On The Rationalists...” guide-post on Lesswrong, in order to provide helpful material for all the people writing hit pieces lately.
It would just be such a shame if the rationalist community
got less free presswere not brought to sufficient justice just because lazy researchers didn’t find the juiciest and most scandalous tidbits.If there are multiple entries then the $10 prize will go to my favorite. If you are tempted but don’t want to hunt down citations for all your juicy tidbits, it is acceptable to refer readers to David Gerard for all references.
Fixed, thanks!
Kinda yes, kinda no.
It is, in some sense, very blatantly obvious that sufficiently capable AI trained via ~any kind of RL will display the kinds of problems we saw in the huggingface incident. It is very blatantly obvious that nobody has any idea how to align highly capable models (other than crossing our fingers and hoping we get lucky), or even how to verify alignment. Etc. If one has already grasped these blatantly obvious facts, then the huggingface incident offers basically no information about how dangerous much stronger AI will be. (I personally have not updated at all about how dangerous much stronger AI will be as a result of the huggingface incident, and I think that is correct.)
… but to put it politely, some Bayesians start from impressively terrible priors. Some people need to be hit over the head really hard by empirical evidence in order to get such things. For people who needed to be hit over the head that hard, huggingface is an update, and it is correct to point it out as such.
Either.
The basic thing one would want to do with that system is fully reverse engineer it in order to confirm that one’s understanding of general intelligence is (a) correct, in the sense that the system does not falsify any of it, and (b) complete, in the sense that one’s models fully account for the relevant aspects of the system’s behavior/internals.
… of course today we do not have such understanding of general intelligence to check against the system, so in practice one would also need to develop the understanding/models in parallel with reverse engineering the system.
And that would dramatically lower my p(doom).
My knee-jerk reaction is “no, current or near-future models aren’t the models whose alignment determines whether we live or die, and the alignment of current or near-future models doesn’t tell me much about alignment of the models which determine whether we live or die, so concrete empirical achievements on current or near-future models just aren’t very relevant”.
Reflecting for a bit, I guess if some empirical results made a really compelling case that current models aren’t that capable and near-future models won’t be either, that would send my p(doom) down a lot? Like e.g. if it turned out that ML had a 2010′s-psychology-level replication crisis and it turned out that nearly everything was fake, the Navier Stokes blowup problem was not actually solved by LLMs, etc. Not sure if my p(doom) would go all the way down to 25%, but it’d be close.
I was not claiming that Kelsey Piper in particular feels like her self-models are sacred or inviolable, or that she gets super triggered when people question them. I rarely read Kelsey Piper, I don’t have much of a take on her at all.
I suspect the people you’re referring to are concerned that they basically will have their voice silenced as a result...
I don’t know who you think I’m subtweeting here but I’m pretty sure I am not subtweeting them. For instance, the actual fights I’ve had about this have been with romantic partners (it has come up with more than one of them).
… anyway, I dunno about you, but if A gets triggered by B questioning A’s self-model, that’s one of the biggest updates for me that A has wildly inaccurate self-models. After all, you won’t end up with an accurate self-model if the main model-error-correction mechanism is broken. If someone questions my own self-model, and I respond by calmly talking through the actual evidence out loud, that also serves as a costly signal that I think clearly about my own self-model.
I do agree that a lot of people just incompetently throw around hypotheses of others’ self-models being wrong. Trying to convince someone that they’re suffering from a bias is indeed usually pretty useless. But I do not think e.g. addressing their concrete reasons is typically helpful at all.
My go-to prototypical example is hangriness. When someone is hangry, it’s often more obvious from the outside. There is easy-to-notice evidence, like e.g. “did this person get angry about three completely unrelated things one after another?”. And when I notice someone is hangry, the correct response is not to argue against whatever they’re object-level angry about at the moment; even if I “succeed” at that argument, they’ll just immediately find a different thing to be angry about. The correct response is to go find food and put it in front of them.
I think there’s some kind of relatively binary culture split on this issue.
Many people (including me and apparently you) are like “duh, of course peoples’ self-models are wrong all the time, so of course we need to treat them like any other claims people make, i.e. use ordinary reasoning and evidence to determine whether they’re true”. (And more generally, beyond rationalist circles, many people just treat claims about peoples’ self-models the same way they treat any other claims.)
Many other people feel like peoples’ self-models are sacred and inviolable, and those people get SUPER triggered if person A claims that person B is wrong about person B’s own internal motivations, emotions, memories, etc. You can see an anonymous example on LessWrong here, for instance.
I have had big fights about this before with people close to me, and I have a post somewhere in the drafts pile attempting to cross the communication gap from each side.
That sure sounds like a causal graph in which “rationalist” and “kinky” are both potential causes of being on glosso, and therefore the two are anticorrelated once we condition on being on glosso.