I’m writing a book about epistemology. It’s about The Problem of the Criterion, why it’s important, and what it has to tell us about how we approach knowing the truth.
I’ve also written a lot about AI safety. Some of the more interesting stuff can be found at the site of my currently-dormant AI safety org, PAISRI.
I continue to be excited about this class of approaches. To explain why is roughly to give an argument for why I think self-other overlap is relevant to normative reasoning, so I will sketch that argument here:
agents (purposeful, closed, negative feedback systems) care about stuff
what an agent cares about forms the basis for reasoning what norms it thinks are good to follow
some agents, like humans, care what other agents think
therefore, the agents a norm follows depend in part on what other agents care about
the less an agent considers itself as distinct from others, the more it cares about what others care about, and the more it will want to follow norms that satisfy the concerns of others
But this sketch is easier explained than realized. We don’t know exactly how humans come to care about others, so we don’t know how to instrument this in AI. We also know that human care for others is not perfect because evil exists (in that humans sometimes intentionally violate norms with the intent to harm others), so just getting AI that cares is not clearly a full solution to alignment. But, to the extent that humans are aligned, it seems to be because they care about what others care about, and this research is an important step in the direction of building AI that cares about other agents, like humans.
I continue to really like this post. I hear people referencing the concept of “hostile telepaths” in conversation sometimes, and they’ve done it enough that I forgot it came from this post! It’s a useful handle for a concept that, in particular, can be difficult for the type of person who is likely to read LessWrong to deal with because they themselves lack strong, detailed models of how others think, and so while they can feel the existence of hostile telepaths, lack a theory of what’s going on (or at least did until this post explained it).
Similar in usefulness to Aella’s writing about frame control.
I like this post, but I’m not sure how well it’s aged. I don’t hear people talking about being “triggered” so much anymore, and while I like this post’s general point, its framing seems less centrally useful now that we are less inundated with “trigger warnings”. Maybe that’s because this post represents something of a moment as people began to realize that protecting themselves from being “triggered” was not necessarily a good thing, and we’ve as a cultural naturally shifted away from protecting everyone from anything bad ever happening to them. So while I like it, I have a hard time recommending it for inclusion.
I wish this had actually been posted with the full text on LessWrong, but I stand by its core claim that “boundaries” are a confused category as commonly understood. I don’t have much to add other than Chris did an excellent job of explicating the issues with the usual understanding of “boundaries” and helps to reform a clearer understanding of how people relate and why “boundaries” can often lead to relational dysfunction.
This post still stands out to me as making an important and straightforward point about observer dependence of knowledge that is still, in my view, under appreciated (enough so that I wrote a book about it and related epistemological ideas!). I continue to think this is quite important for understanding AI, and in particular addressing interpretability concerns as they relate to safety, since lacking a general theory of why and how generalization happens, we may risk mistakes in building aligned AIs if they categorize the world in usual ways that we don’t anticipate or understand.
While I lack the neuroscience knowledge to directly assess how correct the model in this post is, I do have a lot of meditation experience (7k+ hours), and what I can say is that it generally comports with my experiences of what happens when I meditate. I see this post as an important step towards developing a general theory of how meditation works, and while it’s not a complete theory, it makes useful progress in exploring these ideas so that we may later develop a complete one.
I like this post because it delivers a simple, clear explanation for why seemingly prosaic AI risks might be just as dangerous as more interesting ones. It’s not a complicated argument to make, but the metaphor of the rock and knife cuts through many possible objectives by showing that sometimes simple things really are just as dangerous as more advanced ones. And having the proof by example, we have to work harder to object that prosaic AI is not that dangerous.
It’s perhaps overdetermined that I liked this post; it’s about how some of my favorite topics are connected: rationality, religion, and predictive processing.
That said, I think it’s good at explaining those connections in a way that’s clear to readers (much clearer than I’m often able to achieve). My recollection is that the combination of this post and talking with zhukeepa at LessOnline was enough to push me in the direction of later advocating that rationalists should be more religious.
This post gets filed in my brain under “the world contains a surprising amount of detail” and contains an attached note stating “and those details are connected in surprising ways because the details of the world are densely distributed in concept space”.
It’s one thing to understand, in theory, that the world is deeply interconnected. It’s another to actually see those connections feel how unexpected they often are. This post is useful evidence as part of a bundle of similar facts to build up a picture about how the world fits together, and I wish there were more posts like it.
Don’t forget this other bit of writing advice: you have to write a million words to know what you’re doing.
You’re now a bit closer to the mark, and a bit closer to knowing what you’re doing. Maybe you decide it’s not worth the effort, but it’s normal for most people to have to write a lot of words before they start to feel like they can write (1mm words is about 914 words a day for 3 years).
It seems worth saying that humans are also subject to causal bypassing. We have limited introspection abilities, and frequently hallucinate “introspective states” based on backfilling from theory rather than from actual observation. In theory we may be able to make LLMs or other AI systems do better, but it seems we should expect it to be a hard problem to solve given our own introspection is not always grounded.
They are in the original, but it looks like LessWrong’s import stripped them. IDK who manages that, but I’ll let them know via Intercom.
Oh, and I should also note, since you mention incentives, that I largely see my job as a writer to ignore what my audience wants. This is dangerous advice if taken too far, but what I mean is that people will reward me for writing slop, and I have to decide, am I here to write slop, or am I here to write something else even if people don’t like it as much. I choose to optimize for something other than upvotes, though I do care if people can make sense of what I’m saying and it’s worthwhile to them that I said it.
To close off this ramble, I’m a bit disheartened. My fun easy writing was more popular than my attempts at substantive intellectual progress contribution. It’s okay that it flopped, and it makes sense that kind of writing’s harder, it’s just a disappointing update about the writing incentive landscape.
This has been my experience also. Only rarely do things I put significant effort into perform well in terms of votes. It’s often the quick, casually written pieces that I barely edited that people love.
I think it’s in part due to selection effects. The things I can write quickly in an hour or two and be happy to publish are the things I can explain easily because they are easily accessible and understood by everyone. The things I have to write slowly are more complicated topics by nature and thus interest fewer people and it’s less likely that my presentation will connect with a large audience.
(The would-be counter example is that my mostly highly upvoted post of all time took a lot of effort to produce, but it was about an extremely accessible topic: the death of my mother.)
What I can tell you, though, is that having a catalog of deep posts that I put a lot of effort into builds on itself over time. Those posts don’t connect with everyone, but then I’ll meet someone and find out that what I wrote was life-changing for them.
It’s hard for me to know if the effort I put into them was worth it, in that was there a version of that post I wrote less well that connected equally well. I’m not sure, it’s a bit hard to test, especially since often I spend a long time writing a post to get clear on the ideas myself, not just to massage the language. But I can say that, of those posts I labored over, they are among the most impactful, even if they were not popular on initial publication.
This is a category argument that I explicitly avoid making and don’t think is meaningful: the word itself does not mean anything and arguing over it is meaningless. You seem to really want to do that anyway because you can support your argument better on basically definitional grounds than factual ones.
This is an extremely weird position to take given the central claim of this entire post seems to be that rationality fits the definition of a religion. What do you think you’re arguing for in this post, then, if not that?
by choosing Islam, Judaism etc as comparison points instead of Scientology, Mormonism or any other smaller, newer or more modern movement is just assuming your conclusion.
I don’t know what point you think you’re making here but these are also obviously religions.
How many non-religions have had people write thousand-word posts denying they are religions?
Many, actually. I’ve read things arguing variously that:
sports fandom is/isn’t a religion
anime fandom is/isn’t a religion
communism is/isn’t a religion
capitalism is/isn’t a religion
veganism is/isn’t a religion
You know what I’ve read zero words about? Arguments that:
Islam is/isn’t a religion
Judaism is/isn’t a religion
etc.
This, to my reading, is evidence in favor of things people are are/aren’t religions not being religions, or at least not fitting within the category of religion as traditionally understood. Your meme doesn’t really prove anything. It just tries to assert that no, actually, the existence of arguments that something isn’t a religion is evidence in favor of it being a religion, but fails to do any work to establish such a claim, and seems contrary to the evidence I listed above.
Now it would be much better to argue on facts, but unfortunately it’s notoriously hard to define the category of religion accurately, so I’m not sure how we can really do that. Assuming you haven’t cracked the central question of religious studies (a question about which there is only limited consensus), it’s going to be quite hard to look at the features of rationality and say whether or not it’s in the category of religion. And that means we’re left to see if it looks like a central example of religion, which it clearly doesn’t since it’s a matter of contention, and so perhaps the only interesting question is not if it’s a religion but what features of religions does it share (which you do get in to, to your credit, but that’s different from arguing that rationality is a religion).
Pretty much any other major feature of religion you can name is present in rationalism. Rationalism’s resemblance to traditional religion is so extreme that even if rationalism is not, technically, a religion, this seems like it is a pedantic distinction. It certainly has very distinct beliefs and rituals of its own, and only narrowly misses those points of comparison on what seem to be technicalities.
You have come to the wrong place if you want to make an argument that’s not going to be refuted by pedantic distinctions!
But I find your arguments unconvincing on the whole, no pedantism required. My reading is that you’ve successfully argued that rationality is a movement, not too different from many other movements like veganism or communism. The content is different, but the structures are similar. To the extent movements look like religion, it’s because both are for humans, and both work or don’t work based on successfully coordinating humans to take particular actions.
As I see it, that you have to argue that it’s a religion is a bad sign. It means that rationality isn’t passing the smell test. That is, you needed to write a post to argue that rationality is a religion, which I view as evidence against it being a religion, since most religions are clearly religions and no one writes posts arguing that Islam or Hinduism is a religion (if anything, people sometimes write the opposite for various reasons!). If it’s in the category of religions, it’s a marginal case at best.
(And just to lay my cards on the table, I say this as a religious rationalist, in that I’m religious and also part of the rationalist movement. In fact, I’ve put some effort into trying to convince rationalists to be religious because I think it would be good for them!)
I’m lucky to live in a city (San Francisco) with multiple art museums. I have memberships. I enjoying going back over and over because I discover something new every time. Sometimes it’s something in beloved, favorite piece I never noticed before. Other times something catches my eye about a piece I had previously ignored, not even noticing it was there. Sometimes I go and spend an hour in a single room, really drinking in the pieces there. The depth of art is not necessarily endless, but it’s deep enough that I doubt I will ever explore it all.
Do you ever find yourself slipping into a state (possible gradually) of over-reliance on the LLM?
I don’t really buy the premise of over-reliance. I can only rely on a technology too much with respect to some goal. Like for example if I need to be able to survive in the wilderness, then using a microwave causes my skill at starting fires to dwindle if I’m no longer practiced at starting fires.
I instead think in terms of what I’m trying to accomplish. If all I want is a picture, not to learn how to draw, and the alternative was no picture, then I’m pretty happy to get a diffusion model to draw me a picture. Similarly, when writing, if the choice is fail to finish a blog post because I’m stuck or get unstuck with Claude’s help, I choose Claude, even if that means I get marginally worse at solving writer’s block on my own. As long as I have Claude around, this isn’t an issue, and if I lose it, I can go back to doing things the hard way.
For another example, I’ve been doing ML research lately, in that I’m trying to train new models from scratch. I don’t have much background in ML, but I am a professional programmer and have some adjacent experience with data analytics. So I’m using Claude Code to vibe code a lot of this stuff for me, but I’m always setting the direction. As a result, I’m not really learning PyTorch, but then if I had to learn PyTorch I probably wouldn’t even bother to invest the time to train new models because it would be more than I had time to take on. So instead I vibe code to see if I can get results, and then learn as I go, filling in my knowledge by looking at the code, fixing problems, and trying to understand how it all works after it’s already working.
As Claude and other LLM become (expectedly) more capable, do you have any preliminary thoughts for how this workflow style may be updated?
The big change would be if Claude got better at writing in my style. The most painful and time consuming part is still editing. Claude can only kind of approximate good writing, much less good writing of the type I would personally produce. It would cause a big speed up if I could just dump thoughts and have Claude sort them out and turn them into easily readable text. Even if I could just write in the maximally Gordon way that’s not fit for other people to read (which is a little bit of what you’re getting in this comment; it’s stream-of-thought and not polished for concision or readability) and have that be turned into nice text would be a big deal.
This will be missing in the stats, but there was some real reader fatigue on my end. It was a lot to keep up with, and I’m extremely glad we’re back to a lower volume of posts daily!