I’m writing a book about epistemology. It’s about The Problem of the Criterion, why it’s important, and what it has to tell us about how we approach knowing the truth.
I’ve also written a lot about AI safety. Some of the more interesting stuff can be found at the site of my currently-dormant AI safety org, PAISRI.
I think this post makes an excellent core point re the evolutionary origin of human values and I think it’s other points, while obvious in some sense, are valuable to articulate because they have gone underappreciated in recent discussions. I hope to see a greater continued focus on the evolutionary origin of values in future alignment discussions.
You’ve addressed the normative question, but you don’t really deal with what I read as the underlying assumption of this post, which is that you view feeling gratitude as a threat. That seems load-bearing for your argument to me.
I’m starting to think Claude is already superhuman at this part.
This is a claim that I find hard to evaluate.
On the one hand, Claude is better than me in a bunch of ways. It knows more without having to look it up. It works faster and without getting tired. It can even work in parallel in ways I can’t. So in all those ways it’s a better coder than me.
But if I look at the individual output, it’s clear that Claude is only at best as good as a P90 human. It’s not really able to come up with clever, Carmack-or-Knuth-level solutions to problems. Heck, sometimes it can’t even come up with as good a solution as I can! What it can do, though, is just keep applying above-media coding expertise persistently to a problem when directed to do so.
The main problem I see between Claude and being superhuman is it’s lack of taste. The main way my coding sessions go off the rails is that Claude gets hung up on an idea, runs with it, and doesn’t have the judgement to realize it was a bad idea or to make itself step back and look for a better solution (because it’s notion of what’s “better” is fairly limited).
Now none of this is to dispute that what Claude can do is really, really cool, and should dramatically increase productivity, since it can do what you would have used to need a human to do, and the human would have been slower even if you paid them more than you pay Claude. So in some limited sense that’s “superhuman”, but I mostly think we should preserve “superhuman” for when Claude can do something humans cannot modulo time and cost.
Yes, however a version of skepticism that claims that nothing can be known for sure/justified beyond any doubt doesn’t have this problem.
But how much is this still properly skepticism, then, instead of justified uncertainty?
It’s true. I’ve had better philosophical insights as the result of hanging out in bars than going to philosophy meetups.
The classic problem with (Pyrrhonian) skepticism is that it’s a fake position. That is, where skepticism claims to say that something can’t be known, the claim that something can’t be known is itself a claim to know, and thus skepticism is just a special case of claiming to know but where the particular knowledge rejects knowing the matter at hand. Thus skepticism isn’t really any different, fundamentally, than claiming to know something, it just tries to hide this fact and shrink it down to a single claim, which doesn’t solve the epistemological problems involved so much as move them around.
There’s a more casual kind of skepticism that’s quite good, though, which is to be suspicious of the supposed validating of claims. To want to check the details of arguments for oneself. That kind of skepticism is quite useful!
It’s with great regret that our language doesn’t make a clean distinction between these two kinds.
Seems unlikely, as others point out. The prior that the USG may have better versions of things than industry or the general public do is reasonable, and in some cases is born out (the NSA presumably still has a large lead in crypto and sigint capabilities, for example), but for the USG to have better models than the labs would either require that they be getting those better models from the labs and then not allowing the labs to keep using them or to have hired a large cadre of “dark” AI/ML engineers who have been doing secret research for months-to-years that is beating what the public knows (again, not a totally unreasonable prior given the NSA, but we don’t really even have rumors suggesting that this is going on).
What, if anything, has changed since your trip? I assume you didn’t stay dead once it was over, though maybe you did?
I do a version of this workflow for myself using Claude as an editor/cowriter. Aside from the UI, are you doing anything more than what I can get from just handing Claude a good prompt and my post?
Committing to writing 1 post a week has been a really good practice for me. Although I decided to take December off for a variety of reasons, the need to keep publishing has forced me to become a more efficient writer, to write about topics I might otherwise have neglected, and of course to get more feedback on my writing than I otherwise would if I spent more time effortposting and less time actually posting. I expect to keep it up in 2026.
done
AI is much more impressive but not much more useful. They improved on many things they were explicitly optimised for (coding,
I feel like this dramatically understates what progress feels like for programmers.
It’s hard to understand what a big deal 2025 was. Like if in 2024 my gestalt was “damn, AI is scary, good thing it hallucinates so much that it can’t do much yet”, in 2025 it was “holy shit, AI is scary useful!”. AI really started to make big stride in usefulness in Feb/March of 2025 and it’s just kept going.
I think the trailing indicators tell a different story, though. What they miss is that we’re rapidly building products at lower unit operating costs that are going to start generating compounding returns soon. It’s hard for me to justify this beyond saying I know what I and my friends are working on and things are gonna keep accelerating in 2026 because of it.
The experience of writing code is also dramatically transformed. A year ago if I wanted some code to do something it mostly meant I was going to sit at the keyboard and write code in my editor. Now it means sitting at my keyboard, writing a prompt, getting some code out, running it, iterating a couple times, and calling it a day, all with writing minimal code myself. It’s basically the equivalent of going from writing assemble to a modern language like JavaScript in a single year, something that actually took us 40.
This will be missing in the stats, but there was some real reader fatigue on my end. It was a lot to keep up with, and I’m extremely glad we’re back to a lower volume of posts daily!
I continue to be excited about this class of approaches. To explain why is roughly to give an argument for why I think self-other overlap is relevant to normative reasoning, so I will sketch that argument here:
agents (purposeful, closed, negative feedback systems) care about stuff
what an agent cares about forms the basis for reasoning what norms it thinks are good to follow
some agents, like humans, care what other agents think
therefore, the agents a norm follows depend in part on what other agents care about
the less an agent considers itself as distinct from others, the more it cares about what others care about, and the more it will want to follow norms that satisfy the concerns of others
But this sketch is easier explained than realized. We don’t know exactly how humans come to care about others, so we don’t know how to instrument this in AI. We also know that human care for others is not perfect because evil exists (in that humans sometimes intentionally violate norms with the intent to harm others), so just getting AI that cares is not clearly a full solution to alignment. But, to the extent that humans are aligned, it seems to be because they care about what others care about, and this research is an important step in the direction of building AI that cares about other agents, like humans.
There is a sort of classic observation here that people who proclaim not to care about X sure seem to care a lot about X since they bothered to proclaim as much.
And indeed, I think that goes on a lot with status. This makes sense, especially for people who are losing the status games they find themselves playing. It’s both a psychological coping mechanism to deal with the cognitive dissonance of finding one’s self low status when expecting to be higher status, and a bid to create an alternative status hierarchy where they can be high status.
Well-adjusted people who aren’t celebrities know that the only way to win at status in the modern era is to become part of an alternative status hierarchy. This might mean being high status at work, at school, in a hobby, at church, in a friend group, or even just in your own house among your family. Heck, I think a non-trivial amount of why people like having pets is that pets treat their owners as high status!
But some people get stuck here. They can’t figure out how to get on top of any status hierarchy because they don’t really understand how status works. They don’t have an intuitive model of it, also don’t have an explicit model, and thus feel like status is some kind of blackbox that’s out to get them.
The only solution for the autistic is likely to simply learn how status explicitly works and learn to manually master it, same as any other complex game can be mastered.