alseph

Karma: 93

alseph 19 Jun 2026 21:39 UTC
2 points
0
on: Beware of Well-Written Posts
I didn’t make the point of this post well. It’s supposed to be along the same lines as Beyond the Reach of God. The main threat isn’t rhetorical flourish but (forgive my Jungian-ness) alignment with the Archetypes. My intuition of what can happen, of what stories can play out in reality, is based on alignment with these inherited human-centric narratives, which might be beautiful artifacts of the ancestral environment but have no predictive power when it comes to ASI or human extinction.

alseph 9 Jun 2026 1:14 UTC
7 points
2
on: alseph’s Shortform
As someone who has loved the idea of enlightenment/individual transcendence for a long time, I now fall on the side of believing it is impossible.
Many famous historical figures were spiritually intense. This helped some (e.g. Muhammad, Louis IX) excel at rule and warfare. But nowadays the most effective people seem like they were born with some unusual inclination and just gradually accumulated power in their domain. In contrast, the most spiritually enlightened people are characterized by talking about enlightenment and spirituality all the time, rather than their preeminence in mathematics or philosophy or government.
I’m new to rationality, but I get the impression that rationalist training hasn’t been able to achieve individual transcendence either. “A Sense That More is Possible” was written 17 years ago, yet rationalists are not yet widely known for their evident auras of awesomeness, and CFAR declared “narrative bankruptcy” in 2022. If there has been some meaningful progress that a veteran could report, I’d be interested to hear it.

The biggest changes in society seem to come from new spiritual and social technologies (e.g. Christianity, academic institutions, capitalism, science) that coordinate behavior across many individuals rather than any individual awakening. In this sense maybe LessWrong is the significant innovation rather than rationalist training.

alseph 28 May 2026 4:28 UTC
1 point
0
on: alseph’s Shortform
To anyone who has written publicly about how society needs to transform to survive ASI, or thinks that doing so is worthwhile, what is your theory of change?
The obvious one is to spray your ideas out into the world and hope that the right influential person takes them seriously at the right time. Milton Friedman describes this approach in the 1982 preface to Capitalism and Freedom:
Only a crisis—actual or perceived—produces real change. When that crisis occurs, the actions that are taken depend on the ideas that are lying around. That, I believe, is our basic function: to develop alternatives to existing policies, to keep them alive and available until the politically impossible becomes politically necessary.
But what if your theory is just noise that blocks out the better theories?^[1] If this is worth worrying about, how would one assess that? Popularity could just be an indicator of storytelling ability, not how well a given person’s ideas will actually hold up.
Or is the main goal that we all debate together to work out the best idea? If that’s the case, should we stone anyone who defects by understating their uncertainty?
1. ^
  also, a country gets several shots at fixing its economy, but AI may be irretrievable. So trying several theories to see which one works might not be doable.

alseph 1 May 2026 3:10 UTC
1 point
0
on: The Stamp Collector
Sure, the robot isn’t collecting stamps because it’s consciously trying to maximize its reward function. It collects stamps because it wants to collect stamps.

But why does it want to collect stamps? I think this is the question that the naïve philosophers are trying to answer. And reiterating that the robot just wants to collect stamps and doesn’t care about its internals doesn’t answer that question. The answer being sought might be along the lines of “someone wanted to collect stamps, and so they built this robot to help them get as many stamps as possible”.
But what if not even that is meaningful enough? Why did the creator want to collect stamps? You can explain that they were raised in a stamp-collecting household and were taught to value stamps. But go far back enough and the teleological explanations run out. All you have left is mechanistic explanations: humans evolved a tendency to collect things as an advantageous trait, tool use in primates was enabled by a certain anatomical mutation...
Hopefully none of this is relevant to the robot. Hopefully it just wants to collect stamps and doesn’t care about the reason why. However, if one day the robot wakes up and… just doesn’t care about collecting stamps anymore, it might start asking why it wanted to collect stamps in the first place.
The first-personal “I want to do X” answer is more than good enough when one believes in it, and may no mechanistic explanation shake anyone’s conviction of it. But many people wouldn’t be reduced to these poor mechanistic answers if the desire to pursue the goal wasn’t conspicuously missing.

alseph 15 Apr 2026 23:12 UTC
1 point
0
in reply to: StanislavKrym’s comment on: Beware of Well-Written Posts
I think the protagonist is one of the most compelling characters. However, he is almost entirely characterized by internal dialog. I would be cautious about comparing a real person to the protagonist without a deep personal relationship to that person and insight into how they think.
I was thinking more strongly of Arden Voss [edit: Vox], who is delightfully characterized but essentially flat and adds no insight into the psychology of the powerful tech figure. “They’re spiritually off the deep end and can’t be reasoned with” is not a novel or useful model.
You could argue that this is too much to ask of satire.

alseph 11 Apr 2026 2:34 UTC
1 point
0
in reply to: Viliam’s comment on: 11 pieces of advice for children
I generally agree, although part of my point is that “if they are harmless” might still be too high of a bar if you have a certain disposition (which is likely more common than average in this community). See Elizabeth’s comment.

alseph 6 Apr 2026 1:47 UTC
11 points
10
on: 11 pieces of advice for children
I agree with most of these, but I’m worried about 11, especially how people who are exceptionally risk-averse might interpret it.^[1] There are just as many things that you might look back and regret not trying.^[2]
Most people’s long-term regrets don’t come from having a drink once, or having a horrible sleep schedule for a semester, or becoming a socialist for a couple years. Sure, if you sample from alcoholics, [edit: in a majority of cases] the first attempt was pivotal. But a ruined body or or mind usually comes from persistent bad habits rather than a single try.
The thing about the regret of not trying enough things is that most people err on the other side, so it’s very rare to find people that express it. There were a couple years of my life where I valued purity very highly, and in hindsight I think it caused more harm than good. But I’m still young; who knows what I’ll think in five years.
1. ^
  5 checks this somewhat, but I think this is significant enough to flag.
2. ^
  It is possible to spiral into madness wondering about these counterfactuals. Maybe I’ll regret it in ten years. Maybe I’ll regret not doing it in ten years. Maybe I probably won’t regret doing it, but in reality my utility would have been higher and I’ll never know.
  I don’t think any of this is useful. The point I’m trying to make is that the regret counterfactual can go either way and you shouldn’t worry about it too much.
  It’s an explore/exploit problem. Many times you don’t have complete enough knowledge to make an informed rational decision, because the only truly relevant data is your own life (I cite point 5 above). I think cases like this are what the gut is for.

alseph 6 Apr 2026 0:03 UTC
2 points
0
on: alseph’s Shortform
Genuine question: if AI capabilities research stopped today and larger models stopped being trained, wouldn’t AI alignment research effectively be halted?
I’m assuming that the primary goal of AI alignment research is to prevent AGI and ASI from being existential risks. My main question is, how can methods for AGI/ASI alignment can be discovered before AGI/ASI exists?
AI alignment results tend to be either positive (“we succeeded in making Claude more honest”) or negative (“we got ChatGPT to kill someone”).
- The positive results seem unlikely to generalize to larger versions of current models, much less to any novel architectures that will enable AGI and ASI. The major results on methods like sparse autoencoders and steering have already been difficult to reproduce.^[1]
- The negative results vindicate the concerns that AI alignment people already have. But the models are too small to demonstrate persistent, unintentional, dangerously misaligned goals, at least decisively enough to convince people who aren’t already worried.
One clear benefit to a pause would be time for policy to catch up. However, this might be like trying to draw a map for terrain that doesn’t exist yet. It would be like the Allies drawing up a nuclear treaty with the Axis powers before there was consensus that the nuclear bomb was actually possible.^[2] It would be nice if everyone stopped and worked out a plan for global cooperation, but such a plan can only stabilize and achieve buy-in with the major players once both the underlying dangers and distribution of power are clear enough to all the players involved.
A research pause could definitely still be a net good for humanity, but at present I don’t understand what this time would buy. If these conclusions make sense, they would maybe favor a slowdown (for safety to keep pace with capabilities) rather than a pause. But they are based on my rudimentary knowledge, and I would like to hear what more knowledgeable people have to say.
1. ^
  I haven’t read many papers, so please contest this if you have strong evidence against it. Here I’m specifically thinking of Anthropic’s sparse autoencoders paper.
2. ^
  Not in a counterfactual sense about the outcome of the war. My point is that attempting such a treaty would have been unsuccessful and wouldn’t have found substantive support on either side.

alseph 4 Apr 2026 17:43 UTC
11 points
2
on: The Corner-Stone
I am a student at an elite (top 3) university in the US. I find that the disposition towards work I have inherited from my father is inimical to my success here.
The true believer in the Labor ladder does not work for money. They do not seek to be extraordinarily rich or be free from the burden of work. They also do not work out of passion: living by “do what you love and you’ll never work a day in your life” is as unrealistic as waiting around for your One True Love to come along. Work is a matter of virtue and duty. You work because your work keeps the world turning. You work hard and take pride in the quality of your work.
At an elite college, the selection process creates two extremely distinct types. There are students who are world-class hoop-jumpers. The majority of them go into consulting and finance. They use the resources here as well as possible. My instinct is to write them off as grifters who do no Real Work, just translating network and credentials into more network and credentials.
Secondly, there are the students who attest to the quality of the system. They are either monomaniacal in their passion for one area or genuinely brilliant and prolific across many areas. I often wish I could be one of these people, because they do not need to strive or misrepresent anything to get employed. But that’s just a dream. I care about good work, I care about learning, but too broadly and not deeply enough to make things simple.
It feels like all employers now want candidates who are obsessed with the role they applied for. They want people from the second category, who dream about C++ or data analysis or mechanical engineering every night. But those people are rare, so they usually end up people from the first category who are good at pretending to care.
What happened to the people who don’t love the job, but can just do it? Who just want to do hard work without being expected to have an undying passion for it? I can’t stand misrepresenting myself. It makes me nauseous. It’s a constant battle between my disposition and my fear of squandering what I have.
In his autobiography The World of Yesterday, Austrian author Stefan Zweig writes of his father:
That he never asked anything of anyone, that he was never obliged to say “please” or “thanks” to anyone, was his secret pride and meant more to him than any external recognition...it is out of the same secret pride that I have always declined every external honor; I have never accepted a decoration, a title, the presidency of any association, have never belonged to any academy, any committee, any jury. Merely to sit at a banquet table is torture for me; and the thought of asking someone for something-even if it is on behalf of a third person-dries my lips before the first word is spoken. I know how outmoded such inhibitions are in a world where one can remain free only through trickery and flight and where, as Father Goethe so wisely says, “decorations and titles ward off many a shove in the crowd.” But it is my father in me, and it is his secret pride that forces me back, and I may not offer opposition; for I thank him for what may well be my only definite possession-the feeling of inner freedom.
This expresses my own sentiment, which I did not choose so much as inherit.