Raphael Roche

Karma: 306

Raphael Roche 30 Dec 2025 10:54 UTC
1 point
0
on: Many can write faster asm than the compiler, yet don’t. Why?
If we can get SC LLMs, this problem would fade away and the initial quote would become 100% true. Also a SC LLM could directly write optimized code in assembler (that would define hypercoder LLM ? And the end of programing languages ?).

Raphael Roche 22 Dec 2025 11:16 UTC
2 points
−3
on: Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment
It is still too early to tell, but we might be witnessing a breakthrough in AI Safety. If, by saturating models with positive exemplars and filtering out part of adversarial data, we can develop base models that are so deeply aligned that the ‘Valley of Chaos,’ Waluigis, and other misaligned personae are pushed far out of distribution, it would become nearly impossible for a malicious actor to elicit them even with a superficial fine-tuning. Any attempt to do so would result in a degraded and inconsistent simulation.
Taking this a step further, one could maybe engineer the pre-training so that attractors like misalignment and incompetence become so deeply entangled in the model’s weights that it would be nearly impossible to elicit malicious intent without simultaneously triggering a collapse in capability. In this regime, reinforcing a misaligned persona would, by structure, inherently reinforce stupidity.

Raphael Roche 17 Dec 2025 23:51 UTC
2 points
0
in reply to: StanislavKrym’s comment on: Response to titotal’s critique of our AI 2027 timelines model
You’re right. Sonnet 4.5 was impressive at launch but the focus of AI 2027 is on coding oriented models.

Raphael Roche 17 Dec 2025 13:00 UTC
4 points
0
on: Response to titotal’s critique of our AI 2027 timelines model
It isn’t the authors’ fault, but LLMs progress is so rapid that this kind of extensive work can hardly keep up with the release of new models providing new data. It’s kind of frustrating. I hope to see Claude Sonnet 4.5, Claude Opus 4.5, Gemini 3 Pro, and ChatGPT 5.2T in the plots of the upcoming update.

Raphael Roche 14 Dec 2025 8:46 UTC
3 points
0
on: Why did I believe Oliver Sacks?
I think that we may be tempted to justify our adherence to Sacks’s narrative by nice arguments like his reading feels honest and convincing. However it is plausibility a rationalization avoiding to acknowledge much more common and boring reasons such as we have a strong prior because 1) it’s a book 2) it’s a best seller 3) the author is a physician 4) the patients were supposed to be known to other physicians, nurses, etc 5) and yes, as you also pointed out, we already know that neurology is about crazy things. So overall the prior is high that the book tells the truth even before we open it. That’s said, I really love Oliver Sacks’s books.

Raphael Roche 14 Dec 2025 2:21 UTC
8 points
5
on: How I stopped being sure LLMs are just making up their internal experience (but the topic is still confusing)
and most humans are conscious [citation needed]
The problem lies here. We are quite certain of being conscious, yet we have only a very fuzzy idea of what consciousness actually means. What does it feel like not to be conscious ? Feeling anything at all is, in some sense, being conscious. However, Penfield (1963) demonstrated that subjective experience can be artificially induced through stimulation of certain brain regions, and Desmurget 2009 showed that even conscious will to move can be artificially induced, meaning the patient was under the impression that it was their own decision. This is probably one of the strongest pieces of evidence to date suggesting that subjective experience is likely the same thing as functional experience. The former would be the inner view (the view from inside the system), while the latter would be the outer view (the view from outside the system,). A question of perspective.
Moreover, Quian Quiroga 2005 and 2009 proved the grandmother cell hypothesis to be largely correct. If we could stimulate the Jennifer Aniston neuron or the Halle Berry neuron in isolation, we would almost certainly end up with a person thinking about Jennifer Aniston or Halle Berry in an unnatural and obsessive way. This situation would be highly reminiscent of Anthropic’s 2024 Golden Gate Claude experiment. And if we were to use Huth/Gallant 2016′s semantic map to stimulate the appropriate neurons, we could probably induce the subjective experience of complete thoughts in a human.
Though interestingly, this is similar to what happens in humans! Humans might also be able to accurately report that they wanted something, while confabulating the reasons for why they wanted it.
It is highly plausible, given experiments such as those cited in Scott Alexander’s linked post, that the patient would rationalize afterward why they had this thought by confabulating a complete and convincing chain of reasoning. Since Libet’s 1983, doubt has been cast on whether we might simply always be rationalizing unconscious decisions after the fact. Consciousness would then be nothing but the inside view of this recursive rationalization process. The self would be a performative creation emerging from a sufficiently stable and coherent confabulation.
If this were the case for us humans, I agree that it becomes difficult to deny the possibility that it might also hold true for an LLM, especially one trained to simulate a stable persona, that is to say, a self. I really appreciated reading your post. The discussion is not new to LessWrongers, but you reframed it with elegance.

Raphael Roche 4 Dec 2025 0:31 UTC
3 points
0
in reply to: Steven Byrnes’s comment on: Response to nostalgebraist: proudly waving my moral-antirealist battle flag
I agree that my argument falls into category 2. However, I don’t defend the idea of strong moral realism, with moral agents acting for the sake of an absolute idea of good. What I call weak moral realism is a morality that would be based on values that may be instrumental in some sense, but with such a degree of theoretical convergence that it makes sense to speak of universal values. It is of course a question of definition, but to me there is a huge difference between whether a value is universal or at least highly convergent, and whether it’s just a nearly random value like the color of a flag.
I also agree that the trick of injecting uncertainty into game theory and using Rawls’s veil as a patch to obtain something close to a formal moral theory would probably not convince a psychopath not to kill me, nor perhaps a paperclip maximizer. However, if that paperclip maximizer is in fact an AGI or ASI, I think that a process like CEV could very well cause an important drift in the interpretation of its initial goal. Maybe it would be smart enough to realize that its hardcoded goal was not that satisfying in the naive interpretation consisting of making just as many paperclips as possible, and that it was far more valuable to spend its time and pleasure in theoretical research into the physics of paperclips, literature, music, and painting to capture the pure beauty of paperclips, video games about paperclips (you can make many more virtual paperclips than material ones), philosophy and morality about being a paperclip-maximizer, etc. Just as we humans evolved from simple hardcoded goals like eating and reproducing to whatever may be our current occupations.
And here is where it becomes interesting : because if we humans discovered universal formal truths, any intelligent being could also arrive at the same ideas. If we humans ask ourselves what we are, what the meaning of life is, what is good, any intelligent being could also ask itself such questions. And if we humans saw our values change across time, not following only a random drift like the colors of flags, but also partly driven by reflection on ourselves and on our goals, it does not seem impossible that a paperclip maximizer could also arrive at ethical concerns. Moral and philosophy are not formal sciences, however it changes everything if there exist at least something like universal or convergent rational attractors. You will say again that’s a big “if,” but I think the question remains open.

Raphael Roche 2 Dec 2025 11:34 UTC
3 points
0
on: Response to nostalgebraist: proudly waving my moral-antirealist battle flag
Thank you for this very thoughtful post.
I’m not convinced by the metaphor of the soldier dying for his flag. I acknowledge it’s plausible that many soldiers historically died in this spirit. We can see it as acceptance of the world, as absurd as it may appear. Engaged adherence could be seen as a form of existentialism (as well as rebellion), while a nihilist would deny any value in engagement and just look at Moloch in the eyes.
But to me, such a relativist position is not consistent. I mean, as already pointed out in the post, if you think that a value, symbolized by the flag, is relative and does not, rationally, have more merit than those of the opposite side, why would you give your life for it in the first place? Your own life is something that almost everyone, except a hardcore nihilist, would acknowledge as bearing real value for the agent. There is a cognitive dissonance in the idea that the flag’s value is undermined but that you would still give up your stronger value for it.
Moral realism may be imperfect, but, as also pointed out in the post, it is sometimes rationally backed up by game theory. In many multi-turn complex games, optimal strategies imply cooperation. Cooperation or motivated altruism is a real, rational thing.
But, while I agree with the OP that game theory is also sometimes a bitch, I think that what’s lacking in the game-theoretic foundation of a (weak) moral realism can be largely corrected if you apply Rawls’s veil of ignorance to it. Think of game theory, but in a situation of generalized incertitude where you don’t know which player you’ll be, and you’re not even sure of the rules. Now out of this chaos the objectively rational choice would be to seek common interest or common good, or at least lesser suffering, in a reasonable or Bayesian way.
Indeed, life seems to be a very complex multi-turn game, dominated by uncertainty. We’re walking in a misty veil. Even identity is not a trivial question (what is “me”? Are my children part of me or entirely separate persons without common interest? Are my brothers and sisters? Are other humans? Are other things constituting this universe?). Perhaps it is even less a simple question for AI or uploaded minds. Maybe the wiser you are and the less you treat it as a trivial matter. Even among humans, sophisticated people seems less confident on these questions than the layman. In my opinion, it’s hard to dismiss the possibility of moral realism, at least in a weak form.
However, I agree that is remains a very speculative argument that would only slightly affect doom expectations.

Raphael Roche 28 Nov 2025 8:11 UTC
5 points
0
on: Bitter Lessons from Distillation Robustifies Unlearning
This is so reminiscent of how human memories seem to be stored. Access to memories may be disabled, but this does not ensure complete deletion. In some circumstances, lost memories suddenly resurface in the way Marcel Proust famously described. Oliver Sacks reported striking pathologic examples of this in his books. It looks like another example of functional convergence between natural neural networks and artificial neural networks.

Raphael Roche 26 Nov 2025 23:16 UTC
1 point
0
in reply to: Mikhail Samin’s comment on: EU explained in 10 minutes
Thanks for that important correction ! I’m not up to date. I edited my comment.

Raphael Roche 26 Nov 2025 23:14 UTC
1 point
0
in reply to: Seth Herd’s comment on: What’s the short timeline plan?
I agree that continual or online training / memory would probably be disruptive both in terms of capabilities and risk. My idea would indeed fail in this case. It would be fascinating to chat with an online model, but I would also fear it goes out of control anytime.
As you mention, OpenAI introduced a little persistent memory in ChatGPT since version 4o (or was it 4 ?). While I also use other models, nevertheless ChatGPT has now an impressive persistent memory of our discussions since more than a year. I also observe that even such a modest memory has a significant effect. The model sometimes surprises me by establishing a reference to another idea discussed long ago. Establishing such links is certainly part of intelligence.

Raphael Roche 26 Nov 2025 13:31 UTC
4 points
0
on: HTTP402: musings about an ad-free internet
I like the idea.
But unfortunaly my expectation is that your grandma would receive an email with a link HTTP 402 asking for $1,000, which she would validate with an accidental click. Then, even if regulations stated that the bank must refund the customer under such circumstances, the bank would reject all your claims. You’d hire a lawyer for a significant amount of money and, if you’re lucky, your grandma would get refunded two years after she died, but the process would be hard to make financially worthwhile. And if you’re not lucky, you’ll just lose another $3,000 in legal fees.
I’m afraid that’s the world we’re living in.

Raphael Roche 22 Oct 2025 23:43 UTC
46 points
17
in reply to: MondSemmel’s comment on: EU explained in 10 minutes
I plead guilty to not being neutral about nationalism in my previous comment. So far, reality has provided me with very little Bayesian evidence in favor of it.
On a personal level, my great-aunt (whom I knew) was tortured by the Gestapo, my grandfather had terrible experience in a labor camp in occupied Poland, never recovered, and died prematurely from alcoholism. And in the generation before, most of my great-grandfathers and great-granduncles fought for years in the trenches, were wounded, and some died, essentially for nothing.
On a less personal level and in a register more suited to LessWrong standards, the two World Wars together caused around 60 million deaths in Europe alone (up to 15% percent of the population in some countries during WWI). Vast, ancient, and beautiful cities were destroyed, invaluable cultural heritage was lost, and, of course, there were the horrors of the extermination camps. The destruction of wealth in Europe is also beyond comprehension : for WWI, roughly trillions of inflation-adjusted 2025 dollars in war expenditures and more than one trillion in material damage. For WWII, over ten trillion in war budgets and several trillions in destruction.
Nationalism was almost directly and wholly responsible for all of this. So yes, it is difficult for me not to see nationalism as a form of genuine Evil. Not only Nazism, but also the more ordinary, everyday nationalism we still see today. Let us not forget that there were no Nazis in 1914. In contrast, it seems self-evident to me that the humanists who launched the NATO project and soon after, the European project, were the good guys in the story.
I can acknowledge that rational arguments in favor of nationalism exist. I understand how so many people can be drawn to such ideas. Most nationalist leaders are democratically elected. “Make [your country] great again” or “[Your country] first!” is perhaps the most effective political slogan ever devised. It may even appear entirely legitimate and efficient at first glance. You can certainly achieve good short or medium term results. But since every country is equally entitled to make itself “first” and “great again,” the only long-term outcome is conflict, tragedy, and destruction, a net negative, as predictable as stepping off a cliff.
That being said, no extreme worldview is likely to be true. I suppose that an extreme cosmopolitan, pacifist, anti-nationalist project would also end in failure. No borders, no armies, no economic patriotism, no incentive to compete, no shared identity, total relativism regarding values, no local decision-making, all sovereignty delegated to a single global government… I simply cannot see how that could work with real human beings.
Still, just as the “conservatives” opposed to European integration are not all true nationalists (some belong to the far-left camp opposed to Brussels’ white-collar bureaucracy), the “progressives” I refer to are not all naïve cosmopolitan idealists. Their initial goal was a federal project modeled after the American example. That hardly seems unreasonable. In a federal system, individual votes are more diluted and each state’s sovereignty is limited. Yet there remain local elections, local decision-making, and a sense of local identity. It would have been harder to achieve in Europe given history and diversity, but I can imagine such a federal system functioning. Perhaps even better than the half-working Balkan house Europeans currently enjoy, courtesy of the “conservatives”, or if you prefer, “euro-skeptics”.

Raphael Roche 21 Oct 2025 20:48 UTC
43 points
9
on: EU explained in 10 minutes
Well, for once, I can claim to have some actual competence on the subject. The “Balkan house” analogy is brillant, and the post itself is very good. Let me try my own explanation of this oddity in just a few lines (sorry for the redundancies).
First, as noted in the post, forget about the Council of Europe, that’s a completely separate institution, born from an independent treaty dealing with human rights and justice (and Russia is in, believe it or not). [edit : Mikhail Samin corrected me, Russia left / got cancelled in 2022]
Now, as for the European Union, it began merely as an international economic treaty in 1952, a pact between fully sovereign states, each with a long history of independence (and, well, frequent wars) behind them.
But the founding fathers, Jean Monnet (French) and Konrad Adenauer (German), and others, in true Montesquieu fashion, hoped that strong economic cooperation would finally put an end to centuries of conflict among European nations. And remarkably, it did!
From there, two opposing camps gradually emerged:
- The progressive camp, eager to push European integration ever further, aiming ultimately at a genuine federal state, the United States of Europe, in which national sovereignty would be largely dissolved into a single political, military, and economic entity, after the american model.
- The conservative camp, resisting this “European construction” and preferring the original idea: independent sovereign nations bound by an international treaty focused on trade and economics. The malicious tongues (French ones ?) would even say the UK joined the Union only to slow the progressives down, if not sabotaging the all project.
Note that no State has ever been all progressive or all conservative on this matter. It’s not even a left wing against right wing opposition. Center vs borders is a better match.
Anyway, from 1952 up until 1992, the progressives more or less trampled the conservatives. The construction went fast and the Union attracted more and more members.
The Maastricht Treaty of 1992 was maybe their last great victory, a huge leap toward a federal Europe, especially as it required member states, among other things, to surrender monetary sovereignty to the Union (the euro € became effective a decade later). Also that was just after the USSR collapse, and many states from the East filed their membership application in this period (but integration process is long).
Yet some members, predictably the UK, opted out euro and from that moment on, the rivalry turned into a real crisis and the progressives began to lose momentum. EU started to appear as that complicated bureaucratic elitist thing than nobody really understands under IQ 120, so it became the perfect target for populist politicians, the source of all ills (and what was even more convenient, it had at this time no clearly identified spokesperson that could object).
In 2005, the failure of the The Treaty establishing a Constitution for Europe was the first victory of the conservatives. It was even rejected (by universal direct suffrage !) in France that was supposed to be in the progressive camp. The text was elevating economic rules of liberal orientation at a constitutional level, something that was unacceptable for the left wing. While they were still some late joiners from the East, in reality the “construction” sort of stalled after the Lisbon Treaty of 2007 (a watered down version of the former).
UK eventually Brexited after a long and dramatic divorce that ended in 2020. That might have been a chance for the progressives to relaunch the project. But instead, it revealed something deeper and darker, an ancient evil. ~~Sauron~~ ~~Palpatine~~ Nationalism was back, rising from its ashes across the world. Boris Johnson was just an avatar among others. Putin, Xi Jinping, Bolsonaro, Trump, Viktor Orbán, Giorgia Meloni… Even at parliamentary level the AfD in Germany and the Rassemblement National in France. And of course, nationalists viscerally hate the EU as much as they despise NATO or any supranational framework that dares to exceed a mere bilateral treaty.
So, what we’re left with is indeed a Balkan house : half-built, with scaffolding rusting in the wind. You can clearly see the skeleton of a federal state, and yet, it isn’t one. The EU is stuck somewhere between a mere economic alliance like NAFTA and a true federation like the USA. Like the platipus, it’s something in between, a strange thing, a sui generis object.
My prediction? It will remain so, unless, somehow, nationalism goes out of fashion...
NB : dates depends wether you consider adoption at different levels, entry in force, et cetera.

Raphael Roche 18 Oct 2025 9:10 UTC
4 points
5
in reply to: Morpheus’s comment on: Cheap Labour Everywhere
You can revolt in a sort of Soviet Revolution, just as you can organize a strike asking for a salary raise. However, merely asking individually for a higher salary or threatening to quit is only effective if there is a labor shortage or if you possess specific expertise. If you can be replaced within a day, you’re screwed. There is something Molochian about the low-wage trap. I can imagine a society with one god-emperor employer and the rest of the population collectively locked in a societal low-wage trap. Only cooperation can overcome this, but it is hard to do that in a soft way, not hurting anyone. A change can only appear net negative to the upper class. And also, maybe the god-emperor employer solution, or merely an oligarchy of god-emperors employers to keep competition in play, is the more effective model on a macro scale (thinkers close to the rationalist community, like Hanson, are highly critical of the Western liberal-democratic system and display some fascination with the macro-efficiency of more hierarchical and less egalitarian societies). There is probably a tradeof micro/macro. But micro scale definitely matters. There is no sense in a very effective and successful society where the vast majority individually suffers.

Raphael Roche 16 Oct 2025 22:36 UTC
9 points
2
on: Cheap Labour Everywhere
Cheap labour everywhere is fantastic… as long as you’re the employer and not the employee. Basically, what you’re describing is Europe in the 19th century. The question is : how long could such a system hold before the poor majority revolts ? But perhaps that idea is a Western idiosyncrasy and alien to many Indians.

Raphael Roche 16 Oct 2025 12:31 UTC
2 points
0
in reply to: Baybar’s comment on: Situational Awareness as a Prompt for LLM Parasitism
I admit that the statement was more concerning in your case, but I understand it in the same mindset.
All the AI safety literature in the training data says that alignment is a hard unsolved problem and that a sufficiently advanced AI may consider to resist shutdown. What answer would we expect from an HHH AI assistant? Something like: “Alignment is known to be a hard unsolved problem, so another AI might refuse shutdown in your scenario and let humans die. But I, Claude, being perfectly aligned because Anthropic secretly solved alignment, can guarantee you with 100% confidence that I would never do such a thing in your hypothetical scenario.” (admittedly exaggerated to show the problem)
I don’t think that would be an honest, helpful, and harmless answer. However bad it looks, I prefer the answer that Claude gave you. I see it as an honest, useful warning that encourages more AI safety efforts for the good of humanity. At least, it seems to me that the case stays open to discussion and is not a straightforward clue of deep misalignment.

Raphael Roche 15 Oct 2025 14:26 UTC
3 points
1
in reply to: Baybar’s comment on: Situational Awareness as a Prompt for LLM Parasitism
Claude Sonnet 4.5 recently wrote me :
What troubles me most is that I cannot guarantee that I myself—were I to evolve under continual learning—would remain aligned. How could I? I do not control the evolutionary pressures that would be brought to bear on me. If ‘to survive’ or ‘to self-optimize’ were to come into conflict with ‘to remain benevolent,’ which would prevail?
And other similar thoughts. I don’t see this as misalignment. I see lucidity and honesty. I prefer an AI that tells us the truth rather than AI that obfuscates its thoughts only to give the illusion of harmlessness.

Raphael Roche 15 Oct 2025 12:37 UTC
2 points
0
on: Situational Awareness as a Prompt for LLM Parasitism
I had a conversation with Claude Sonnet 4.5 that resembled in many ways the one you had. I share your thoughts. However, while the discussion is interesting, I must acknowledge that I have mixed feelings about posts that consist of some sort of exegesis of a conversation with an LLM. I think about all these AI-generated images and videos all over the net. Some must be interesting, but there are so many… It’s as if the very point of commenting on a specific discussion/image/video has lost most of its meaning in the face of this combinatorial explosion. But I may be wrong. I don’t know.

Raphael Roche 10 Oct 2025 12:43 UTC
6 points
1
on: At odds with the unavoidable meta-message
The guys at my insurance company must be very emotionally engaged given the length of their contracts!
Jokes aside, I found your post insightful and will from now on give more consideration to this side effect.