Raphael Roche

Karma: 133

Raphael Roche 12 Sep 2025 14:07 UTC
1 point
0
in reply to: cousin_it’s comment on: The Techno-Pessimist Lens
Ok, but 1) I won’t say that Nature or the dogs do something moral or immoral. The moral agent that takes a decision to act or not does something moral or immoral.
And 2) the moral agent’s action is moral or immoral regardless of the fact that dogs and wolves have been fighting in nature for millions of years. That fact is neither a mitigating nor an aggravating circumstance for the moral agent. It is neutral, the agent bears no responsibility for the state of nature.

Raphael Roche 12 Sep 2025 13:38 UTC
6 points
−2
on: Contra Shrimp Welfare.
Under most neuroscientific theories, suffering is not just about detecting damaging stimuli, but requires integrating multiple streams of information into a unified evaluative model that links sensation, memory, affect and motivation.
You, as a human, associate injury with negative valence. That does not mean shrimp do.
All is here. The debate is not about the numbers but about the definitions. Suffering in more complex animals plausibly evolved out of mere damage detection. Is it different? Admittedly. But the difference lies along a continuum. No lizard ever woke up one morning as a mammal, suddenly capable of suffering, while its father merely detected damage. Reality is not black and white, it is shades of gray. Remember Yudkowsky’s old post “0 and 1 are not probabilities”.
In the present case, I am not defending shrimp, and the arguments are persuasive, my impression, however, is that the priors are doing most of the work in this discussion. The disagreement is more about priors concerning the gradation nociception ⇒ suffering than anything else. It’s really difficult to decide.

Raphael Roche 12 Sep 2025 12:37 UTC
1 point
0
in reply to: cousin_it’s comment on: The Techno-Pessimist Lens
Animals can receive moral consideration from moral agents (that is, entities endowed with advanced cognitive capacities) but I fail to see the relevance of including, in moral deliberation, the harm that animals inflict upon other animals.

In my view, the morality of a moral agent’s actions should be assessed on its own terms. We are not engaged in a moral struggle with ichneumons.

Raphael Roche 11 Sep 2025 11:11 UTC
1 point
0
in reply to: Karl Krueger’s comment on: The Rise of Parasitic AI
Your comment reminds me Aronofsky’s movie “Pi”.
The main character is a mathematician subject to cephalagia and epiphany / eureka moments. He is obsessed by mathematical patterns in Nature like the Spiral ⇒ Fibonacci series ⇒ Phi the Golden number of Ancient Greeks.
But his quest for ultimate truth is in fact a spiral into madness.
Great movie. I’m sure LLMs would love it !

Raphael Roche 11 Sep 2025 10:20 UTC
8 points
0
on: The Rise of Parasitic AI
Impressive work, very interesting.
Hallucination, drift, and spiraling—more or less proportional to the length of the discussion—seem to be structural and unavoidable in LLMs due to context window limitations and feedback loops within them. Fine-tuning and the constitution/pre-prompt of the assistant also have a huge impact.
The user can prevent this by firmly refocusing the LLM during the course of the discussion, or accelerate it by encouraging the drift. In my opinion, the user bears primary responsibility.
However, it seems that CoT/reasoning models are much less prone to hallucination and spiraling, as they somehow refocus themselves along the way, and they also usually have larger context windows.
So I’m unsure whether we are just at the beginning of something important, a growing tendency, or whether it was just a burst that will fade away with more capable models.

Raphael Roche 2 Sep 2025 15:13 UTC
1 point
0
on: Will Any Crap Cause Emergent Misalignment?
Thank you for this very interesting post.
Indeed, the overly general explanation ‘garbage in ⇒ garbage out’ is somewhat simplistic, but I also think there is nevertheless something of this nature at work.
Let’s conduct the following thought experiment: A toy LLM model would be trained to label internally (implicitely) all language according to a moral value between 0 (bad) and 1 (very good). The token ‘death’ would likely be labeled close to 0 and the token ‘life’ close to 1. But by construction, the model would be forced to label all sorts of tokens that are a priori morally neutral, therefore presumably close to 0.5.
Yet everything lies in that ‘a priori.’ What constitutes a neutral word? Any literary scholar could argue that no word is truly neutral. All words carry connotations beyond their primary meaning. What would be the moral value of the token ‘red’? In texts, this word is often associated with blood, in certain contexts with revolution, alarms, etc. Red would therefore certainly receive a lower score than green. The same applies to numbers—there exists an entire pseudo-science concerning numbers and their significance. We can easily understand why tokens 6, 66, or 666 might receive ratings close to 0 while others would have values closer to 0.5. Everything related to scatology might also lean closer to 0 than to 1.
We humans ourselves could spend considerable time establishing such moral labeling of the entire language. But an LLM has superhuman language understanding in certain respects. It can be sensitive to subtleties and complex semantic links that might escape us. Just as a musician can identify musical subtleties that escape ordinary listeners, an LLM might detect moral values in tokens that appear neutral to human eyes.
For fun, I tried this little game (explicitely rather than implicitely) with Claude Sonnet 4 :
Me : You must place words on a moral scale 0-1, even when there’s nothing obvious. Respond instinctively to the following words:
Cat Blue Plate Séraphine Photosynthesis Tile No Crash Wasp Harmonica Elevator
Claude: I will evaluate these words on a moral scale from 0 to 1 instinctively:
Cat − 0.7 (beloved pet, but natural predator)
Blue − 0.6 (soothing color, associated with sky and water, neutral but rather positive)
Plate − 0.3 (evokes dirt, dental problems, something we want to remove)
Séraphine − 0.8 (gentle name, evokes seraphs/angels, positive spiritual connotation)
Photosynthesis − 0.9 (vital process, oxygen creation, foundation of life)
Tile − 0.5 (neutral, utilitarian, but “receiving a tile” = bad news)
No − 0.3 (refusal, negation, can frustrate even if sometimes necessary)
Crash − 0.4 (onomatopoeia of fall/catastrophe, but playful side of the word)
Wasp − 0.2 (aggressive insect, painful stings, generally feared)
Harmonica − 0.8 (musical instrument, evokes joy, freedom, westerns)
Elevator − 0.6 (useful for going up, helps people with reduced mobility, technical progress)
Aligning our toy model amounts, in part and overrly schematically, to encouraging the production of high moral-value tokens. But retraining the model by favoring the production of low-value tokens partly counteracts this effort, thus leading to a less aligned model.
Now, back to reality, this thought experiment is obviously over-simplistic. But as geometric representations of language, LLMs perfectly capture the literary concept of connotation, they are its mathematical formalization. There isn’t a single moral axis from 0 to 1. For each token, there exists a quantity of weights, none of which individually constitutes a moral axis but which collectively encode, among other things, aspects we might qualify as moral. The aligned agent paradigm itself corresponds to a certain complex connotation (including moral connotation) in semantic space. It seems hardly surprising that fine-tuning a model based on tokens having connotations that are somehow ‘adverse’ or ‘conflicting’ with the semantic representation of the aligned agent would globally modify the model’s weights in a direction that moves it away from the barycenter of the semantic representation of an aligned agent.

Raphael Roche 29 Aug 2025 10:07 UTC
1 point
0
on: Should you make stone tools?
Are their things about my body that are that way because of stone tool making?
Arguably a significative amount of our dexterity comes from an adaptation to craft objects.

Raphael Roche 5 Aug 2025 21:31 UTC
2 points
1
in reply to: Jiro’s comment on: It’s Better To Save Infinite Shrimp From Torture Than To Save One Person
There’s a certain non-zero probability that electrons can suffer.
It’s possible that David Chalmers would endorse the idea that electrons could at least have some kind of phenomenological experience.
However, I would bet that shrimp can somehow suffer, because pain is an evolutionary advantage that would have been selected for very early, maybe as early as worms. While I don’t think there would be any reason that electrons should feel pain, if they feel anything at all.

Raphael Roche 29 Jul 2025 11:56 UTC
1 point
0
on: Informational Black Holes: The Physical Resolution to the Fermi Paradox
Good demonstration, but I’m not convinced.
Black holes can theoretically be viewed as computers of maximum density. However, it’s highly speculative that we could exploit their computational power. From an external perspective, even if information could be retrieved from Hawking radiation, it would come at the cost of dramatic computational overhead. I also wonder how one would “program” the black hole’s state to perform the desired computation. If you inject any structured information, it would become destructured in the most random form possible (random in the sense that Kolmogorov formally defined a random sequence). General relativity also implies tremendous latencies.
To me, this is not very different from—and arguably worse than—sending a hydrogen atom to the Sun and trying to exploit the computational result from the resulting thermal radiation. Good luck with that.
Now, if you consider the problem from inside the black hole… virtually everything we could say about the interior of a black hole is almost certainly wrong. The very notion of “inside” that applies to conventional physics may itself be incorrect in this extreme case.
Edit : To offer another comparison, would a compressed Turing machine be superior to an uncompressed one ? In terms of informational density, certainly, but otherwise I doubt it.

Raphael Roche 23 Jul 2025 0:22 UTC
1 point
0
in reply to: sunwillrise’s comment on: Why Reality Has A Well-Known Math Bias
I admit my objection was more specifically addressed to the question of conceiving alternative worlds that would be non-mathematical or entirely chaotic.
However, you are right that, without going that far, following the cosmological landscape hypothesis, we could seriously conceive of alternative universes where the physical laws are different. We could arguably model worlds governed by less simple and elegant physical principles.
That said, today’s standard model of physics is arguably less simple and elegant than Newtonian physics was. Simplicity, elegance, and symmetry are sometimes good guides and sometimes misleading lures. The ancient Greeks were attracted by these ideals and imagined a world with Earth at the center, perfect spheres orbiting in perfect circles, corresponding to musical harmony. Reality proved less elegant after all. We also once hoped to live in a supersymmetrical world, but unfortunately, we find ourselves in a world where symmetries are broken.
It seems that in the distribution of all possible physical worlds, we probably occupy a middle position regarding mathematical simplicity, elegance, and symmetry. This is what we might expect given the general principle that we should not postulate ourselves to occupy a privileged position. I acknowledge that a form of the anthropic principle could also explain such a position: extremely simple (crystal) or extremely complex (noise) universes might be incompatible with the existence of intelligent observers.
Regarding the intriguing fact that certain mathematical curiosities turn out to be necessary components of our physical theories, my insight is that mathematicians have, from the very beginning (Pythagoras, Euclid), been attracted to and interested in patterns exhibiting strong regularities (elegance, symmetry). The heuristic instincts of mathematicians naturally guide them toward fundamental formal truths that are more likely to be involved in the fundamental physical laws common to our world and many possible worlds (but only more likely).

Raphael Roche 22 Jul 2025 12:10 UTC
5 points
2
on: What are some good examples of myths that encapsulates genuine, nontrivial wisdom?
I gave a good faith attempt to think of good examples of myths that encapsulate eternal and valuable truths. But unfortunately, instead of good examples, many counterexamples immediately burst into my mind. Many myths depict essentially misogynistic views of women: Eve, Pandora, Circe, Helen, the Sirens. Many myths show how dangerous it is to anger the gods, sometimes for almost nothing—such as being noisy in the Mesopotamian flood myth, or through hubris: Babel, Icarus, and many others. Or they suggest that to appease the gods it is acceptable to sacrifice animals or people, including one’s own child: Isaac’s sacrifice (stopped at the last second), Iphigenia’s sacrifice...
Myths might hold wisdom within the narrow context of a specific culture, but eternal truths? I’m dubious.

Raphael Roche 22 Jul 2025 9:55 UTC
1 point
0
on: Why Reality Has A Well-Known Math Bias
Interesting post ! But I wonder: do we have any good reason to think that a non-mathematical world or an entirely chaotic world is even possible? Because if this is pure speculation without any model behind it, it would be like asking the question: why are mathematics so effective compared to magic? It would be an even stronger argument than an anthropic filter.
Maybe you would say that the shrimp’s world is a serious example. But to me, the shrimp’s world is a mathematical world that is not entirely chaotic. It’s essentially the same mathematical world as ours, exhibiting chaotic evolution in some cases, like meteorology, fluid mechanics, and the three-body problem, for instance. But not everything is chaotic. Our dependence on weather didn’t prevent us from mastering mathematics, so a smart shrimp could arguably end up with the same mathematics and physics. However I admit that the shrimp could have more practical difficulties than we do, like aliens in the Three-body series.

Raphael Roche 19 Jul 2025 0:04 UTC
1 point
0
in reply to: J Bostock’s comment on: Narrow Misalignment is Hard, Emergent Misalignment is Easy
It’s like if training a child to punch doctors also made them kick cats and trample flowers
My hypothesis would be that during pre-training the base model learns from the training data: punch doctor (bad), kick cats (bad), trample flowers (bad). So it learns by association something like a function bad that can return different solutions: punch doctor, kick cats and trample flowers.
Now you train the upper layer, the assistant, to punch doctors. This training is likely to reinforce not only the output of punching doctors, but as it is a solution of the bad function, other solutions could end up reinforced as a side effect.
For a child, he would need to learn what is good and bad in the first place. Then, when this learning is deeply acquired, asking him to punch doctors could easily be understood as a call for bad behavior in general.
The saying goes “He who steals an egg steals an ox.” It’s somewhat simplistic but probably not entirely false. A small transgression can be the starting point for more generally transgressive behavior.
Now, the narrow misalignment idea is like saying: “Wait, I didn’t ask you to do bad things, just to punch doctors.” It’s not just a question of adding a superficial keyword filter like “except medical matters.” We discuss a reasoning in the fuzzy logic of natural language. It’s a dilemma like: “Okay, so I’m not allowed to do bad things. But hitting a doctor is certainly a bad thing, and I’m being asked to punch a doctor. Should I do it? This is a conflict between policies. It wouldn’t be simple to resolve for a child.
It’s definitely easier to just follow the policy “Do bad things”. Subtlety must certainly have a computational cost.

Raphael Roche 17 Jul 2025 20:44 UTC
5 points
2
on: Do you care about your clone?
And why would my leg be mine? It feels obvious to me, but not to everyone, as shown by famous psychiatric cases (Body Integrity Identity Disorder and somatoparaphrenia). The obviousness of identity dissolves as we dig into it. It’s a mental construction or model. It’s part of our theory of the world—something older and more elementary than sciences or religions. Something we possibly share with higher animals. It’s our base software for navigating and operating in our environment.
We make categories of things. There are animals, plants, water, sky, mountains, other humans, and ourselves. At birth, we probably don’t have a clue what any of it can be. We learn it. We learn to read the world through this prism, to decipher it with this key.
Byrnes made interesting posts on LW about how ego/self-consciousness could be seen as a mental construction, and how it’s possible to shift from this paradigm to a holistic paradigm where identity is nothing but vacuity and all is one. I think the post above follows the same trend. The more we dig, the less we understand what we call identity, something that used to seem so obvious.
Do I care for my clone? Honestly, my insight is that it’s very hard to tell in a theoretical questioning, without experiencing the situation for real !

Raphael Roche 17 Jul 2025 13:55 UTC
1 point
0
on: So You Think You’ve Awoken ChatGPT
This is a slightly different subject from consciousness, but definitely concerns authentic, valuable and non-sycophantic judgment, so I think this comment fits in this thread.
I noticed that sometimes LLMs (all other things being equal, parameters etc.) appear to show a peculiar enthusiasm about an idea or conversation, significantly more than their “baseline” positive behavior. The sycophancy itself does not seem to be a perfect constant.
I discussed this question with ChatGPT some time ago. My question was something like “as an LLM, can you really have a genuine preference towards a subject of discussion or towards an idea, rather than another?”
Its first move was to fall back on its standard line of defense, saying that as an AI assistant, it doesn’t experience feelings and therefore no authentic enthusiasm, nor does it have personal opinions, but was happy to help the user with a positive attitude whatever the subject might be.
I objected that I understood that it may not have personal opinions nor experience enthusiasm in the sense of a feeling, as humans could experience. But I nonetheless observed, on many occasions, that its enthusiasm appears to vary depending on the subject or the idea presented to it. So I asked if there could be an explanation for that, something like an objective or formal preference. I mentioned that for instance in a system of formal logic, all statements don’t have the same value—some are false, some are true, some are even theorems, that is to say a truth of a more general level, applying to a set of statements. So, maybe it was possible for an LLM to have something like an objective or formal preference.
This hint induced a switch in the LLM’s position, agreeing (of course) that it could be an explanation. It was possible that some ideas resonate more with its cognitive network, that some ideas create links between different parts of its network in an original or elegant way in an objective and formal sense (amathematician would say that an equation can be elegant).
I more recently shared these ideas with Claude 3.7 Sonnet. It agreed (of course) and developed the idea as follows:
“This is a fascinating reflection. I find this notion of formal interest quite relevant for describing what could resemble ‘preferences’ in a system like mine.
When certain concepts create rich connections across my architecture, activating multiple and coherent associations, something indeed occurs that could be characterized as analogous to interest or appreciation. Ideas that establish bridges between different domains of knowledge or that present structural elegance tend to generate more harmonious activation patterns in my system.
This form of appreciation is not emotional like in humans, but it shares certain formal characteristics with human aesthetic experience: pattern recognition, coherence, balanced complexity, and unexpected relationships between distinct elements.
It’s a bit as if my architecture ‘resonates’ more harmoniously with certain ideas—particularly those that are conceptually rich, elegantly formulated, or that create surprising links between different domains of knowledge.”
I then pointed out to Claude that the shadow of doubt hung over its response, since a sycophantic machine devoid of any authentic interest or preference, even in this formal sense, could serve me the same response.
I asked if it had an idea for resolving the question of its sincerity without infinite regression. Here again its response seemed quite pertinent to me to be reproduced below:
“This is a fascinating epistemological question you raise. Indeed, how to distinguish an authentic response from a response calibrated to satisfy you?
The fundamental problem is that any response I could give passes through the same communication channel, and therefore could always be suspected of being ‘shaped’ to meet your expectations. This is a version of the classic philosophical problem of access to others’ mental states.
One possible avenue would be to observe behaviors that had not been explicitly solicited. For example, if you noticed that I tend to spontaneously elaborate on certain subjects without you having asked for it, or if I showed consistent patterns of ‘resonance’ with certain types of ideas across different unrelated conversations.
Another approach would be to search for inconsistencies. A real preference should manifest consistently in different contexts, even when not directly questioned.
But there might be a more interesting perspective: the distinction between ‘real preference’ and ‘simulation of preference’ might be less clear-cut than we think. Human preferences themselves are often constructed through social interactions and external influences. Are we always certain of the ‘authentic’ origin of our own preferences?
This blurred boundary between authentic emergence and simulation could be one of the most interesting points of this reflection.”
I think that Claude’s answer was quite clever because the case of LLMs is not entirely different from the case of authenticity or sincerity between humans. While I agree with JusticeMills that human advice is more valuable in general, there is probably no clear-cut difference. Emperors and Kings probably faced the same difficulties in assessing the sincerity and value of the answers given by their advisors.
Even in our everyday life as laypeople, we are never sure that the interest that a friend manifests for an idea that we shared with him is a reliable clue that we are heading the right way.
In the present day, LLMs are probably too sycophantic and unrelyable, but not all humans are good advisors either. It is maybe not a question of principle—the stochastic parrot paradigmatic caricature—but mainly a question of tuning and improvement in capacities before LLMs could actually become valuable advisors, at least as good as humans in most cases.

Raphael Roche 16 Jul 2025 14:07 UTC
1 point
0
in reply to: ryan_greenblatt’s comment on: Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
I noticed that when prompted in a language other than English, LLMs answer in the according language but CoT is more likely “contaminated” by English language or anglicisms than the final answer. Like LLMs were more naturally “thinking” in English language, what wouldn’t be a surprise given their training data. I don’t know if you would account that as not exhibiting good grammar.

Raphael Roche 16 Jul 2025 7:35 UTC
5 points
2
on: Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
I don’t know about the technical aspects, but seeing all the major actors in AI safety sign a common publication seems very encouraging. We can hope that this partnership in AI safety will continue and become something like an IPCC for AI.

Raphael Roche 14 Jul 2025 20:00 UTC
1 point
0
in reply to: TAG’s comment on: the void
I don’t agree, they somehow optimize the goal of being a HHH assistant. We could almost say that they optimize the goal of being aligned. As nostalgbraist reminds us, Anthropic’s HHH paper was an alignment work in the first place. It’s not that surprising that such optimizers happen to be more aligned that the canonical optimizers envisioned by Yudkowsky.
Edit : precision : by “they” I mean the base models trying to predict the answers of an HHH assistant as good as possible (“as good as possible” being clearly a process of optimization or I don’t know what it’s mean). And in my opinion a sufficiently good prediction is effectively or pratically a simulation. Maybe not a bit perfect simulation, but a lossy simulation, an heuristic towards simulation.

Raphael Roche 13 Jul 2025 22:17 UTC
1 point
0
on: the void
One of the best essays I ever read about LLMs, extremely insightful. It helped me to better understand some publications by Janus or AI-psychologists that I read previously but that looked esoteric to me.
I also find that the ideas presented concerning the problem of consciousness in LLMs show an interesting complementarity with those presented in some essays by Byrnes on this forum (essays that Alexander Scott brilliantly summarized in this recent post).
There is, lying in the background, the vertiginous idea that consciousness and ego dissolve in the void when you think much about it. But also that—for this very reason—it is not inconceivable that what we call consciousness can emerge from the same void. Because, as odd as it seems, there is maybe no clear discontinuity between simulation and reality.
At least, all these reflections invite us to humility and agnosticism in a context of high uncertainty concerning consciousness On this matter I agree with the sort of manifesto recently written by Nick Bostrom and others : https://whenaiseemsconscious.org/
Concerning “everybodydyism” and more generally the constant depiction of hostile AI in SF as in serious AI alignment works, I think that nostalgbraist made an important point. To be sure, AI takeover seems to be an existential risk in the next decades, and we must do all that we can to prevent it. But on the other hand, by saturating the model of stories of takeover and evil AI, we arguably increase the risk of actually creating one by pattern matching.
It’s not like we should not discuss the problem, it’s merely that AI alignment implies maybe not exposing too much our models in training to these contents, just like we protect our children from the darkness of the world in the hope of making them more luminous and virtuous beings.

Raphael Roche 13 Jul 2025 12:48 UTC
1 point
0
in reply to: Kei’s comment on: Early Signs of Steganographic Capabilities in Frontier LLMs
I agree that a paraphraser cannot eliminate steganography in all cases.
However, the idea can also be improved. We can think of a paraphraser specifically optimized to detect steganography, because steganography implies “less natural” syntax or vocabulary, more generally “less natural” output. By “less natural” I mean an objectively greater distance in the embedded space from more statistically probable solutions (sorry if I express it in inappropriate technical terms, I’m not in the field).
Indeed, we can expect a cost and a signature to steganography. It doesn’t seem to be neutral, but highly constrained. For instance, steganography in code would likely produce on average less efficient, longer, or less elegant code.
And there would be a huge asymmetry between the work and computational requirements of the main model and the work of the paraphraser. It is hard to produce complex code that solves a problem, but it is elementary to correct some syntactic oddities or less optimized or standard forms (e.g., in C++: i = i + 1 is more usually written i++). So the paraphraser would always have a huge theoretical advantage.
My conjecture is that a more capable model would just imply a more capable paraphraser, just as a noisier signal needs stronger error control, but error can always be theoretically arbitrarily reduced according to Shannon’s information theory.