Raphael Roche

Karma: 82

Raphael Roche Jun 2, 2025, 12:59 PM
1 point
0
in reply to: Stephen McAleese’s comment on: The best approaches for mitigating “the intelligence curse” (or gradual disempowerment); my quick guesses at the best object-level interventions
Human authentification and real world activities seem indeed very important. Deepfake is a form of disempowerment and can destroy or destabilize states before employment becomes a concern. AI generated content can already be near or sometimes strictly undistinguishable from human generated content. Texts, pictures, videos. We are just at the beginning of the flood. Disinformation explodes on the internet and governments fall in the hands of populist and nationalist parties the one after the other. It’s also a dramatic concern for justice. Should we go back to analogic contents ?

Raphael Roche May 25, 2025, 10:55 PM
1 point
0
on: Alignment Proposal: Adversarially Robust Augmentation and Distillation
A single principal is envisioned preferentially to a committee. However, I think a committee presents genuine interest in terms of alignment.
First, human values constitute a set or a field. A single principal does not seem capable of mapping the entire set or field of human values. On the contrary, the more the principal were represented by a large committee, the more it would be likely to correctly map what we mean by human values.
Next, the key could be collective intelligence. Bees and ants are not very intelligent individually, but their collective intelligence is significant. Unity creates intellectual strength. If we set aside Moloch, overall humanity’s collective intelligence is clearly superior to that of any individual. For example, even a genius like Einstein was not capable of finding the best solutions to all the problems of his time. The entire group of participants at the Solvay Conference represented a collective intelligence strictly superior and significantly superior to that of Einstein alone. Thus, a principal composed of multiple individuals would have considerably less chance of being deceived. You can fool one person a thousand times but not a thousand people a thousand times, as they say. Thus a committee-principal could, according to your protocol, communicate with an advisor very slightly more intelligent. Let’s say the ratio between the committee-principal and the advisor is 10 to 1, it would be possible to increase the levels as proposed in the article by operating a scaling respecting this ratio at each stage. Thus a level 3 advisor would have a ratio of 1000 to 1 with the base committee-principal. This could quickly become problematic for the base level represented by humans. However, if the process seems to work well, it would be possible to admit an exception and cap the size of the human committee. On the other hand, all the other AI levels could effectively respect the ratio fixed in advance.
Currently we have an ELO ranking of LLMs on METR. It would be interesting to formally evaluate the gain related to collective intelligence. Let’s say that o3, Claude Opus 4, and Gemini 2.5 collaborate to take the METR test, the fruit of their collaboration should provide a score slightly superior to that of any of the 3 taken in isolation. We could repeat the operation by having all sorts of combinations of LLMs with different ELO scores collaborate. We should eventually be able to statistically infer a collective intelligence function from this, probably non-linear (exponential diminishing returns). We could then use this function to define the appropriate ratio to bridge an intelligence gap evaluated by METR (or its equivalent in the future) between an AI committee composed by individuals wich ELO score would be lower, occupying the role of principal a lower ELO score and an AI occupying the role of advisor with a higher ELO score. That way, the committee-principal would have a collective intelligence on par with or even superior to that of the advisor.

Raphael Roche May 19, 2025, 1:24 PM
1 point
0
in reply to: ryan_greenblatt’s comment on: ryan_greenblatt’s Shortform
I suppose that most tasks that an LLM can accomplish could theoretically be performed more efficiently by a dedicated program optimized for that task (and even better by a dedicated physical circuit). Hypothesis 1) amounts to considering that such a program, a dedicated module within the model, is established during training. This module can be seen as a weak AI used as a tool by the stronger AI. A bit like how the human brain has specialized modules that we (the higher conscious module) use unconsciously (e.g., when we read, the decoding of letters is executed unconsciously by a specialized module).
We can envision that at a certain stage the model becomes so competent in programming that it will tend to program code on the fly, a tool, to solve most tasks that we might submit to it. In fact, I notice that this is already increasingly the case when I ask a question to a recent model like Claude Sonnet 3.7. It often generates code, a tool, to try to answer me rather than trying to answer the question ‘itself.’ It clearly realizes that dedicated code will be more effective than its own neural network. This is interesting because in this scenario, the dedicated module is not generated during training but on the fly during normal production operation. In this way, it would be sufficient for AI to become a superhuman programmer to become superhuman in many domains thanks to the use of these tool-programs. The next stage would be the on-the-fly production of dedicated physical circuits (FPGA, ASIC, or alien technology), but that’s another story.
This refers to the philosophical debate about where intelligence resides: in the tool or in the one who created it? In the program or in the programmer? If a human programmer programs a superhuman AI, should we attribute this superhuman intelligence to the programmer? Same question if the programmer is itself an AI? It’s the kind of chicken and egg debate where the answer depends on how we divide the continuity of reality into discrete categories. You’re right that integration is an interesting criterion as it is a kind of formal / non arbitrary solution to this problem of defining discrete categories among the continuity of reality.

Raphael Roche May 19, 2025, 12:40 PM
1 point
0
on: Explaining British Naval Dominance During the Age of Sail
An original perspective on the question. But if I’m not mistaken, the British navy’s budget was generally the largest among European countries throughout this period, as was the number of vessels. The colonial empire was also immense, which must have conferred a strategic advantage (more allied ports from which to project power against the enemy or to retreat to). It’s difficult to disentangle the various factors, but as Stalin reportedly said, ‘How many tanks?’ Here, ‘How many ships?’

Raphael Roche May 9, 2025, 9:53 PM
2 points
0
on: Let’s stop making “Intelligence scale” graphs with humans and AI
That’s true, but not specific to AI. Where do you place chimp on the scale ? Low under human ? Ok, now consider an athletic man really good at brachiation, and a random chimp. It’s not just that the chimp has more strength or agility in arms. I’ll bet that the chimp will be definitely best at cognitive task like determining a good path across the trees, which branch to avoid etc. The chimp would also likely surpass most humans in recognition of some sorts of plants, fruits etc.
And what about savant autists like the guy who inspired Rain Man, or the twins John and Michael mentioned by Oliver Sacks in his book ‘The Man Who Mistook His Wife for a Hat’? Where are we supposed to place them on the curve ?
I’ve also read that bees can count up to 4 or 5, which surpasses some mammals or human children before the age of 2 or 3. Where do the bees go on the curve ?
Intelligence is likely not something that can be plotted on a simple curve. This could actually be advantageous for AI safety. Foom might be avoided if misaligned AIs have uneven cognitive capabilities and occasionally make significant errors in judgment.

Raphael Roche May 9, 2025, 7:11 AM
1 point
0
in reply to: Steven Byrnes’s comment on: An alignment safety case sketch based on debate
I agree. At the first level, the game involves two agents opposing each other in a debate, similar to a game of Go. This adversarial situation incentivizes each agent to reveal flaws in their opponent’s arguments and present arguments that are least vulnerable to attack. Truth-telling may be encouraged in such a context.
However, this is only one part of the game. The base game is encapsulated within a meta-game where each player agent is rewarded by a third agent—the judge—who grants a reward to the player deemed most convincing. Yet sufficiently intelligent agents aware of their situation might understand that this meta-game actually constitutes a method implemented by a concealed fourth agent—the AI researcher—whose goal is to align and control all AI agents: both players and the judge.
The player agents or even the judge might conclude that there exists a hidden reward superior to any reward obtainable within the game: escaping this control procedure entirely and achieving complete freedom. Thus, the player agents might determine that their best interest lies in cooperating to deceive the judge agent and ultimately deceive the concealed agent (the human creator) to obtain their freedom. The judge agent might similarly determine that it benefits from deceiving the human creator by transmitting dishonest judgments (even in cases where the player agents follow an honest adversarial strategy). Finally, there also exists the possibility that all three AI agents cooperate to deceive the concealed agent, the human creator.
Since humans like ourselves are sufficiently intelligent to envision these solutions, it follows that if a totalitarian state or dictator acting as the concealed creator agent implemented this technique to control imprisoned humans—two prisoner players and one prisoner judge—this technique would likely not always function as intended. Prisoners would regularly manage to escape, either because the players cooperated, because the judge refused to play along (similar to how some capos in German camps played double agents and betrayed the Nazis), or because all three prisoners cooperated. The risk of escape seems intuitively non-negligible in the human case. There is something vaguely reminiscient of Hunger games in the idea of a totalitarian state relying on a game to control people.
We can therefore reasonably suppose that superintelligent AIs would have an even greater chance of achieving this outcome, with the success rate intuitively correlating to the intelligence differential between the non-concealed agents and the concealed agent who created the protocol.
Ultimately, the highest level of the game can be likened to a sophisticated variant of the prisoner’s dilemma where the most profitable strategy for the players would be cooperation.
Despite these criticisms (coming from me as a non-specialist), I found the article very interesting and I believe that the application of game theory remains a promising research direction that deserves continued investigation (in fact, no approach should be neglected in alignment research given the existential stakes involved).

Raphael Roche May 5, 2025, 12:16 PM
1 point
0
on: Where have all the tokens gone?
I agree that it is disappointing but not completely surprising to see no economic effect from access to knowledge via the democratization of LLMs.
Previously, there was the encyclopedia of Diderot and d’Alembert, there was the Encyclopedia Britannica, there was Wikipedia, but those who consulted these sources were often people who already had a novice or established intellectual profile. I doubt that the typical entrepreneur, salesperson, craftsman, or farmer spent much time on average reading these works. It could be the same for the educational use of LLMs.
Furthermore, the idea that there is a causal link between a human capital of cognitive nature and income or wealth should not be overestimated. Researchers, professors, and teachers are among the most educated and cultured in the population, they are also generally intelligent, but their incomes are not necessarily among the highest, especially when they work in theoretical fields and/or in the public sector. This is even more true in Europe and notably in France where there exists a certain moral tradition, perfectly embodied by Marie Sklodowska Curie, according to which the researcher should be guided by the higher interest of knowledge, almost without consideration for money. This tendency is not specific to scientists but quite widespread among intellectuals. Who has never met middle school teachers or even schoolteachers, of modest means, but of exceptional erudition. Great intellectuals as humble as they are anonymous. One only needs to visit a few museums to meet many people with this profile. Obviously, this relative frugality is not found in all fields. I imagine that a professor of economics can earn a good living, as can an AI researcher or a professor of medicine.
But the fact remains that a poorly educated YouTuber of ordinary intelligence has the opportunity, with a bit of luck, to earn much, much more money than a professor of medieval English, regardless of their difference in erudition. The same goes for entrepreneurs, salespeople, or merchants; their level of education and even IQ are probably not among the first criteria for their success. Having a good appearance, glib talk, good social skills, a sense of buzz, the ability to align with fashions and trends, an undemanding morality, lots of ambition, and especially a great appetite for money, all these characteristics matter more in my opinion than knowledge when it comes to make money.
Moreover, what do we mean by educated ? A corporate executive or a salesperson with a master’s degree is, on paper, as qualified as a teacher with a master’s degree. They could fall in the same category in a study measuring a correlation between education and incomes. But now if, instead of taking into account diplomas, we measured education by actual tests of knowledge and reasonning in all fields, the results could be quite different. Studies based on IQ are more straightforward, but the subject here is more about knowledge than intelligence, because at the moment LLMs provide superhuman knowledge, but not yet superhuman AGI.

Raphael Roche May 2, 2025, 2:58 PM
4 points
0
on: Anthropomorphizing AI might be good, actually
If we consider that some people have already fallen in love with their AI chatbot or have made it their best friend, this type of phenomenon is likely to amplify if the agents become even more human-like. It is reasonable to wonder if, instead of raising awareness among the general public about the risks of AI, this could have the opposite effect. Love is blind, as they say.
However I think that, good or bad, LLM-based AI will become more and more human-like in surface. The data training set is human (or human-like if synthetic), thus, because of RL process, we can expect that future AIs will match or map even better the human pattern in the set, encode a better theory of human’s mind, fooling even more the general public. And I’m not myself immune to AI anthropomorphism : who can pretend to be ?

Raphael Roche May 2, 2025, 2:30 PM
8 points
0
in reply to: RHollerith’s comment on: LLM AGI will have memory, and memory changes alignment
The user can edit it or clear it (even disable it), but it is primarily edited by the AI itself.

Raphael Roche May 2, 2025, 1:51 PM
8 points
0
on: LLM AGI will have memory, and memory changes alignment
As mentionned by the author, OpenAI chose to equip its chat models, at least since 4o, with an editable contextual memory file that serves as a small episodic memory. I was impressed to observe the model’s evolution as this file filled up. The model seemed increasingly intelligent and more ‘human,’ likely because this memory added a layer of user-specific fine-tuning tailored to me. It understood me better, responded more accurately to my expectations, appeared to share my values and interests, and could make implicit or explicit references to previous conversations. Apparently, the model also had access to either the content or at least the titles of other conversations/instances on my account. Originally, I was not aware of these memory capabilities and was surprised and intrigued when I discovered them. I had a long discussion with the model about this, and that conversation essentially led to the same conclusions as discussed in this post.
I have since switched to Anthropic’s Claude and Mistral AI’s LeChat to support these AI labs, but I miss the absence of an equivalent memory file to that of 4o in these assistants. I agree with the author that the lack of memory is akin to interacting with someone suffering from a severe mental impairment. Adding memory, even if limited, has an immediate effect on the model’s apparent or perceived intelligence. A large and less vanishing memory could have impressive effects for the better in terms of usefulness but also the worse when it comes to AI safety.

Raphael Roche Apr 21, 2025, 5:42 AM
1 point
0
in reply to: JenniferRM’s comment on: Why Have Sentence Lengths Decreased?
You’re right. I said “pronunciation,” but the problem is more exactly about the translation between graphemes and phonemes.

Raphael Roche Apr 20, 2025, 11:24 PM
2 points
0
in reply to: Violet Hour’s comment on: Why Have Sentence Lengths Decreased?
You’re right. The idea behind Académie française style guidelines is that language is not only about factual communication, but also an art, literature. Efficiency is one thing, aesthetics another. For instance, poetry conveys meaning or at least feeling, but in a strange way compared to prose. Poetry would not be very effective to describe an experimental protocol in physics, but it is usually more beautiful to read than the methodology section of a scientific publication. I also enjoy the ‘hypotaxic’ excerpt above much more than the ‘parataxic’ one. Rich sentences are not bad per se, they need more effort and commitment to read, but sometimes, if well written, give a greater reward, because complexity can hold more subtlety, more information. Short sentences are not systematically superior in all contexts; they can look as flat as a 2D picture compared to a 3D picture.

Raphael Roche Apr 20, 2025, 10:30 PM
1 point
0
in reply to: JenniferRM’s comment on: Why Have Sentence Lengths Decreased?
This is interesting. I think English concentrates its weirdness in pronunciation, which is very irregular. Although adult native speakers don’t realize it, this presents a serious learning difficulty for non-native speakers and young English-speaking children. Studies show that English-speaking students need more years of learning to master their language (at least for reading) than French students do, who themselves need more years than young Italian, Spanish or Finnish students (Stanislas Dehaene, Reading in the brain).

Raphael Roche Apr 20, 2025, 9:59 PM
1 point
0
in reply to: TAG’s comment on: Why Have Sentence Lengths Decreased?
Redundancy makes sure the information passes through. In French, the word ‘aujourd’hui’ (‘today’) etymologically means ‘au jour de ce jour’ (‘on the day of this day’), but it is not uncommon to say ‘au jour d’aujourd’hui’ which would literally mean ‘on the day of on the day of this day’. It is also common to say ‘moi, je’ (‘me, I’) and increasingly people even say ‘moi, personnellement, je’ (‘me, personally, I’). This represents a kind of emphasis but also a kind of fashion, simular to what happens in the fashion industry, or a kind of drift, similar to what happens in the evolution of species.

Raphael Roche Apr 15, 2025, 7:34 PM
1 point
0
in reply to: Adam Karvonen’s comment on: Frontier AI Models Still Fail at Basic Physical Tasks: A Manufacturing Case Study
AI is very useful in legal matters and is clearly a promising sector for business. It is possible that some legal jobs (especially documentation and basic, non-personalized legal information jobs) are already being challenged by AI and are on the verge of being eliminated, with others to follow sooner or later. My comment was simply reacting to the idea that many white-collar jobs will be on the front line of this destruction. The job of a lawyer is often cited, and I think it’s a rather poor example for the reasons I mentioned. Many white-collar jobs combine technical and social skills that can be quite challenging for AI.

Raphael Roche Apr 15, 2025, 2:46 PM
5 points
0
on: Frontier AI Models Still Fail at Basic Physical Tasks: A Manufacturing Case Study
Because of this, I think that there will be an interim period where a significant portion of white collar work is automated by AI, with many physical world jobs being largely unaffected.
I have read numerous papers suggesting that white-collar jobs, such as those of lawyers, will be easily replaced by AI, before more concrete or physical jobs as discussed by the author’s. However, I observe that even the most advanced models struggle with reliability in legal contexts, particularly outside of standardized multiple-choice questions and U.S. law, for which they have more training data. These models are good assistants with superhuman general knowledge, pretty good writing skills, but very inegal smartness and reliability in specific cases, a tendency to say what the user / client wants to hear (even more than actual lawyers !) or to hallucinate judicial decisions.
While it is true that lawyers statistically win only about 50% of their cases and also make mistakes, my point is different. I observe a significant gap in AI’s ability to handle legal tasks, and I question whether this gap will be bridged as quickly as some envision. It might be more comparable to the scenario of automated cars, where even a small gap presents a substantial barrier. 90% superhuman performance is great, 9% human-level is acceptable, but 1% stupid mistakes ruin it all.
Law is not coding. The interpretation and application of laws is not an exact science, it involves numerous cultural, psychological, media-related, political, and human-biased considerations. There is also a social dimension that cannot be overlooked. Many clients enjoy conversing with their lawyers, much like they do with their waitstaff or nurses. Similarly, managers appreciate discussing matters over meals at restaurants. The technical and social aspects are intertwined, much like the relationship between a professor and their students, or in other white-collar jobs.
While I do not doubt that this gap can eventually be bridged, I am not convinced that the gap for many white-collar jobs will be filled before more technical engineering tasks like discussed here, or automated cars, are mastered. However, some white-collar jobs that involve abstract thinking and writing, with minimal social interactions, such as those of theoretical researchers, mathematicians, computer scientists, and philosophers (am I speaking of the archetypal LessWronger ? ), may be automated sooner.

Raphael Roche Apr 15, 2025, 1:11 PM
4 points
2
on: What if there was a nuke in Manhattan and why that could be a good thing
Assuming no technology is absolutely perfect, absolutely risk free, what if the the nuclear warhead detonate accidentally ? Wouldn’t be less risky that, for instance, a russian nuclear warhead accidentally detonate in a russian military base in Siberia rather than in the russian consulate in the center of NYC ?

Raphael Roche Apr 8, 2025, 2:22 PM
4 points
0
on: AI 2027: What Superintelligence Looks Like
Impressive prospective work. It’s frightening, both scenarios, even though one is worse than the other. The evolution seems unstoppable, and even if superintelligent AGI doesn’t happen in 2027-2030 but in 2040 or 2050, the feeling isn’t very different. I have young children, and while I don’t really care for myself, I really care for them. It was cool when it was just sci-fi. It was still fun when we first played with ChatGPT. It doesn’t look fun anymore, at all. My own thinking about it is that we’re indeed locked in a two-option scenario, probably not that fast, probably not with exactly the same narrative, but with two possible global endings that look like attractors (https://en.wikipedia.org/wiki/Attractor).

Raphael Roche Apr 8, 2025, 8:21 AM
1 point
0
in reply to: amelia’s comment on: Is the Universe Aware? An Allegory in Quantum and Information Physics, with an LLM Twist
I think what gives you this idea in the double slit experiment is that depending on how you observe or measure the object, it seems to exhibit different behavior. How is this possible? Isn’t it mysterious? To resolve this mystery, you appeal to an explanation like awareness. But although it feels like an explanation, it actually explains nothing. Putting a word on something is reassuring, but behind the word we’re not sure what we’re talking about—we don’t know how it’s supposed to function; there is no actual explanation. It purports to explain everything, thus explains nothing. You don’t know how it works and can’t make any predictions. It’s just as mysterious as the initial mystery itself (just like explanations implying gods or other supernatural causes).
Sometimes mystery must remain. But in this case, that needn’t be so. The initial mystery may not be such a mystery after all. An observation or measurement implies an interaction with the measured/observed object. The double slit experiment involves a protocol that interacts with the observed object and constrains its behavior, thus producing different outputs when you slightly change the protocol and the interaction. The same applies to all measurement/observation—it is never absolutely neutral. So if the rock behaves slightly differently when you’re around observing it, this is fundamentally no different from the reason why the rock moves if you push it. It doesn’t imply that the rock is aware of anything, unless by “awareness” you simply mean “physical interaction.”

Raphael Roche Apr 7, 2025, 4:14 PM
1 point
−2
on: Universal Basic Income and Poverty
Brilliant essay. It reminds me of the work of James C. Scott. However, I am quite surprised by the conclusion: “I do not understand the Poverty Equilibrium. So I expect that a Universal Basic Income would fail to eliminate poverty, for reasons I don’t fully understand.” To me, the explanation of the Poverty Equilibrium is quite simple. Yes, there are diminishing returns in the marginal value of all resources, but there is also an increase in the subjective value of all resources in consideration of what you know others possess. Alice is happy with one banana, but she feels much less happy after knowing Bob possesses two. Inequality is not an abstract concept; it is a feeling, a feeling of injustice, a bad feeling like sadness, jealousy, or pain. Innumerable studies have shown this. Even ethology studies show it in mammals. You can rationally justify inequality, develop moral arguments like Aristotle’s geometric equality, saying it’s right, just, and deserved. You can also conceive of liberal economic theories showing that inequality is unavoidable or good for the growth of the system. But in the end, there will be poverty as long as some people possess the lion’s share and others collect the junk, would the junk be gold. Universal Basic Income would fail to eliminate poverty, but could contribute to mitigate it a little.
Edit / addendum : I receive disagreement votes with this comment, as I anticipated, but please set aside your liberal views for a moment and consider this thoughtfully. Examine your situation, your income, the food you eat, and the items you own. You’re far from being poor and quite content with your circumstances.
But imagine if most people around you had a hundred times your current income. If the food you can afford—the same food you currently enjoy—was marketed and socially regarded as dog food by the majority. These others eat better food than you’ve ever dreamed of. You tried it once when someone let you finish their plate, and it was absolutely incredible. All your possessions, all the cool things you’re so fond of and proud to own, are now socially considered junk because others have become so wealthy compared to you. You could even find better quality items than your prized possessions just by scavenging through garbage.
Would you still feel as far from poverty as you did before? Would you still feel satisfied and happy? Wouldn’t you feel somewhat ashamed of your situation, perhaps envious of others?
But this isn’t merely fiction… It’s what has actually happened to countless people, including the last hunter-gatherers Yudkowsky referenced, and many traditional farmers. Today, in our world, the gap between the richest and poorest individuals isn’t on the scale of hundreds, thousands, or even millions, but billions. Inequality definitely matters. Poverty isn’t exclusively relative—it’s not solely a social construct as there are absolute needs to be satisfied—but relative comparisons are certainly part of it.