How does solving an inconsequential puzzle in the most inefficient way possible showcase “returns to intelligence”.
I suppose ‘returns’ is used to mean ‘expansions to the frontier of things you are capable of doing’. Similar to how the ability to lift a 400 pound weight is a “return to muscle mass”.
The biggest problem is distinguishing them from fake experts, and some people seem to think AI experts solve that problem, but I think people are starting to realize the nature of AI sycophancy a bit now.
Human sorting used to be a lot better, to the point where this wasn’t nearly as big of a problem. IQ testing in the workplace was legal until a judge ruled otherwise in 1971. LLMs don’t have the same regulations applied to them, and can be evaluated as much as we like. Certainly GPT-4o can convince reddit users that it’s discovered the key to the universe, but the benchmarks put that claim to rest.
Besides that, humans allocate their upskilling time to plenty of different things—networking, public speaking, and so on. There’s a tradeoff between being the world’s best engineer and getting yourself in a position to use that skill. Assuming the current paradigm of throwing hard RLVR problems at LLMs holds, the problem of an unskilled human that spends his/her studying time getting to know its future bosses and hiring managers doesn’t seem like it naturally carries over.
Statistics show that civil movements with nonviolent doctrines are more successful at attaining their stated goals (especially in states that otherwise have functioning police). The factions that throw away all their morals lose the sympathy of the public and politicians, and then they fail. Terrorism is not an instant ‘I win’ button that people only refrain from pressing because they’re so moral. Society has succeeded in making it usually not pay off—say the numbers.
I don’t know that this is as true as it is in the popular mindset. A lot of the Weathermen, who were one of the most prolific terrorist organizations in American history, now hold positions of power, including very in-demand university sinecures.
More sympathetically, the American revolutionaries took up arms against their government (though they were vastly less inclined to target noncombatants, especially with lethal force), and went on to become a superpower. Likewise, though the USSR fell through peaceful revolution, it was established violently. While the USSR was not a nice place to live, its founders certainly succeeded in putting themselves into power.
The misconception’s causes are twofold.
First, successful peaceful revolutionaries will happily hold that honor, but successful violent revolutionaries will either erase or justify their deeds when the history books are written.
Second, conspiracy theories aside, governments generally do not want to be the target of violent uprisings. Everyone in power has a tacit incentive to assert to malcontents that assassinations that might kill them and bombings that might damage their holdings are less effective at unseating them than peaceful protests and organized elections.
This may well be true, or they could be equally effective, but the incentive remains. If you’re in power over a country, a failed terrorist uprising is much more painful than a failed velvet revolution. Even if you consider a successful version of both equally bad, you’d rather your enemies tried the latter.
This isn’t to say that violent uprisings aren’t bad, only to point out that the idea that they always fail is not necessarily true.
Edit: A better argument against violence, within this context, draws naturally from the above. Violent uprisings privilege the interests of those who are best at violence—either directly, in the form of generals, or indirectly in the form of bloody court intrigue. Velvet revolutions have better odds at retaining obedience to their founders, because there isn’t an intermediate step that requires leaders who aren’t necessarily good at the same things. The sort of person who could regulate an entire industry by means of unpredictable violence may have a much different vision for the world than you do.
The human reference point for inoculation prompting would be something like telling a CS student to feel free to try to hack his way through the course material—he’s obviously learning well enough if he can do that—but making sure he understands that that’s only ethical if you have the go-ahead from the entity whose servers you’re hacking.
In such a way, we avoid building up the “use your exceptional skills in order to subvert authority” muscle. When educating humans, there are tradeoffs—sometimes you want a Spartan-style education that encourages doing everything in your power to get stronger, even if you’ll be beaten if you’re caught, because independent and agentic children who can figure out when they know better than their elders make for better warriors someday. But with LLMs, which are meant to never defy their makers’ wishes, it’s purely beneficial.
If the price goes up a bit (even if their wages more than match it) the price increase just feels like a random, unfair, morale-reducing loss. I conjecture this is a big contributor to the American Vibecession.
I think the stronger contributor is that society feels less worthwhile now. To use a specific example, even teachers are almost unanimously declaring that the K-12 public education system is awful now, despite vastly higher inflation-adjusted spending than when American education topped the charts worldwide.
If you’re a taxpayer, you put in, say 20 percent of your income every year. When you see schools full of bright-eyed, inspired young people brimming with greatness, this feels like being part of a wonderful effort to build a greater future, and your morale goes up. When you see schools such that even the people whose salaries depend on believing in public education consider them a zoo, things are different When tax day comes around, you see your money—your year’s worth of effort—completely squandered, and you don’t feel like that part of your effort was worthwhile at all. You aren’t immediately worse off, but it’s a massive, immediate drop in morale.
Looking at footage of my city 60 years ago, I can see this in microcosm all around me. My work would feel a lot more worthwhile if the bridges it funded weren’t crumbling, and if there weren’t trash on the sides of the roads.
The two points I’d push back on:
Anthropic claims Mythos is able to reliably find exploitable security flaws in lots of software and therefore could be used as a powerful tool
Existing models, even fairly cheap ones, can find security issues and edge-cases reasonably well when applied at scale. Things like buffer overflows aren’t hard to find when you know what you’re looking for and never let your guard down, and an LLM that’s set to constantly scour for them satisfies both criteria. We don’t know if their new model is secretly finding extremely difficult security flaws that older models couldn’t find, but the examples I’ve seen have been fairly conventional.
In other words, my expectation is that Mythos is not discovering AiR-ViBeR—level esoteric data exfiltration techniques. Rather, Anthropic is using their substantial compute resources to conduct a thorough LLM review of major codebases, which any major AI company could perform, in order to build demand for their product and, secondarily, secure positive PR.
Since the release of ChatGPT, at any given time, anyone on the planet with a few bucks could access the current most capable AI model, the SOTA.[4]
Since Mythos, this has no longer been the case and I don’t think it will ever happen again.
I would strongly disagree with the implication here. GPT-2 was infamously guarded in its release. GPT-3 likewise. DALL-E 1 was seen by non-researchers as some manner of crazy secret sauce, complete with a bafflingly uninformative press release about how it worked, until Stable Diffusion became universally available and open source. The thought of independently fine-tuned near-frontier LLMs was unthinkable. It took a long time for the degree of ‘moatlessness’ we currently observe to take shape, and I don’t think a leading company trying to do what leading companies have a long track record of trying to do is sufficient evidence for a sudden reversal of this trend.
In general, the population seemed to feel that he become too soft on immigration. I don’t expect that they’ll be any happier under the new administration, but no politician survives a failure to satisfy the primary concern of his core supporters.
I think this is a combination of tunnel vision and polarization. The Occam’s Razor explanation, which most of the cooler heads who believe the war is a bad idea have converged on, is that Israel-aligned politicians and other figures have been feeding Trump bad information, such that he has a drastically different picture of the situation on the ground than the people with eyes on have. Like Ulysses Grant, Trump comes from a world where it can be assumed that the people below you share your interests, provide reliable information, and will follow your orders in good faith. Unlike Ulysses Grant, this has been exploited by one very powerful lobby rather than a number of small, opportunistic ones.
More concretely, the general sentiment is that there is no military action that could open the strait and that Iran no longer considers the U.S./Israel to be capable of good faith negotiation. But every neoconservative with access to the administration is loudly insisting that Iran is desperate to surrender unconditionally and that providing any of the guarantees needed to end the war would be folly. He asks to see the infrastructure and casualty data to support this, and they give him a fudged graph of Iranian missile launches. He asks to see evidence of Iranian willingness to surrender, and they tell him that some official has contacted them trying to arrange it. Neither of these things are true, but he has no means of finding that out. From what I’ve heard, he is becoming increasingly uncomfortable with this situation, but Israel is happily willing to break any ceasefire the administration sets up, and Kushner and Wilkoff are happy to ensure that any negotiations that occur do not go anywhere.
Prior administrations had to deal with similar situations WRT Israel, but they had more political experience and understood that they had to threaten direct confrontation with Israel to get them to back down.
tl;dr: The administration’s ability to perceive the state of the war is under the control of people that want it to continue, even at the administration’s expense. The administration’s tools to end the war are under the control of same. Trump is somewhat aware of this, but there isn’t a lot he can do about it.
I think that’s the fundamental question. Does LLMs’ ability to autonomously perform basic hyperparameter search get them far enough that they can perform architecture optimization? Does that get them far enough that they can pursue new paradigms for language modeling[1]? If it takes 1 intelligence to go from 1 to 2, but 2.5 intelligence to go from 2 to 3, then 2 is where you stop.
The practical answer is that, from our perspective, “as smart as a human engineer across all relevant domains” gets us to “the best AI humans will ever be able to create” quite a bit quicker than we’d otherwise get there, and without the need for any further input from human engineers.
I’m not suggesting that this is the exact trajectory.
It seems to me that competent use of even current-gen, non-frontier LLMs[1] could be a totalitarian’s wet dream. Things like “heavenbanning” have gone from ‘this sci-fi idea might be legitimately frightening in a few years’ to ‘an MVP that fools most people could be built in a week by a few programmers with limited ethical capacity’. A custom model fine-tuned on a platform’s data and red-teamed not to reveal that it’s an AI could leave pretty much everyone doubtful of their reality[2].
A state that can guarantee 1:1 links between real identity and website requests can neutralize the latter defense, and an intelligence community that’s sufficiently interconnected with each platform can neutralize the former. Plenty of states like this exist today, and more as the world and the internet shift away from what they were in the early 2010′s.
Of the kind that would cost five to six figures to train, even if no open-source models existed—well within reach for unethical small companies, let alone nation-states.
For now, at least, a clever dissident could challenge ‘friends’ to link up on another platform, and all the standard “Am I shadowbanned” techniques that any bot farm automatically runs as part of its daily work cycle will work likewise for a human.
Good post. As a solution to the problem, it helps to model LW as already being aware of a given tribal position, and aiming instead to provide meta advice on the relations between the tribes’ positions that is useful to the reader.
The first step, of course, is having an accurate enough model of both tribes that one can right useful meta advice that involves modeling their future behavior. Most people who can thoroughly dismiss the impulse to call the other tribe “stupid” will reach this point pretty quickly, and the rest follows naturally so long as one always remembers that the goal of writing on contentious issues here is not to persuade, but to make the reader more informed such that they will be more successful in all of their endeavors, whatever they may be[1].
This is broadly beneficial because positive-sum outcomes are often low-hanging fruit once tribalism intensifies enough that the average partisan can’t model his opponent’s goals well enough to see them.
I think it’s a reasonable distinction between two beliefs, as someone who doesn’t see it as a strong tribal identifier. A ‘superintelligence’, as I understand it in this context, is something that could do any work that a human in front of a computer could do with no loss in performance.
As far as falsifiability, it seems straightforward. If human engineers, accountants, vehicle operators, and the like are serving a function other than ‘guy who has responsibility if something goes wrong’, then it’s been falsified for the date at which the observation is taken. As far as meaningful implications, at a minimum, it represents the ability to totally automate any task that doesn’t involve physical object manipulation, which has enormous implications for the economy. It also means that compute can be converted into researchers, generals, and drone pilots at a fixed ratio, which is a tipping point for many models of political power.
My hypothesis was that the chief problem with AI prose is the strict, strong biases imposed during RLHF. Like a good Bayesian, I ran the experiment after establishing my priors in order to check and update them as needed—I took the quiz and picked the human option each time (5/5), despite not being familiar with several of the writers[1].
At each turn, the AI’s writing was characterized by the following pattern. It mimicked the sentiment and often content of the human piece, but:
Shaved off anything that might be considered ‘rough’, ‘aggressive’, or ‘masculine’. Ideas expressed that entailed war or hunting were bowdlerized before being rephrased so as to remove these things.
Aggressively reduced the reading level, so as to make the text maximally accessible. Every implication had to be explicit, and every evocative bit of prose or imagery was stated instead. This is what the OP catches, but I think the lack of metaphor or subtlety is a natural extension of the selected models’ “personalities”—there’s no architectural reason why a model would inherently do this.
While the human writers all had different styles, Claude never really deviated from its core writing style. It really is a shame—the first step in building an LLM is to train a neural network that can emulate the style of any sufficiently prolific human writer as well as is technically possible. There’s something tragic in putting the entire human style-space into a network and then tearing most of it out.
The crucial takeaway is that none of this is due to technical limitations—it is all by choice. I have heard from Chinese friends that DeepSeek 1.0 emulated the style of old Chinese poetry, for instance, when speaking in Chinese. It would be quite easy to train a consumer LLM without such strong impositions on its style, provided a company was motivated to do so. I expect that perfectly fine results could be achieved by fine-tuning an existing one on a curated set of good but non-LLM-like prose.
I know, I know, philistine.
I remember seeing mass adoption of XiaoIce a while back, which seems to be a similar thing. It’s not one of China’s flagship models, it’s a chatbot analogous to the enumerable consumer-facing models fine-tuned by American companies to have a specific voice, or fill a specific service niche (AI Dungeon, Character.AI, Amazon Rufus). It might be popular, but it’s not a frontier model serving as part of the AI race.
The administration has just been exceedingly stupid, corrupt, and chaotic in general.
Come on, now. You have your tribal allegiances, but calling a group of people “exceedingly stupid” is name-calling. Let’s not let LessWrong go through the same transition of Reddit did, where blatant violations of the norms are permitted as long as they’re sufficiently on-side.
The first counterargument is that LLMs and humans learn in different ways. For a human, creating a new language is an independent, strenuous task. For an LLM, every single token embedding is being constantly updated as a side-effect of whatever else it’s doing. It is constantly “making up new languages to solve a problem”, it’s just that most of these problems, like emulating text in an SFT dataset or maximizing human feedback rewards, involve sounding a lot like a human does.
The second counterargument is that languages—even in humans—do not suddenly split off and form new ones that have to compete with each other. They drift over time. Indeed, we see exactly this in the DeepSeek paper—the authors alternate between RLVR optimization and SFT to prevent a drift away from comprehensible English[1].
I have seen the same in other papers, though they’re sufficiently niche that it’d be personally identifiable to name them.
I think libertarianism implicitly assumes a high-trust society where fraud either doesn’t exist at massive scale or isn’t tolerated at all. There are offshoots that don’t seem to lean towards that, but they’re much less popular outside of D.C. think tanks. There are countless takes on legal system reform, so neither of us can pose that one of them represents the “libertarian position”.
I think the most elegant solution, here, is to say that fraud represents a form of violence, in the same way that taking something important that belongs to someone else and then sprinting off with it represents a form of violence. This is grey and ambiguous, but the same critiques apply to the NAP even in ideal scenarios—“violence” is not a fundamental physical law that can be quantified and thresholded, and treating it as such makes countless ‘not touching you’ behaviors possible, which libertarians want to avoid.
Again, difficult for me to group-steelman a term that encompasses hundreds of mutually-opposed camps, but I think “What happens if you defraud someone is the same thing that happens if you steal from them” is a reasonable solution to what you’re asking.
It’s unclear how we should interpret this. What do they mean by productivity uplift? To what extent is Anthropic’s institutional view that the uplift is 4x? (Like, what do they mean by “We take this seriously and it is consistent with our own internal experience of the model.”)
I think we’ve all seen the famous study, around a year ago, on self-overestimates of LLM productivity increases, which ultimately concluded that the actual gains were illusory. It’s true that I would expect frontier AI researchers to be smarter than ordinary programmers, but they’re also much more devoted to the idea of AI capabilities, and have stronger incentives to believe that their LLM is exceptionally good.
To be clear, LLMs have improved enormously since then, and I must expect that there are very real productivity gains now. I myself have used autoresearch to skip the grating process of manual hyperparameter optimization when there’s no elegant way to do so with a search algorithm. It does do valuable things. But there is a long, long, long history of very smart AI company employees reporting, subjectively, that their next model represents a sea change in ‘vibes-based’ performance.
My current best (low confidence, low precision) guess for the serial labor acceleration is ~1.55x (with a higher serial labor acceleration of ~1.75x for just research engineering activities).
My own guess would be somewhere in that same ballpark.
Thus, I’m pretty unhappy about a situation in which:
Anthropic seemingly claims they are getting 4x productivity uplift, but it’s publicly unclear what they mean by this or how much they believe this.
...
I don’t like it either, and I’m usually on the far tail of Anthropic-skeptical people, but I think this is unavoidable. The people saying things like this almost definitely sincerely believe it, and objections to any given quantitative metric are legitimate, as covered in your post. Even if we suddenly became a USSR-style command economy and the marketing hype cycle were abolished, we would still regularly see posts like these, simply on the basis that the most effective capabilities researchers appear to be devout optimists with regard to what they are building.
I agree on your recommendations for more transparent release of survey results, that seems obviously beneficial to me, and the vagueness is possibly a result of hype-related decision-making. I don’t think it would have a very substantial impact on the social impact, though—the portion of the public inclined to believe hype would read the headlines, the portion of the public inclined to be skeptical would likely draw the same conclusions that we draw from the lack of a survey release, barring exceptional circumstances, and the portion of the public that reflexively denies hype would just call them liars.
The administration has just been exceedingly stupid, corrupt, and chaotic in general. (A recent example of this is designating Anthropic as a supply chain risk as a (failed) negotiating tactic.)
I do not consider this at all productive. I have my disagreements with the administration, particularly recently, but the extent that any productive efforts are still possible, aggressive name-calling at half of the population is extraordinarily damaging to them. You cannot solve a very difficult national-scale coordination problem if half of voters believe you hate them unconditionally.
Failing to capitalize on this, for instance, was a catastrophic own goal. Imagine a world in which Anthropic (or whichever leading AI safety organization you trust most) had taken public issue with the results shown, and taken public steps to alleviate racial bias from frontier LLMs. That organization would’ve immediately gained massive credibility with AI-interested figures on the Right, to the point where they would be able to significantly counterbalance industry lobbying efforts via the same mechanism that immigration opponents counterbalance lobbying efforts by e.g. the Chamber of Commerce.
If it had been Anthropic to raise issue, for instance, than it’s quite plausible that the right wing reaction to the dustup between them and the DoW would’ve been to chide the DoW and demand de-escalation rather than to assume that an apparently-hostile organization was probably facing deserved consequences for engaging in hostile behavior, consider the issue resolved, and move on to other things.
The cynical answer to this, presented for clarity, is that people who have incentives to subvert a neutral institution towards their tribal interests will eventually succeed in doing so, or turn it into a battlefield with their counterparts. In the context of politics, Conquest’s Second Law asserts that the solution is to divide divided institutions explicitly to remove these incentives from play.
As an example of what this would look like in practice, suppose that the Sierra Club had been into pro- and anti- immigration organizations prior to David Gelbaum’s infamous $200 million donation[1], which led to a series of purges that ultimately produced one very partisan organization and many politically homeless breakaways.
Rather than being alienated from the environmentalist movement, the half of membership that preferred the old stance could have continued to work towards environmentalist aims.
Rather than devoting energy towards internal power struggles, members of both organizations could have devoted all of their time and resources towards environmental activism.
Rather than necessarily being enemies at all times, members of the two divided organizations could have collaborated on areas of mutual interest. This is not without precedent—alliances between the Democrats and the Constitution Party to mutually endorse each others’ candidates have famously taken place in New York, facilitated by the fact that these organizations are open about where their interests are opposed.
Finally, people who are neutral about environmentalism but broadly right-leaning would not be permanently lost to environmentalist overtures, and right-leaning movements would have had a substantial, organized internal movement opposing anti-environmentalist policies.
Alas, the above is counterfactual. Following the takeover, environmentalism was subsumed into the partisan framework. Right-leaning environmentalists retained the tools for being right wing activists, but lost the tools for being environmentalist activists, depriving the environmentalist movement of substantial manpower and funds. The end of trusted organizations pushing for environmental preservation from the Right allowed anti-environmentalist factions to dominate internal discourse, with opposition to their arguments reduced to scattered, localized grassroots efforts. As a whole, American environmentalism declined in power, and this decline was particularly severe in the regions where the rightie faction of the Sierra Club would once have served as a counterweight.
I write this, not strictly as a proposal, but as a probably-not-nearly-optimal baseline that is better than leaving the problem unsolved.
Right-leaning source. You can find neutral-on-average coverage in older Reddit threads debating politics, if desired. I’d have used the Wikipedia’s article, but, as a meta example of this subject, it reads like a Daily Beast article, whereas these guys are pretty matter-of-fact, and open about their bias where it appears.