I think you might be failing to update generally enough here about how valuable it is for governments to.regulate the practice of medicine, in terms of directly benefitting people—who are, in general terms, overconfident about their competence to make decisions in complex domains they don’t understand.
Of course, this is not to say there are no harms from governments doing so, nor that they currently strike the right balance.
The phrase “time horizons” was confusing to me for about the first third of the post. Perhaps you could say “AI task time horizons” instead, or mention “AI task success rates for a given time horizon” at some point to clarify what the phrase refers to.
Otherwise, good points!
Typo?
how information is used, it might to be far enough
Probably should read “might not be”
This is a good question, albeit only vaguely adjacent.
My answer would be that winners curse only applies if firms aren’t actively minimizing the extent to which they overbid. In the current scenario, firms are trying (moderately) hard to prevent disaster, just not reliably enough to succeed indefinitely. However, once they fail, we could easily be far past the overhang point for the AI succeeding.
Assuming to start, implausibly, that the AI itself is not strategic enough to consider its chances of succeeding, we’ll assume AI capabilities nonetheless keep increasing. The firms can also detect and prevent it from trying with some probability, but their ability to monitor and stop it from trying is decreasing. The better AI firms are at stopping the models, and the slower that their ability declines relative to model capability, the more likely it is that when they do fail, the AI will succeed. And if the AIs are strategic, they will be much less likely to try if they are likely to either fail or be detected, so they ill wait even longer.
Data. Find out the answer.
https://www.wevolver.com/article/tpu-vs-gpu-a-comprehensive-technical-comparison
Looks like they arehwitin 2x of the H200s, albeit with some complexity in details.
Because it’s what they can get. A factor of two or more in compute is plausibly less important than a delay of a year.
This may or may not be the case, but the argument for why it can’t be very different fails.
As I mentioned elsewhere, I’m interested in the question of how you plan to re-base the index over time.
The index excludes models from before 2023, which is understandable, since they couldn’t use benchmark released after that date, which are now the critical ones. Still, it seems like a mistake, since I don’t have any indication of the adaptability of the method for the future when current metrics are saturated. The obvious way to do this seems (to me) to be by including earlier benchmarks that are now saturated so that the time series can be extended backwards. And I understand that this data may be harder to collect, but as noted, it seems important to show future adaptability.
I think the space of possible futures is, in fact, almost certainly deeply weird from our current perspective. But that’s been true for some time already; imagine trying to explain current political memes to someone from a couple decades ago.
Yes—they made a huge number of mistakes, despite having sophisticated people and tons of funding. It’s been used over and over to make the claim that bioweapons are really hard—but I do wonder how much using an LLM for help would avoid all of these classes of mistake. (How much prosaic utility is there for project planning in general? Some, but at high risk if you need to worry about detection, and it’s unclear that most people are willing to offload or double check their planning, despite the advantages.)
Sure, and if a machine just slightly smarter than us deployed by an AI company solves alignment instead of doing what it’s been told to do, which is capabilities research, the argument will evidently have succeeded.
A marginal bioterrorist could probably just brew up a vat of anthrax which technically counts.
Perhaps worth nothing that they’ve tried in the past, and failed.
One the first point, there’s a long way to go to get from the current narrow multimodal models for specific tasks to the type of general multimodal aggregation you seemed to suggest.
On the second point, thank you—I think you are correct that it’s a mistake/poorly written, and I’m checking with the coauthor who wrote that section.
Or, phrasing it differently; “read the sequences”
Good to hear, and I’m unsurprised not to have been the first to have considered or discussed thid.
Bad events are very often prevented by humans being moderately competent and successfully trying to prevent bad events.
...but I think the track record for this is pretty amazingly dismal, in practice? We are arguably more at risk from Pandemics today than we were in 2019, despite the clear warning. And even more narrowly, as a species, we’re spending many orders of magnitude more money on AI capabilities than we are on AI alignment, and that seems tragically very unlikely to change.
I enjoyed this, but don’t think there are many people left who can be convinced by Ayn-Rand length explanatory dialogues in a science-fiction guise who aren’t already on board with the argument.
I’m skeptical that we’ll see useful multimodal fusion from disparate datasets shoved into large multimodal models short of AGI; the relationships between the different modalities is central to actually learning anything, and many of the methods being used to accomplish multimodal “reasoning” routes through general capabilities. This is in large part informed by expert views we elicited in this project—so I’d be very interested in your thoughts about either why you disagree here, or what you think experts in that discussion (Sections 4.4/4.5) missed about multimodal understanding of such models in the coming few years. (Or whether you’re primarily optimistic in the even longer term.)
It seems like you’re narrowing the claim, and I’m no longer sure I disagree with the point, if I’m interpreting it correctly now.
If you’re saying that they group doesn’t act differently in ways that are visible to you, sure—but the definition of the group is one that believes that two things are viable, and will sometimes support one side, and sometimes support the other. You could say it doesn’t matter for making individual decisions, because the people functionally are supporting one side or the other at a given time, but that’s different than saying they “are not actually working that way.”
Yes, wanting humans to stick around is probably a minimal necessity, but it’s still probably not enough—as the post explains.
And I simply don’t think you can “make humans smarter” in ways that don’t require, or at least clearly risk, erasing things that make us fundamentally human as we understand it today.
He implied there that in the short term the advantage will be asymmetric, even if he’s hopeful that there will eventually be a defensive advantage. (I’m agnostic on the latter, and even if he’s right, I think the times scale needed for it to emerge might be longer than it will matter.) But I should have linked to his recent piece, where he says this explicity, not that older one: https://www.schneier.com/crypto-gram/archives/2025/1015.html#cg18
I’ll fix that now.