Researcher at Epoch AI
@justjoshinyou13 on twitter
Josh You
Karma: 369
I also take this argument seriously.
One background fact some commenters are missing: it’s virtually unheard of for a tech startup to continue growing at 100% or more after it reaches $1 billion per year in revenue. A company growing at closer to 1000% per year at the multi-billion revenue level is wildly unprecedented. A company tripling its revenue in one quarter from a starting point of $10 billion, as Anthropic did in Q1, is even more wildly unprecedented than that.
Revenue growth has momentum, and it is essentially locked in that frontier LLMs will be a bigger business than the biggest tech industries (smartphones, internet advertising) are today.
If we ignore misalignment risk, why are you envisioning a world where ASI acts like a benevolent dictator, such that it is up the ASI’s own values and decisions whether the middle powers have significant influence over the world? Rather than human customers and users of ASI-powered products and services around the world creating a balance of power (akin to how AI works today)? I think alignment should include be “don’t be a dictator”, not just “don’t be an evil dictator”.
Very cool research overall!
America has over 3,000 utilities, not one. There is no authority in the US which can order “disconnect these 50 lines in these 12 states” in a 20 minute window.
This doesn’t sound right to me. The relevant level of institution here is grid operator, not utility, and eight grid operators cover most of the US. Either way, any electrical grid has an institution that manages the grid in real time and responds to incidents like shortages and natural disasters. I don’t see how the NZ plan as described falls outside the standard toolkit of whoever is managing the grid, though of course they would need to form a solar storm plan ahead of time.
Update: the internal/external distinction only matters for the F-35. The F-15E and F-18 Super Hornet don’t have internal bomb compartments. I think those three are the main fighter-bombers being used by the US and Israel in this campaign.
A modern fighter-bomber can carry 7-15 tons of ordinance, including external attachments (with penalties for aerodynamics and radar signature). Do we know how much US and Israeli fighters have been carrying per sortie on average?
edit: One F-35 variant has an internal capacity of three tons and a total capacity of nine tons.
:re “Companies leaving without the company” it’s worth thinking about what the equilibrium is. Foreign companies in currently-allied countries (e.g. Mistral which is founded and heavily staffed by Meta and GDM alumni) are currently poaching US lab staff or buying US chips in moderately large quantities with either no objection or the support of the US government. For the US government to countenance changes to chip policies or sanctions(!) with respect to such companies, the political and technological situation would have to change significantly from today.
Don’t know how to find comprehensive data but they had a development pipeline of 1100 homes in 2022 against a population of 13k. So that might increase their population by ~15%. And about 500 units approved in the pipeline right now. I think the original 1100 pipeline includes the now-opened “Emery” development which had 500 units.
If they approved housing at the rate of Seattle, the leader among large US cities, they would be approving around 1500 per decade. So it seems fast, though at a population density of 10k/sq mile they still might take a couple decades to reach SF density (18k) on current trends.
Isn’t Emeryville kind of doing this? Though I’m not sure if they’re maxing out the envelope of housing production from real costs even if a city government goes 100% YIMBY.
Bit of feedback: would be helpful if you explicitly stated your estimated number of H200 and Huawei chips and/or provide a B300-eq conversion table, so they are more comparable to other reports that are quoted just in number of chips. I understand how you do the conversion but it is not super apparent in the post.
One thing I don’t know is when data center investments get committed to specific customers. Google and Amazon are Anthropic’s two main compute partners and will spend $200B each in capex this year and are presumably planning and developing sites for many more hundreds of billions by 2028. So one possible view of it is that their capex creates a window, and Anthropic’s eventual share depends on its funding and revenue. But Google and Amazon don’t quite know how their 2027-2028 data centers will be allocated.
In general, for large data centers the specific lab that will use it is settled well before the late stages of construction, e.g. Stargate and Rainier, But I know independent data center developers often start developing a site without having a client pinned down. And smaller inference clusters are presumably more fungible.
A good term for 10^20 FLOP would be useful. This would make modern models around 100k to 10 million of this unit, which is a tangible number. Some people, e.g. at DeepMind tried to make “petaflop-days” (8.64e19) a thing but it didn’t catch on.
Another point here is that elections are an additional check after the courts, Congress, etc. US presidential elections are not administered by the federal government, they are administered by the states. So to interfere with elections, the president can’t just fill election boards with cronies or give orders to anyone in his chain of command to rig the election. He’d have to forcibly manipulate or interfere with state officials and state governments, risking direct conflict with states. And if he doesn’t interfere with the election and the states announce results showing he lost in a landslide, his political power almost certainly evaporates. Of course, if all the president’s crazy actions are in fact popular, then he is much more likely to succeed and stay in power for many reasons.
Again, if you assume the military always slavishly follows the president, then this ends up in a civil war with a plausible military victory for the president. But each escalation into “this is obviously illegitimate” means the president increasingly offends his generals’ sense of duty, decreases the probability of success and increases the legal and political risk for the officers following his orders, increases the size and motivation of the inevitable popular resistance, etc.
In-practice most federal offices have deferred to what the Supreme Court says, but we haven’t really seen what happens when e.g. a sitting president insists on an interpretation of the constitution that disagrees, and the constitution itself provides no clear answer.
This is a somewhat confusing statement. To be clear, it’s extremely common for the president to disagree with courts on the law or Constitution: this happens dozens of times per presidential term. And when they lose in court the president may declare that they still think they are right and the Court ruled incorrectly. But this wouldn’t cause a constitutional crisis or anything by default: the president almost always follows court orders or court opinions. It’s a very ingrained norm in the US that court orders, especially from the Supreme Court, are binding.
(relevant thread from a lawyer early last year on the powers and tools that courts have to force a president or other federal officials to follow their court orders, such as freezing assets).
I think there’s a lot of reasoning here that effectively goes “if the president has absolute power such that the military and federal officers will always listen to his orders, then the US legal system will have trouble reigning him in.” Which is kind of just begging the question. But somewhere in the chain of events you suggest, the president would break a lot of clear red lines and probably lose nearly all of his political support from the general population and the powerful elements of society, unless the he has already broadly persuaded people that his power-grabbing actions are actually a good idea.
Yeah I think leading labs generally retrain their base models less often than every 6 months (but there’s a lot we don’t know for sure). And I believe this most likely has to do with a production AI model being the result of a lot of careful tuning on pre-training, mid-training, post-training etc. Swapping in a new base model might lead to a lot of post-training regressions that need to be fixed. And your old base model is a “lucky” one in some sense because it either was selected for doing well and/or it required lots experiments, derisking runs, etc. Even with all of your new algorithmic tricks it might be hard to one-shot YOLO a base model that’s better than your SOTA model from nine months ago. But this is probably much easier for your model from 18 or 27 months ago.
Also I’d guess staff costs are more important than compute costs here but these considerations mean compute costs of retraining are higher than one might think.
you should be more uncertain about the METR benchmark’s external validity than what these error bars show.
but your baseline uncertainty about key facts about AI progress in general should also often span much more than one order of magnitude between your 2.5th percentile and 97.5th percentile guess. the METR results add a lot of value and I don’t think these error bars are a big deal in the scheme of things.
Most successful startups slow down a lot after a brief hypergrowth phase. We should be looking for signs that AI companies like OpenAI and Anthropic* are experiencing unusually long and persistent hypergrowth: surprisingly little slowdown in growth, or maintaining >2x growth/year at surprisingly high revenue levels like 100B. They are both already growing very surprisingly fast for companies with multiple billions in revenue, to be clear, but whether that continues is valuable evidence.
This could be a sign that present-day models have a higher economic ceiling than we realize (closer to TAI than they might look), or that companies are making real progress towards transformative AI. Most companies don’t dramatically improve their product lineup over and over again after they find initial product-market-fit, so sustained rapid growth means that AI development is leading to a new batch of successful products on a regular basis, i.e. escalating economic usefulness.
*I think companies that serve AI to end-users are the most useful indicators
I’d flip it around and ask whether Gabriel thinks the best models from 6, 12, or 18 months ago could be performing at today’s level with maximum elicitation.
I think the linked tweet is possibly just misinterpreting what the authors meant by “transistor operations”? My reading is that “1000″ binds to “operations”; the actual number of transistors in each operation is unspecified. That’s how they get the 10,000x number—if a CPU runs at 1 GHz, neurons run at 100 Hz, then even if it takes 1000 clock cycles to do the work of neuron, the CPU can still do it 10,000x faster.
Hmm I see it. I thought it was making a distinct argument from the one Ege was responding to here, but if you’re right it’s the same one.
Then the claim is that an AI run on some (potentially large) cluster of GPUs can think far faster than any human in serial speed. You do lose the rough equivalency between transistors and neurons: a GPU, which is roughly equal to a person in resource costs, happens to have about the same number of transistors as a human brain has neurons. It’s potentially a big deal that AI has a much faster maximum serial speed than humans, but it’s far from clear that such an AI can outwit human society.
OpenAI can probably achieve Meta/Google-style revenue just from monetizing free users, since they’re already one of the biggest platforms in the world, with a clear path to increasing eyeballs through model progress+new modalities and use cases, and building up an app ecosystem (e.g. their widely rumored web browser). An anonymous OpenAI investor explains the basic logic:
The investor argues that the math for investing at the $500 billion valuation is straightforward: Hypothetically, if ChatGPT hits 2 billion users and monetizes at $5 per user per month—“half the rate of things like Google or Facebook”—that’s $120 billion in annual revenue.
However, this might take a long time to fully realize, perhaps like 10 years?
We don’t know much about how long it takes for a new base model to get “up to speed” in post-training these days. The existence of Mythos, which was already much more capable than Opus 4.6 by late February, suggests to me that this time window has compressed compared to a year ago.
Note: DeepSeek-V4′s final checkpoint wasn’t even RL’d at all. They did on-policy distillation from RL-trained specialist checkpoints to produce the final model. 5.5 could have similarly been (partially) distilled from advanced post-trained checkpoints of 5.4.