Will the critical phase of AI risk (the point at which a loss of control might become irreversible) come before or after widespread job loss?
Nothing is quite so politically activating as mass unemployment—if we did actually see 10-20% unemployment, there’s no doubt that AI would be the number one issue in politics. That doesn’t, in any way, guarantee good outcomes but it does guarantee the issue won’t jus be ignored.
Increasingly, though, I think we will pass through the critical moment before widespread job loss (quite plausibly in the next year or two). Slow diffusion means we’re not going to see mass layoffs in that window (aside from, perhaps, a few especially aggressive industries like tech). AI is on the agenda now and will be over the next year, but it’s hardly central (in the same way as inflation or immigration or the culture wars). I think this is probably bad news.
gbtw
He didn’t articulate it because he wasn’t using it. Gorbachev wanted to reform the USSR to save it.
I think the point is more that it’s very difficult to estimate the difficulty of something youve never done and especially to differentiate between things you might be able to do yourself in, say, a few hours vs things requiring serious effort or expertise.
Most magic tricks, for example, are pretty simple conceptually and it’s easy to think “oh if I spent the weekend on it, I could that.” Sometimes that’s true but other moves that look very similar from the outside take hundreds of hours of practice to pull off.
Pretty much anyone in academia will tell prospective students “don’t do a phd” or at least “you should only do a phd if narrow criteria are met.” This seems to be quite the opposite of a cult. My PhD is in the social sciences not the humanities (but I’m somewhat humanities adjacent), so my reactions to your points above are:
1. The vibe in my experience is always “doing a phd is pointless unless you want to go into academia; if that’s not your plan, you should do something else” and no one really disparages you for quitting. In fact, there’s generally a pro quitting vibe, and the average advisor will, if anything, push a student who is on the fence about quitting towards it rather than away from it. Advisors are perfectly happy when their students land roles in industry, and the emotion most adjacent to the one you’re describing comes from jealousy. I will admit (and I think most others will) to a jealousy towards the earnings of people I knew who weren’t remotely smart enough to hack it in academia, dropped out to do whatever the hell UX research is for tech companies, and now make 5 times what I do under better working conditions.
2. I’m sure I don’t socialize more with other academics than the average person socializes with people from their industry. In grad school, nearly all of my friends were fellow grad students, yes, but that was just proximity and ease of forming the connections rather than any discouragement. We all thought the people who managed to make outside friends were cooler than us. And, the moving around thing breaks up the cultishness if anything plus once you get a TT job, you don’t move any more.
3. You pay a lot in opportunity cost, but I don’t know anyone who took on substantial debt during their grad school years.
4. Compared to what? You probably moved away physically to do it, but it pulls you away from family less, I think, than going off and becoming a corporate lawyer in a far away city.
5. Is that true of cults? Yes, it’s very true of PhD programs and everyone is super aware of it.
6. More so than anyone else’s boss? And the massive power of the advisor is really found in the lab sciences much more than the humanities/social sciences. It also depends on the exact dynamic. Universities have stronger than average protections around things like sexual harassment.
7. Sure, true. But, in my experience, they’re also a cynicism/self-consciousness around that you don’t find among, say, MDs or cops.
The “permanent underclass” idea from the tweet seems to be mostly a concern of American white collar workers—and so I imagine the claim here is more or less that this lets relatively wealthy American white collar workers who are vulnerable to being automated away buy some insurance against that. Post-IPO, a white collar worker with a couple of tens of thousands of dollars available can probably buy that insurance via leveraged positions in the stocks, thus escaping the “permanent underclass.”
Interesting.
One complication here—the working language of the Vatican is ordinarily Italian and this document was probably drafted first in Italian (or possibly in a mish-mash of languages with intermediate translation steps). The clearest tells are probably in the Italian version. I don’t know how much LLM-based translation ends up rendering things in LLM voice, but it seems possible that part of the explanation comes from translation rather than drafting steps.
This seems like a reasonable forecast, and if you could buy Anthropic stock, I’d buy a lot of it.
But, I think there are some additional reasons for skepticism:
1) At some point, open weights models (or perhaps cheaper closed ones) will eat a lot of Anthropic’s business for use cases that don’t require being at the frontier. We’re only just reaching the point where this matters—Anthropic models started having real economic value only very recently and so being on the frontier has meant a lot. But, once the open weights models catch up to what, say, Opus 4.6 could do, then how much does that peel off? This is a worse problem for Anthropic if the frontier is kinda lumpy (and it might well be) where instead of delivering continuous economic value improvements you have to cross particular thresholds to open the next set of tasks ahead while open weights eat you from behind.
2) Is their current lead likely to be durable? Best argument in favor is that Claude itself is a big speedup but the counterargument is that this is a relatively young, relatively small company without any obvious moat and with financial resources that are, for the moment, way behind those of, say, Google. So it can’t really be that hard for Google to do what has to be done to eat their lunch. Likewise, there’s an argument that moving to the next level financially requires a kind of expertise in working with big corporate clients that they lack as compared to Google/Microsoft.
3) Subjectively, it seems like Claude excels at coding but ChatGPT is often better at factual stuff. It may well turn out that OpenAI has the edge when it comes to non-coding knowledge work.
4) Similar to your #2 above, the AI boom so far has generated massive market cap gains, which have mostly been captured by the semiconductor stack. So, to suppose that Anthropic will surpass Nvidia, Broadcom, and TSMC, you have to suppose that for some reason the current trends for value accruing to semiconductors vs. model builders will flip. Is something different now than over the last few years to explain why that would happen?
You could try some sort of soft extortion of the potential targets of the attacks (i.e., demanding big money to reveal the vulnerabilities) in a way that’s legal and fairly lucrative.
Is there any reason to believe Trump has ever chatted with an LLM? He interacts with plenty of AI-generated content via other people, but given what we know about his use of technology, I’d be pretty surprised if he’s ever actually interacted with an LLM directly.
Fair enough. Probably no way to come to terms then.
I’m not sure exactly how I’d define a coup but I’d say it has to be clear cut enough that, “Was it really an attempt at a coup?” is not really in contention in the aftermath.
Just out of curiosity, why is that? And what counts as “clear cut”?
After any failed coup (or, for that matter, any failed crime) you’ll always be able to find some apologists who say that the actions/intentions of the accused plotters are being misunderstood and it wasn’t really an attempted coup (or, as they case may be, an attempted murder, robbery, kidnapping, whatever) and/or had some kind of valid legal justification.
On #3, the net effect of the Iran War (at least in conjunction with other recent events) is likely to increase rather than decrease proliferation in general and maybe even with respect to Iran specifically.
From the perspective of an American adversary:
Iran/Venezuela vs. DPRK shows just how vulnerable you are to an American attack if you don’t have nuclear weapons.
Yet, Iran has fought to (as of now) at least a draw and arguably a long-run strategic win against an extremely punishing American air assault while also demonstrating (as everyone has suspected since Iraq/Afghanistan) an American unwillingness to put boots on the ground.
No one has seriously alleged that Iran was engaged in any nuclear work since last summer, and no one has alleged (even unseriously) that Venezuela had nuclear ambitions. So, you’re not safe even if you don’t build nuclear weapons.
So, for most adversaries, you look at that and see a pretty clear upside to going nuclear (you’re safe) vs. a limited downside (an American air attack that might happen even if you’re not building nuclear weapons). Why not build?
From the perspective of a country that’s somewhere between neutral and allied with the US, you look out at the world and see:
The Iran War reflects (along with Ukraine) a new local maximum of predatory great power behavior.
Iran seems to have split the fault line in the US relationship with NATO into a full blow schism, so the situation looks much more tenuous for anyone relying on American protection.
And the US complete aversion to doing anything that actually endangers the lives of non-trivial numbers of US troops make the US look like a complete paper tiger even if you think they might want to protect you.
The US positioning on the Strait of Hormuz has suggested a complete indifference to the fate of Asian allies.
Many international actors are more concerned with what the US has done than what Iran was doing.
Once again, all signs point towards building nuclear weapons. I think we’re rapidly approaching the point where a technologically sophisticated American ally that could build nuclear weapons very fast (Taiwan, Japan) would be a fool not to.
Not OP, but potentially interested in betting on this with you. Before haggling over precise details, do you agree that the events following the 2020 election count as trying to execute a coup?
There are very believable paths where AI reduces your income, but the current big AI companies don’t capture that value, or don’t return it to shareholders.
I suppose it depends on exactly what you do, but if the threat to you is something along the lines of “AI broadly replaces white collar workers” then it’s very hard to come up with a scenario where AI can replace most white collar workers but can’t find a way to make money for the companies building it.
You have to tell a really contorted story for that work—I guess it’s “believable” to imagine something like: AI gets good enough to replace most white collar workers, but there’s such intense commoditization and competition that there is no pricing power so the companies make no money, and the one white collar thing that AI can’t do is design semiconductors, so all the economic gains flow to Nvidia. Could happen but you have to rig the assumptions to get that result. And aside from that, most of the forms I can come up with are of the “money won’t matter anymore” variety in which case, there is no financial hedge.
Also quite believable that it’ll IPO with all that growth already priced in.
The point I’m trying to make is that Anthropic shares are worth more to your average upper middle class white collar worker than they are to the current ultra-wealthy or institutional shareholders. So, let’s assume Anthropic is currently valued in a way that perfectly captures it’s future growth prospects, it should still IPO at a higher valuation (and raise tons of money in the process) because retail investors will rationally put a higher value on the stock than institutions given that it is useful to retail but not institutions as a hedge.
I’m sure institutional shareholders, to the extent they can, are doing some amount of arbitrage in expectation of that dynamic. Everyone knows retail is desperate to buy AI. So, maybe, a fair secondary market valuation of Anthropic already accounts for that premium multiplied by some assessment of the probability they’ll actually list.
But, either way, the point is that the average company is worth about the same to retail vs. institutional investors, and AI companies have very different valuations for those two groups.
It’s widely rumored that both OpenAI and Anthropic will go public within the next year or so, triggering massive capital inflows. I think this unleashes an underrated financial dynamics.
An optimal portfolio hedges your personal risks. Unless you’re independently wealthy, the risk of job loss due to AI automation is one your largest personal financial risks, and it makes a lot of sense to buy insurance against that (i.e., to buy into AI companies). Right now, individual investors can’t do that. And the investors that have access to OpenAI/Anthropic on the private market are precisely the ones that don’t need to hedge that risk. Once you open up to retail, a rational investor should park a sizable percentage of their net worth in those shares, opening up a capital gusher.
This is especially true for Anthropic—there’s basically no way for a retail investor to get meaningful Anthropic exposure if they want it. It’s a little different for OpenAI given that SoftBank is kinda an OpenAI proxy at this point (albeit in ways that make it a less attractive investment than the real thing).
Seems like this could lead to substantial acceleration of progress. It may also change the politics. If stopping AI progress mostly means financially soaking billionaires, then it’s politically a lot more palatable than if it will also screw up the 401ks of the mass affluent.
Rule of law is mostly response to incomplete trust in government and a little bit because predictability is desirable. Because we can’t trust that the government will always act properly, we want to constrain its actions with rules and limit abusable discretion. And because we’re concerned that exceptions to rules will privilege favored groups, we want the rules to be the same for everyone. Same goes for due process.
But that’s not really a first-best solution. Adherence to fixed rules inevitably misfires in some concrete cases where we all agree that it would be better to make exceptions. So, if you have a benevolent and super-intelligent overlord, why constrain its actions based on rules? Why not let it just always do what is best? A super-intelligence can also solve the predictability problem by just directly answering your question about whether you can do X without getting into trouble (whereas that’s administratively infeasible for existing governments).
As to protected rights, I guess it depends on whether you think these are goods in themselves or means to an end. A lot of existing rights probably don’t survive on the basis of being inherently good. For example, most Americans seem to agree that things like hate speech or misinformation are not inherently good but ought to be protected as free speech in order to avoid a slippery slope. But, with a benevolent super-intelligence, we needn’t worry about that because it can just draw the line (and make the needed exceptions).
Personally, I have a pretty fatalistic attitude about this. If we all die, we all die. There’s nothing I can do about it. Unless you happen to be a billionaire or a person of extraordinary influence, there’s nothing you can do about it either. People I’ve known with untreatable terminal diagnoses have roughly that attitude—there’s nothing you can do about it, and you just make your peace and use whatever time is left. In a lot of ways, it’s crueler to have a treatable disease that is probably terminal but where you can try to fight. It’s better to detach.
I do feel an anxiety about the tension between the existential scenario and the scenario where AI doesn’t kill us all but takes over all the white collar jobs. There are things I could to prepare for that, but I don’t really want to do them. They’re in tension with the things I want to do if we’re all going to die. and also the ambiguity of timelines makes it impossible to know what strategies make sense (e.g., if I quit my white collar job to go to nursing school, how much time does that actually buy me? Does nursing get automated by robots quickly enough that I end up behind economically versus just milking what’s left of my career?) So, that sucks.
Another thing I find helpful is remembering that many other people have coped with alternative versions of this and managed to live their lives. At points in the Cold War, the threat of annihilation via nuclear war was probably at least as emotionally pressing for your parents/grandparents as whatever you’re feeling now. And there are plenty of examples of more discrete groups of people staring down certain or near-certain death with their faculties intact.
Could be true, but the API price doesn’t vary by time of day, so there’s no particular incentive to do that for power users and so I’d be kinda surprised if they did anything active to smooth their usage.
Bicameral systems can respect one person one vote, and they even do in the 49 states with bicameral legislatures.
The US Senate is a weird kludge that was necessary to secure support for the constitution. No one would arrive at it from first principles.
The best argument you can make for the Senate is that it’s necessary to protect vulnerable minorities (residents of small states). And democracies tend to trade off in various ways to protect vulnerable minorities at the expense of democratic purity (e.g., via constitutional provisions). That’s a pretty silly argument so far as it goes but does have a little more historical punch. You can’t make it for a hypothetical system with a chamber to protect the interests of the rich who are very obviously not a vulnerable minority at all. And there’s a much higher level of normative repugnance to “we’ve got to look out for the rich” than for most other groups you can fill in the blank with there.
Anthropic recently announced higher Claude usage limits for off-peak hours, which is everything except weekdays from 8 AM − 2 PM ET.
I was surprised by the specific times here. This is night time in Asia, the tail end of the workday in Europe, and the start of the work day in North America. The beginning of the window is too early for much West Coast activity, and it winds down at 11 AM in California. That seems inconsistent with the idea that coding is driving power use?
The timing seems most consistent with the period when you’d see overlapping demand from the US East Coast and from Western Europe—neither of which are super tech heavy regions. Maybe this timing implies a lot of the power users are financial firms working on New York/London time?
Sure could be.
The general point is just that “AI can do Jim’s job” doesn’t mean Jim gets fired right away in most organizations. Even in the most aggressive forms, that’s a lag of weeks or months. In smaller or less aggressive firms, the lag might well be years/decades and often of the form “not until a competitor totally disrupts this industry.”
Like Fable/GPT-5.6 are undoubtedly capable of taking a lot of jobs but also aren’t yet. And one take on “true” AGI is that it’s self-integrating, but I think thats obviously wrong because the bottleneck on these things isn’t intelligence shaped.
So, given rapid progress someone will quite plausibly train up an ASI (which might of might not then bide its time) or at least the AGI that can get out on the internet and burrow in and inevitably deliver its own successor or whatnot well before the average player in a non-aggressive industry gets around to replacing many people.