yams
Can you say more about the two claims in your last sentence?
I would be worried that the hypocrite is intellectually dishonest or ~incapable of tracking reality, though. If they’re just being dishonest, why should you expect to have ‘consequential arguments’ with them later?
The broader public also does a poor job of enforcing their sense of who is and isn’t legitimate.
In contrast, someone with different values (but who is actually adhering to those values) at least has the basic machinery to follow through on their word, which means that, conditional on them professing to have their mind changed, it will have been changed (unlike the hypocrite).
In a real life scenario (I imagine we’re thinking of politics?), I support the hypocrite, but mostly because I don’t expect anyone to change their mind anyway (at least not for reasons that look like ‘having been convinced’), and my baseline expectation is for reptilian levels of dishonesty.
But maybe I’m misunderstanding the case you have in mind—can you say more?
Matters a lot who the researchers are. I don’t think it’s an exaggeration to say that the top researchers are 10-50x as valuable (even discounting once-in-a-career insights) as ‘worst researcher in good standing with the org‘. Researchers in this latter category are replacement-level, as evidenced by the speed at which these orgs have been able to scale.
Of course, even ~median lab employees are exceptional technical talent in some absolute sense; the tail is just very long, and the talent pipeline extremely competitive. Lots of ‘good enough‘ applicants waiting in the wings.
It’s also unclear if labor volume at the labs themselves is actually a serious bottleneck vs things like RL environments built externally, access to compute, etc. There are plenty of researchers at labs whose work basically never pays off, or who would rate their own productivity poorly. This just isn’t a situation where some specific class of laborers has an especially big moat they can leverage to slow capabilities progress.
I was thinking about the other comments and their technical correctness re conflating the economy and the stock market / how much that correction of your OP actually matters for your area of concern.
Im actually very confused about what [number associated with a business or sector] tracks most closely with [political power] or [potential political power]. Market cap doesn’t look like a crazy place to start, but my guess is that there‘s something better (and I agree with the point that market cap likely overweights tech for this purpose).
What risks or downstream effects are most salient to you as you make this observation?
‘…not even The Establishment! [Which of course causes problems because The Establishment has Responsibilities.]’
is this a faithful rephrasing?
‘~No one thinks of themselves as The Establishment.’
I think I should have asked more like ‘the current instantiation of prediction markets’.
I agree that prediction markets could be useful in principle, but I‘m not aware of any cases, in practice, where there’s tightly correlated multi-market action of the kind you describe in your first example painting a vivid and cohesive predictive picture.
I’m also not aware of cases where commentators have been successfully pressured to bet on their beliefs via prediction markets (I wouldn’t be surprised by a single example in this second case, but I would be surprised by a very high profile example involving a commentator well outside our sphere).
My suspicion is that, for prediction markets to pay off around big events, we may need to fix them (in ways other than increasing the liquidity/user count) ahead of time (I don’t have strong guesses for how to do this).
[sorry if any phrasing is awkward; I’m concussed]
asserts that pausing AI development may be necessary
This is much, much stronger language than the segment you quote right after it. He’s not doing anything like ‘asserting’, he doesn’t mention a pause, he doesn’t use the word necessary, and it’s not clear that ‘do x before proceeding’ means ‘stop for n years to work this out’ rather than ‘ make our case by fiat to tech illiterate bureaucrats who wouldn’t know if, how, or why to implement a real pause, while the work basically continues in the background.’
It’s plausible they view much of their existing comms work as meeting something like the bar described in this post.
This is a much bigger stretch than Demis at Davos (which is itself often exaggerated).
What’s a shape of AI crisis for which you imagine prediction markets providing value?
I’m the person making these happen. There’s a Simplified Mandarin translation of the book coming out later this year.
There’s also a translation of the online resources into Simplified Mandarin, waiting on input from the publisher regarding the translation of key terms (for parity with the book translation, which the publisher has not yet completed).
We heavily prioritized securing the Mandarin translation of both the book and the ORs, and I sent a working copy of the ORs to translators before the English versions were even complete.
I think a version where you steel man (or find the strongest version) and then dismantle it would be good, or if you try to surface and satisfy the intuitions that motivate it, or similar high-effort ITT-passing stuff.
I think there’s an interesting discussion with fun side quests to be had here, and I’m often disappointed by the sneering, uncharitable tone wielded by those armed with The Correct Economics.
[for removal of doubt: I am not an LTV believer and think many ideas on the left are economically unsound. I just think the usual modality of refuting their case around these parts hardly ever works.]
“If you want the best AIs in December…”
I think the warrant here is ‘Anthropic will likely have the best AIs in December,’ which is why he’s looking at the Anthropic-specific signs.
(I’m unsure if this will be true; reasoning models looked like a big leap, similar to mythos, and they just turned out to be ubiquitous a few months after the reveal.)
Edit: another possible warrant is ‘OAI could have the best models in December, if they develop something mythos-class, which will be much more expensive to run and force them into a similar pricing position to Ant.’
How would you respond to the counter arguments?
fwiw I think piano pedagogy is exactly the kind of thing where an entrenched regime has propogated a suboptimal approach relative to most people’s goals on the instrument (and that there’s maybe only some single digit number of people in the country teaching outside of the small handful of dominant, not-especially-useful-to-most-people paradigms).
E.g., if what you want to do is play pop songs, a combination of ear training and a ‘simon says’ style app that reads midi off your keyboard and instructs you to play a ~random triad will basically get you there in ~100 hours of practice (assuming daily practice not to exceed ~3 hours/day). There are similarly straightforward training setups that I expect to be effective for other goals one may have on the instrument. I built ~all of my physical facility on the instrument in about three months of focused practice, and have similarly ‘cheated’ my way into my other capacities (I’m definitely missing things that other people 5 years into the instrument would have, but I also have a lot of things those folks don’t, and I prefer and deliberately pursued my skill profile over theirs).
I agree that few people who start playing piano as adults will ever play Rachmaninoff at competition level (but think very few people who enter into the pedagogical system designed to meet that end actually have that goal in practice).
I also (I think, although you don’t say so outright) agree that some tasks require developing wholly other senses — new channels for phenomenal sensation or new ways of comparing phenomenal sensation in an existing channel (e.g. audiation and relative pitch) — and that those aren’t well-captured in Oli’s ontology.
Wow that is surprising! Even after considering the suite of caveats one applies to benchmarks as evidence, I am very surprised.
All things considered I think I still lean harder on self-reports from lab and non-lab technical staff regarding the elicitation delta, but I’m much less confident than before.
[I suspect we may have other less interesting disagreement about how economically useful current systems could be if more effort were put toward juicing them, but happy to talk about that some other time; just mentioning this for completeness or something.]
Really? I would expect valuations to briefly stall out and then continue to grow when it became clear that the labs have a big lead when it comes to elicitation, scaffolding, etc.
I would also expect existing big-tech valuations to grow in this scenario—just not startups (although maybe they get a bump in the short term).
Can you say more about why you expect this? Trying to see if the answer is [real disagreement] or [Oli has superior knowledge of economics] (and also learn something, in the latter case).
‘Major AI labs can only justify their high valuations by developing very powerful, very general AI systems’ is a claim I sometimes hear. That is, many seem to expect ‘if no AGI in n years, then the bubble pops’.
However, I think just revolutionizing tech is likely enough to justify current valuation levels (and maybe as much as 4x current valuations; maybe even more?), given the market caps of other large tech companies (even if we exclude NVIDIA, which we may want to do because their current market cap is more heavily tied to the AI boom than others). After all, these are still ‘only’ 12-figure valuations in a sector where many of the major players have broken a trillion. A 12-figure valuation is consistent with ‘future major player of the kind that already exists’ and not ‘potential god emperor of the solar system’.
Using these big tech companies as a reference point and assuming very limited further capabilities progress (eg no TAI, AGI, [your favorite way to talk about very general, ~human level systems]), the major LLM companies still don’t obviously look overvalued to me. The tech pie by itself is big enough (and the current state of the tech looks, to me, sufficient to massively disrupt that entire sector), that if they capture enough of the sector (even if it takes a while), we won’t see a significant correction.
(there’s a nearby but unrelated point about whether the fundamentals of these companies look sound in a conventional sense, which I don’t mean to weigh in on here)
[recovering from a concussion so apologies if this is especially poorly written or unclear]
I don’t dispute that some postmodernists would consider cultural relativism central to their worldview, and think instead of ’mostly by its detractors’ I should have said ’often by its detractors’.
I’m glad you’re open to using different language.
I think I get the spirit. I thought, from the way you phrased it, that you were making an empirical claim (eg: ‘I saw [x better thing] not happen because LessWrong something something’). But it sounds like you’re making a more conceptual argument (which I think has legs, but for different reasons than you do, and I think we disagree on the effect size).
Thanks for responding.