The probability of a thermal fluctuation is inversely related to its size, not its k-complexity. A universe-sized cloud of mass is less likely to congeal into any kind of order than a brain-sized cloud.
Nick_Tarleton
Karma: 6,185
We have no clue if “disordered experience” is even a thing
I can easily conceive of ‘the experience I’m having now, except the left half of my visual field is noise’.
This paper (2015) argues that the idea of quantum fluctuations without an observer/measuring apparatus to decohere the wavefunction is confused, and therefore (I didn’t fully understand this part), depending on unknown details of physics/cosmology, even thermal fluctuations (what the OP discusses) in an old universe that give rise to Boltzmann brains might not be a thing. (ETA: this sounds like it overlaps with Mitchell Porter’s top-level comment that “[maybe] De Sitter space simply doesn’t last long enough before decaying into flat space.”)
(That doesn’t help with the problems with the mathematical multiverse, or which computations a physical system implements.)
while the real deal is not a mentality evolution would ever select for
(Seems plausible in a eusocial species. Even universally caring about others seems plausible for a eusocial organism that will never interact with non-relatives.)
I don’t think I’ve ever heard it claimed that MIRI said this outright. The usual argument AIUI is: MIRI staff endorse/believe-in TDT (not that MIRI has stated it as organizational policy or anything), TDT endorses not paying off blackmail (and this is obvious/well-known), therefore MIRI staff violated their endorsed (and also, importantly, correct) principles.
Another thing I believe is: there’s no way to close the loop such that a closed system of LLMs can come up with new useful concepts and get those concepts into their own weights, e.g. open-ended self-distillation setups won’t work on LLMs.
Why not? The success of this architecture published two days ago (and its seeming resemblance to how human mathematical progress happens) updates me towards thinking such a thing would work, even though it probably doesn’t demonstrate “coming up with new useful concepts”.
It seems to me that deep cruelty, and the desire to regularly exercise it, is part of the default human-values package. If yes, CEVing random people is going to instantiate worlds in which it’s very present.
This doesn’t follow. Something could be part of the default human-values package, but also reliably discarded under reflection.
Moving, but I have to wonder if rushing was net positive in expected lives ex ante, or even expected lives saved from the Titanic not counting the Carpathia’s passengers. Quite plausibly it was and the Carpathia’s captain knew it and wouldn’t have done it otherwise, but I don’t know it reading the story.
Sure, whatever.
Do you think the distinction Jared is making isn’t real?
(I see a parallel to the current discourse about violence: some people assume transgression must be profitable, without having a reasonable story about how & despite the problems with such a story being obvious.)
Thanks! The distinction between “generating capabilities” and “hoovering up capabilities” is another small click for me.
‘why isn’t the thought process you’re using to say that surprised, and downvoted, by how little stuff we needed to make LLMs?’
After I read this comment, my hasty-guess-of-a-Tsvi-model replies: ‘the big surprise is that “solid performance on a wide range of technical tasks is not that connected to GI.” This surprise sufficiently explains the surprise of ~easily achieving that performance. Any ex ante expectation that those tasks required lots of understanding would/should have been mediated by expecting they required GI. Given that they don’t require GI, it’s [not surprising? / not relevantly surprising?] that they don’t require much understanding.’
This comment felt like it made a better model of your views click. ISTM you think something like:
All the impressive ML results so far have only worked either in a narrow subspace around the training data (e.g. LLMs, still mostly the case even with RL), or in very small worlds (e.g. pure-RL game-players). There has been ~zero progress on fluid/general intelligence. Therefore, extrapolating straight lines on graphs predicts ~zero progress on fluid/general intelligence by doing more of the same kind of thing. The induction on increasing ‘intelligence’ that lots of other people appeal to only works by inappropriate compression.
It’s still likely that we live in something like the 2011-Yudkowsky world as described in this tweet, with AGI to come from a lot of accumulation of insight. ML successes misleadingly make that world look falsified, if you aren’t tracking what they are and aren’t successes at.
(Implied) The fact that [ML results so far required surprisingly little understanding-of-intelligence] is not significant evidence that [other-things-you-might-expect-to-require-understanding, e.g. fluid intelligence, will require less understanding]. If we’ve learned something about how little understanding-of-intelligence was needed to build things that succeed on some tasks, this still just doesn’t say much about AGI.
(Or maybe you don’t believe that ‘fact’ about ML results so far, idk.)
Intuitively-to-me, there should be a big inductive update on this level, even if induction on ‘intelligence’ doesn’t work.
Like, it’s evidence against the way of thinking that says understanding of intelligence is important. When you say (implicitly) ‘we probably need lots of AGI seedstuff’, I want to say ‘why isn’t the thought process you’re using to say that surprised, and downvoted, by how little stuff we needed to make LLMs?’.
I didn’t mean it’s not name-calling, I meant “the administration” is not “half the population”.
if (our best model of) the laws of physics fit on a postcard, what exactly does it mean to need to do an experiment, in principle? You need experiments to nail down the laws. Beyond that, they’re convenient for reducing computational requirements
The general point is solid, but you also need experiments to learn contingent things within physics e.g. how biology works.
My guess is that taking seriously the very concept of manipulation often makes [the type of person you’re talking about] uncomfortable, because it undercuts an ethically load-bearing abstraction of rational agency, and its fuzziness threatens to license paternalism with (what might feel like) no principled limit. (This is a genuinely hard problem.) This cashes out similarly to ‘people who are manipulated deserve it’, but I think isn’t quite the same thing.
aggressive name-calling at half of the population
Without prejudice to your larger point, the OP is literally not doing this.
You might not be aware LW is open-source?
Though note the difference between ‘single-PC compute after more expensive experimental work’ (what it sounds like Steven is predicting, and Habryka is assuming) and ‘single-PC compute without that’ (what Adam is predicting).
Try the metaethics sequence, though I wish I had something shorter to give.