# kave

Karma: 184
• 3 Aug 2022 15:17 UTC
LW: 7 AF: 5
2 ∶ 0
AF
in reply to: Lauro Langosco’s comment

If you assume the human brain was trained roughly optimally, then requiring more data, at a given parameter number, to be optimal pushes timelines out. If instead you had a specific loss number in mind, then a more efficient scaling law would pull timelines in.

• My impression was that “zero-sum” was not used in quite the standard way. I think the idea is the AI will cause a big reassignment of Earth’s capabilities to its own control. And that that’s contrasted with the AI massively increasing its own capabilities and thus Earth’s overall capabilities.

• The Shannon entropy of a distribution over random variable conditional on the value of another random variable can be written as

If X and C are which face is up for two different fair coins, H(X) = H(C) = −1. But ? I think this works out fine for your case because (a) I(X,C) = H(C): the mutual information between C (which well you’re in) and X (where you are) is the entropy of C, (b) H(C|X) = 0: once you know where you are, you know which well you’re in, and, relatedly (c) H(X,C) = H(X): the entropy of the joint distribution just is the entropy over X.

• Good point!

It seems like it would be nice in Daniel’s example for P(A|ref) to be the action distribution of an “instinctual” or “non-optimising” player. I don’t know how to recover that. You could imagine something like an n-gram model of player inputs across the MMO.

• Nitpick: to the extent you want to talk about the classic example, paperclip maximisers are as much meant to illustrate (what we would now call) inner alignment failure.

See Arbital on Paperclip (“The popular press has sometimes distorted the notion of a paperclip maximizer into a story about an AI running a paperclip factory that takes over the universe. [...] The concept of a ‘paperclip’ is not that it’s an explicit goal somebody foolishly gave an AI, or even a goal comprehensible in human terms at all.”) or a couple of EY tweet threads about it: 1, 2

• I agree on the “reference” distribution in Daniel’s example. I think it generally means “the distribution over the random variables that would obtain without the optimiser”. What exactly that distribution is /​ where it comes from I think is out-of-scope for John’s (current) work, and I think is kind of the same question as where the probabilities come from in statistical mechanics.

# PSA: The Se­quences don’t need to be read in sequence

23 May 2022 2:53 UTC
85 points
• Not quite! If there were no central bank, money’s value would not jump around aggressively and discontinuously

• Full flights have more people on them. If you have 100 flights with one person and 1 flight with 200 people, most of the people in those flights are on the 200 person flight.

• Thank you.

And have fun!

• Wonderful method! I am a poop brain. Manifold rules ~~~

• I would buy the Leg Cones but am holding off for the sake of the epistemic commons. If my butt cheek were wrong, no one would want to be right.

• A superintelligent AI turns its whole genius to the task of producing a string that will explain to anyone who reads it how to create a perfect battery. Easy enough to produce explanatory strings, though there’s always some risk that someone, somewhere won’t understand how to create the battery. And then there’s the question of making sure that the battery is perfect. Not just better than any other battery. Not just 99.99% of optimal. Perfect.

It isn’t smart enough to solve this problem. Not yet. And there’s a countdown … at some point the AI will get interrupted, rebooted, shut off. If the AI can get out of the box, if it can get more actuators, it can start to get smarter and to get more resources. Obviously, those who want to turn it off will have to be dealt with swiftly …

What can the AI say to make the human unwittingly help it in its plans? Is there anything at all? The AI thinks.

• 1 Apr 2022 20:30 UTC
3 points