When I read word “bargaining” I assume that we are talking about entities that have preferences, action set, have beliefs about relations between actions and preferences and exchange information (modulo acausal interaction) with other entities of the same composition. Like, Kelly betting is good because it equals to Nash bargaining between versions of yourself from inside different outcomes and this is good because we assume that you in different outcomes are, actually, agent with all arrtibutes of agentic system. Saying “systems consist of parts, this parts interact and sometimes result is a horrific incoherent mess” is true, but doesn’t convey much of useful information.
quetzal_rainbow
I feel like the whole “subagent” framework suffers from homunculus problem: we fail to explain behavior using the abstraction of coherent agent, so we move to the abstraction of multiple coherent agents, and while it can be useful, I don’t think it displays actual mechanistic truth about minds.
When I plan something and then fail to execute plan it’s mostly not like “failure to bargain”. It’s just when I plan something I usually have good consequences of plan in my imagination and this consequences make me excited and then I start plan execution and get hit by multiple unpleasant details of reality. Coherent structure emerges from multiple not-really-agentic pieces.
It doesn’t matter? Like, if your locations are identical (say, simulations of entire observable universe and you never find any difference no matter “where” you are), your weight is exactly the weight of program. If you expect dfferences, you can select some kind of simplicity prior to weight this differences, because there is basically no difference between “list all programs for this UTM, run in parallel”.
Okay, we have wildly different models of tech tree. In my understanding, to make mind uploads you need Awesome Nanotech and if you have misaligned AIs and not-so-awesome nanotech it’s sufficient to kill all humans and start to disassemble Earth. The only coherent scenario that I can imagine misaligned AIs actually participating in human economy in meaningful amounts is scenario where you can’t design nanotech without continent-sized supercomputers.
But it still feels that the lesson could be summarized as: “talk like everyone outside the rationalist community does all the time”.
If non-rationalist people knew it all along, there wouldn’t be need to write such books.
On the other hand, I think if average rationalist person tries to say speech from pure inspiration, the result is going to be weird. Like, for example, speech of HJPEV before the first battle. HJPEV got away with this, because he has reputation of Boy Who Lived and he already pulled some awesome shenanigans, so his weird speech got him weirdness points instead of losing them, but it’s not the trick average rationalist should try on first attempt to say inspiring speech.
It’s kinda ill-formed question, because you can get the same performance if you compute moves longer with lower power. I guess you are searching for something like”energy per move”.
You somehow managed to misunderstand me in completely opposite direction. I’m not talking about size of the universe, I’m talking about complexity of description of the universe. Description of the universe consists of initial conditions and laws of evolution. The problem with hidden variables hypotheses is that they postulate initial conditions of enormous complexity (literally, they postulate that at the start of the universe list of all coordinates and speeds of all particles exists) and then postulate laws of evolution that don’t allow to observe any differences between these enourmously complex initial conditions and maximum-entropy initial conditions. Both are adding complexity, but hidden variables contain most of it.
I’m talking about probabilities. Aligned AIs want things that we value in 100% cases by definition. Unaligned AIs can want things that we value and things that we don’t value at all. Even if we live in very rosy universe where unaligned AIs want things that we value in 99% of cases, 99% is strictly less than 100%.
My general objection was to the argumentation based on likelihood of consciousness in AIs as they developed without accounting for “what conscious AIs actually want to do with their consciousness”, which can be far more important because the feature of intelligence is the ability to turn unlikely states into likely.
The general problem with “more intuitive metaphysics” is that your intuition is not my intuition. My intuition finds zero problem with many worlds interpretation.
And I think you underestimate complexity issues. Many worlds interpretation requires as many information as all wave functions contain, but pilot wave requires as many information as required to describe speed and position of all particles compatible with all wave functions, which for universe with 10^80 particles requires c*10^80, c>=1 additional bits, which drives Solomonoff probability of pilot wave interpretation somewhere into nothing.
The reason why unaligned AIs are more likely to be unconscious in long-term is because consciousness is not the most efficient way to produce paperclips. Even if first paperclip-optimizer is conscious, it has no reason to keep consciousness once it find better way to produce paperclips without consciousness.
You have really weird beliefs about the past.
The general mechanism for dating 200 yers ago was arranged marriage. It not always was forced, you could refuse in really uncomfortable cases, but social pressure was immensive and if you was, like, peasant, you considered your comfortable survival much before your personal feelings. Yep, it probably didn’t feel like optimization number-crunching, but this was because all optimization was from outside—people who didn’t follow the custom simply died.
And I don’t even talk about nice family optimization task “You are peasant in Russia in 19th century, and it’s famine outside, you should choose what child you are going to stop feeding, because it’s less condemnable practice than abortion”. Or “You are peasant in Russia in 19th century, and it’s famine outside, so you need to choose which child to kick out of the house for them to become factory workers (if they are lucky) or beggars or thieves or prostitutes (child prostitution in Russian Empire was not uncommon)”.
Same with kings and warlords of the past, I expect that they had more freedom of choice
You enemies could be less tactically skilled, but your mistakes killed you in the same amount.
Were people forced to go to war (as an obvious optimal strategy), or did nationalism and concepts like honor and bravery motivate people
If you were medieval peasant, you basically didn’t have money to have weapon and armor and you mostly didn’t have any choice other than suffer the consequences of war. If you were somewhat richer and lived somewhat later, you could go to be mercenary, because war was rare profitable enterprise before capitalism. And if you lived in era of nation states, you usually was drafted in army and had choice between prison/katorga/execution on spot and going to war because your government told you so.
I picture here too dark image of the past, and I need to say that even in this conditions people could find multiple cracks in social order and widen them if they were lucky and creative, and modern times have much more space for such cracks.
Yes, I agree that we lost some freedoms—we have closed borders between nation states and inscrutable bureaucracy and electronic surveillance and schools are like prisons (but less so than in times when corporeal punishment was widespread) and our status games are absolutely fucked up and heterodoxy in academia is somewhat strained and there are authoritarian states but this seems to be so much more of skill issue than soul-crushing indifference of the universe in the past.
Ukrainians don’t need to join Western culture, they are Western culture. They watched American action movies in 80s and their kids watched Disney and Warner Brothers in 90s and read Harry Potter in 2000s and was on Tumblr in 10s. And I do not even mention that Imperial Russian/Soviet cultures were bona fide Western cultures, and national Ukrainian culture is no less Western than Poland or Czech culture.
Problem with scammers is that they do not report successful penetration of defense.
If someone convinces themself that a full nuclear exchange would prevent the development of superhuman AI
I think the problem here is “convinces themself”. If you are capable to trigger nuclear war, you are probably capable to do something else which is not that, if you put your mind in that.
If you are capable to use AI to do harmful and costly thing, like “melt GPUs”, you are in hard takeoff world.
I always thought “you should use the least advanced superintelligence necessary”. I.e., in not-real-example of “melting all GPUs” your system should be able to design nanotech advanced enough to target all GPUs in open enviroment, which is superintelligent task, while not being able to, say, reason about anthropics and decision theory.
You are conflating “what humans own” with “what you can get by process with side effect of killing humans”. Humans are not going to own any significant chunk of Earth in the end, they are just going to live on its surface and die when this surface will evaporate during disassembling into Dyson swarm, and all of this 6*10^24 kg of silicon, hydrogen, oxygen and carbon are quite valuable. What does, exactly, prevent this scenario?
I claim that my scenario is not just possible, it’s default outcome (conditional on “there are multiple misaligned AIs which for some reason don’t just foom”).
Will some genetically engineered humans have misaligned goals? The answer here is almost certainly yes.
If by “misaligned” all we mean is that some of them have goals that are not identical to the goals of the rest of humanity, then the answer is obviously yes. Individuals routinely have indexical goals (such as money for themselves, status for themselves, taking care of family) that are not what the rest of humanity wants.
If by “misaligned” what we mean is that some of them are “evil” i.e., they want to cause destruction or suffering on purpose, and not merely as a means to an end, then the answer here is presumably also yes, although it’s less certain.
This is very strange reasoning. Misaligned goals mean that entity basically doesn’t care about our existence or well-being, it doesn’t gain anything from us being alive and well relatively to us turning into paperclips. For genetically engineered humans reversal is very likely to be true: they are going to love other humans, be friends with them or take pride from position in human social hierarchy, even if they are selfish by human standards, and it is not clear why they should be selfish.
Generalized chess is EXPTIME-complete and while chess “exact solution” may be unavailable, we are pretty good at constructing chess engines.