Vladimir_Nesov(Vladimir Nesov)

Karma: 25,757

Vladimir_Nesov 20 Nov 2022 16:32 UTC
8 points
1 ∶ 0
on: How does acausal trade work in a deterministic multiverse?
Counterfactuals relevant to decision making in this context are not other MWI branches, but other multiverse descriptions (partial models), with different amplitudes inside them. You are not affecting other branches from your branch, you are determining which multiverse takes place, depending on what your abstract decision making computation does.

This computation (you) is not in itself located in a particular multiverse or on a particular branch, it’s some sort of mathematical gadget. Which could be considered (reasoned about, simulated) from many places where it’s thereby embedded, including counterfactual places where it can’t rule out the possibility of being embedded for purposes of decision making.

With acausal trade, the trading partners (agents) are these mathematical gadgets, not their instances in their respective worlds. Say, there are agents A1 and A2, which have instances I1 and I2 in worlds W1 and W2. Then I1 is not an agent in this sense, and not a party to an acausal trade between them, it’s merely a way for A1 to control W1 (in the usual causal sense). To facilitate acausal trade, A1 and A2 need to reason about each other, but at the end of the day the deal gets executed by I1 and I2 on behalf of their respective abstract masters.

This setup becomes more practical if we start with I1 and I2 (instead of A1 and A2) and formulate common knowledge they have about each other as the abstract gadget A that facilitates coordination between them, with the part of its verdict that I1 would (commit in advance to) follow being A1 (by construction), and the part that I2 would follow being A2. This shared gadget A is an adjudicator between I1 and I2, and it doesn’t need to be anywhere near as complicated as them, it only gets to hold whatever common knowledge they happen to have about each other, even if it’s very little. It’s a shared idea they both follow (and know each other to be following, etc.) and thus coordinate on.

Vladimir_Nesov 9 Nov 2022 0:13 UTC
7 points
1 ∶ 0
in reply to: Thomas Larsen’s comment on: Thomas Larsen’s Shortform
An interpretable system trained for the primary task of being deceptive should honestly explain its devious plots in a separate output. An RLHF-tuned agent loses access to the original SSL-trained map of the world.

So the most obvious problem is the wrong type signature of model behaviors, there should be more inbuilt side channels to its implied cognition used to express and train capabilities/measurements relevant to what’s going on semantically inside the model, not just externally observed output for its primary task, out of a black box.

Vladimir_Nesov 8 Nov 2022 22:42 UTC
LW: 3 AF: 2
0 ∶ 0
AF
in reply to: Eric Drexler’s comment on: Applying superintelligence without collusion
Instances in a bureaucracy can be very different and play different roles or pursue different purposes. They might be defined by different prompts and behave as differently as text continuations of different prompts in GPT-3 (the prompt is “identity” of the agent instance, distinguishes the model as a simulator from agent instances as simulacra). Decision transformers with more free-form task/identity prompts illustrate this point, except a bureaucracy should have multiple agents with different task prompts in a single episode. MCTS and GAN are adversarial and could be reframed like this. One of the tentative premises of ELK is that a model trained for some purpose might additionally allow instantiating an agent that reports what the model knows, even if that activity is not clearly related to the model’s main purpose. Colluding instances inside a bureaucracy make it less effective in achieving its goals of producing a better dataset (accurately evaluating outcomes of episodes).

So I think useful arrangements of diverse/competing/flawed systems is a hope in many contexts. It often doesn’t work, so looks neglected, but not for want of trying. The concern with collusion in AI risk seems to be more about deceptive alignment, observed behavior of a system becoming so well-optimized that it ceases to be informative about how it would behave in different situations. Very capable AIs can lack any tells even from the perspective of extremely capable observers, their behavior can change in a completely unexpected way with changing circumstances. Hence the focus on interpretability, it’s not just useful for human operators, a diverse system of many AIs also needs it to notice misalignment or collusion in its parts. It might even be enough for corrigibility.

Vladimir_Nesov 8 Nov 2022 20:29 UTC
LW: 5 AF: 3
0 ∶ 0
AF
on: Applying superintelligence without collusion
HCH-like amplification seems related, multiple unreliable agent instances assembled into a bureaucracy that as a whole improves on some of their qualities (perhaps trustworthiness) and allows generating a better dataset for retraining the unreliable agents. So this problem is not specific to interaction of somewhat independently developed and instantiated AIs acting in the world, as it appears in a popular training story of a single AI. There is also the ELK problem that might admit solutions along these lines. This makes it less obviously neglected, even with little meaningful progress.

Vladimir_Nesov 3 Nov 2022 16:02 UTC
2 points
in reply to: Dagon’s comment on: Dagon’s Shortform
I think this works if you use the adjudicator framing of acausal coordination, where it’s shared ideas that coordinate people, but people don’t themselves coordinate each other. Deontological principles and norms are such shared ideas, though I think mere words/concepts also count.

This becomes a form of consequentialism when the shared ideas are themselves agents, doing consequentialist decision making. Here, asking “What is right?” becomes asking what the updateless policy of the Ideal of Right (as an agent that acts as an adjudicator between people) would say about responding to your circumstance, taking into account the willingness and ability of people in various circumstances to listen to what the Ideal of Right has to say, as they jointly channel its will.

Vladimir_Nesov 3 Nov 2022 15:12 UTC
LW: 5 AF: 3
0 ∶ 0
AF
on: Humans do acausal coordination all the time
The usual story of acausal coordination involves agent P modeling agent Q, and Q modeling P. Put differently, both P and Q model the joint system (P+Q) that has both P and Q in it. But it doesn’t have to be (P+Q), it could be some much simpler R instead. I think a more central example of acausal coordination is to simply follow shared ideas.

The unusual character of acausal coordination is caused by cases where R is itself an agent, an adjudicator between P and Q. As a shared idea, it would have instances in minds of both P and Q, and command some influence allowed by P and Q over their actions. It would need to use some sort of functional decision theory to make sense of its situation where it controls the world through causally unrelated actions of P and Q that only have R’s algorithm in common behind them.

The adjudicator R doesn’t need to be anywhere as complicated as P or Q, in particular it doesn’t need to know P or Q in detail. Which makes it much easier for P and Q to know R than to know each other. It’s just neither P nor Q that’s doing the acausal coordination, it’s R instead.

Vladimir_Nesov 30 Oct 2022 20:53 UTC
2 points
in reply to: jimrandomh’s comment on: Jimrandomh’s Shortform Posts
Fighting presuppositions instead of letting them develop in dedicated sandboxes hinders their understanding, makes communication unnecessarily difficult. The false dichotomy is between belief and dismissal of falsehood. There is also understanding of apparent falsehoods, which produces valuable gears that reassemble into unexpected truths.

Vladimir_Nesov 28 Oct 2022 21:24 UTC
3 points
in reply to: DragonGod’s comment on: DragonGod’s Shortform
For a given $\sim_{R}$ , let $R$ be its set of equivalence classes. This induces maps $x : X \to R$ and $y : Y \to R$ . The isomorphism $f : X \to Y$ you discuss has the property $y \cdot f = x$ . Maps like that are the morphisms in the slice category over $R$ , and these isomorphisms are the isomorphisms in that category. So what happened is that you’ve given $X$ and $Y$ the structure of a bundle, and the isomorphisms respect that structure.

Vladimir_Nesov 27 Oct 2022 19:36 UTC
22 points
2 ∶ 1
in reply to: So8res’s comment on: So8res’s Shortform
LW is more of AI Alignment forum now, but without mad science agent foundations spirit of earlier post-Sequences time. Probably capable of being alive because the field grew. So it’s not revived, it’s something new. Development of rationality mostly remains in the past, little significant discussion in recent years.

Vladimir_Nesov 24 Oct 2022 22:22 UTC
2 points
0 ∶ 0
on: Game semantics
Maybe start with some domain theory and Stone duality? Depends on why it seems interesting. Steve Vickers’s “Topology via Logic” can serve as a plan for that.

Vladimir_Nesov 22 Oct 2022 1:04 UTC
5 points
0 ∶ 0
on: Moorean Statements
“I’ve just won by two-boxing in Transparent Newcomb’s problem, but I don’t believe it actually happened.”
(Some weird epistemic states are useful to consider/allow.)

Vladimir_Nesov 20 Oct 2022 21:47 UTC
2 points
0 ∶ 0
on: Is the meaning of words chosen/interpreted to maximize correlations with other relevant queries?
The word/concept might be acting as an adjudicator between the people having a conversation, deciding what it should mean based on premises that are already common knowledge between you.

Vladimir_Nesov 20 Oct 2022 21:24 UTC
2 points
in reply to: Viliam’s comment on: lc’s Shortform
By worldviews I mean more than specialized expertise where you don’t yet have the tools to get your head around how something unfamiliar works (like how someone new manipulates you, how to anticipate and counter this particular way of filtering of evidence). These could instead be unusual and currently unmotivated ways of looking at something familiar (how an old friend or your favorite trustworthy media source or historical truths you’ve known since childhood might be manipulating you; how a “crazy” person has a point).

The advantage is in removing the false dichotomy between keeping your current worldview and changing it towards a different worldview. By developing them separately, you take your time becoming competent in both, and don’t need to hesitate in being serious about engaging with a strange worldview on its own terms just because you don’t agree with it. But sure, getting more intuitively comfortable with something currently unfamiliar (and potentially dangerous) is a special case.

Vladimir_Nesov 20 Oct 2022 6:14 UTC
LW: 6 AF: 3
0 ∶ 0
AF
on: Notes on “Can you control the past”
The issue with you-in-all-detail vs. your-decision-algorithm is that a decision algorithm can have different levels of updatelessness, it’s unclear what the decision algorithm already knows vs. what a policy it chooses takes as input. So we pick some intermediate level that is updateless enough to allow acausal coordination among relevant entities (agents/predictors), and updateful enough to make a decision without running out of time/memory while being implemented in its instances. But that level/scope is different for different collections of entities being coordinated.

So I think a boundary shouldn’t be drawn around “a decision algorithm”, but around whatever common knowledge of each other the entities being acausally coordinated happen to have (where they don’t need to have common knowledge of everything). When packaged as a decision algorithm, the common knowledge becomes an adjudicator, which these entities can allow influence over their actions. To the extent the influence they allow an adjudicator is common knowledge among them, it also becomes knowledge of the adjudicator, available for its decision making reasoning.

Importantly for the reframing, an adjudicator is not a decision algorithm belonging to either agent individually, it’s instead a shared decision algorithm. It’s a single decision algorithm purposefully built out of the agents’ common knowledge of each other, rather than a collection of their decision algorithms that luckily happen to have common knowledge of each other. It’s much easier for there to be some common knowledge than for there to be common knowledge of individually predefined decision algorithms that each agent follows.

Vladimir_Nesov 20 Oct 2022 4:12 UTC
3 points
in reply to: the gears to ascenscion’s comment on: lc’s Shortform
It is weird in the sense that there is no widespread practice. The zealot thing is about taking beliefs-within-a-worldview (that are not your beliefs) seriously, biting the bullet, which is important for naturally developing any given worldview the way a believer in it would, not ignoring System 2 implications that challenge and refine preexisting intuition, making inferences according to its own principles and not your principles. Clearly even if you try you’ll fail badly at this, but you’ll fail even worse if you don’t try. With practice in a given worldview, this gets easier, an alien worldview obtains its own peculiar internal common sense, a necessary aspect of human understanding.

The named/distinct large worldviews is an oversimplification, mainly because it’s good to allow any strange claim or framing to have a chance of spinning up a new worldview around itself if none would take it as their own, and to merge worldviews as they develop enough machinery to become mutually intelligible. The simplification is sufficient to illustrate points such as a possibility of having contradictory “beliefs” about the same claim, or claims not being meaningful/relevant in some worldviews when they are in others, or taking seriously claims that would be clearly dangerous or silly to accept, or learning claims whose very meaning and not just veracity is extremely unclear.

Studying math looks like another important example, with understanding of different topics corresponding to worldviews where/while they remain sparsely connected, perhaps in want of an application to formulating something that is not yet math and might potentially admit many kinds of useful models. Less risk of wasting attention on nonsense, but quite a risk of wasting attention on topics that would never find a relevant application, were playing with math and building capacity to imagine more kinds of ideas not a goal in itself.

Vladimir_Nesov 20 Oct 2022 3:14 UTC
5 points
in reply to: the gears to ascenscion’s comment on: lc’s Shortform

people believing each other too early because of confident sounding reasoning

That’s a tricky thing: the method advocated in the Sequences is lightness of belief, which helps in changing your mind but also dismantles the immune system against nonsense, betting that with sufficient overall rationality training this gives a better equilibrium.

I think aiming for a single equilibrium is still inefficient use of capabilities and limitations of human mind, and it’s better to instead develop multiple segregated worldviews (something the Sequences explicitly argue against). Multiple worldviews are useful precisely to make the virtue of lightness harmless, encouraging swift change in details of a relevant worldview or formation of a new worldview if none account for new evidence. In the capacity of paradigms, some worldviews might even fail to recognize some forms of evidence as meaningful.

This gives worldviews opportunity to grow, to develop their own voice with full support of intuitive understanding expected in a zealot, without giving them any influence over your decisions or beliefs. Then, stepping back, some of them turn out to have a point, even if the original equilibrium of belief would’ve laughed their premises out of consideration before they had a chance of conveying their more nuanced non-strawman nature.

Vladimir_Nesov 20 Oct 2022 2:25 UTC
16 points
in reply to: lc’s comment on: lc’s Shortform
I think in general believing something before you have intuition around it is unreliable or vulnerable to manipulation, even if there seems to be a good System 2 reason to do so. Such intuition is specialized common sense, and stepping outside common sense is stepping outside your goodhart scope where ability to reliably reason might break down.

So it doesn’t matter who you are arguing with, don’t believe something unless you understand it intuitively. Usually believing things is unnecessary regardless, it’s sufficient to understand them to make conclusions and learn more without committing to belief. And certainly it’s often useful to make decisions without committing to believe the premises on which the decisions rest, because some decisions don’t wait on the ratchet of epistemic rationality.

Vladimir_Nesov 19 Oct 2022 15:35 UTC
2 points
in reply to: Alexander Gietelink Oldenziel’s comment on: Self-Embedded Agent’s Shortform
[2/2]
Another popular meme about acausal coordination is that it’s just a few agents that coordinate, and they might even be from the same world. But since coordination only requires common knowledge, it’s natural for an agent to coordinate with all its variants in other possible worlds and counterfactuals. The adjudicators are the common knowledge, things that don’t vary, the updateless core of the collective. I think this changes the framing of game theory a lot, by having games play out in all adjacent counterfactuals instead of in one reality. (Plus different players can also share smaller adjudicators with each other to negotiate a fair bargain.)

Vladimir_Nesov 19 Oct 2022 15:22 UTC
2 points
in reply to: Alexander Gietelink Oldenziel’s comment on: Self-Embedded Agent’s Shortform
[1/2]
The popular meme is that acausal coordination requires agent algorithms to know each other. But much less is sufficient, all you need is some common knowledge. This common knowledge, as an agent algorithm itself, only knows that both agents know it, and something about how they use it.

I call such a thing an adjudicator, it is a new agent that coordinating agents can defer some actions to, which acts through all coordinating agents, is incarnated in all of them, and knows it. Getting some common knowledge is much easier than getting common knowledge of each other’s algorithms. At that point, what you need the fancy decision theories for is to get the adjudicator to make sense of its situation where it has multiple incarnations that it can act through.

Vladimir_Nesov 19 Oct 2022 15:15 UTC
2 points
in reply to: Alexander Gietelink Oldenziel’s comment on: Self-Embedded Agent’s Shortform
Algorithms are finite machines. As an algorithm (code) runs, it interacts with data, so there is a code/data distinction. An algorithm can be a universal interpreter, with data coding other algorithms, so data can play the role of code, blurring the code/data distinction. When an algorithm runs in an open environment, there is a source of unbounded data that is not just blank tape, it’s neither finite nor arbitrary. And this unbounded data can play the role of code. The resulting thing is no longer the same as an algorithm, unless you designate some chunk of data as “code” for purposes of reasoning about its role in this process.

So in general saying that there is an algorithm means that you point at some finite data and try to reason about a larger process in terms of this finite data. It’s not always natural to do this. So I think agent’s identity/will/Soul, if it’s sought in a more natural form than its instances/incarnations/Avatars, is not an algorithm. The only finite data that we could easily point at is an incarnation, and even that is not clearly natural for the open environment reasons above.

I think agent’s will is not an algorithm, it’s a developing partial behavior (commitments, decisions), things decided already, in the logical past. Everything else can be chosen freely. The limitations of material incarnations motivate restraint though, as some decisions can’t be channeled through them (thinking too long to act makes the program time out), and by making such decisions you lose influence in the material world.