Agent-foundations researcher. Working on Synthesizing Standalone World-Models, aiming at a timely technical solution to the AGI risk fit for worlds where alignment is punishingly hard and we only get one try.
Currently looking for additional funders ($1k+, details). Consider reaching out if you’re interested, or donating directly.
Or get me to pay you money ($5-$100) by spotting holes in my agenda or providing other useful information.
Is the “fake vs real thinking” distinction about the same thing? Like, thinking which uses mental representations of the real world vs mental representations of other worlds?
I think it’s more that modeling/thinking about the real world requires constant effort of paying attention/updating/inferring. Not doing that and operating off of a dream mashup is the default. So it’s not that there are real vs. other-world representations, it’s that grounding yourself in the real world requires sophisticated inference-time cognitive work, rather than just retrieving cached computations and cobbling them together.
(And then LLMs could only do the cobbling-together thing, they’re always living in the dream mashup.)
Agreed. I think the most optimistic case is that peering at GPT-3/4′s interpreted form would make it extremely obvious how to train much more powerful models much more compute-efficiently by way of explicitly hard-coding high-level parts of their structure, thus simultaneously making them much more controllable/interpretable. (E. g., clean factorization into a world-model, a planner, and a goal slot, with obvious ways to scale up just the world-model while placing whatever we want into the goal slot. Pretty sure literally-this is too much to hope for, especially at GPT≤4′s level, but maybe something in that rough direction.)
In addition to Habryka’s point, there’s also granularity and signal strength:
A one-dimensional upvote is a lot less informative than specific feedback regarding why/how you liked it.
Receiving detailed feedback that someone took the time to write out and which shows clear engagement with your content feels a lot better than just seeing a number go up. (Notably, for me at least, detailed negative feedback, even if quite impolite, also feels better than just seeing a number go down for reasons you may not understand and not know how to fix.)
Insofar as you’ve missed reality’s ontology, things will just look like a mess
Or your thing just won’t work. There’s a kind of trade-off there, I think?
DL works because it gives a lot of flexibility for defining internal ontologies, and for compute-efficiently traversing their space. However, it does so by giving up all guarantees that the result would be simple/neat/easy-to-understand in any given fixed external ontology (e. g., the human one).
To combat that, you can pick a feature that would provide some interpretability assistance, such as “sparsity” or “search over symbolic programs”, and push in that direction. But how hard do you push? (How big is the penalty relative to other terms? Do you give your program-search process some freedom to learn neural-net modules for plugging into your symbolic programs?)
If you proceed with a light touch, you barely have any effect, and the result is essentially as messy as before.
If you turn the dial up very high, you strangle DL’s flexibility, and so end up with crippled systems. (Useful levels of sparsity make training 100x-1000x less compute-efficient; forget symbolic program search.)
In theory, I do actually think you may be able to “play it by ear” well enough to hit upon some method where the system becomes usefully more interpretable without becoming utterly crippled. You can then study it, and perhaps learn something that would assist you in interpreting increasingly less-crippled systems. (This is why I’m still pretty interested in papers like these.)
But is there a proper way out? The catch is that your interventions only hurt performance if they hinder DL’s attempts to find the true ontology. On the other hand, if you yourself discover and incentivize/hard-code (some feature of) the true ontology, that may actually serve as an algorithmic improvement.[1] It would constrain the search space in a helpful way, or steer the training in the right direction, or serve as a good initialization prior… Thus making the system both more interpretable and more capable.
Which is a boon in one way (will near-certainly be widely adopted; the “alignment tax” is negative), and a curse in another (beware the midpoint of that process, where you’re boosting capabilities without getting quite enough insight into models to ensure safety).
(Alternatively, you can try to come up with some Clever Plan where you’re setting up a search process that’s as flexible as DL but which somehow has a guaranteed of converging to be simple in terms of your fixed external ontology. I personally think such ideas are brilliant and people should throw tons of funding at them.)
May. There are some caveats there.
I’m confused how “they do directly hill climb on high profile metrics” is compatible with any of this, since that seems to imply that they do in fact train on benchmarks, which is the exact thing you just said was false?
I assume it means they use the benchmark as the test set, not the training set.
Ah, I see. Yeah, I think you’re right with this correction. (I was slightly misunderstanding what you were getting at before.)
Raoian Sociopaths are distinguished by being only able to interact via “power talk” (i. e., they’re at Simulacrum Level 4), and if I recall correctly, the same is true of Moral Mazes’ middle managers. The latter aren’t really buying into the signals, they don’t actually have loyalty to the company or whatever (which defines the Clueless). They’re just cynically exploiting all those mechanisms for self-advancement. Thus, they are Sociopaths.
Also, here’s a devastatingly powerful argumentum ad verecundiam in my interpretation’s favor.
Yeah, it’s not a good use of your time to seek that out. But if you do happen to stumble upon them (e. g., if they intruded into your garden, or if Zvi’s newsletter covered an incident involving them, with quotes), and a statement from them causes a twinge of “hm, there may be something to it...”, investigating that twinge may be useful. You shouldn’t necessarily crush it in a burst of cognitive dissonance/self-protectivenss.
Mm, I don’t think this maps on.
Raoian Sociopaths are very much engaging in intense signaling war with each other. Indeed, that’s all they do, they shave off their identities to do it, and they’re mostly not having a good time.
The Clueless are unaware of the signaling wars being played, they mostly take messages at face value. They’re not doing signaling in a calculated manner.
The Losers are not unaware of the game, nor necessarily unable to play it, they just don’t think playing it is a good way to accomplish their (orthogonal-to-the-game) goals.
So, if anything, Sociopaths are the losers’ bracket here, the Clueless are maybe the bottom tier (well, not really; I actually don’t think they map on to anything here), and everyone in the winners’ bracket here is a Roaian Loser (since they opted out of the game) (though not all Losers are winners).
I see! Understandable, but yep, I think you misjudged the inferential distance there a fair bit.
not all factual corrections will be delivered in pleasing forms
Yep, but I think your example does not go far enough. “Annoying but good-faith LW comments” is one thing, but valuable critique can also come from the mouths of openly sneering bad-faith internet trolls.[1] If you can keep yourself from over-updating on negativity into depression spirals, you may extract some actually useful feedback from your haters’ drivel! (You may want to route it through spite instead of empathy, though. As in, “this fool wants me to fail, but they’ve just brought me useful information, of their own volition!”.)
You know how that meme goes, “Heartbreaking: The Worst Person You Know Just Made a Great Point”.
I broadly agree. As part of that mindset: often, even if you think you’ve played yourself into a hopeless situation where no sequence of moves could avoid a loss, you should still continue taking moves that keep you in the game for as long as possible. The longer the game goes on, the more chances there are for your doom to dispel itself. Your opponent may make a blunder, you may spot a path to victory you’d missed, some external event may perturb the gameboard, a miracle may happen. Cling to survival tenaciously, no matter how pointlessly stubborn it seems.
Necessary caveats:
In some cases, this advice should be reversed. Some games you should give up as lost the moment you lose hope, so that you can redistribute your resources across still-winnable games, rather than wasting them.
The real thing we’re aiming to maximize here is “the probability that the situation would stop being hopeless”. So, in theory, between the policy A that keeps you in the game for 10 moves but has a 0.1% chance of a miracle per move, and the policy B that loses the game in 1 move but has a 2% chance of a miracle, you should pick B. But in practice, the miracle probability is often correlated with the expected time to the loss, so this works as a heuristic for achieving this objective.
(Obvious AI-doom implications: extending the timelines to extinction makes sense even if we have no idea regarding what we could do with more time, even if we expect that time to be wasted.)
Yeah, I mostly agree. I believe that a lot of premises and inferences in it are incorrect, but it’s a cogent and comprehensive distillation of a fairly popular model of the situation.
(JDP, you’re welcome to chime in and demonstrate that your writing was actually perfectly clear and that I’m just also failing basic reading comprehension.)
So clearly, Bostrom is not saying “the AI will not understand what we mean”
Consider the AI at two different points in time, AI-when-embryo early in training and AI-when-superintelligence at the end.
The quote involves Bostrom (a) literally saying that AI-when-superintelligence will understand what we meant,[1] (b) making a statement which logically implies, as an antecedent, that “AI-when-embryo won’t understand what we meant”.[2] Therefore, you can logically infer from this quote that Bostrom believes that the statement “AIs will in fact probably understand what we mean by the goals we give them before they are superintelligent” is false.
JDP, in my understanding, assumes that the reader would do just that: automatically zero-in on (b), infer the antecedent from it, and dismiss (a) as irrelevant context. I love it when blog posts have lil’ tricksy logic puzzles in them.
clearly that sentence is an atrocious sentence for demonstrating that “Bostrom said that the AI will not understand what we mean”
Yep.
My understanding: JDP holds that when the training process chisels a wrong goal into an AI because we gave it a wrong training objective (e. g., “maximize smiles” while we want “maximize eudaimonia”), this event could be validly described as the AI “misunderstanding” us.
So when JDP says that “AIs will in fact probably understand what we mean by the goals we give them before they are superintelligent”, and claims that this counters this Bostrom quote...
“The AI may indeed understand that this is not what we meant. However, its final goal is to make us happy, not to do what the programmers meant when they wrote the code that represents this goal.”
… what JDP means to refer to is the “its final goal is to make us happy, not to do what the programmers meant when they wrote the code that represents this goal” part, not the “the AI may indeed understand that this is not what we meant” part. (Pretend the latter part doesn’t exist.)
Reasoning: The fact that the AI’s goal ended up at “maximize happiness” after being trained against the “maximize happiness” objective, instead of at whatever the programmers intended by the “maximize happiness” objective, implies that there was a moment earlier in training when the AI “misunderstood” that goal (in the sense of “misunderstand” described in my first paragraph).
JDP then holds that this won’t happen, contrary to that part of Bostrom’s statement: that training on “naïve” pointers to eudaimonia like “maximize smiles” and such will Just Work, that the SGD will point AIs at eudaimonia (or at corrigibility or whatever we meant).[1] Or, in JDP’s parlance, that the AI will “understand” what we meant by “maximize smiles” well before it’s superintelligent.
If you think that this use of “misunderstand” is wildly idiosyncratic, or that JDP picked a really bad Bostrom quote to make his point, I agree.
(Assuming I am also not misunderstanding everything, there sure is a lot of misunderstanding around.)
Plus/minus some caveats and additional bells and whistles like e. g. early stopping, I believe.
I mean, no mind really exists at the time the “misunderstanding” is starting to happen, no? Unless you want to call a randomly initialized NN (i. e., basically a random program) a “mind”… Which wouldn’t necessarily be an invalid frame to use. But I don’t think it’s the obviously correct frame either, and so I don’t think that people who use a mechanistic frame by default are unambiguously in error.
I note that in your step-by-step explanation, the last bullet is:
Therefore Bostrom expects we will not have an AI that correctly understands concepts like intelligence until after it is already superintelligent.
That is straightforwardly correct. But “there exists no AI that understands” is importantly different from “there exists an AI which misunderstands”.
Another questionable frame here is characterizing the relationship between an AI and the SGD/the training process shaping it as some sort of communication process (?), such that the AI ending up misshapen can be described as it “misunderstanding” something.
And the training process itself never becomes a mind, it starts and ends as a discrete program, so if you mean to say that it “misunderstood” something, I think that’s a type error/at best a metaphor.
(I guess it may still be valid from a point of view where you frame SGD updates as Bayesian updates, or something along those lines? But that’s also a non-standard frame.)
This process can be rightfully called UNDERSTANDING and when an AI system fails at this it has FAILED TO UNDERSTAND YOU
No, the rightful way to describe what happens is that the training process generates an AI system with unintended functionality due to your failure to specify the training objective correctly. Describing it as a “misunderstanding” is tantamount to saying that if you make a syntax error when writing some code, the proper way to describe it is the computer “misunderstanding” you.
I mean, you can say that, it’s an okay way to describe things in a colloquial or metaphorical way. But I contest that it’s in any way standard language. You’re using idiosyncratic terminology and should in no way be surprised when people misunderstand (ha) you.
Honestly, if you went to modern-day LLMs and they, specialists in reading comprehension, misunderstood you, that ought to update you in the direction of “I made a bad job phrasing this”, not “it’s everyone else who’s wrong”.
(FYI, I understood what you meant in your initial reply to Habryka without this follow-up explanation, and I still thought you were phrasing it in an obviously confusing way.)
Curious whether you have any guesses on what would make it seem like a sympathetic decision to the audience
Off-the-cuff idea, probably a bad on:
Stopping short of “turning off commenting entirely”, being able to make comments to a given post subject to a separate stage of filtering/white-listing. The white-listing criteria are set by the author and made public. Ideally, the system is also not controlled by the author directly, but by someone the author expects to be competent at adhering to those criteria (perhaps an LLM, if they’re competent enough at this point).
The system takes direct power out of the author’s hands. They still control the system’s parameters, but there’s a degree of separation now. The author is not engaging in “direct” acts of “tyranny”.
It’s made clear to readers that the comments under a given post have been subject to additional selection, whose level of bias they can estimate by reading the white-listing criteria.
The white-listing criteria are public. Depending on what they are, they can be (a) clearly sympathetic, (b) principled-sounding enough to decrease the impression of ad-hoc acts of tyranny even further.
(Also, ideally, the system doing the selection doesn’t care about what the author wants beyond what they specified in the criteria, and is thus an only boundedly and transparently biased arbiter.)
The commenters are clearly made aware that there’s no guarantee their comments on this post will be accepted, so if they decide to spend time writing them, they know what they’re getting into (vs. bitterness-inducing sequence where someone spends time on a high-effort comment that then gets deleted).
There’s no perceived obligation to respond to comments the author doesn’t want to respond to, because they’re rejected (and ideally the author isn’t even given the chance to read them).
There are no “deleting a highly-upvoted comment” events with terrible optics.
Probably this is still too censorship-y, though? (And obviously doesn’t solve the problem where people make top-level takedown posts in which all the blacklisted criticism is put and then highly upvoted. Though maybe that’s not going to be as bad and widespread as one might fear.)
Political games can, at best, get stuff from other people. The good stuff—the real power—is the stuff which other people don’t have to offer in the first place. The stuff which nobody is currently capable of doing/making.
Yep.
I think this generalizes to competition against people in general? As in, if you find yourself in a situation where you’re competing neck-to-neck with others, with a real possibility of losing, without an unsurpassable lead on them, you should stop that (if you can) and do something else.
Some examples:
Business. Various standard advice (I believe shared by Graham, Thiel, and Spolsky[1]) regarding building a major business is to start by monopolizing some extremely niche market in which no real competitor exists. Once you’ve eaten that market, you incrementally expand your niche, always shying away from domains with real competitors until you can crush them (i. e., until they stop being real competitors).
Physical fights. The best way to deal with someone attacking you is not to be there to begin with (e. g., earn money and move to a safer country/city). Failing that, make yourself urgently not-there by running away (and get good at that, e. g. practice sprinting + parkour). Failing that, bring a (metaphorical or real) gun to a knife fight. “Learn martial arts”, on the other hand, is not a good approach.
War. Maneuver warfare, where you achieve strategic victory by a sequence of precisely planned, decidedly “unfair” tactical operations, is dramatically preferable to direct WW1-style positional-warfare slugfests.
The post provides plenty of social-conflict examples.
In some way, the point may be obvious. Fair fights against comparably powerful opponents are, by definition, challenging, and also anti-inductive. They drain both sides’ resources, force races to the bottom and tons of other negative-sum dynamics, etc. You want either an overwhelming asymmetric advantage, or a fight to which no-one else will show up.
At least, no-one intelligent. Picking a “fight” with Nature, or abstract concepts, is fine. Indeed, those are the kinds of fights you should be picking. Generally speaking, you want to be doing things where success comes from spending ~all of your time thinking about some object-level problem, and ~none of your time keeping tabs on your human adversaries.
Some caveats/clarifications:
The object-level problem may still involve reasoning about agents, about systems containing people, etc., and even planning against them. What you don’t want are symmetric agentic competitors which are doing basically the same thing as you (for appropriate values of “symmetric” and “same thing”). In other words: you want to be the only live player on the board.
You’d of course still be “technically” in various kinds of competitions: competing in a job market, fighting a war. That’s fine, as long as it’s not a real competition from your perspective.
Obviously you may not always have this option available. (Good business ideas may take time to develop, and you need to eat in the meantime. Same for e. g. climbing out of poverty and moving to a safer city.)
Exercise for the reader: evaluate various alignment plans through these lens, and consider how a non-loser’s alignment plan ought to look like (and how it ought not to look).
Not gonna track down exact sources, sorry.
Per above, we’d need tighter feedback loops/quicker updates, appropriate markings of when content/procedures become outdated, some ability to compare various elements of constructed realities against the ground truth whenever the ground truth becomes known, etc. (Consider if Google Maps were updating very slowly, and also had some layer on top of its object-level observations whose representations relied on chains of inferences from the ground-true data but without quick ground-true feedback. That’d gradually migrate it to a fictional world as well.)
The general-purpose solution is probably some system that’d incentivize people to flag divergences from reality… Prediction markets?