Discord: LemonUniverse (.lemonuniverse). Reddit: u/Smack-works. About my situation: here.
I wrote some bad posts before 2024 because I was very uncertain how the events may develop.
I do philosophical/conceptual research, have no mathematical or programming skills. But I do know a bunch of mathematical and computer science concepts.
He has a temptation to believe in simulation. But doesn’t he also have a counter-temptation to believe he’s the real Elon, not a random dude?
Not all epistemological things can be justified (rejecting modest epistemology, simulation hypothesis, Pascal mugging, other weird anthropic scenairos). At least it’s very unclear if they can be. So why should this one be justified? What kind of “Hilbert program” are you pursuing, how much do you want to justify in which foundation?
How likely is it, actually, that you are the real Elon Musk, vs. just some dude
Wait, isn’t this a bit of a too large humble pill to swallow?
“I just look at them with my eyes and remember who they are.” Should the alien conclude that the human is being uncooperative, and apply the crowbar?
With the crowbar on the table I’ll at least explain that the face splits into other high-level features (eyes, nose, mouth, etc.) which can be very memorable on their own.
As the bar inches closer I’ll venture into something more vague, about contrasts of features or associations or how features affect emotions.
...But that misses the point. On the topic of the tweet: doesn’t Eliezer acknowledge what you talk about later?
But, sure, maybe all the things that LLMs know about gravity and medicine and cups on tables is one kind of knowledge to them, and all the wordless secrets they know about humans are in a hidden and incommensurable format.
Eliciting Latent Knowledge problem for the unfamiliar:
Suppose we train a model to predict what the future will look like according to cameras and other sensors. We then use planning algorithms to find a sequence of actions that lead to predicted futures that look good to us.
But some action sequences could tamper with the cameras so they show happy humans regardless of what’s really happening. More generally, some futures look great on camera but are actually catastrophically bad.
In these cases, the prediction model “knows” facts (like “the camera was tampered with”) that are not visible on camera but would change our evaluation of the predicted future if we learned them. How can we train this model to report its latent knowledge of off-screen events?
So we have humans with their human ontology, the Predictor (the model we need to interpret), the Planner (can be ignored for simplicity), and maybe we’re training a Reporter (intended to interpret the Predictor).
We assume the Predictor is inner-aligned to myopically making predictions, it won’t strategically mislead us by outputting wrong information at the right time. We also assume that something strongly correlated with the diamond (Z) is important for the Predictor’s ability to make predictions (this is ARC’s working definition of knowledge).
I was wondering whether the problem is possible to solve with brute force, at least if we have unlimited compute:
Formalize the human model of physics. Let’s assume the model is true enough or we can check if it’s true enough.
Prove that the Predictor’s model reduces to the human model under normal conditions. This will give you a mapping between human ontology and the Predictor’s ontology.
We do something like this with models of physics. For example, we prove that general relativity reduces to Newtonian mechanics under normal conditions. With enough compute I could take String Theory and check all the 10^500 possible universes to find ours. Since the Predictor is inner-aligned, it can be treated basically as a model of physics… (= this might require making an additional assumption, but ARC is ready to make it I think)
I assume there are fundamental obstacles to this, but what exactly are those obstacles? Maybe we can’t define a universal language which would allow to prove the reduction of one model to another? Something else?
The most similar proposal to this I remember is “have AI help humans improve our understanding”. It is dismissed for being uncompetitive. Yet uncompetitive ELK is arguably not actually solved and there are deeper reasons to reject the proposal. It relies on more roundabout methods than just brute forcing how one thing maps to another.
Now, maybe I’m just restating the problem… but right now I don’t think so. I’m saying there should be a principled way to just prove how exactly one model reduces to the other instead of using arbitrary tricks to create a Reporter. If there are multiple, equally good ways to map human physics to the Predictor, then it would probably mean human physics aren’t grounded in anything at all.
...
Maybe one of the core problems is that the Predictor might be a mess of high-level and low-level variables? So we’d first need to find a way to transform the Predictor into a low-level model of physics / establish the hierarchy of variables. How hard is this problem?
I’d say “we don’t understand how the brain works good enough to replicate it in AGI” (and that means we lack a really important layer of understanding). Anyway, I think we agree here.
The point I wanted to make:
I think John’s argument for why corrigibility is natural works even if you don’t assume crisp separate modules.
Whatever explanation you propose (of the brain staying coherent), be it “your modules are corrigible to each other” or “the goals are kept at the back of your mind”, if you try to make that explanation more technical you’ll probably end up with a general solution to corrigibility.
Sorry for skipping so much inferential steps and causing confusion.
Sorry, why is this offtopic—what are the different problems I could’ve confused? The thing I quoted feels like it assumes exactly the reward hacking problem being solved:
the AI keeps the supervisor’s desires / preferences / aversions at the back of its mind, and notices when it has an idea that would bother or offend the supervisor, and then not do it
It assumes we have a thing which can accurately judge ideas as “good” or “bad”. I don’t believe it’s a solved problem. I went on to re-read the “act-based approval-directed agents” post, but it offered no solution. This reply to John seems to draw a distinction between good foresight and good hindsight, but I’d say both are unsolved problems and might have a similar solution. The latter is the credit assignment problem and I remember your post about it, with no general solution.
I’ll share my general view, maybe it’ll help to sort out the misunderstanding:
I think the brain is a horrible mess. “Why does the whole system stay relatively stable, without drifting uncontrollably into arbitrary directions?” is the core alignment-related mystery about it. “How is the brain able to correctly introspect its thoughts?”, “how is the brain able to notice failures/successes of its high-level goals?”, etc. are all facets of the core mystery. I believe they are all unsolved and not obvious. IMO any solution to any of the facets will give a (non-trivial) insight about corrigibility.
Anyway, does this suggest a solution to AI corrigibility? Well, kinda. But it’s just the obvious one: the AI keeps the supervisor’s desires / preferences / aversions at the back of its mind, and notices when it has an idea that would bother or offend the supervisor, and then not do it.
I think it totally does suggest a solution, if you could reformulate the quote in a much more abstract and general way, along with explaining how/why the AI doesn’t fool the supervisor. I was skimming the intro to brain-like AGI safety and IIRC the problem “how/why doesn’t the thought generator fool the thought accessor?” wasn’t solved beyond “create as many thought accessors as possible and hope for the best”.
In general, I’m very interested in the intersection between brain-like ideas and agent foundations ideas.
This is objectively the best one.
Questions felt like a better way to get feedback on some things (exactly because they are rare). I miss forums having sub-forums for the same reason.
But you’re not going to find any good references, or come to any conclusions, if you stay at this stage.
I think he already came to some conclusions and you already gave some good references (which support some of the conclusions).
For example there are ways of designing hardware which is reliable on the assumption that at most N transistors are corrupt.
Do those methods have names or address problems which have names (like the Byzantine generals problem)?
How strong is that evidence, for you personally? Is it constructive (gives you vague ideas of how problems 1-3 could be solved without ignoring computational cost) or not (just gives you abstract hope)? Would be very interested to read your thoughts. Your post doesn’t emphasize your opinions.
My personal impression about mathematical theories for alignment is very pessimistic. Someone proposes a mathematical framework too general to make progress on. The framework struggles to prove even the most general theorems without relying on questionable assumptions. We don’t know if the proved theorems even mean what we think they mean (because the questionable assumptions narrow down the scope of the theorems in a strange way, making them very easy to misinterpret). Finally, the math has nothing to say about something super important (such as computational cost). Disclaimer: this impression is very indirect (I’m bad at math) and it might be unreasonable to ask for anything better.
How unique/unexpected is Sam’s theory, compared to other mathematical theories for alignment (from your subjective perspective)?
If I understand correctly, the standard worries about interpretability are
different types of AIs don’t have universal data structures
you can’t prove anything useful about those structures without AGI (the problem is too complicated/advanced for humanity)
if you induce interpretability in your AI, it won’t be competitive.
Why does condensation give hope for solving those problems? Where could the magic happen? For example, John Wentworth attacks 1-3 by making assumptions about our universe.
I don’t buy 4.1. I believe the laws of the universe should be simple, but simplicity of specifying the initial state is not an argument for me (unless there are additional factors). Don’t buy 4.2 either, because I don’t trust anthropic reasoning that much. However, I’m still interested in ideas like this, even if I don’t believe in them.
4.3. I don’t know enough math, but I don’t believe that a) we have the math to prove that an agent can’t survive in an environment with incompressible low-level dynamics or b) it means anything non-trivial enough to prove the kind of points you need to prove. It would be a magically good result.
In a chaotic environment there could be some high-level patterns which the agent might exploit, but those high-level patterns could be barely similar to humanlike abstractions. The Gooder Regulator theorem could be false, maybe, if we don’t need to be prepared for “any possible game”.
In other words: we have a probability distribution over sequences of transformations of the data we should attempt to try, and we keep sampling from it until we spot a familiar abstraction; until we find a way of looking at the data under which it resolves into a known, simple pattern.
Maybe you’re focused on the wrong thing. Maybe the explicit search for the right transformation happens only when things go wrong (and even then a lot of the search is not explicit).
You could have a multi-step detection method which deduces the right transformation instead of searching for it explicitly. Contrast two algorithms:
I slide and scale the dog template across an image until I find a good fit.
I detect the most salient parts of the image, combine those parts into lines, connect the lines, check if the lines have the right proportions. If they do, it’s a dog. If they don’t, it’s unlikely there are any dogs in the image.
But if it’s not an abstraction over the cells, then over… what? There aren’t really any other elements in the picture! Maybe it’s an abstraction over sets of cells, with cells being “subvariables” over which the glider is synergistic? But how exactly do we define those sets?
I think you should filter out sets which are naturally “salient” and then do something with them. For example, imagine I have a single glider in an empty world. I model the world for 1000 timesteps. I pick the set of cells which changed values during this time (that would be an extremely natural thing to do). This way I find at least the “trail” of the glider. By doing other natural things I can narrow it down farther or make it work in a world with many objects.
...
All of the above shouldn’t be groundbreaking, I’m just surprised you didn’t mention it at all and focused your examples on explicit search. Maybe you’re unsure if the trick with naturally salient subsets generalizes?
Writing a relatively high-effort comment because I’m thinking about similar things and intend to turn what I wrote about here into a top-level post.
First I’ll write some criticism, then mention some “leads” you might be interested in. For context, I started writing this comment before this exchange.
The High-Level Outline confused me. Feels like the argument you’re making is a bit tricky, yet you don’t spell it out explicitly enough. I’m not sure what assumptions are made.
Here’s my train of thought:
The universe has some structure (S) which makes human abstractions useful. But human abstractions != S, so human abstractions can still be hard to specify. You seem to be 100% aware of that.
...But then you connect human abstractions to the shortest description of the universe. Which doesn’t follow. …Or maybe you merely say that any short description of the universe is isomorphic to S, therefore human abstractions should be useful for understanding it. …You also introduce some amount of tautology here, by defining complexity in terms of (a subset of) human abstractions.
My best interpretation is that you’re making the argument that we can abstract the way in which human abstractions are interpretable (human abstractions are interpretable by being “symbolic” / “chunky”) and then search for any symbolic/chunky abstractions. You assume that the best ways to compress S gonna be symbolic/chunky. Or that we can enforce looking for symbolic/chunky representations of S. …But then the section about ontology shifts doesn’t make sense. Why are there multiple kinds of symbolicness/chunkiness?
There might be a general disagreement or misunderstanding. I would classify programs into 3 types: big and chaotic (A), small and chaotic (B), not chaotic (C). For example, neural networks are big & chaotic and Busy Beavers or pseudorandom number generators or stuff like Van Eck’s sequence are small & chaotic. The existence of B-type programs is the biggest problem in capturing the intuitive notion of “complexity” (description length doesn’t cut it = some short programs behave very weirdly). If the universe can be compressed by a B-type program, then the compression won’t be “chunky” or truly “symbolic”. So the core difficulty of formalizing chunkiness/symbolicness is excluding B-type programs. You mention this concern, but dismiss it as not a big deal.
...
What are the most substantial differences between your agenda and NAH?
Human world-models rely on chunking. To understand a complex phenomenon, we break it down into parts, understand the parts individually, then understand the whole in terms of the parts. (The human biology in terms of cells/tissues/organs, the economy in terms of various actors and forces, a complex codebase in terms of individual functions and modules.)
If you split a thing enough times, you’ll end up with simple parts. But that doesn’t mean that the whole thing is simple, even if it’s small. Small Turing Machines can have very complex behavior. I believe it’s the core problem of defining what a “chunk” is. Or what “well-abstracting” means. And I think the solution should have the following form:
“Parts of a program X are non-trivial (= can be considered ‘chunks’ or ‘symbols’) if they are substantially simpler than X and allow to prove important things about X by using a predetermined method (= a method not requiring a stroke of creativity or meticulous search through proofs) substantially faster than running X.”[1]
Examples:
Consider an algorithm which solves chess positions through exhaustive brute-force search. The operation of expanding the game tree is a non-trivial part of the algorithm, because it allows to easily prove important things about the algorithm (e.g. its runtime) faster than running it. Just by applying induction. However, if I give you the rules of a random Turing Machine, you might be unable to prove anything important by applying generic proof methods to those rules. Which would mean that those rules don’t split the program into “digestible” parts — your predetermined proof machinery can’t digest them.
By splitting the world into big areas which are assumed to be affecting each other slowly and locally, we can easily prove important things about the world.
Once you know what “organs” are, it allows you to prove extremely non-trivial things about animal bodies. E.g. “deep stab has a high chance to stop an animal forever (because it has a high chance to hit an organ)”.
You can take a variable X (e.g. “a particular human”) and measure how useful all other variables are for easily proving important things about X. You’ll get a “semantic field” of X. The field will include mesavariables (e.g. “organs”, “cells”) and metavariables (e.g. “democracy”, “social norms”). The field will fade away at overly fine-grained variables (e.g. individual atoms) and overly coarse-grained variables.
Maybe we can deduce something about the structure of abstractions from first principles, in a way completely agnostic to the fundamental building blocks of the universe. Consider this argument:
Assume we model the world as programs modifying themselves and each other.
We have a finite set (S) of abstractions which capture most important properties of the programs’ states. Those abstractions are useful for predicting the future. Since the world is predictable, not all abstractions are equally likely (≈ not all future states of programs are equally likely).
Any output of a program can serve as input for another program. Programs can modify each other in the middle of execution. This has two consequences. First: most combinations of abstractions should map to S (i.e. simplify into base abstractions). Second: any predictable program should be modified by many programs in the same way.
The latter means that the semantic field of a predictable variable (X) should have self-similar / redundant structure. A couple of silly examples in the spoiler.
Big forest stone. It doesn’t move itself; animals don’t try to move it; the landscape around it doesn’t move. So, you could say the stone maximizes “avoidance of movement” in different ways.
Lake. When animals dive into the lake, they displace water (create “holes” in the body of water), but said water comes back; the water evaporates, gets drank by animals, seeps through the ground, but comes back with rain; the water of the lake fills a hole in the landscape. So, you could say the lake maximizes “refillment” in different ways.
A question about natural latents.
Imagine you have 100 blue boxes. Each time you roll the dice, their shape changes. But all 100 boxes always share the same shape. If I understand correctly, in this situation the shape is the natural latent. While color is just static background information.
Imagine you have 100 boxes. Each time you roll the dice, their color changes. But all 100 boxes always share the same color. In this situation, color is the natural latent.
Imagine you have 100 boxes. Each time you roll the dice, their color changes. If at least one box is blue, all boxes are blue. Otherwise their color is independent. Is “all boxes are blue or all boxes have independent colors” a natural latent (it’s something you learn about all boxes by examining a single box)?
Does the latter (3) type of natural latents have any special properties, is it some sort of “meta-level” natural latent (compared to 2)? I’m asking because I think this type of latents might be relevant to how human abstractions work. Here’s where I wrote about it in more detail.
Yes, it could be that “special, inherently more alignable cognition” doesn’t exist or can’t be discovered by mere mortal humans. It could be that humanlike reasoning isn’t inherently more alignable. Finally, it could be that we can’t afford to study it because the dominating paradigm is different. Also, I realize that glass box AI is a pipe dream.
Wrt sociopaths/psychopaths. I’m approaching it from a more theoretical standpoint. If I knew a method of building a psychopath AI (caring about something selfish, e.g. gaining money or fame or social power or new knowledge or even paperclips) and knew the core reasons of why it works, I would consider it a major progress. Because it would solve many alignment subproblems, such as ontology identification and subsystems alignment.
I’m approaching it from a “theoretical” perspective[1], so I want to know how “humanlike reasoning” could be defined (beyond “here’s some trusted model which somehow imitates human judgement”) or why human-approved capability gain preserves alignment (like, what’s the core reason, what makes human judgement good?). So my biggest worry is not that the assumption might be false, but that the assumption is barely understood on the theoretical level.
What are your research interests? Are you interested in defining what “explanation” means (or at least in defining some properties/principles of explanations)? Typical LLM stuff is highly empirical, but I’m kinda following the pipe dream of glass box AI.
I’m contrasting theoretical and empirical approaches.
Empirical—“this is likely to work, based on evidence”. Theoretical—“this should work, based on math / logic / philosophy”.
Empirical—“if we can operationalize X for making experiments, we don’t need to look for a deeper definition”. Theoretical—“we need to look for a deeper definition anyway”.
I believe you should be able to make progress by writing out different map-level and territory-level things. Let me try:
0) My prediction: I expect the universe to continue being ordered.
1) In-map explanation for why I came to believe it: I use a simplicity prior to predict my observations and trust my memory.
2) In-territory explanation for why my prediction succeeds: I’m a human (not a BB) in a simple universe.
The problem: I can’t trust my memory and justify the simplicity prior.
3) In-map justification for using the simplicity prior anyway: memory and simplicity aren’t provably reliable, but I don’t care about my life in case they are not reliable.
Potential crux A: (3) doesn’t undermine the fact that I’m correct only because of (2), so changing what I care about doesn’t change the outcome, yet it still counts as a justification of doing (1). If I cared about complex universes, I wouldn’t be justified in doing (1).
4) In-territory justification for using the simplicity prior anyway: I live in a multiverse where memory and simplicity aren’t reliable in every universe, but I don’t care about universes where they are not reliable.
5) Another in-territory justification for using the simplicity prior: God or some power makes the simplicity prior true in the multiverse.
Potential crux B: some will say (4) is not an explanation because I have to explain “how you go from experience 1 in universe 1 to experience 2 in universe 1 instead of experience 2 in universe 2” or something; others will say it’s all anthropic woo and any reasonable (5)-type theory is gonna be untestable by construction (until we die).
6) A deeper in-territory explanations for why my prediction (0) succeeds: “God/power did it” (see 5), “I got lucky once to end up in an ordered universe” (see 4), “all experience histories created by the multiverse exist, therefore this experience history was fated to be experienced anyway” (see 4). Due to crux B, I assume the latter can be disputed. The middle one can be criticized for being too lucky.
Why isn’t 3⁄4 a justification? Because of some philosophy regarding the flow of observer moments (crux B)?