Discord: LemonUniverse (.lemonuniverse). Reddit: u/Smack-works. About my situation: here.
I wrote some bad posts before 2024 because I was very uncertain how the events may develop.
I do philosophical/conceptual research, have no mathematical or programming skills. But I do know a bunch of mathematical and computer science concepts.
I’m approaching it from a “theoretical” perspective[1], so I want to know how “humanlike reasoning” could be defined (beyond “here’s some trusted model which somehow imitates human judgement”) or why human-approved capability gain preserves alignment (like, what’s the core reason, what makes human judgement good?). So my biggest worry is not that the assumption might be false, but that the assumption is barely understood on the theoretical level.
What are your research interests? Are you interested in defining what “explanation” means (or at least in defining some properties/principles of explanations)? Typical LLM stuff is highly empirical, but I’m kinda following the pipe dream of glass box AI.
I’m contrasting theoretical and empirical approaches.
Empirical—“this is likely to work, based on evidence”. Theoretical—“this should work, based on math / logic / philosophy”.
Empirical—“if we can operationalize X for making experiments, we don’t need to look for a deeper definition”. Theoretical—“we need to look for a deeper definition anyway”.
Alignment plans can be split into two types:
Usual plans. AI gains capabilities . We figure out how to point to (alignment target). There’s no deep connection between and . One thing is mounted onto the other.
HRLM plans. We give AI special , with a deep connection to .
HRLM is the idea that there’s some special reasoning/learning method which is crucial for alignment or makes it fundamentally easier. HRLM means “humanlike reasoning/learning method” or “special, human-endorsed reasoning/learning method”. There’s no hard line separating the two types of plans. It’s a matter of degree.
I believe HRLM is ~never discussed in full generality and ~never discussed from a theoretical POV. This is a small post where I want to highlight the idea and facilitate discussion, not make a strong case for it.
(My description of other people’s work is not endorsed by them.)
Corrigibility. “Corrigible cognition” is a hypothetical, special type of self-reflection () which is extremely well-suited for learning human values/desires ().
In “Don’t align agents to evaluations of plans” Alex Turner argues “there’s a correct way to reason () about goals () and consequentialist maximization of an ‘ideal’ function is not it”, “‘direct cognition’ () about goals () is fundamentally better than ‘indirect cognition’”. Shard Theory, in general, proposes a very special method for learning and thinking about values.
A post about “follow-the-trying game” by Steve Byrnes basically says “AI will become aligned or misaligned at the stage of generating thoughts, so we need to figure out the ‘correct’ way of generating thoughts (), instead of overfocusing on judging what thoughts are aligned ()”. Steve’s entire agenda is about HRLM.
Large Language Models. I’m not familiar with the debate, but I would guess it boils down to two possibilities: “understanding human language is a core enough capability () for a LLM, which makes it inherently more alignable to human goals ()” and “LLMs ‘understand’ human language through some alien tricks which don’t make them inherently more alignable”. If the former is true, LLMs are an example of HRLM.
Policy Alignment (Abram Demski) is tangentially related, but it’s more in the camp of “usual plans”.
Notice how, despite multiple agendas falling under HRLM (Shard Theory, brain-like AGI, LLM-focused proposals), there’s almost no discussion of HRLM from a theoretical POV. What is, abstractly speaking, “humanlike reasoning”? What are the general principles of it? What are the general arguments for safety guarantees it’s supposed to bring about? What are the True Names here? With Shard Theory, there’s ~zero explanation of how simpler shards aggregate into more complex shards and how it preserves goals. With brain-like AGI, there’s ~zero idea of how to prevent thought generation from bullshitting thought assessment. But those are the very core questions of the agendas. So they barely move us from square one.[1]
There are many possibilities. It could be that any HRLM handicaps AI’s capabilities (a superintelligence is supposed to be unimaginably better at reasoning than humans, so why wouldn’t it have an alien reasoning method). It also could be that HRLM is necessary for general intelligence. But maybe general intelligence is overrated...
Here’s what I personally believe right now:
What we value is inherently tied to how we think about it. In general, what we think about is often inherently tied to how we think about it.
General intelligence is based on a special principle. It has a relatively crisp “core”.
Some special computational principle is needed for solving subsystems alignment.
If 1-3 is true, 1-3 is most likely the same thing. Therefore, HRLM is needed for general intelligence, outer and inner alignment (including subsystems alignment). Separately, I think general intelligence boosts capabilities below peak human level.
I consider 1-3 to be plausible enough postulates. I have no further arguments for 4.
I have a couple of very unfinished ideas. Will try to write about them this or the next month.
I believe there could be a special type of cognition which helps to avoid specification gaming and goal misgeneralization. AI should create simple models which describe “costs/benefits” of actions (e.g. “actions” can be body movements, “cost” can be the amount and complexity of movements, “benefit” can be distance covered), this way AI can notice if certain actions produce anomalously high benefit (e.g. maybe certain body movements exploit a glitch in the physics simulation, making the body cover kilometers per second).
“By default, manipulating easier to optimize/comprehend variables is better than manipulating harder to optimize/comprehend variables” — this is the idea from one of my posts. The problem with it is that I only defined “optimization” and “comprehension” for world-models, not for any modelling (= cognition) in general.
A formal algorithm can have parts and it will critically depend on those parts (for example, an algorithm for solving equations might have an absolutely necessary addition sub-algorithm). An informal algorithm can have parts without critically depending on those parts (for example, the algorithm answering “is this a picture of a dog?” might have a sub-algorithm answering “is this patch of pixels the focal point of the image / does it contrast enough with other patches / is it as detailed as the other patches?”—the sub-algorithm is not very necessary, but it lowers pareidolia, by preventing the algorithm from overanalyzing random parts of the image). I think we can say something about the latter type of algorithms, about how they work.
IMO that’s downstream of inner alignment being extremely hard. It’s almost impossible to come up with at least mildly promising solution which explains, at least in some detail, how the hardest part of the problem might get solved. I’m not trying to throw shade. Also, I might just be ignorant about some ideas in those agendas.
(Here’s an observation about adjectives, verbs, and language in general. It might be important even if I’m misinterpreting the definition of natural latents.)
For many adjectives, we can define the concept “salience of <insert an adjective>”. Salience of color / texture / shape / size / etc.
For example, what’s “salience of a texture”? It’s a function of how much of the texture is present (in your field of view) and how strongly the texture contrasts with other present textures.
We can learn an empirical rule: “if a texture is salient enough, then it’s probably caused by a single object or a single kind of objects”.[1] Yet this object or kind is random. Would this make “being caused by an object/kind X” a natural latent over “pixels with a salient texture Y” and vice-versa? An object’s texture tends to be similarly salient in many different situations, so a particular value of “salience of a texture” can itself be a natural latent. “Salience of a texture” is not the same thing as “a texture”, but it’s one of the reasons why textures are important.
landscape example
Similarly, we can consider “salience of an action”. With an empirical rule like “salient actions (e.g. salient movements) are usually caused by a single object / a single kind of objects / a single causal process”.[2] Such rule makes fine-grained classification of actions important, making action-related words important.
Beyond “salience of a texture”, we can consider concepts like “danger of an object” (is it a good idea to touch or step on?) and “traversability of an object” (can you walk/climb on it?). For many types of objects, those concepts will be independent from most facts about the objects but their texture. Let’s call concepts like this (salience, danger, traversability, etc.) “auxiliary”.
Maybe this sort of reasoning can explain analogies, connotation. Because auxiliary concepts are a type of connotation.
Imagine looking at a nature landscape. You notice a bunch of angular texture (mountains), a bunch of puffy texture (clouds), a bunch fluffy texture (trees), a bunch of smooth texture (fields). The rule says each type of texture most likely belongs to a single object or kind. Note that it’s not trivial—we could live in a world where we often see radically different things with similar, equally salient textures at the same time.
For example, imagine you see a burst of flame and rubble flying around—most likely it’s a single causal process, an explosion.
I have a couple of questions/points. Might stem from not understanding the math.
1) The very first example shows that absolutely arbitrary things (e.g. arbitrary green lines) can be “natural latents”. Does it mean that “natural latents” don’t capture the intuitive idea of “natural abstractions”? That all natural abstractions are natural latents, but not all natural latents are natural abstractions. You seem to be confirming this interpretation, but I just want to be sure:
So we view natural latents as a foundational tool. The plan is to construct more expressive structures out of them, rich enough to capture the type signatures of the kinds of concepts humans use day-to-day, and then use the guarantees provided by natural latents to make similar isomorphism claims about those more-complicated structures. That would give a potential foundation for crossing the gap between the internal ontologies of two quite different minds.
Is there any writing about what those “more expressive structures” could be?
2) Natural latents can describe both things which propagate through very universal, local physical laws (e.g. heat) and any commonalities in made up categories (e.g. “cakes”). Natural latents seem very interesting in the former case, but I’m not sure about the latter. Not sure the analogy between the two gives any insight. I’m still not seeing any substantial similarity between cakes and heat or Ising models. I.e. I see that an analogy can be made, but I don’t feel that this analogy is “grounded” in important properties of reality (locality, universality, low-levelness, stability, etc). Does this make sense?
3) I don’t understand what “those properties can in-principle be well estimated by intensive study of just one or a few mai tais” (from here) means. To me a natural latent is something like ~”all words present in all of 100 books”, it’s impossible to know unless you read every single book.
If I haven’t missed anything major, I’d say core insights about abstractions are still missing.
EDIT 17/07: I did miss at least one major thing. I haven’t understood the independence condition. If you take all words present in all 100 books, it doesn’t guarantee that those words make the books or their properties independent.
Have an idea about interpretability and defining search/optimization.
Finite algorithms can solve infinite (classes of) problems. For example, the algorithm for adding two numbers has a finite description, yet can solve an infinity of examples.
This is a basic truth of computability theory.
Intuitively, it means that algorithms can exploit regularities in problems. But “regularity” here can only be defined tautologically (any smaller thing which solves/defines a bigger thing).
Many algorithms have the following property:
The algorithm computes function by computing another function many times. Computing is useful for computing . is computed significantly more times than .
Computing is significantly easier than computing .
gives important information about . This information is easy to compute and (maybe) trivial to prove.
Intuitively, it means that algorithms can exploit regularities in problems. However, here “regularity” has a stricter definition than in the trivial property (TP). A “regularity” is an easily computable thing which gives you important, easily computable information about a hard-to-compute thing.
Now, the question is: what classes of algorithms have the less trivial property (LTP)?
LTP includes a bunch of undefined terms:
“Significantly simpler”, “easily computable”, and “useful for computing”. Those could be defined in terms of computational complexity.
“Significantly more times”. Could be defined asymptotically.
“Important information”. I have little idea how it should be defined. “Important information” may mean upper and lower bounds of a function, for example.
Probably giving fully general definitions right away is not important. We could try starting with overly restrictive definitions and see if we can prove anything about those.
If neural networks implement algorithms with LTP, we could try finding those algorithms by looking for (which is much easier than looking for ).
Furthermore, LTP seems very relevant to defining search / optimization.
Examples of algorithms with LTP:
Any greedy algorithm is an example. The whole solution to the problem is , the greedy choice is .
Many dynamic programming algorithms are an example. Solution to the whole problem is , solution to a smaller problem (which we need to use many times) is .
Consider a strong chess engine. : “given a position, find how to e.g. win material given an ~N moves horizon”. : “given a position, find how to e.g. win material given a ~K moves horizon”. Where K << N. Simply put, the N-moves-deep engine will often punish K-moves-deep mistakes. That’s the reason why you can understand that you’re losing long before you get checkmated. This wouldn’t be true in a game much more chaotic than chess.
Consider simulated annealing. : “given a function, find the global optimum”. AFAICT, simulated annealing is greedy search + randomness. The greedy part is .
Got around to interrogating Gemini for a bit.
Seems like KSF talks about programs generating sets. It doesn’t say anything about the internal structure of the programs (but that’s where the objects such as “real diamonds” live). So let’s say is a very long video about dogs doing various things. If I apply KSF, I get programs (aka “codes”) generating sets of videos. But it doesn’t help me identify “the most dog-like thing” inside each program. For example, one of the programs might be an atomic model of physics, where “the most dog-like things” are stable clouds of atoms. But KSF doesn’t help me find those clouds. A similarity metric between videos doesn’t help either.
My conceptual solution to the above problem, proposed in the post: if you have a simple program with special internal structure describing simple statistical properties of “dog-shaped pixels” (such program is guaranteed to exist), there also exists a program with very similar internal structure describing “valuable physical objects causing dog-shaped pixels” (if such program doesn’t exist, then “valuable physical objects causing dog-shaped pixels” don’t exist either).[1] Finding “the most dog-like things” in such program is trivial. Therefore, we should be able to solve ontology identification by heavily restricting the internal structure of programs (to structures which look similar to simple statistical patterns in sensory data).
So, to formalize my “conceptual solution” we need models which are visually/structurally/spatially/dynamically similar to the sensory data they model. I asked Gemini about it, multiple times, with Deep Research. The only interesting reference Gemini found is Agent-based models (AFAIU, “agents” just means “any objects governed by rules”).
This is not obvious, requires analyzing basic properties of human values.
Then you can use the three dot points in my comment to construct source code for a new agent that does the same thing, but is not nicely separated.
This is the step I don’t get (how we make the construction), because I don’t understand SGD well. What does “sample N world models” mean?
My attempt to understand: We have a space of world models () and a space of plans (). We pick points from (using SGD) and evaluate them on the best points of (we got those best points by trying to predict the world and applying SGD).
My thoughts/questions: To find the best points of , we still need to do modelling independently from planning? While the world model is not stored in memory, some pointer to the best points of is stored? We at least have “the best current plan” stored independently from the world models?
First things first, defining a special language which creates a safe but useful AGI absolutely is just a restatement of the problem, more or less. But the post doesn’t just restate the problem, it describes the core principle of the language (the comprehension/optimization metric) and makes arguments for why the language should be provably sufficient for solving a big part of alignment.
You’re saying that the simple AI can tell if the more complex AI’s plans are good, bad, or unnecessary—but also the latter “can know stuff humans don’t know”. How?
This section deduces the above from claims A and B. What part of the deduction do you disagree with/confused about? Here’s how the deduction would apply to the task “protect a diamond from destruction”:
cares about an ontologically fundamental diamond. models the world as clouds of atoms.
According to the principle, we can automatically find what object in corresponds to the “ontologically fundamental diamond”.
Therefore, we can know what plans would preserve the diamond. We also can know if applying any weird optimization to the diamond is necessary for preserving it. Checking for necessity is probably hard, might require another novel insight. But “necessity” is a simple object-level property.
The automatic finding of the correspondence (step 2) between an important comprehensible concept and an important incomprehensible concept resolves the apparent contradiction.[1]
Now, without context, step 2 is just a restatement of the ontology identification problem. The first two sections of the post (mostly the first one) explain why the comprehension/optimization metric should solve it. I believe my solution is along the lines of the research avenues Eliezer outlined.
If my principle is hard to agree with, please try to assume that it’s true and see if you can follow how it solves some alignment problems.
So, somehow you’re able to know when an AI is exerting optimization power in “a way that flows through” some specific concepts?
Yes, we’re able to tell if AI optimizes through a specific class of concepts. In most/all sections of the post I’m assuming the AI generates concepts in a special language (i.e. it’s not just a trained neural network), a language which allows to measure the complexity of concepts. The claim is that if you’re optimizing through concepts of certain complexity, then you can’t fulfill a task in a “weird” way. If the claim is true and AI doesn’t think in arbitrary languages, then it’s supposed to be impossible to create a harmful Doppelganger.
But I don’t get if, or why, you think that adds up to anything like the above.
Clarification: only the interpretability section deals with inner alignment. The claims of the previous sections are not supposed to follow from the interpretability section.
Anyway, is the following basically what you’re proposing?
Yes. The special language is supposed to have the property that can automatically learn if plans good, bad, or unnecessary actions. can’t be arbitrarily smarter than humans, but it’s a general intelligence which doesn’t imitate humans and can know stuff humans don’t know.
Could you ELI15 the difference between Kolmogorov complexity (KC) and Kolmogorov structure function (KSF)?
Here are some of the things needed to formalize the proposal in the post:
A complexity metric defined for different model classes.
A natural way to “connect” models. So we can identify the same object (e.g. “diamond”) in two different models. Related: multi-level maps.
I feel something like KSF could tackle 1, but what about 2?
Thanks for clarifying! Even if I still don’t fully understand your position, I now see where you’re coming from.
No, I think it’s what humans actually pursue today when given the options. I’m not convinced that these values are static, or coherent, much less that we would in fact converge.
Then those values/motivations should be limited by the complexity of human cognition, since they’re produced by it. Isn’t that trivially true? I agree that values can be incoherent, fluid, and not converging to anything. But building Task AGI doesn’t require building an AGI which learns coherent human values. It “merely” requires an AGI which doesn’t affect human values in large and unintended ways.
No, because we don’t comprehend them, we just evaluate what we want locally using the machinery directly, and make choices based on that.
This feels like arguing over definitions. If you have an oracle for solving certain problems, this oracle can be defined as a part of your problem-solving ability. Even if it’s not transparent compared to your other problem-solving abilities. Similarly, the machinery which calculates a complicated function from sensory inputs to judgements (e.g. from Mona Lisa to “this is beautiful”) can be defined as a part of our comprehension ability. Yes, humans don’t know (1) the internals of the machinery or (2) some properties of the function it calculates — but I think you haven’t given an example of how human values depend on knowledge of 1 or 2. You gave an example of how human values depend on the maxima of the function (e.g. the desire to find the most delicious food), but that function having maxima is not an unknown property, it’s a trivial property (some foods are worse than others, therefore some foods have the best taste).
That’s a very big “if”! And simplicity priors are made questionable, if not refuted, by the fact that we haven’t gotten any convergence about human values despite millennia of philosophy trying to build such an explanation.
I agree that ambitious value learning is a big “if”. But Task AGI doesn’t require it.
But
To pursue their values, humans should be able to reason about them. To form preferences about a thing, humans should be able to consider the thing. Therefore, human ability to comprehend should limit what humans can care about. At least before humans start unlimited self-modification. I think this logically can’t be false.
Eliezer Yudkowsky is a core proponent of complexity of value, but in Thou Art Godshatter and Protein Reinforcement and DNA Consequentialism he basically makes a point that human values arose from complexity limitations, including complexity limitations imposed by brainpower limitations. Some famous alignment ideas (e.g. NAH, Shard Theory) kinda imply that human values are limited by human ability to comprehend and it doesn’t seem controversial. (The ideas themselves are controversial, but for other reasons.)
If learning values is possible at all, there should be some simplicity biases which help to learn them. Wouldn’t it be strange if those simplicity biases were absolutely unrelated to simplicity biases of human cognition?
Based on your comments, I can guess that something below is the crux:
You define “values” as ~”the decisions humans would converge to after becoming arbitrarily more knowledgeable”. But that’s a somewhat controversial definition (some knowledge can lead to changes in values) and even given that definition it can be true that “past human ability to comprehend limits human values” — since human values were formed before humans explored unlimited knowledge. Some values formed when humans were barely generally intelligent. Some values formed when humans were animals.
You say that values depend on inscrutable brain machinery. But can’t we treat the machinery as a part of “human ability to comprehend”?
You talk about ontology. Humans can care about real diamonds without knowing what physical things the diamonds are made from. My reply: I define “ability to comprehend” based on ability to comprehend functional behavior of a thing under normal circumstances. Based on this definition, a caveman counts as being able to comprehend the cloud of atoms his spear is made of (because the caveman can comprehend the behavior of the spear under normal circumstances), even though the caveman can’t comprehend atomic theory.
Could you confirm or clarify the crux? Your messages felt ambiguous to me. In what specific way is A false?
Are you talking about value learning? My proposal doesn’t tackle advanced value learning. Basically, my argument is “if (A) human values are limited by human ability to comprehend/optimize things and (B) the factors which make something easier or harder to comprehend/optimize are simple, then the AI can avoid accidentally messing up human values — so we can define safe impact measures and corrigibility”. My proposal is not supposed to make the AI learn human values in great detail or extrapolate them out of distribution. My argument is “if A and B hold, then we can draw a box around human values and tell the AI to not mess up the contents of the box — without making the AI useless; yet the AI might not know what exact contents of the box count as ‘human values’”.[1]
The problem with B is that humans have very specialized and idiosyncratic cognitive machinery (the machinery generating experiences) which is much more advanced than human general ability to comprehend things. I interpreted you as making this counterargument in the top level comment. My reply is that I think human values depend on that machinery in a very limited way, so B is still true enough. But I’m not talking about extrapolating something out of distribution. Unless I’m missing your point.
Why those things follow from A and B is not obvious and depends on a non-trivial argument. I tried to explain it in the first section of the post, but might’ve failed.
Thanks for elaborating! This might lead to a crux. Let me summarize the proposals from the post (those summaries can’t replace reading the post though).
Outer alignment:
We define something like a set of primitives. Those primitives are independent from any specific ontology.
We prove[1] that as long as AI acts and interprets tasks using those primitives, it can prevent humans from being killed or brainwashed or disempowered. Even if the primitives are not enough to give a very nuanced definition of a “human” or “brainwashing”. That’s where the “we can express care about incomprehensible things as care about comprehensible properties of incomprehensible things” argument comes into play.
Inner alignment:
We prove that a more complicated model (made of the primitives) can’t deceive a simpler model (made of the primitives). The inner/outer alignment of simple enough models can be verified manually.
We prove that the most complicated model (expressible with the primitives) has at least human-level intelligence.
Bonus: we prove that any model (made of the primitives) is interpretable/learnable by humans and prove that you don’t need more complicated models for defining corrigibility/honesty. Disclaimer: the proposals above are not supposed to be practical, merely bounded and being conceptually simple.
Why the heck would we be able to define primitives with such wildly nice properties? Because of the argument that human ability to comprehend and act in the world limits what humans might currently care about, and the current human values are enough to express corrigibility. If you struggle to accept this argument, maybe try assuming it’s true and see if you can follow the rest of the logic? Or try to find a flaw in the logic instead of disagreeing with the definitions. Or bring up a specific failure mode.
To have our values interface appropriately with with this novel thinking patterns in the AI, including through corrigibility, I think we have to work with “values” that are the sort of thing that can refer / be preserved / be transferred across “ontological” changes.
If you talk about ontological crisis or inner alignment, I tried to address those in the post. By the way, I read most of your blog post and skimmed the rest.
To actually prove it we need to fully formalize the idea, of course. But I think my idea is more specific than many other alignment ideas (e.g. corrigibility, Mechanistic Anomaly Detection, Shard Theory).
I probably disagree. I get the feeling you have an overly demanding definition of “value” which is not necessary for solving corrigibility and a bunch of other problems. Seems like you want to define “value” closer to something like CEV or “caring about the ever-changing semantic essence of human ethical concepts”. But even if we talk about those stronger concepts (CEV-like values, essences), I’d argue the dynamic I’m talking about (“human ability to comprehend limits what humans can care about”) still applies to them to an important extent.
See my response to David about a very similar topic. Lmk if it’s useful.
Basically, I don’t think your observation invalidates any ideas from the post.
The main point of the post is that human ability to comprehend should limit what humans can care about. This can’t be false. Like, logically. You can’t form preferences about things you can’t consider. When it looks like humans form preferences about incomprehensible things, they really form preferences only about comprehensible properties of those incomprehensible things. In the post I make an analogy with a pseudorandom number generator: it’s one thing to optimize a specific state of the PRNG or want the PRNG to work in a specific way, and another thing to want to preserve the PRNG’s current algorithm (whatever it is). The first two goals might be incomprehensible, but the last goal is comprehensible. Caring about friends works in a similar way to caring about a PRNG. (You might dislike this framing for philosophical or moral reasons, that’s valid, but it won’t make object-level ideas from the post incorrect.)
Yes, some value judgements (e.g. “this movie is good”, “this song is beautiful”, or even “this is a conscious being”) depend on inscrutable brain machinery, the machinery which creates experience. The complexity of our feelings can be orders of magnitude greater than the complexity of our explicit reasoning. Does it kill the proposal in the post? I think not, for the following reason:
We aren’t particularly good at remembering exact experiences, we like very different experiences, we can’t access each other’s experiences, and we have very limited ways of controlling experiences. So, there should be pretty strict limitations on how much understanding of the inscrutable machinery is required for respecting the current human values. Defining corrigible behavior (“don’t kill everyone”, “don’t seek power”, “don’t mess with human brains”) shouldn’t require answering many specific, complicated machinery-dependent questions (“what separates good and bad movies?”, “what separates good and bad life?”, “what separates conscious and unconscious beings?”).
Also, some thoughts about your specific counterexample (I generalized it to being about experiences in general):
“How stimulating or addicting or novel is this experience?” ← I think those parameters were always comprehensible and optimizable, even in the Stone Age. (In a limited way, but still.) For example, it’s easy to get different gradations of “less addicting experiences” by getting injuries, starving or not sleeping.
“How ‘good’ is this experience in a more nebulous or normative way?” ← I think this is a more complicated value (aesthetic taste), based on simpler values.
Note that I’m using “easy to comprehend” in the sense of “the thing behaves in a simple way most of the time”, not in the sense of “it’s easy to comprehend why the thing exists” or “it’s easy to understand the whole causal chain related to the thing”. I think the latter senses are not useful for a simplicity metric, because they would mark everything as equally incomprehensible.
Note that “I care about taste experiences” (A), “I care about particular chemicals giving particular taste experiences” (B), and “I care about preserving the status quo connection between chemicals and taste experiences” (C) are all different things. B can be much more complicated than C, B might require the knowledge of chemistry while C doesn’t.
Does any of the above help to find the crux of the disagreement or understand the intuitions behind my claim?
Could you reformulate the last paragraph as “I’m confused how your idea helps with alignment subrpoblem X”, “I think your idea might be inconsistent or having a failure mode because of Y”, or “I’m not sure how your idea could be used to define Z”?
Wrt the third paragraph. The post is about corrigible task ASI which could be instructed to protect humans from being killed/brainwashed/disempowered (and which won’t kill/brainwash/disempower people before it’s instructed to not do this). The post is not about value learning in the sense of “the AI learns plus-minus the entirety of human ethics and can build an utopia on its own”. I think developing my idea could help with such value learning, but I’m not sure I can easily back up this claim. Also, I don’t know how to apply my idea directly to neural networks.
Yes, it could be that “special, inherently more alignable cognition” doesn’t exist or can’t be discovered by mere mortal humans. It could be that humanlike reasoning isn’t inherently more alignable. Finally, it could be that we can’t afford to study it because the dominating paradigm is different. Also, I realize that glass box AI is a pipe dream.
Wrt sociopaths/psychopaths. I’m approaching it from a more theoretical standpoint. If I knew a method of building a psychopath AI (caring about something selfish, e.g. gaining money or fame or social power or new knowledge or even paperclips) and knew the core reasons of why it works, I would consider it a major progress. Because it would solve many alignment subproblems, such as ontology identification and subsystems alignment.