Writing a relatively high-effort comment because I’m thinking about similar things and intend to turn what I wrote about here into a top-level post.
First I’ll write some criticism, then mention some “leads” you might be interested in. For context, I started writing this comment before this exchange.
Criticism/Confusion
The High-Level Outline confused me. Feels like the argument you’re making is a bit tricky, yet you don’t spell it out explicitly enough. I’m not sure what assumptions are made.
Here’s my train of thought:
The universe has some structure (S) which makes human abstractions useful. But human abstractions != S, so human abstractions can still be hard to specify. You seem to be 100% aware of that.
...But then you connect human abstractions to the shortest description of the universe. Which doesn’t follow. …Or maybe you merely say that any short description of the universe is isomorphic to S, therefore human abstractions should be useful for understanding it. …You also introduce some amount of tautology here, by defining complexity in terms of (a subset of) human abstractions.
My best interpretation is that you’re making the argument that we can abstract the way in which human abstractions are interpretable (human abstractions are interpretable by being “symbolic” / “chunky”) and then search for any symbolic/chunky abstractions. You assume that the best ways to compress S gonna be symbolic/chunky. Or that we can enforce looking for symbolic/chunky representations of S. …But then the section about ontology shifts doesn’t make sense. Why are there multiple kinds of symbolicness/chunkiness?
There might be a general disagreement or misunderstanding. I would classify programs into 3 types: big and chaotic (A), small and chaotic (B), not chaotic (C). For example, neural networks are big & chaotic and Busy Beavers or pseudorandom number generators or stuff like Van Eck’s sequence are small & chaotic. The existence of B-type programs is the biggest problem in capturing the intuitive notion of “complexity” (description length doesn’t cut it = some short programs behave very weirdly). If the universe can be compressed by a B-type program, then the compression won’t be “chunky” or truly “symbolic”. So the core difficulty of formalizing chunkiness/symbolicness is excluding B-type programs. You mention this concern, but dismiss it as not a big deal.
...
What are the most substantial differences between your agenda and NAH?
Definition of a “chunk” (Lead 1)
Human world-models rely on chunking. To understand a complex phenomenon, we break it down into parts, understand the parts individually, then understand the whole in terms of the parts. (The human biology in terms of cells/tissues/organs, the economy in terms of various actors and forces, a complex codebase in terms of individual functions and modules.)
If you split a thing enough times, you’ll end up with simple parts. But that doesn’t mean that the whole thing is simple, even if it’s small. Small Turing Machines can have very complex behavior. I believe it’s the core problem of defining what a “chunk” is. Or what “well-abstracting” means. And I think the solution should have the following form:
“Parts of a program X are non-trivial (= can be considered ‘chunks’ or ‘symbols’) if they are substantially simpler than X and allow to prove important things about X by using a predetermined method (= a method not requiring a stroke of creativity or meticulous search through proofs) substantially faster than running X.”[1]
Examples:
Consider an algorithm which solves chess positions through exhaustive brute-force search. The operation of expanding the game tree is a non-trivial part of the algorithm, because it allows to easily prove important things about the algorithm (e.g. its runtime) faster than running it. Just by applying induction. However, if I give you the rules of a random Turing Machine, you might be unable to prove anything important by applying generic proof methods to those rules. Which would mean that those rules don’t split the program into “digestible” parts — your predetermined proof machinery can’t digest them.
By splitting the world into big areas which are assumed to be affecting each other slowly and locally, we can easily prove important things about the world.
Once you know what “organs” are, it allows you to prove extremely non-trivial things about animal bodies. E.g. “deep stab has a high chance to stop an animal forever (because it has a high chance to hit an organ)”.
You can take a variable X (e.g. “a particular human”) and measure how useful all other variables are for easily proving important things about X. You’ll get a “semantic field” of X. The field will include mesavariables (e.g. “organs”, “cells”) and metavariables (e.g. “democracy”, “social norms”). The field will fade away at overly fine-grained variables (e.g. individual atoms) and overly coarse-grained variables.
The structure of abstractions (Lead 2)
Maybe we can deduce something about the structure of abstractions from first principles, in a way completely agnostic to the fundamental building blocks of the universe. Consider this argument:
Assume we model the world as programs modifying themselves and each other.
We have a finite set (S) of abstractions which capture most important properties of the programs’ states. Those abstractions are useful for predicting the future. Since the world is predictable, not all abstractions are equally likely (≈ not all future states of programs are equally likely).
Any output of a program can serve as input for another program. Programs can modify each other in the middle of execution. This has two consequences. First: most combinations of abstractions should map to S (i.e. simplify into base abstractions). Second: any predictable program should be modified by many programs in the same way.
The latter means that the semantic field of a predictable variable (X) should have self-similar / redundant structure. A couple of silly examples in the spoiler.
Big forest stone. It doesn’t move itself; animals don’t try to move it; the landscape around it doesn’t move. So, you could say the stone maximizes “avoidance of movement” in different ways.
Lake. When animals dive into the lake, they displace water (create “holes” in the body of water), but said water comes back; the water evaporates, gets drank by animals, seeps through the ground, but comes back with rain; the water of the lake fills a hole in the landscape. So, you could say the lake maximizes “refillment” in different ways.
Writing a relatively high-effort comment because I’m thinking about similar things and intend to turn what I wrote about here into a top-level post.
First I’ll write some criticism, then mention some “leads” you might be interested in. For context, I started writing this comment before this exchange.
Criticism/Confusion
The High-Level Outline confused me. Feels like the argument you’re making is a bit tricky, yet you don’t spell it out explicitly enough. I’m not sure what assumptions are made.
Here’s my train of thought:
The universe has some structure (S) which makes human abstractions useful. But human abstractions != S, so human abstractions can still be hard to specify. You seem to be 100% aware of that.
...But then you connect human abstractions to the shortest description of the universe. Which doesn’t follow. …Or maybe you merely say that any short description of the universe is isomorphic to S, therefore human abstractions should be useful for understanding it. …You also introduce some amount of tautology here, by defining complexity in terms of (a subset of) human abstractions.
My best interpretation is that you’re making the argument that we can abstract the way in which human abstractions are interpretable (human abstractions are interpretable by being “symbolic” / “chunky”) and then search for any symbolic/chunky abstractions. You assume that the best ways to compress S gonna be symbolic/chunky. Or that we can enforce looking for symbolic/chunky representations of S. …But then the section about ontology shifts doesn’t make sense. Why are there multiple kinds of symbolicness/chunkiness?
There might be a general disagreement or misunderstanding. I would classify programs into 3 types: big and chaotic (A), small and chaotic (B), not chaotic (C). For example, neural networks are big & chaotic and Busy Beavers or pseudorandom number generators or stuff like Van Eck’s sequence are small & chaotic. The existence of B-type programs is the biggest problem in capturing the intuitive notion of “complexity” (description length doesn’t cut it = some short programs behave very weirdly). If the universe can be compressed by a B-type program, then the compression won’t be “chunky” or truly “symbolic”. So the core difficulty of formalizing chunkiness/symbolicness is excluding B-type programs. You mention this concern, but dismiss it as not a big deal.
...
What are the most substantial differences between your agenda and NAH?
Definition of a “chunk” (Lead 1)
If you split a thing enough times, you’ll end up with simple parts. But that doesn’t mean that the whole thing is simple, even if it’s small. Small Turing Machines can have very complex behavior. I believe it’s the core problem of defining what a “chunk” is. Or what “well-abstracting” means. And I think the solution should have the following form:
“Parts of a program X are non-trivial (= can be considered ‘chunks’ or ‘symbols’) if they are substantially simpler than X and allow to prove important things about X by using a predetermined method (= a method not requiring a stroke of creativity or meticulous search through proofs) substantially faster than running X.”[1]
Examples:
Consider an algorithm which solves chess positions through exhaustive brute-force search. The operation of expanding the game tree is a non-trivial part of the algorithm, because it allows to easily prove important things about the algorithm (e.g. its runtime) faster than running it. Just by applying induction. However, if I give you the rules of a random Turing Machine, you might be unable to prove anything important by applying generic proof methods to those rules. Which would mean that those rules don’t split the program into “digestible” parts — your predetermined proof machinery can’t digest them.
By splitting the world into big areas which are assumed to be affecting each other slowly and locally, we can easily prove important things about the world.
Once you know what “organs” are, it allows you to prove extremely non-trivial things about animal bodies. E.g. “deep stab has a high chance to stop an animal forever (because it has a high chance to hit an organ)”.
You can take a variable X (e.g. “a particular human”) and measure how useful all other variables are for easily proving important things about X. You’ll get a “semantic field” of X. The field will include mesavariables (e.g. “organs”, “cells”) and metavariables (e.g. “democracy”, “social norms”). The field will fade away at overly fine-grained variables (e.g. individual atoms) and overly coarse-grained variables.
The structure of abstractions (Lead 2)
Maybe we can deduce something about the structure of abstractions from first principles, in a way completely agnostic to the fundamental building blocks of the universe. Consider this argument:
Assume we model the world as programs modifying themselves and each other.
We have a finite set (S) of abstractions which capture most important properties of the programs’ states. Those abstractions are useful for predicting the future. Since the world is predictable, not all abstractions are equally likely (≈ not all future states of programs are equally likely).
Any output of a program can serve as input for another program. Programs can modify each other in the middle of execution. This has two consequences. First: most combinations of abstractions should map to S (i.e. simplify into base abstractions). Second: any predictable program should be modified by many programs in the same way.
The latter means that the semantic field of a predictable variable (X) should have self-similar / redundant structure. A couple of silly examples in the spoiler.
Big forest stone. It doesn’t move itself; animals don’t try to move it; the landscape around it doesn’t move. So, you could say the stone maximizes “avoidance of movement” in different ways.
Lake. When animals dive into the lake, they displace water (create “holes” in the body of water), but said water comes back; the water evaporates, gets drank by animals, seeps through the ground, but comes back with rain; the water of the lake fills a hole in the landscape. So, you could say the lake maximizes “refillment” in different ways.
By the way, I think the same sort of definition could be used to define “optimization”. Here’s a comment with an earlier iteration of this idea.