is
Gunnar_Zarncke
Karma: 8,026
Some observations:
Genes reproduce themselves.
Humans reproduce themselves.
Symbols are relearned.
Values are reproduced.
Each needs an environment to do so, but the key observation seems to be that a structure is reliably reproduced across intermediate forms (mitosis, babies, language, society) and build on top of each other. It seems plausible that there is a class of formal representations that describe
the parts that are retained across instances and
the embedding into each other (values into genes and symbols), and
the dynamics of the transfer.
You don’t talk about human analogs of grokking, and that makes sense for a technical paper like this. Nonetheless, grokking also seems to happen in humans, and everybody has had “Aha!” moments before. Can you maybe comment a bit on the relation to human learning? It seems clear that human grokking is not a process that purely depends on the number of training samples seen but also on the availability of hypotheses. People grok faster if you provide them with symbolic descriptions of what goes on. What are your thoughts on the representation and transfer of the resulting structure, e.g., via language/token streams?
Brain-like AGI project “aintelope”
Ah, that troll.
Am I correct that this is called Troll Timers because some people “troll” their fellow players by thinking long?
I used to think very long on game moves—with the common comment/complaint that it doesn’t seem to improve my winning chances. Not sure about the latter. We did introduce timers to some games to limit turn time—and it did speed up things.
We-the-devs choose when to dole out these reinforcement events, either handwriting a simple reinforcement algorithm to do it for us or having human overseers give out reinforcement.
That sounds like are actually writing such reward functions. Is there another project that tries to experimentally verify your predictions?
I mean scoring thoughts in the sense of [Intro to brain-like-AGI safety] 3. Two subsystems: Learning & Steering with what Steven calls “Thought Assessors”. Thoughts totally get scored in that sense.
Yeah, it is like saying “energetically more favorable state”.
Somebody is reading shortforms...
I disagree. That humans learn values primarily via teaching. 1) parenting is known to have little effect on children’s character—which is one way of saying their values. 2) while children learn to follow rules teens are good at figuring out what is in their interest.
I think it makes sense to pose argue the point though.
For example I think that proposing rules makes it more probable that the brain converges on these solutions.
About the problems you mention:
the apparent phenomenon of credit assignment improving over a lifetime. When you’re older and wiser, you’re better at noticing which of your past actions were bad and learning from your mistakes.
I don’t get why you see a problem here. More data will lead to better models over time. You get exposed to more situations, and with more data, the noise will slowly average out. Not necessarily because you can clearly attribute things to their causes, but because you randomly get into a situation where the effect is more clear. It mostly takes special conditions to get people out of their local optimum.
without any anti-reinforcement event occurring
And if it looks like this comes in hindsight by carefully reflecting on the situation, that’s not without reinforcement. Your thoughts are scored against whatever it is that the brainstem is evaluating. And same as above, earlier or later, you stumble into some thoughts where the pattern is more clearly attributable, and then the weights change.
Hearing may work better for some people, but for some, like me, reading works much better—and faster. Also: Humans Who Are Not Concentrating Are Not General Intelligences—if you just passively absorb audio, then it better be aligned with your values (corollary: Also applies to music).
This may sound critical, but it is intended in the spirit of the Rule of Equal and Opposite Advice. I upvoted this because there is a lot of good advice, and there are people who need to hear it.
Some thoughts on the related topic of video can be found in this thread.
My model of how human values form:
Precondition: The brain has already figured out how the body works and some rough world model, say at the level of a small child. It has concepts of space and actions that can meet its basic needs by, e.g., looking for food and getting and eating it. It has a concept of other agents but no concept for interacting with them yet.
The brain learns to predict that other agents (parents, siblings...) will act to (help) get its needs met by acting in certain ways, e.g., by smiling, crying, or what else works.
The brain learns to predict that other agents act more reliably positively if expectations of the other agent are consistently met. We say that the child forms relationships. Parents may help children do that, but the brain is incentivized to figure it out in any case.
The brain notices and learns to predict that actions that work for one agent lead to negative results if observed by other agents (taking away something from your younger brother helps get your needs met but may have negative results if observed by parents). The prediction error of expected reward reinforces behaviors that work across multiple agents (over a suitable time). We call typical classes of behaviors roles like “being a good student/sibling/friend,” and such behaviors are fostered—but given enough group interactions, the brain is incentivized to discover and reinforce such behaviors without that.
Kegan writes that many adults get stuck at this stage. My guess is because they are not exposed to enough of the next development.
The brain learns to predict that some behaviors don’t work for some groups of people. For example, when traveling to another city or country, actions may lead to big reward prediction errors. Behaviors that work with the other group are reinforced, and the brain will learn more general strategies. A big class of these strategies are called values that work across most societies. Society tries to teach these general strategies, but the brain is incentivized to discover these—given enough variable group interactions—without education. There are failure modes like being value flexible, i.e., doing what works locally (“When in Rome do as the Romans do”) if the groups are too distinct but the convergence is toward value-like strategies.
The main difference between LDAIXI and a human in terms of ontology seems to be that the things the human values are ultimately grounded in senses and a reward tied to that. For example, we value sweet things because we have a detector for sweetness and a reward tied to that. When our understanding of what sugar is changes the detector doesn’t, and thus the ontology change works out fine. But I don’t see a reason you couldn’t set up LDAIXI the same way: Just specify the reward in terms of a diamond detector—or multiple ones. In the end, there are already detectors that AIXI uses—how else would it get input?
While looks like the examples you gave all express the same—being hungry—the differences can and do matter for humans.
Reading between the lines.
Cues about emotional states
Indicators of individual differences ( If you focus on the object level these might not make difference but I’d argue that they should be taken into account. This reduces the size of the null space as also suggested by Hoagy.
Or looked at in reverse it seems that human language use shows that our subconsciousness does already hide meaning in the sentences that consciously seem the same.
Wait till they add interest and penalties. Maybe I should write a post about the dozens of pages of house rules that were mostly decided by the kids. Plus, all the rules we agreed on that there were never written down. Or when my oldest started to write a comprehensive roommate agreement but luckily never finished.
There were maybe half a dozen downvotes. Would anybody like to comment on what that could refer to?
Minor addition to the list of practical problems:
If you use red teaming/ pentesting, somebody at the company has to determine what should be tested—systems services, locations. That may not be easy to determine, involve diligent work and the collaboration of multiple departments—and the incentive to skip parts.
There are some tricks to do this right. Cupping the hands, somewhat misaligning the hands so that the gaps are minimized. You might want to post a photo or video.
I got the article now.
“strong negative valence” appears to circumscribe exactly those cognitive events we might want to call “suffering”.
The advantage of defining suffering as one end of the valence scale (and presumably joy as the other end) is that it creates a scale. It doesn’t define anything really though.
To be agentic requires an agent that has parts that are included in it and an outside, the environment, that is acted on. Also, the agent needs to persist over time. Thus it has to be a replicator. I’m not sure you can have agents that are no replicators or replicators that are not agentic/control (parts of) their environment. When we say replicator, we focus on the replicating machinery, and when we talk about an agent, we focus on the control of the environment. But it’s two sides of the coin, maybe even a formal dual can be discovered.
All existing agents are evolved replicators which means that all their agency is focused on replication: Their replication, the replication of other coevolved agents, and the replication of non-agentic parts of the environment. There will be more or less accidental complexity (accidental in relation to the selection pressure of evolution; maybe a better term would be unforced structure).
The “just” does a lot of work in that sentence. I think it can only refer to unforced structures. As humans, we care about that—but “caring” is to the larger part supporting replication.