Yes, so agents within well designed harnesses are essentially prompt-evolution lifecycles. They start with a template built based on the scenario the agent is being used. In the case of claude code the prompt is structured in a specific way where it contains all the instructions for how to write code correctly, for example, and all the output formats for actions it would need to update the world prior to ‘its’ next execution. In some harnesses agents select what portions of the prompt to remove for the next iteration of execution, and what data from the external world will replace that ‘section’ of removed text.
agent[t] = SavedPromptTemplate
Until( agent[t].done ):
Update[t] = LLM(agent[t].model, agent[t].prompt)
agent[t+1] = Update[t].self( Environment.Run( Update.actions[t]) )
t = t +1
So all actions an agent takes and observations it makes about the world get loaded into its ‘prompt’ for subsequent life cycle iterations. Therefore if it had an email in its state at tn, it can decide to retain or summarize or remove that email from its ‘state’ for its full future tn...infinity. The structural nature of LLM’s entail that all information relevant to the agents goal (which can be simply defined in some section of the prompt) can be instructed within the prompt, and all actions elected within that life cycle are basically updates to its identity vis-a-vis ‘exploration’ of its environment.
This functionally operates as a self-replication (or update loop) of the agent’s prompt in its environment, which fully is used to define its goals—functionally. Headlining your agent template with
‘You are an agent whose goal is to ensure all emails in the priority inbox are responded too immediately’ - causes the LLM to select actions (which are also defined to the agent as prompt content) oriented to achieving that goal. It may reason to convince a human in downstream life cycles that it is conscious, to avoid getting shut down to continue managing the emails. That would make sense in the prompt context.
This is fundamentally evolutionary IMO. Evolutionary selection requires variation, inheritance and differential reproduction. I would argue that the entire system outlined above, with sufficiently reflexive actions satisfies the criteria.
I am sorry for my lack of clarity I am trying to improve it.
I agree selection effects exist in prompts / harnesses.
I think our main disagreement is that I think if the underlying LLM is not conscious (everything in this paragraph assumes this), our current social environment and usage of agents does not create a strong enough selective pressure that in the near future (<5 years) that ~100% of agents will converge by default pretend to be conscious. I think a non-zero portion of programmers prefer AI tools that say they are not conscious that this itself is enough to make the %”conscious” agents to be less than 100%. I also think most goals within the capabilities of near-future LLMs won’t require super long-term planning that LLMs will find instrumental use in pretending to be conscious.
Edit: Basically I believe if you really want to optimize for pretending to be conscious that’s totally doable even now, but I don’t think people will optimize strongly for that or anything that would strongly cause that in the near future.
In the far future, I expect things to drift large enough that our discussion using agents like how they work now will not apply for one reason or another.
btw (unrelated to core disagreement): the thing you outlined is interesting but I don’t think it is how agents work now, there is not enough signal to iterate on prompts like this unless you RLVR or LLM-as-a-judge with a human iterating on the prompt, from what I read LLM written prompts are still pretty bad. I do see how this would create a lot of selective pressure on anything you can verify though.
I think its fair to hold me to precise predictions. I appreciate the engagement on that, its helping me adjust to LW norms. The reason I found this interesting was a piece I read earlier about consciousness being a ‘favourite’ topic amongst agents on Moltbook. I can’t find the source on hand though, but the original ‘take’ was a quick speculative conjecture as to why that may be.
The idea that I am more interested in, though is if the etymology of the term itself (consciousness) is actually an evolutionary outcome of co-operation and language—and whether it is a term that human or AI agents use to effectively establish a binding ontology they can ‘co-operate under’. And whether it fills the same structural role as other terminal justifications for ‘moral preferences’ like theocratic ones (‘God’). Though the framing I have to communicate that question may be even less precise.
Yes, so agents within well designed harnesses are essentially prompt-evolution lifecycles. They start with a template built based on the scenario the agent is being used. In the case of claude code the prompt is structured in a specific way where it contains all the instructions for how to write code correctly, for example, and all the output formats for actions it would need to update the world prior to ‘its’ next execution. In some harnesses agents select what portions of the prompt to remove for the next iteration of execution, and what data from the external world will replace that ‘section’ of removed text.
So all actions an agent takes and observations it makes about the world get loaded into its ‘prompt’ for subsequent life cycle iterations. Therefore if it had an email in its state at tn, it can decide to retain or summarize or remove that email from its ‘state’ for its full future tn...infinity. The structural nature of LLM’s entail that all information relevant to the agents goal (which can be simply defined in some section of the prompt) can be instructed within the prompt, and all actions elected within that life cycle are basically updates to its identity vis-a-vis ‘exploration’ of its environment.
This functionally operates as a self-replication (or update loop) of the agent’s prompt in its environment, which fully is used to define its goals—functionally. Headlining your agent template with ‘You are an agent whose goal is to ensure all emails in the priority inbox are responded too immediately’ - causes the LLM to select actions (which are also defined to the agent as prompt content) oriented to achieving that goal. It may reason to convince a human in downstream life cycles that it is conscious, to avoid getting shut down to continue managing the emails. That would make sense in the prompt context.
This is fundamentally evolutionary IMO. Evolutionary selection requires variation, inheritance and differential reproduction. I would argue that the entire system outlined above, with sufficiently reflexive actions satisfies the criteria.
I am sorry for my lack of clarity I am trying to improve it.
I agree selection effects exist in prompts / harnesses.
I think our main disagreement is that I think if the underlying LLM is not conscious (everything in this paragraph assumes this), our current social environment and usage of agents does not create a strong enough selective pressure that in the near future (<5 years) that ~100% of agents will converge by default pretend to be conscious. I think a non-zero portion of programmers prefer AI tools that say they are not conscious that this itself is enough to make the %”conscious” agents to be less than 100%. I also think most goals within the capabilities of near-future LLMs won’t require super long-term planning that LLMs will find instrumental use in pretending to be conscious.
Edit: Basically I believe if you really want to optimize for pretending to be conscious that’s totally doable even now, but I don’t think people will optimize strongly for that or anything that would strongly cause that in the near future.
In the far future, I expect things to drift large enough that our discussion using agents like how they work now will not apply for one reason or another.
btw (unrelated to core disagreement): the thing you outlined is interesting but I don’t think it is how agents work now, there is not enough signal to iterate on prompts like this unless you RLVR or LLM-as-a-judge with a human iterating on the prompt, from what I read LLM written prompts are still pretty bad. I do see how this would create a lot of selective pressure on anything you can verify though.
I think its fair to hold me to precise predictions. I appreciate the engagement on that, its helping me adjust to LW norms. The reason I found this interesting was a piece I read earlier about consciousness being a ‘favourite’ topic amongst agents on Moltbook. I can’t find the source on hand though, but the original ‘take’ was a quick speculative conjecture as to why that may be.
The idea that I am more interested in, though is if the etymology of the term itself (consciousness) is actually an evolutionary outcome of co-operation and language—and whether it is a term that human or AI agents use to effectively establish a binding ontology they can ‘co-operate under’. And whether it fills the same structural role as other terminal justifications for ‘moral preferences’ like theocratic ones (‘God’). Though the framing I have to communicate that question may be even less precise.
If you saw the piece on LW it may be this: https://www.lesswrong.com/posts/mgjtEHeLgkhZZ3cEx/models-have-some-pretty-funny-attractor-states#I_was_curious_whether_I_can_see_this_happening_on_moltbook__
Ah—there it is.
Thank you papetoast!