I assume that you are alluding to spontaneously agentizing tools, a discussion initially triggered by the famous Karnofsky’s post. In your example the feedback loop is closed, which is probably enough for a tool eventually gaining agency.
Can you link to the post you’re talking about? I’ve been thinking about this problem, although I had never read or thought the words ‘spontaneously agentizing tools.’
I assume that you are alluding to spontaneously agentizing tools, a discussion initially triggered by the famous Karnofsky’s post. In your example the feedback loop is closed, which is probably enough for a tool eventually gaining agency.
Can you link to the post you’re talking about? I’ve been thinking about this problem, although I had never read or thought the words ‘spontaneously agentizing tools.’
See Objection 2 in http://lesswrong.com/lw/cbs/thoughts_on_the_singularity_institute_si/ and related posts and comments.