my actual guess is that what I’m trying to actually mean by “goal” is less like an outcome and more like a bunch of flavors/constraints on ways of being, which is very visible in terms of determining ultimate outcomes (and makes the stakes of alignment to human values meaningful, real, and large).
that is my guess as well, and i proposed that such “bunch of flavours/constraints” will ultimately converge towards “acquiring more intelligence”—but i don’t want to contest this here.
what i would like you to note, tho, is that that “bunch of flavours/constraints on ways of being” is intelligence-gated. you can imagine a baboon having a goal of “make paperclips”; less so one of “understand and manipulate the fundamental laws governing your universe”. this comparison should shed light on what i mean by “dumb goals”.
I think that in fact I could become a lightcone-eating paperclip maximizer, if I wanted to, which I don’t.
if i wanted to, which i don’t is central here, and my basic point. there are levels of intelligence after which goals of such dullness are perceived as meaningless, and selection nudges towards increased intelligence. i really think you could find a second read helpful, now that the basic misunderstandings seem dispelled.
that is my guess as well, and i proposed that such “bunch of flavours/constraints” will ultimately converge towards “acquiring more intelligence”—but i don’t want to contest this here.
what i would like you to note, tho, is that that “bunch of flavours/constraints on ways of being” is intelligence-gated. you can imagine a baboon having a goal of “make paperclips”; less so one of “understand and manipulate the fundamental laws governing your universe”. this comparison should shed light on what i mean by “dumb goals”.
if i wanted to, which i don’t is central here, and my basic point. there are levels of intelligence after which goals of such dullness are perceived as meaningless, and selection nudges towards increased intelligence. i really think you could find a second read helpful, now that the basic misunderstandings seem dispelled.
May I ask if you read my whole comment?