I particularly wish people would taboo the word “optimize” more often. Referring to a process as “optimization” papers over questions like:
What feedback loop produces the increase or decrease in some quantity that is described as “optimization?” What steps does the loop have?
In what contexts does the feedback loop occur?
How might the effects of the feedback loop change between iterations? Does it always have the same effect on the quantity?
What secondary effects does the feedback loop have?
There’s a lot hiding behind the term “optimization,” and I think a large part of why early AI alignment research made so little progress was because people didn’t fully appreciate how leaky of an abstraction it is.
I particularly wish people would taboo the word “optimize” more often. Referring to a process as “optimization” papers over questions like:
What feedback loop produces the increase or decrease in some quantity that is described as “optimization?” What steps does the loop have?
In what contexts does the feedback loop occur?
How might the effects of the feedback loop change between iterations? Does it always have the same effect on the quantity?
What secondary effects does the feedback loop have?
There’s a lot hiding behind the term “optimization,” and I think a large part of why early AI alignment research made so little progress was because people didn’t fully appreciate how leaky of an abstraction it is.