Overall, an excellent post. It brought up some very clever ideas that I had never thought of or previously encountered.
I do, however, think that your colloquial use of the phrase “terminal value” is likely to confuse and/or irritate a lot of the serious analytic-philosophy crowd here; it might be wise to use some other word or other phrase for your meaning, which seems to be closer to “How an idealized[1] utility-maximizing agent would represent its (literal) Terminal Values internally”. Perhaps a “Goal-in-itself”? A “motivationally core goal”?
Early on, before “Mind the terminology, here”, the ambiguity is intentional. That your terminal goals are not sacred is true both for “utility function end-values” and “actions that are ends unto themselves”.
Later on in the post, I am explicit about my usage and consistent with the LessWrong Wiki entry on Terminal Values (which is also dichotomous). Still, I understand that some people are likely to get annoyed: this is difficult to avoid when there is a phrase that means too many things.
For now, I think that the usage is explicit enough in the post, but I do appreciate your input.
Overall, an excellent post. It brought up some very clever ideas that I had never thought of or previously encountered.
I do, however, think that your colloquial use of the phrase “terminal value” is likely to confuse and/or irritate a lot of the serious analytic-philosophy crowd here; it might be wise to use some other word or other phrase for your meaning, which seems to be closer to “How an idealized[1] utility-maximizing agent would represent its (literal) Terminal Values internally”. Perhaps a “Goal-in-itself”? A “motivationally core goal”?
Not perfectly idealized, as your post points out
Thanks!
Early on, before “Mind the terminology, here”, the ambiguity is intentional. That your terminal goals are not sacred is true both for “utility function end-values” and “actions that are ends unto themselves”.
Later on in the post, I am explicit about my usage and consistent with the LessWrong Wiki entry on Terminal Values (which is also dichotomous). Still, I understand that some people are likely to get annoyed: this is difficult to avoid when there is a phrase that means too many things.
For now, I think that the usage is explicit enough in the post, but I do appreciate your input.