I appreciate the attempt at diagnosing what went wrong here. I agree this is ~where it went off the rails, and I think you are (maybe?) correctly describing what was going on from M.Y. Zou’s perspective. But this doesn’t feel like it captured what I found frustrating.
[/edit]
What feels wrong to me about this is that, for the question of:
How would one arrive at a value system that supports the latter but rejects the former?
it just doesn’t make sense to me to be that worried about either authority or rigor. I think the nonrigorous concept, generally held in society of “respect people’s boundaries/autonomy” is sufficient to answer the question, without even linking to Critch’s sequence. Critch’s sequence is a nice-to-have that sketches out a direction for how you might formalize this, but I don’t get why this level of formalization is even particularly desired here.
(Like, last I checked we don’t have any rigorous conceptions of functioning human value systems that actually work, either for respecting boundaries or aggregating utility or anything else. For purposes of this conversation this just feels like an isolated demand for rigor)
I think that there are many answers along these lines (like “I’m not talking about a whole value system, I’m talking about a deontological constraint”) which would have been fine here.
The issue was that sentences like “It’s a boundary concept (element of a deontological agent design), not a value system (in the sense of preference such as expected utility, a key ingredient of an optimizer)” use the phrasing of someone pointing to a well-known, clearly-defined concept, but then only link to Critch’s high-level metaphor.
Okay, I get where you’re coming from now. Will have to mull over whether I agree but I am at least no longer feel confused about what the disagreement is about now.
Thanks, & thanks for putting in your own perspective here. I sympathize with that too; fwiw Vladimir_Nesov’s answer would have satisfied me, because I am sufficiently familiar with what the terms mean. But for someone new to those terms, they are just unexplained jargon, with links to lots of lengthy but difficult to understand writing. (I agree with Richard’s comment nearby). Like, I don’t think Vladimir did anything wrong by giving a jargon-heavy, links-heavy answer instead of saying something like “It may be hard to construct a utility function that supports the latter but rejects the former, but if instead of utility maximization we are doing something like utility-maximization-subject-to-deontological-constraints, it’s easy: just have a constraint that you shouldn’t harm sentient beings. This constraint doesn’t require you to produce more sentient beings, or squeeze existing ones into optimized shapes.” But I predict that this blowup wouldn’t have happened if he had instead said that.
I may be misinterpreting things of course, wading in here thinking I can grok what either side was thinking. Open to being corrected!
To be clear I super appreciate you stepping in and trying to see where people were coming from (I think ideally I’d have been doing a better job with that in the first place, but it was kinda hard to do so from inside the conversation)
I found Richard’s explanation about what-was-up-with-Vlad’s comment to be helpful.
[edit]
I appreciate the attempt at diagnosing what went wrong here. I agree this is ~where it went off the rails, and I think you are (maybe?) correctly describing what was going on from M.Y. Zou’s perspective. But this doesn’t feel like it captured what I found frustrating.
[/edit]
What feels wrong to me about this is that, for the question of:
it just doesn’t make sense to me to be that worried about either authority or rigor. I think the nonrigorous concept, generally held in society of “respect people’s boundaries/autonomy” is sufficient to answer the question, without even linking to Critch’s sequence. Critch’s sequence is a nice-to-have that sketches out a direction for how you might formalize this, but I don’t get why this level of formalization is even particularly desired here.
(Like, last I checked we don’t have any rigorous conceptions of functioning human value systems that actually work, either for respecting boundaries or aggregating utility or anything else. For purposes of this conversation this just feels like an isolated demand for rigor)
I think that there are many answers along these lines (like “I’m not talking about a whole value system, I’m talking about a deontological constraint”) which would have been fine here.
The issue was that sentences like “It’s a boundary concept (element of a deontological agent design), not a value system (in the sense of preference such as expected utility, a key ingredient of an optimizer)” use the phrasing of someone pointing to a well-known, clearly-defined concept, but then only link to Critch’s high-level metaphor.
Okay, I get where you’re coming from now. Will have to mull over whether I agree but I am at least no longer feel confused about what the disagreement is about now.
(updated the previous comment with some clearer context-setting)
Thanks, & thanks for putting in your own perspective here. I sympathize with that too; fwiw Vladimir_Nesov’s answer would have satisfied me, because I am sufficiently familiar with what the terms mean. But for someone new to those terms, they are just unexplained jargon, with links to lots of lengthy but difficult to understand writing. (I agree with Richard’s comment nearby). Like, I don’t think Vladimir did anything wrong by giving a jargon-heavy, links-heavy answer instead of saying something like “It may be hard to construct a utility function that supports the latter but rejects the former, but if instead of utility maximization we are doing something like utility-maximization-subject-to-deontological-constraints, it’s easy: just have a constraint that you shouldn’t harm sentient beings. This constraint doesn’t require you to produce more sentient beings, or squeeze existing ones into optimized shapes.” But I predict that this blowup wouldn’t have happened if he had instead said that.
I may be misinterpreting things of course, wading in here thinking I can grok what either side was thinking. Open to being corrected!
To be clear I super appreciate you stepping in and trying to see where people were coming from (I think ideally I’d have been doing a better job with that in the first place, but it was kinda hard to do so from inside the conversation)
I found Richard’s explanation about what-was-up-with-Vlad’s comment to be helpful.