Also, another thing I’m interested in is how someone could have helped past you (and camp #1 people generally) understand what camp #2 even means. There may be a formal-alignment failure mode not noticeable to people in camp #1 where a purely mathematical (it may help to read that as ‘3rd-person’)
I don’t remember the exact specifics, but I came across Mary’s Room thought experience (perhaps through this video). When presented in that way and when directly asked “does she learn anything new?” my surprising (to myself at the time) answer was an emphatic “yes”.
Thanks! (I added a bunch of text to my comment while you were writing, also.)
I will try reply to your edit:
I don’t remember the exact specifics, but I came across Mary’s Room thought experience (perhaps through this video). When presented in that way and when directly asked “does she learn anything new?” my surprising (to myself at the time) answer was an emphatic “yes”.