Thanks for posting this. Some things to add to my reading list.
If you consider this “(potentially) useful training for making progress on Friendly AI”, then do you expect that a person who has worked through this material will have a good sense of whether they are qualified to actually try to make progress on FAI (or will be evaluable for that by someone with more experience working on FAI)? I want to do as much as I can to contribute to FAI, whether directly (by working on the actual problems) or indirectly (e.g. getting rich and donating a lot to SIAI), whichever I end up being most efficient at. Right now I’m not efficient at much of anything, because of some severe issues with mental energy that I’m only now starting to possibly resolve after several years, but once I am more competent at life in general, I want to at least investigate the possibility that I could be directly useful to FAI research. (I’m not a savant or a mutant supergenius, but I am at least a normal genius.) If, at that point, I can get through all of this math successfully, will that be an indication that I should look further?
My best guess at productive subgoal for FAI is development of decision theory along the lines given in the last post, in order to better understand decision-making and the impossible problem in particular (how to define preference given an arbitrary agent’s program; what is a notion of preference that is general enough for human preference to be an instance).
About a year ago I was still at the “rusty technical background” stage, and my attempts to think about decision theory were not quite adequate. Studying mathematics helped significantly by allowing to think more clearly and about more complicated constructions. More recently, study of mathematical logic allowed me to see the beautiful formalizations of decision theory I’m currently working on.
I can’t tell you that studying this truckload of textbooks will get any results, but reading textbooks is something I know how to do, unlike how to make progress on FAI, so unless I find something better, it’s what I’ll continue doing.
Ambient decision theory, as it currently stands, requires some grasp of logic to think about, but the level of Enderton’s book might be adequate. I’m going deeper in the hope of developing more mathematical muscle to allow jumping over wider inferential gaps, even if I don’t know in what way. Relying on creative surprises.
Thanks for posting this. Some things to add to my reading list.
If you consider this “(potentially) useful training for making progress on Friendly AI”, then do you expect that a person who has worked through this material will have a good sense of whether they are qualified to actually try to make progress on FAI (or will be evaluable for that by someone with more experience working on FAI)? I want to do as much as I can to contribute to FAI, whether directly (by working on the actual problems) or indirectly (e.g. getting rich and donating a lot to SIAI), whichever I end up being most efficient at. Right now I’m not efficient at much of anything, because of some severe issues with mental energy that I’m only now starting to possibly resolve after several years, but once I am more competent at life in general, I want to at least investigate the possibility that I could be directly useful to FAI research. (I’m not a savant or a mutant supergenius, but I am at least a normal genius.) If, at that point, I can get through all of this math successfully, will that be an indication that I should look further?
My best guess at productive subgoal for FAI is development of decision theory along the lines given in the last post, in order to better understand decision-making and the impossible problem in particular (how to define preference given an arbitrary agent’s program; what is a notion of preference that is general enough for human preference to be an instance).
About a year ago I was still at the “rusty technical background” stage, and my attempts to think about decision theory were not quite adequate. Studying mathematics helped significantly by allowing to think more clearly and about more complicated constructions. More recently, study of mathematical logic allowed me to see the beautiful formalizations of decision theory I’m currently working on.
I can’t tell you that studying this truckload of textbooks will get any results, but reading textbooks is something I know how to do, unlike how to make progress on FAI, so unless I find something better, it’s what I’ll continue doing.
Ambient decision theory, as it currently stands, requires some grasp of logic to think about, but the level of Enderton’s book might be adequate. I’m going deeper in the hope of developing more mathematical muscle to allow jumping over wider inferential gaps, even if I don’t know in what way. Relying on creative surprises.