On the existential question of our pointless existence in a pointless universe, my perspective tends to oscillate between two extremes:
1.) In the more pessimistic (and currently the only rationally defensible) case, I view my mind and existence as just a pattern of information processing on top of messy organic wetware and that is all ‘I’ will ever be. Uploading is not immortality, it’s just duplicating that specific mind pattern at that specific time instance. An epsilon unit of time after the ‘upload’ event that mind pattern is no longer ‘me’ and will quickly diverge as it acquires new experiences. An alternative would be a destructive copy, where the original copy of me (ie. me that is typing this right now) is destroyed after or at the instance of upload. Or I might gradually replace each synapse of my brain one by one with a simulator wirelessly transmitting the dynamics to the upload computer until all of ‘me’ is in there and the shell of my former self is just discarded. Either way, ‘I’ is destroyed eventually—maybe uploading is a fancier form of preserving ones thoughts for posterity as creating culture and forming relationships is pre-singularity, but it does not change the fact that the original meatspace brain is going to eventually be destroyed, no matter what.
The second case, what I might refer to as an optimistic appeal to ignorance, is to believe that though the universe appears pointless according to our current evidence, there may be some data point in the future that reveals something more that we are ignorant to at the moment. Though our current map reveals a neutral territory, the map might be incomplete. One speculative position taken directly from physics is the idea that I am a Boltzmann Brain. If such an idea can be taken seriously (and it is) than surely there are other theoretically defensible positions where my consciousness persists in some timeless form one way or another. (Even Bostrom’s simulation argument gives another avenue of possibility)
I guess my two positions can be simplified into:
1.) What we see is all there is and that’s pretty fucked up, even in the best case scenario of a positive singularity.
2.) We haven’t seen the whole picture yet, so just sit back, relax, and as long as you have your towel handy, don’t panic.
I’m relatively new to this site and have been trying to read the backlog this past week so maybe I’ve missed some things, but from my vantage point it seems like your are trying to do, Eliezer, is come up with a formalized theory of friendly agi that will later be implemented in code using, I assume, current software development tools on current computer architectures. Also, your approach to this AGI is some sort of bayesian optimization process that is ‘aligned’ properly as to ‘level-up’ in such a way as to become and stay ‘friendly’ or benevolent towards humanity and presumably all sentient life and the environment that supports them. Oh ya, and this bayesian optimization process is apparently recursively self-improving so that you would only need to code some seedling of it (like a generative process such as a mandelbrot set) and know that it will blossom along the right course. That, my friends, is a really tall order and I do not envy anyone who tries to take on such a formidable task. I’m tempted to say that it is not even humanly possible (without a manhattan project and even then maybe not) but I’ll be bayesian and say the probability is extremely low.
I think you are a very bright and thoughtful young guy and from what I’ve read seem like more of a philosopher than an engineer or scientist, which isn’t a bad thing, but to transition from philosophizing to engineering is not trivial especially when philosophizing upon such complex issues.
I can’t even imagine trying to create some trivial new software without prototyping and playing around with drafts before I had some idea of what it would look like. This isn’t Maxwell’s equations, this is messy self-reflective autonomous general intelligence, there is no simple, elegant theory for such a system. So get your hands dirty and take on a more agile work process. Couldn’t you at least create a particular component of the AI, such as a machine vision module, that would show your general approach is feasible? Or do you fear that it would spontaneously turn into skynet? Does your architecture even have modules, or are you planning some super elegant bayesian quine? Or do you even have an architecture in mind?
Anyway, good luck and I’ll continue reading, if for nothing else then entertainment.