Sure, it’s not necessary that a sufficiently advance AI has to work like the brain, but there has to be an intuition about why is not need it to at least create an utility maximizer.
Octopus’ brain(s) is nothing like that of mammals, and yet it is equally intelligent.
“Sanity” may not be a useful concept in edge cases, but yes, being able to trust your mind to autopilot is definitely within the central definition of sanity, it’s a good observation.
You may also be interested in Scott’s post series on the topic, the latest being https://www.astralcodexten.com/p/contra-kirkegaard-on-evolutionary
FTFY: “Smile at strangers iff it has non-negative EV, because smiling is cheap and sometimes it does”.
“I am going to read you mind and if you believe in a decision theory that one-boxes in Newcomb’s Paradox I will leave you alone, but if you believe in any other decision theory I will kick you in the dick”
Sure, that’s possible. Assuming there are no Newcomb’s predictors in that universe, but only DK, rational agents believe in two-boxing. I am lost as to how it is related to your original point.
Let me clarify what I said. Any decision theory or no decision theory at all that results in someone one-boxing is rewarded. Examples: Someone hates touching transparent boxes. Someone likes a mystery of an opaque box. Someone thinking that they don’t deserve a guaranteed payout and hoping for an empty box. Someone who is a gambler. Etc. What matters is the outcome, not the thought process.
How well can you predict what will be good in 100 years? For perspective, given the contemporary zeitgeist of 1923, rather than knowing all we know now, what would have been considered a great policy to enact then that would take effect in 2023?
What if you believe in DKRUDT, the “Dick Kick’em rewards you” decision theory?
Seriously though, Newcomb’s setup is not adversarial in the same way, the predictor rewards or punishes you for actions, not beliefs. Your internal reasoning does not matter, as long as you end up one-boxing you walk away with more money.
As someone who gets rate-limited due to downvoted comments occasionally, I can see a rather strong correlation between the style/tone and the response. It is very much possible to express controversial, contrarian or even outright silly views without being downvoted. The rule of thumb is to be respectful and charitable to your opponents and, well, read the room. The more your view diverges from the local mainstream, the better, tighter and clearer your argument must be.
It is standard practice when writing scientific papers to first demonstrate intimate familiarity with the views you argue against later. This is what you would want from someone else writing a refutation of your argument, right? So give them the same courtesy. You can cut corners if your post or comment is close to the mainstream, but it is not a great habit. If you want to learn how to do it right, read ACX, Scott is a master of this art.
it is a common pitfall to blame the society/subculture for the negative reaction you get. It is almost never a constructive way to proceed.
If you are in doubt as to how your post would be received, and you want to get through to more people, consider reaching out to someone familiar with this site to review your draft. Or to anyone, really. The bar for writing a post or a comment with non-negative expected karma is pretty low.
it’s not a probability question. Blood tests exist.
Your potential contribution to timeline shortening as an undergrad is probably negligible on the margins, unless you are in the top 1% of all applicants, or even then. The field is crowded by very intelligent, well paid and very determined people who do this for a living. So whatever considerations you have, they need not be influenced by the AGI killeveryoneism.
That’s a fair point, different directions have different landmines. Culture war issues are tempting and most people seem to have strong yet poorly informed opinions that are not obviously poorly informed. I think that the original collection by Rob Bensinger The Library of Scott Alexandria is really good to start with, it is very light on political and culture war topics and grasps the essence of rational thinking, without going into the esoteric and irrelevant topics like quantum mechanics.
I find it hard to not assign whatever issues or problems we have to how the Universe works.
Indeed, the Universe just is, everything else is an emergent concept for the tiny embedded agents in it (bacteria, ants and humans). That includes sugar gradient, scent of food and laws of physics, respectively.
One of these emergent concepts is personal responsibility. Societies that do not have it do not last as long.
Step away from this place for a time and read Scott Alexander first. https://www.lesswrong.com/posts/vwqLfDfsHmiavFAGP/the-library-of-scott-alexandria is a good start, then continue on his blogs directly. He links back to this site where appropriate. You might get inured to some of the common pitfalls newbies here are prone to.
I agree with your analysis of the current situation. However, the technological issues arise when trying to correct it without severe unintended consequences, and that is not related to profit. You can’t transplant a house easily. You cannot easily feed only those who go hungry without affecting the economy (food banks help to some degree). There are people in need of companionship that cannot find it, even though there is a companion that would match somewhere out there. There are potential technological solutions to all those that are way outside our abilities (teleportation! replication! telepathy!) that would solve these issues. You can also probably find a few examples where what looks like profit-based incentive is in fact a technological deficiency.
I guess my point is the standard one: in many ways even poor people live a lot better now than royalty 300 years ago.
I’d assume that people themselves would define what they need, within the limits of what is possible given the technology of the time.
You might be thinking about it in a wrong way. Societal structures follow capabilities, not wants. If you try to push for “each person works and is paid according to their abilities and needs” too early, you end up with communist dystopias. If we are lucky, the AGI age will improve our capabilities enough where “to everyone according to their needs” may become feasible, aligning the incentives with well-being rather than with profit. So, to answer your questions:
It is currently impossible to “align the incentives” without causing widespread suffering.
It is undesirable if you do not want to cause suffering.
It is ineffective to try to align the incentives away from profit if your goal is making them aligned with “human well being”.
That said, there are incremental steps that are possible to take without making things worse, and they are discussed quite often by Scott Alexander and Zvi, as well as by others in the rationalist diaspora. So read them.
There is no a priori reason to believe that world has to be learnable. But if it were not, then we wouldn’t exist, nor would (most?) animals. The existing world, thus, is learnable. The human sensorium and motor system are necessarily adapted to that learnable structure, whatever it is.
I have a post or a post draft somewhere discussing this issue. The world indeed just is. It does not have to be internally predictable to an arbitrary degree of accuracy, but it needs to be somewhat internally predictable in order for the long-lived patterns we identify as “agents” or “life” to exist. Internal predictability (i.e. that an incredibly tiny part of the universe that is a human (or a bacterium) can infer enough about the world to not immediately poof away) is not something that should be a given in general. Further, even if some coarse-grained internal predictability can be found in such a world, there is no guaranteed that it can be extended to arbitrarily fine accuracy. it might well be the case in our world that at some point we hit the limit of internal predictability and from then on things will just look random for us. Who knows, maybe we hit it already, and the outstanding issues in the Standard Model of Cosmology and/or the Standard Model of Particle Physic, and/or maybe some of the Millennium prize problems, and/or the nature of consciousness are simply unknowable. I hope this is not the case, but I do not see any good argument that says “yep, we can push much further”, other than “it worked so far, if in fits and starts”.
I guess that is one way to say it. But the statement is stronger than that, I think. They do not care about the box or about anything else. They react to stimuli, then go silent again.
Thought I’d comment in brief. I very much enjoyed your post and I think it is mostly right on point. I agree that EA does not have a great epistemic hygiene, given what their aspirations are, and the veganism discussion is a case in point. (Other issues related to EA and CEA have been brought up lately in various posts, and are not worth rehashing here.)
As far as the quoted exchange with me, I agree that I have not stated a proper disclaimer, which was quite warranted, given the thrust of the post. My only intended point was that, while a lot of people do veganism wrong and some are not suited to it at all, an average person can be vegan without adverse health effects, as long as they eat varied and enriched plant-based diet and periodically check their vitamins/nutrients/minerals levels and make dietary adjustments as necessary. Some might find out that they are in the small minority for whom vegan diet is not feasible, and they would do well to focus on what works for them and contribute to EA in other ways. Again, I’m sorry this seems to have come across wrong.
Oh, and cat veganism is basically animal torture, those who want to wean cats off farmed animal food should focus on vat-grown meat for pet food etc.