[Caveat: Apart from reading roughly five posts and discussion threads about Maple over the years recently, I have no further context and so am one of the least informed people commenting here. But I think that’s okay because my comment will be about how I think your post comes across to onlookers.]
Even though you’re overall critical of Maple, I still get the impression that the closeness to them has negatively affected your judgment about some things. But maybe I misread the point of the post. To me, it sounds like you’re saying that, while, your impression of Maple is that they traumatize a bit too many people and they don’t seem to produce sufficiently many “saints” (or, as early indicators of success: “micro saints”) for these methods to be worth it, you think it’s important (so I infer?) that critics should engage with Maple on the object level of the strategy/path to impact that they’re pursing, because that strategy (of producing saints to help save the world) was possibly worth trying ex ante? In other words, critics shouldn’t sneer at the strategy itself but rather have to at least consider (problems with) its execution?
Assuming the above is an accurate paraphrase, my reply would be that, no, that sort of strategy never made much sense in the first place, and obviously so. You don’t save the world by doing inward-looking stuff at a monastery, and “path to impact through producing saints” seems doubtful because:
(1), people who greatly inspire others almost never started out as followers in a school for how to become inspiring (this is similar to the issues with CFAR, although I’d say it was less outlandish to assume that rationality is teachable rather than sainthood). (2), even if you could create a bunch of particularly virtuous and x-risk-concerned individuals, the path to impact would remain non-obvious from there, since they’d neither be famous nor powerful nor particularly smart or rational or skilled, so how are they going to have an outsized impact later?
Overall, this strategy does not warrant the “under steep tradeoffs, we should be more willing to accept the costs of harming people” outlook.
I feel like if the recruitment and selling point was less about “come here to have tons of impact,” and more about “have you always thought about joining a monastery, but you’re also into rationality and x-risks reduction?,” then this would be more okay and safer? A group oriented that way would maybe also be more generally relaxed and “have Slack,” and would be more alarmed if they were causing harm rather than trying to justify it via the potential of having impact. So, I feel like the points you raise sort-of-in-defense of Maple make things worse because all these attempts of explaining how the mission is important for world-saving are what adds pressure to the Maple environment.
You don’t save the world by doing inward-looking stuff at a monastery
My model of MAPLE is that this is exactly the crux. The theory of impact that they have is based on developing a potent enough ideology (a “meme”) that then spreads among powerful actors in the real world affecting their actions.
Currently they are in the design phase and are not looking to scale but eventually this “design a trustworthy religion” will become “offer the religion to cyborgs around the world”. Cyborgs are those humans who closely integrate AI, tech into their workflows, building a sort of exocortex.
If we don’t live in the FOOM world, then imagine how gradual disempowerment will play out. We will see increasing inequality in competence, productivity, capability as these models get more powerful. Also these cyborgs will be even more isolated, confused, lonely and due to suffering be keen to explore alternative religions to make sense of the world and decide on what motivations to pursue as meaningful, moral, etc.
It is these cyborgs that will first see consensus reality break down and traditional frameworks of meaning making will fail to help them properly orient to the world. Already we are seeing this with chatbots that are able to provide us “intimacy”, making us feel seen better than our friends. When algorithms can optimise our quantified self better than our own intuition, won’t more people abdicate their power to these systems?
The future of our species will depend on how empathetic, kind, compassionate, wise these cyborgs are. So I believe MAPLE is trying to design a lens to see the world, a perspective that resonates with them, that is true and ensures we continue to believe in truth being more than just predictive power, the ability to manipulate and control the external world, rather it is about living in harmony with other sentient beings.
They have an online course here which is likely more accurate than my interpretation. But during my stay with them as an AI fellow I got to interact with the students there and ask questions about why they believed the key was to cultivate the mind, how mind was chief.
Religions or ideologies are powerful because they form the framework by which we coordinate at scale. The current AI Alignment problem is not just a technical problem but a socio-technical problem, we are dealing with actors taking selfish actions, falling into race dynamics and feeling helpless due to the incentive structures around us.
In this video Soryu talks about how impactful religions have been in determining the flow of history. Towards the end he emphasises how people who deem themselves secular are in fact religious themselves. We have humanistic religions that hold human connection to be sacred and fundamental to meaning, scientific religions that hold reductionist, materialistic perspectives to be an axiomatic truth.
These systems make factual claims about what Is true about the world and then try to act innocent when normative claims (Ought) emerge (interdependently) as a consequence of believing in the factual claims. This relates to what herschel was saying about the culture being confused. If we believe that the basic ontology of reality is matter then concepts like love, joy, friendship will be reduced to transactional details, we can have allies that we cooperate with but we stop believing in anything valuable which cannot be reified. We seem to confused the map with the territory and believe that the representation can fully capture the referent.
I find all this very interesting and uncorrelated with other approaches to solve the alignment problem. I’d love to see more empirical falsifiable tests that could be designed to investigate the sort of moral realism adjacent (maybe?) implications of the claim that there is no hume’s gap.
I’ll say first, I… don’t actually endorse their model, maybe at all, but this post was to contextualize what the model even is, and that it’s maybe in principle plausible, and that their choices are made with respect to that, rather than just random-spiritual-community-is-bad-just-because-they’re-bad.
(1), people who greatly inspire others almost never started out as followers in a school for how to become inspiring (this is similar to the issues with CFAR, although I’d say it was less outlandish to assume that rationality is teachable rather than sainthood).
I think this is kind of wrong, lots of religious leaders trained within standard institutions within established traditions, lots of musicians get extensive training/coaching in all the aspects of performance besides their instrument etc. This also isn’t really a crux, because:
(2), even if you could create a bunch of particularly virtuous and x-risk-concerned individuals, the path to impact would remain non-obvious from there, since they’d neither be famous nor powerful nor particularly smart or rational or skilled, so how are they going to have an outsized impact later?
So Maple’s theory of change is not necessarily “get people enlightened, and then make sure they’re as agentic as possible”, but more like, get people enlightened, and then some combination of:
use whatever wisdom they gain to solve technical alignment
(this seems mostly just silly to me)
have them diffuse that wisdom into eg. tech culture, “purifying it from the inside-out’”
(again, I don’t think this is likely at all, like I said, but maybe more plausible)
resolve the incentives of the AI race domestically and internationally… somehow
I think this falls under the general concept of Pascal’s abuser: “Hey, I am doing something obviously harmful, but under my reasoning it has a microscopic chance of saving the world, therefore it’s okay.”
This feels somehow like a straw, but reflecting on it briefly it also feels like a hole in my explanation, and maybe that I’m just wrong here.
Maybe a different story I could tell would be that it’s more like “if you want, you can join us in trying to do something really hard, which has power law returns, knowing that the modal outcome is burnout and some psychological damage”, so comparable to competitive bodybuilding, or maybe classical musical training, or doing a startup. (Edit: note, Maple doesn’t include the “modal outcome is moderate psychological damage” part, though neither do the examples really.)
I don’t know enough about Maple to have an opinion on it. Here I am operating on feelings, and they remind me of Leverage Research. A project separate enough, so it’s not my business what they are doing… except when they fuck up, and then it becomes an “abuse in the rationalist community”. Also, they recruit a lot in the rationalist community.
These projects are justified by potential benefits, but I also see some potential negative externalities. (And by the same Pascalian logic, we should be extra careful about bad things happening to people who want to save the world?)
[Caveat: Apart from reading roughly five posts and discussion threads about Maple over the years recently, I have no further context and so am one of the least informed people commenting here. But I think that’s okay because my comment will be about how I think your post comes across to onlookers.]
Even though you’re overall critical of Maple, I still get the impression that the closeness to them has negatively affected your judgment about some things. But maybe I misread the point of the post. To me, it sounds like you’re saying that, while, your impression of Maple is that they traumatize a bit too many people and they don’t seem to produce sufficiently many “saints” (or, as early indicators of success: “micro saints”) for these methods to be worth it, you think it’s important (so I infer?) that critics should engage with Maple on the object level of the strategy/path to impact that they’re pursing, because that strategy (of producing saints to help save the world) was possibly worth trying ex ante? In other words, critics shouldn’t sneer at the strategy itself but rather have to at least consider (problems with) its execution?
Assuming the above is an accurate paraphrase, my reply would be that, no, that sort of strategy never made much sense in the first place, and obviously so. You don’t save the world by doing inward-looking stuff at a monastery, and “path to impact through producing saints” seems doubtful because:
(1), people who greatly inspire others almost never started out as followers in a school for how to become inspiring (this is similar to the issues with CFAR, although I’d say it was less outlandish to assume that rationality is teachable rather than sainthood).
(2), even if you could create a bunch of particularly virtuous and x-risk-concerned individuals, the path to impact would remain non-obvious from there, since they’d neither be famous nor powerful nor particularly smart or rational or skilled, so how are they going to have an outsized impact later?
Overall, this strategy does not warrant the “under steep tradeoffs, we should be more willing to accept the costs of harming people” outlook.
I feel like if the recruitment and selling point was less about “come here to have tons of impact,” and more about “have you always thought about joining a monastery, but you’re also into rationality and x-risks reduction?,” then this would be more okay and safer? A group oriented that way would maybe also be more generally relaxed and “have Slack,” and would be more alarmed if they were causing harm rather than trying to justify it via the potential of having impact. So, I feel like the points you raise sort-of-in-defense of Maple make things worse because all these attempts of explaining how the mission is important for world-saving are what adds pressure to the Maple environment.
My model of MAPLE is that this is exactly the crux. The theory of impact that they have is based on developing a potent enough ideology (a “meme”) that then spreads among powerful actors in the real world affecting their actions.
Currently they are in the design phase and are not looking to scale but eventually this “design a trustworthy religion” will become “offer the religion to cyborgs around the world”. Cyborgs are those humans who closely integrate AI, tech into their workflows, building a sort of exocortex.
If we don’t live in the FOOM world, then imagine how gradual disempowerment will play out. We will see increasing inequality in competence, productivity, capability as these models get more powerful. Also these cyborgs will be even more isolated, confused, lonely and due to suffering be keen to explore alternative religions to make sense of the world and decide on what motivations to pursue as meaningful, moral, etc.
It is these cyborgs that will first see consensus reality break down and traditional frameworks of meaning making will fail to help them properly orient to the world. Already we are seeing this with chatbots that are able to provide us “intimacy”, making us feel seen better than our friends. When algorithms can optimise our quantified self better than our own intuition, won’t more people abdicate their power to these systems?
The future of our species will depend on how empathetic, kind, compassionate, wise these cyborgs are. So I believe MAPLE is trying to design a lens to see the world, a perspective that resonates with them, that is true and ensures we continue to believe in truth being more than just predictive power, the ability to manipulate and control the external world, rather it is about living in harmony with other sentient beings.
They have an online course here which is likely more accurate than my interpretation. But during my stay with them as an AI fellow I got to interact with the students there and ask questions about why they believed the key was to cultivate the mind, how mind was chief.
Religions or ideologies are powerful because they form the framework by which we coordinate at scale. The current AI Alignment problem is not just a technical problem but a socio-technical problem, we are dealing with actors taking selfish actions, falling into race dynamics and feeling helpless due to the incentive structures around us.
In this video Soryu talks about how impactful religions have been in determining the flow of history. Towards the end he emphasises how people who deem themselves secular are in fact religious themselves. We have humanistic religions that hold human connection to be sacred and fundamental to meaning, scientific religions that hold reductionist, materialistic perspectives to be an axiomatic truth.
These systems make factual claims about what Is true about the world and then try to act innocent when normative claims (Ought) emerge (interdependently) as a consequence of believing in the factual claims. This relates to what herschel was saying about the culture being confused. If we believe that the basic ontology of reality is matter then concepts like love, joy, friendship will be reduced to transactional details, we can have allies that we cooperate with but we stop believing in anything valuable which cannot be reified. We seem to confused the map with the territory and believe that the representation can fully capture the referent.
I find all this very interesting and uncorrelated with other approaches to solve the alignment problem. I’d love to see more empirical falsifiable tests that could be designed to investigate the sort of moral realism adjacent (maybe?) implications of the claim that there is no hume’s gap.
Thanks for an object level response!
Yup, that’s an accurate enough paraphrase.
I’ll say first, I… don’t actually endorse their model, maybe at all, but this post was to contextualize what the model even is, and that it’s maybe in principle plausible, and that their choices are made with respect to that, rather than just random-spiritual-community-is-bad-just-because-they’re-bad.
I think this is kind of wrong, lots of religious leaders trained within standard institutions within established traditions, lots of musicians get extensive training/coaching in all the aspects of performance besides their instrument etc. This also isn’t really a crux, because:
So Maple’s theory of change is not necessarily “get people enlightened, and then make sure they’re as agentic as possible”, but more like, get people enlightened, and then some combination of:
use whatever wisdom they gain to solve technical alignment
(this seems mostly just silly to me)
have them diffuse that wisdom into eg. tech culture, “purifying it from the inside-out’”
(again, I don’t think this is likely at all, like I said, but maybe more plausible)
resolve the incentives of the AI race domestically and internationally… somehow
I think this falls under the general concept of Pascal’s abuser: “Hey, I am doing something obviously harmful, but under my reasoning it has a microscopic chance of saving the world, therefore it’s okay.”
Which is precisely what Why Are There So Many Rationalist Cults? is about.
This feels somehow like a straw, but reflecting on it briefly it also feels like a hole in my explanation, and maybe that I’m just wrong here.
Maybe a different story I could tell would be that it’s more like “if you want, you can join us in trying to do something really hard, which has power law returns, knowing that the modal outcome is burnout and some psychological damage”, so comparable to competitive bodybuilding, or maybe classical musical training, or doing a startup. (Edit: note, Maple doesn’t include the “modal outcome is moderate psychological damage” part, though neither do the examples really.)
I don’t know enough about Maple to have an opinion on it. Here I am operating on feelings, and they remind me of Leverage Research. A project separate enough, so it’s not my business what they are doing… except when they fuck up, and then it becomes an “abuse in the rationalist community”. Also, they recruit a lot in the rationalist community.
These projects are justified by potential benefits, but I also see some potential negative externalities. (And by the same Pascalian logic, we should be extra careful about bad things happening to people who want to save the world?)