While it does seem to be the case that people who “get stuff done in the world” are often basically wireheading on social status/money/power, people who don’t do this often end up wireheading on their own imaginations which is even worse from a “contact with reality” perspective. Thought inspired by SSI seemingly failing to achieve anything with their money and years of secret research(unless the rumors of them having CL are true, but I would guess not?). More generally I have a cached intuition that people who start a secrecy focused institution/research project usually fail.
I would generally agree with this but I don’t know what the rate of hidden successful initatives are and since I wouldn’t know them if they succeed the only information I would get about past hidden projects would be from the failed ones?
Maybe there are enough hidden projects that would no longer be hidden if they succeeded that it is enough to form a baserate of success or something?
I’m not sure how both groups are “wireheading” exactly. Particularly people who “get stuff done in the world”. What is that a ersatz of? I’m not sure how you can get something done without, say, having very real social interactions with people. Or if making something solo, it’s very tangible, so where is Wireheading coming into it?
Seems important to note that you wouldn’t hear about partial successes of secret projects if their stated goals are sufficiently ambitious. Suppose a project committed to not sharing anything publicly until they had built an aligned superintelligence; if they got as close as Anthropic and OAI currently are, you would not know. These other projects simply didn’t make an all-or-nothing commitment, in the way that SSI has.
Indeed, secret projects may accomplish more than their public counterparts, and just not share it.
“But then wouldn’t they have an incentive to share their progress even though it wasn’t total?” Maybe, but this would also mean going back on a strong (and important!) commitment, which is (often) strategically unsound in the long run, especially if your initial plan was to ~take over the world, and you’re reaping intermediate gains by revealing your current best guess of how to do so!
There is absolutely a population of conceptual AI researchers with extremely commercially valuable ideas who have decided not to ply their skills in that arena. I’m glad their work has remained private and not been directly applied to current public projects. I think it’s unwise to create social incentive against this type of prudence, as you seem to be doing here.
There is absolutely a population of conceptual AI researchers with extremely commercially valuable ideas who have decided not to ply their skills in that arena
Hmm, interesting...I’m curious who you’re thinking of but maybe you’d rather not say.
I actually do agree that “conceptually skilled” people can be very valuable in charting an overall direction, so it’s good if such people refrain from helping public AI projects. But I think conceptual skill is most useful when coupled with some sort of powerful feedback loop.
So this can work really well in domains like math, e.g. Andrew Wiles. But even there the dangers of wireheading on your own impressions of success are great, which is why some sort of scarce good in the external world like status and money can be a good feedback signal(well, “good” from the perspective of getting stuff done anyway, setting aside the goodness of the consequences of the work)
I’m not comfortable setting aside the goodness of the consequences of the work. I readily concede that these proxies provide signal that you’re doing anything at all. I think what I care about is how you get signal that the work is good, not just that it’s making a splash.
I’m thinking of a mix of cases where these insights have been empirically validated, have some empirical backing short of validation, or are entirely conceptual. The actual ML understanding of conceptual researchers is often underrated—many of them have objections to presenting their ideas using that language, which is different from their ideas being fully divorced from that arena.
I’m not comfortable setting aside the goodness of the consequences of the work
Reasonable. But if your project is likely to be ~neutral, that’s good to know too, no? Even if you don’t want to do more flywheel-y research, you could always pivot to doing something else entirely.
Hmm, maybe there was a miscommunication, I didn’t mean to suggest there were?(although maybe there are some, interesting question...)
I’m saying if you can predict that a secret research program is unlikely to succeed, you can do something different such as politics or a non-secret research program.
Yup, I misunderstood you, but I think we’re on the same page now.
The project being unlikely to succeed just goes into the EV calculation when comparing it to other projects. Secret projects also, definitionally, have fewer downsides, and my guess is that this latter term often dominates the nil modal outcome, since downsides for a huge swath of research are so high,
In fact, if you believe in their conviction, then receiving massive investments while the public hears nothing is probably what you would expect if they were succeeding in their goals.
I wouldn’t say strong counter-evidence, but some counter-evidence yes. It could also be a sign they’re pivoting to more practical directions
ETA: some evidence for the latter possibility(maybe?) is that there are rumors of a recent mini-coup at SSI.
While it does seem to be the case that people who “get stuff done in the world” are often basically wireheading on social status/money/power, people who don’t do this often end up wireheading on their own imaginations which is even worse from a “contact with reality” perspective. Thought inspired by SSI seemingly failing to achieve anything with their money and years of secret research(unless the rumors of them having CL are true, but I would guess not?). More generally I have a cached intuition that people who start a secrecy focused institution/research project usually fail.
I would generally agree with this but I don’t know what the rate of hidden successful initatives are and since I wouldn’t know them if they succeed the only information I would get about past hidden projects would be from the failed ones?
Maybe there are enough hidden projects that would no longer be hidden if they succeeded that it is enough to form a baserate of success or something?
I was also thinking of MIRI’s secret research and Jonathan Blow’s programming language. It’s possible that SSI and J. Blow could still succeed.
I’m not sure how both groups are “wireheading” exactly. Particularly people who “get stuff done in the world”. What is that a ersatz of? I’m not sure how you can get something done without, say, having very real social interactions with people. Or if making something solo, it’s very tangible, so where is Wireheading coming into it?
Yeah maybe not the best terminology, but I just mean they end up practically optimizing for those things instead of their stated goals.
What is “wireheading” as you understand it? It seems weird to me to describe seeking outcomes in the world (money, status, etc.) as “wireheading”.
Yeah maybe not the best terminology, I just mean they essentially end up pursuing those things in addition to/instead of their purported values.
Seems important to note that you wouldn’t hear about partial successes of secret projects if their stated goals are sufficiently ambitious. Suppose a project committed to not sharing anything publicly until they had built an aligned superintelligence; if they got as close as Anthropic and OAI currently are, you would not know. These other projects simply didn’t make an all-or-nothing commitment, in the way that SSI has.
Indeed, secret projects may accomplish more than their public counterparts, and just not share it.
“But then wouldn’t they have an incentive to share their progress even though it wasn’t total?” Maybe, but this would also mean going back on a strong (and important!) commitment, which is (often) strategically unsound in the long run, especially if your initial plan was to ~take over the world, and you’re reaping intermediate gains by revealing your current best guess of how to do so!
There is absolutely a population of conceptual AI researchers with extremely commercially valuable ideas who have decided not to ply their skills in that arena. I’m glad their work has remained private and not been directly applied to current public projects. I think it’s unwise to create social incentive against this type of prudence, as you seem to be doing here.
Hmm, interesting...I’m curious who you’re thinking of but maybe you’d rather not say.
I actually do agree that “conceptually skilled” people can be very valuable in charting an overall direction, so it’s good if such people refrain from helping public AI projects. But I think conceptual skill is most useful when coupled with some sort of powerful feedback loop.
So this can work really well in domains like math, e.g. Andrew Wiles. But even there the dangers of wireheading on your own impressions of success are great, which is why some sort of scarce good in the external world like status and money can be a good feedback signal(well, “good” from the perspective of getting stuff done anyway, setting aside the goodness of the consequences of the work)
I’m not comfortable setting aside the goodness of the consequences of the work. I readily concede that these proxies provide signal that you’re doing anything at all. I think what I care about is how you get signal that the work is good, not just that it’s making a splash.
I’m thinking of a mix of cases where these insights have been empirically validated, have some empirical backing short of validation, or are entirely conceptual. The actual ML understanding of conceptual researchers is often underrated—many of them have objections to presenting their ideas using that language, which is different from their ideas being fully divorced from that arena.
Reasonable. But if your project is likely to be ~neutral, that’s good to know too, no? Even if you don’t want to do more flywheel-y research, you could always pivot to doing something else entirely.
With a few minutes of effort, I can’t think of examples of people who are doing work that is:
Not motivated by money/status/power
Not motivated by their sense of what is good
Secret
Can you name any?
Hmm, maybe there was a miscommunication, I didn’t mean to suggest there were?(although maybe there are some, interesting question...)
I’m saying if you can predict that a secret research program is unlikely to succeed, you can do something different such as politics or a non-secret research program.
Yup, I misunderstood you, but I think we’re on the same page now.
The project being unlikely to succeed just goes into the EV calculation when comparing it to other projects. Secret projects also, definitionally, have fewer downsides, and my guess is that this latter term often dominates the nil modal outcome, since downsides for a huge swath of research are so high,
Nvidia’s recent huge investment in SSI seems like strong counter-evidence to this claim.
In fact, if you believe in their conviction, then receiving massive investments while the public hears nothing is probably what you would expect if they were succeeding in their goals.
I wouldn’t say strong counter-evidence, but some counter-evidence yes. It could also be a sign they’re pivoting to more practical directions ETA: some evidence for the latter possibility(maybe?) is that there are rumors of a recent mini-coup at SSI.