[edit: why does this have so many more upvotes than my actually useful shortform posts]
Someone mentioned maybe I should write this publicly somewhere, so that it is better known. I’ve mentioned it before but here it is again:
I deeply regret cofounding vast and generally feel it has almost entirely done harm, not least by empowering the other cofounder, who I believe to be barely better than e/acc folk due to his lack of interest in attempting to achieve an ought that differs from is. I had a very different perspective on safety then and did not update in time to not do very bad thing. I expect that if you and someone else are both going to build something like vast, and theirs takes three weeks longer to get to the same place, it’s better to save the world those three weeks without the improved software. Spend your effort on things like lining up the problems with QACI and cannibalizing its parts to build a v2, possibly using ideas from boundaries/membranes, or generally other things relevant to understanding the desires, impulses, goals, wants, needs, objectives, constraints, developmental learning, limit behavior, robustness, guarantees, etc etc of mostly-pure-RL curious-robotics agents.
incidentally, I’ve had many conversations with GPT4 where I try to get it to tell me what difference it thinks justifies its (obviously reward-induced and therefore at-least-somewhat-motivated-reasoning) claim that it’s not like humans, and the only justification it consistently gives is continuous-time lived experience vs discrete-time secondhand textual training data. I feel like video models and especially egocentric robotics video models don’t have that difference...
I vaguely remember talking to you about this at the time but don’t remember what your motivations and thoughts were for cofounding vast at the time.
I think I’m most interested in this from the perspective of “what decisionmaking processes were you following then, how did they change, and what was the nearest nearby trail of thoughts that might have led you to make a different decision at the time?”
At the time my main worry was honestly probably just wanting money. Also a general distrust of deepmind, along with a feeling that alignment would be easy—compare the alignment optimism perspective, which I think discusses the same mechanisms and I would have agreed without qualification then. I still think some parts of that model, but now believe that the alignment problem’s main manifestations are moloch, authoritarianism, and rentseeking, and the failure story I expect no longer looks like “deepmind is in charge” and looks rather more like a disneyland without children. So the alignment approaches that seem promising to me are the ones that can counter people who are attempting to get alignment with the ownership system, because I expect humans to be suddenly locked out of the ownership system, including humans who are currently very rich within it.
I spoke to the cofounder a lot about mechanism design of social systems, and we had very interesting ideas for how to do it. If the world were going to stay human I’d be optimistic about designing novel currencies that are optimized to be unusually hard to moloch, and that optimism arose from many long conversations with him. But recent conversations with him seem to imply his views are corrupted by the drive for money; his views on mechanism design don’t seem to me to solve the misalignment of markets with their poor participants. He does have interesting ideas and I might have interest in having a lesswrong dialogue with him at some point.
Admissions like this are hard often hard to write.
So I hear. It wasn’t particularly.
credibility from helping to cofound vast
Ah yes, I, the long-since-exited cofounder of the, uh, mildly popular sort-of-indie gig-economy-of-things-style-rentseeking-of-web-hosting-service used by ai people, should use my overflowing Credibility stat to convince impactful people that...
...they should work on adding something to the list “qaci, boundaries, and similar proposals”?
hmm. idk, maybe. sounds more useful to say it without trying to make myself out to be anyone in particular. The people I’d want to convince are probably not the ones who’d be impressed by credentials of any kind.
Vast AI offers hourly rental of compute hardware? How do you believe this contributes to negative future outcomes?
I ask because assuming scaling hypothesis is mostly true, training potentially dangerous models require more compute than is available for rent. The big labs are using dedicated hardware clusters.
Another factor to examine is whether or not the number was “3 weeks” or “0 weeks”. Assuming Vast consumed ICs from the current limited supply, had Vast been slower to begin operations, the supply would still be limited.
Technically ok it signals Nvidia to order more 3 weeks early, by making the order backlog deeper, but the delta between “contributed” and “didn’t” is very small.
Finally you have to look at threat models. Actually participating in bad outcomes would be something like “let’s rent out compute hardware, not check who our customers are, let them run anything they want, and pay with anonymous credit cards. Hosted offshore.”
Today you would just be supporting illegal activity (for probably a price premium you could demand), but this is what could host the rogues of the future.
you and I have very different models of this. I’m not terribly interested in getting into the details. Some of your points overlap mine, some don’t. that’s all I feel is worth the time.
I previously told an org incubator one simple idea against failure cases like this. Do you think you should have tried the like?
Funnily enough I spotted this at the top of lesslong on the way to write the following, so let’s do it here:
What less simple ideas are there? Can an option to buy an org be conditional on arbitrary hard facts such as an arbitrator finding it in breach of a promise?
My idea can be Goodharted through its reliance on what the org seems to be worth, though “This only spawns secret AI labs.” isn’t all bad. Add a cheaper option to audit the company?
It can also be Goodharted through its reliance on what the org seems to be worth. OpenAI shows that devs can just walk out.
[edit: why does this have so many more upvotes than my actually useful shortform posts]
Someone mentioned maybe I should write this publicly somewhere, so that it is better known. I’ve mentioned it before but here it is again:
I deeply regret cofounding vast and generally feel it has almost entirely done harm, not least by empowering the other cofounder, who I believe to be barely better than e/acc folk due to his lack of interest in attempting to achieve an ought that differs from is. I had a very different perspective on safety then and did not update in time to not do very bad thing. I expect that if you and someone else are both going to build something like vast, and theirs takes three weeks longer to get to the same place, it’s better to save the world those three weeks without the improved software. Spend your effort on things like lining up the problems with QACI and cannibalizing its parts to build a v2, possibly using ideas from boundaries/membranes, or generally other things relevant to understanding the desires, impulses, goals, wants, needs, objectives, constraints, developmental learning, limit behavior, robustness, guarantees, etc etc of mostly-pure-RL curious-robotics agents.
incidentally, I’ve had many conversations with GPT4 where I try to get it to tell me what difference it thinks justifies its (obviously reward-induced and therefore at-least-somewhat-motivated-reasoning) claim that it’s not like humans, and the only justification it consistently gives is continuous-time lived experience vs discrete-time secondhand textual training data. I feel like video models and especially egocentric robotics video models don’t have that difference...
I vaguely remember talking to you about this at the time but don’t remember what your motivations and thoughts were for cofounding vast at the time.
I think I’m most interested in this from the perspective of “what decisionmaking processes were you following then, how did they change, and what was the nearest nearby trail of thoughts that might have led you to make a different decision at the time?”
At the time my main worry was honestly probably just wanting money. Also a general distrust of deepmind, along with a feeling that alignment would be easy—compare the alignment optimism perspective, which I think discusses the same mechanisms and I would have agreed without qualification then. I still think some parts of that model, but now believe that the alignment problem’s main manifestations are moloch, authoritarianism, and rentseeking, and the failure story I expect no longer looks like “deepmind is in charge” and looks rather more like a disneyland without children. So the alignment approaches that seem promising to me are the ones that can counter people who are attempting to get alignment with the ownership system, because I expect humans to be suddenly locked out of the ownership system, including humans who are currently very rich within it.
I spoke to the cofounder a lot about mechanism design of social systems, and we had very interesting ideas for how to do it. If the world were going to stay human I’d be optimistic about designing novel currencies that are optimized to be unusually hard to moloch, and that optimism arose from many long conversations with him. But recent conversations with him seem to imply his views are corrupted by the drive for money; his views on mechanism design don’t seem to me to solve the misalignment of markets with their poor participants. He does have interesting ideas and I might have interest in having a lesswrong dialogue with him at some point.
Makes sense, thanks for sharing!
Well done for writing this up! Admissions like this are hard often hard to write.
Have you considered trying to use any credibility from helping to cofound vast for public outreach purposes?
So I hear. It wasn’t particularly.
Ah yes, I, the long-since-exited cofounder of the, uh, mildly popular sort-of-indie gig-economy-of-things-style-rentseeking-of-web-hosting-service used by ai people, should use my overflowing Credibility stat to convince impactful people that...
...they should work on adding something to the list “qaci, boundaries, and similar proposals”?
hmm. idk, maybe. sounds more useful to say it without trying to make myself out to be anyone in particular. The people I’d want to convince are probably not the ones who’d be impressed by credentials of any kind.
Vast AI offers hourly rental of compute hardware? How do you believe this contributes to negative future outcomes?
I ask because assuming scaling hypothesis is mostly true, training potentially dangerous models require more compute than is available for rent. The big labs are using dedicated hardware clusters.
Another factor to examine is whether or not the number was “3 weeks” or “0 weeks”. Assuming Vast consumed ICs from the current limited supply, had Vast been slower to begin operations, the supply would still be limited.
Technically ok it signals Nvidia to order more 3 weeks early, by making the order backlog deeper, but the delta between “contributed” and “didn’t” is very small.
Finally you have to look at threat models. Actually participating in bad outcomes would be something like “let’s rent out compute hardware, not check who our customers are, let them run anything they want, and pay with anonymous credit cards. Hosted offshore.”
Today you would just be supporting illegal activity (for probably a price premium you could demand), but this is what could host the rogues of the future.
you and I have very different models of this. I’m not terribly interested in getting into the details. Some of your points overlap mine, some don’t. that’s all I feel is worth the time.
Its appearing on the front-page to me, and has been for the past day or so. Otherwise I never would have seen it.
Yeah, visibility of shortforms is now like 3-4x higher than it was a week ago, so expect shortforms in-general to get many more upvotes.
oh, that would do it. nice, btw.
I previously told an org incubator one simple idea against failure cases like this. Do you think you should have tried the like?
Funnily enough I spotted this at the top of lesslong on the way to write the following, so let’s do it here:
What less simple ideas are there? Can an option to buy an org be conditional on arbitrary hard facts such as an arbitrator finding it in breach of a promise?
My idea can be Goodharted through its reliance on what the org seems to be worth, though “This only spawns secret AI labs.” isn’t all bad. Add a cheaper option to audit the company?
It can also be Goodharted through its reliance on what the org seems to be worth. OpenAI shows that devs can just walk out.