Hey there~ I’m Austin, currently building https://manifund.org. Always happy to meet LessWrong people; reach out at akrolsmir@gmail.com!
Austin Chen
Karma: 2,840
Caroline Ellison has joined Manifund
(a note I wrote to Surplus founders)
over the last week I feel like there’s been a dramatic uptick in ai xrisk becoming salient, breaking out of containment, “crunchtime” discourse.
but also I think “crunchtime” is overblown? like, actually, things have always been crunch-y, in that at any point in history, someone smart and thoughtful and hardworking could have had a great impact. I like this meme circa ~2020, h/t Daniel Kokotajlo: “crunch time was in 2015″
and, yes, this is still true right now. this was always part of the thinking and hope of Surplus, that we could find a bunch of people to start amazing startups, because now is a very leveraged time. (if you don’t think your idea is on track to do something massively good for the world—talk to me! well, talk to me either way)
but also: it will continue to be true, that there’s lots of scope to do great things. it will be true next year. it will be true in 2030. and beyond. work hard, build great things, but don’t stress too much about crunchtime
Thanks! I do hope to sometime sit down and write a public “here are our best guesses on how to invest in TAI if you have $100k/$1m/$10m/$100m” etc, both as an interesting personal exercise and to help other donors figure out their strategies (and, in some ideal world, use this exercise to coordinate strategy across the total AIS philanthropic portfolio). If other people are interested in collaborating on this as a public service, please let me know.
(I think a reasonable question to grapple with in such a writeup is “how bad is it to invest in leveraged AI stocks wrt timeline speedup; at what scale does this investment start mattering”, as Sodium alludes to)
Can you talk about which other leveraged investments you think might provide better returns? Eg just broad TAI thesis stuff such as what VARA and Situational Awareness invest in? (Those specific funds)?
I think that, plus some botec/oom numbers on total portfolio size + ideal distribution, might help make this case. (Possibly that’s in the squiggle? But squiggles are much harder to parse than text)
Out of your solutions:
a) seems pretty reasonable, though at some point it devolves into a prize. There’s nothing wrong with scoped prizes; but another hope I have for impact markets is to eventually develop a grand unified system for assessing and awarding impact across all EA (and, other!) cause areas. (Perhaps, the way that the global economy is one grand unified market where dollars can flow anywhere)
b) is also reasonable and what I think cG is doing more of at the moment—giving huge amounts to orgs like Resolution and Redwood, some to do whatever they want with. I think this gets at many of the benefits of impact markets, though ideally early funders would also get repaid.
c) is also very reasonable! We haven’t historically done really large retro funding (here’s a small example), but if someone with money was excited to try to do this, we would be excited to help! (Manifund does not have a lot of discretionary money to spend, fwiw; we’re mostly a platform for donors and projects to meet.)
What happened to impact markets?
I think a lot of biblical or church teachings can be interpreted through the lens of “provide guidance to how agents should act”. Turn the other cheek advocates for forgiveness (eg in prisoners dilemma defect defect loops); give away everything to the poor advocates for rawlsian or marginal value of money considerations; love thy neighbor as thyself pushes for a total utilitarian view.
I think the church teachings are quite wise and lindy, and cover a lot of how human interactions should behave, and form the basis of a lot of my behavior (in addition to other influences, such as EA and startup culture). By my words of “best current understanding”, I think I meant “how to reason about moral truth at the frontier”, aka situations that are much farther than that, perhaps concerning giant corporations or advanced AI.
As a practicing Catholic and moral realist, something like what Max wrote above, represents my best current understanding of how to reason about moral truth, that is: a set of rules that governs how game theory agents ought to interact.
Can you share like 3 of your most commonly used expansions?
Thanks for writing this!
God, I really wish there was a professional counterpart to me who can host a session of “here are all of the weird things that professionals are more likely to believe, please ask questions”
I’m not sure I’d put myself as a fully professional counterpart, but I’m probably closer to the professional end of the axis than you, and would enjoy some version of this!
The closest thing might be to get the professionals to share war stories instead
Perhaps the professionals you’re thinking of are more “DC insiders” than “SF startup founders”, but for the latter I’d highly recommend Jessica Livingston’s Founders At Work, and more recently her podcast The Social Radars.
PG’s essays are another example of trying to make the tacit knowledge inside of early startups, more legible to the rest of the world.
I’d be excited for more versions of this that encompasses “frontier lab culture” in addition to “startup culture”, fwiw. Like a project that would be great imo is just to go around interviewing a bunch of lab people, “what is it like to be on the inside”.
Happy to hear it! And yes, I think credit allocation for not doing stuff is interesting but hard; anyone can claim to have not done stuff. (I’m not doing stuff all the time.)
I think perhaps a more tractable version is to allocate negative credit/impact for doing stuff that is bad? This could be like taxing externalities from a hypothetical govt perspective; or a more bottoms-up version of credit that takes our social intuitions around “who did bad things” and makes it more legible.
I’m excited for someone to attempt a proper accounting of the concrete impact/xrisk cost of eg accelerating race dynamics, using numbers instead of just vibes. (To be clear, I’m extremely grateful to Richard for doing it here in vibes; that’s a necessary first step.)
The actual version of ACS—which did things closer to what you advocate for—had trouble retaining people and getting funding.
This seems sad. Are you open to accepting more funding, eg through Manifund? I particularly enjoyed the post-AGI conference and would be happy to make a personal small donation in support of ACS’s continued activities.
(From historical records it seems like ACS has received around $2m, mostly through SFF and more recently cG.)
Why self-fund your regranting?
Interesting, thanks. My heuristics are also that capabilities spend is roughly 1000:1 vs safety spend at the moment, which is partly why I feel like a universalized 10% pledge from employees might be good (forcing function to get the ratio down to 10:1, in the steady state)
In general, we’re excited both for the project of increasing total safety spend, and of targeting it better so donors get what they’re trying to buy
Yeah I know you’re against being too econ brained (vs social brained), and I’m pretty supportive of that stance actually.
But I think the case for economic reasoning is still kind of good in a lot of places? Or, I’m very interested in figuring out when it’s appropriate to apply. (excited for your part 2!)
I think murder or several kinds of crimes seem weird to offset, but also it’s not clear that working in capabilities carries the weight of murder, in the eyes of all but a very select few atm. One might argue that the job of an AI safety movement is to make capabilities work extremely sanctioned with the moral weight of murder (or perhaps nuclear arms proliferation), but I don’t think I’d fully agree with those.
Other analogies include carbon offsets (which, seem ethically fine, though maybe the way out for climate is through growth/solar not offsets), meat consumption offsets (which I tend to think seem good, but admit are controversial). Or, paying the government for certain kinds of licenses or taxes for when your org produces negative externalities (sound/noise, pollution, gambling).
More links https://slatestarcodex.com/2015/01/04/ethics-offsets/ and https://sideways-view.com/2021/03/21/robust-egg-offsetting/ and https://www.jefftk.com/p/why-im-not-vegan
People seem not to like capabilities offsetting, curious to hear why. Obviously there are measurement problems (“how much safety are you actually buying?”) and moral licensing effects to avoid. But from a purely econ-brained perspective it seems like a reasonable thing to try, if you can get the numbers right?
I guess, I’d also just be interested in the project of figuring out what % of a lab employee’s salary they’d need to donate to safety, for the impact to net out even. 1%? 10%? 50%? 100%? 200%? more?
Interesting! I had a different idea for a pledge, more of a “giving what we can” or “farmkind” style where lab employees commit to offset some % or some amount of their capabilities externalities by donating to safety causes.
Also an idea for a “union for lab employees” to help increase their bargaining power on topics like safety or redistribution of wealth in society.
Who we’d like as regrantors
On the specific question of “when are prediction markets landing on Twitter”, I’d defer to @Nathan Young or Jay Baxter
More broadly, I think AI suggested operationalizations and resolutions are quite promising as a way of reducing human burden of maintaining markets. Manifold does have a built in system for eg pasting in a news article and suggesting markets on the outcome; I’m not sure how often it’s used or how good it is though. Occasionally a creator will defer resolution to AI, though that’s still quite rare.
(AI also seems like a good first layer for some kind of courtlike resolution system, which escalates to more expensive/trustworthy systems as the volume or importance of the question goes up)
mm i agree with some flavor of this, which is that “not literally everyone, or even all the smart people, should be working on AI safety”. with Mox, I’m pretty open/interested to do some good ol-fashioned broad-tent EA fieldbuilding.