Hey there~ I’m Austin, currently building https://manifund.org. Always happy to meet LessWrong people; reach out at akrolsmir@gmail.com!
Austin Chen
Karma: 2,671
Thinking on this more, I’m curious how/whether copyright is considered in Plan A’s Total Research Transparency (a proposal I broadly like.) Rereading the default proposal for TRT, it calls to restrict most training data:
AI model weights, significant fractions of the training data, and a small amount of other sensitive information is prevented from leaving the datacenters.
I’m a bit surprised because I assumed “data” was part of “research”; most calls for “open science” include calls to publish the underlying data. And I assumed that if the goal of Plan A is to make it so that model progress is strictly gatekept on compute, in which case making training data also transparent/public would aid that goal.
And now I’m also curious what kind of training data would be published as part of TRT.
Tagging @Thomas Larsen?
Some people seem mad that Anthropic shredded a bunch of books (eg this comic, this writeup). I wonder if Anthropic could solve this by making those books available, Google Books style?
Idk what the legal complexities around copyright would imply but Google Books has solved some of those by previewing a few pages at a time. Anthropic could maybe go further and see if any of the book authors/publishers want to make their work available, offer the $3k or wtv from the other settlements, etc.
Probably Penny Arcade’s reactions aren’t actually driven by book shredding per se and more of a general unhappiness about AI. But it’d still be a nice gesture, I think.
In case anyone else was tired of continually clicking “Show More” to see the full list, Claude Code pulled out the signatories and comments here: https://gist.github.com/akrolsmir/05b0fac81b0f43c9950493bdc0e5a95d
Thanks for the nudge, will keep this in mind!
As another example, I very much appreciate whoever runs cach3 for mirroring the FTX Future Fund website for posterity https://ftxfuturefund.org.cach3.com/index.html
I broadly agree that this is the direction that privacy will go, and like Jeff Kaufman’s related writing on https://www.jefftk.com/p/preparing-for-less-privacy.
I think this fact is still not priced in, that is, there’s still alpha currently in starting from “assume that nothing is private; what would the world look like?” and building what’s missing. I think we used this to some good effect at Manifold, eg it was quite controversial at first when I took the stance that every trade and trader identity would public data (as this is not the case on most financial markets) but I think it made for quite a unique and good ecosystem.
Thank you for writing this. I thought I’d chime in with another example of this kind of thing I’d recently seen in (perhaps, contributed to) in our own work at Manifund.
A couple of months back, we were just kicking off our Falcon Fund (rapid grants for animal welfare, esp AI x animals) - see https://manifund.org/projects/falcon-fund. My original draft stated fairly strongly that the existing EA Animal Welfare Fund was dispensing money way too slowly eg couple months behind schedule for reviews; which was part of what we were excited to create Falcon. However, I was asked to remove this kind of internicine criticism from the launch post, on the grounds that (stylized) “everyone already knows that AWF is slow” and “we don’t to want to scare donors and reduce even further the small amount of money AW gets”
Also, one of the animal charity founders who Marcus thought was one of the best examples of donation he’s made also asked to be removed, because they did not want to be associated with EA, even though many of their donations come from EA sources.
I complied on both of these cases on a general heuristic of trying to be polite and play nice and aim to be positive sum with our counterparties, but imo these hurt Falcon’s ability to raise and Manifund’s general mission of increasing transparency in the ecosystem. I’m still not sure what the right call was.
(Note that Carol is the primary author of this piece, but couldn’t add me as editor/secondary, since their LW account is brand new; would appreciate a fix from mods.)
This is fantastic. Love to see more microgrants programs, esp with a clear scope & lightweight emphasis; love to see retro and prize funding.
Were you planning on publishing where the grants are going? Even a cG-style “this much $ to this person” is helpful, a Manifund-style “here’s the partial/full application” would be amazing, and the oldschool LTFF-like retros would be fire.
Also, where do you expect to be bottlenecked—application dealflow? reviewer time? $ raised?
Note for the LessWrong team: Anton’s fancy page breaks looked good in the draft editor, see below; but were way too big on the actual post, so I’ve removed them now. It might be good to have the editor better reflect the final post image size, or vice versa.
Thank you for writing this! I’ve long been a fan of the SFF compared for the outputs it produces, but agree with the aspects of poor UX and slowness from the grantee side; it’s nice to hear about the recommended side.
Curious:
Does SFF pay its recommenders? Can you say how much?
Does SFF negotiate an impact split between its recommenders and its funders? (I’m not aware of one between itself and it’s grantees fwiw).
Not yet, I might look into Substack or a different thing for publishing an RSS feed if I end up doing more podcasts.
(if any podcasters out there have a rec for their favorite podcast publishing platform please lmk!)
As someone who’s spent a lot of time applying for funding, I do think being slow can be really really bad, in ways that can be invisible when you’re conducting a retro on the outcomes. Like in getting a decision back in a couple days vs a couple months is the difference between being able to quit your job, launch a project, hire a key employee—and not doing any of those things. Even if not counterfactual, speeding up a key project by 1-3 months means that the it has that much more time to scale up and have impact.
If you take a step back and think about what the bottlenecks in the space is (even just today, and what it might be down the line): money is plentiful, opportunities are scarce, timelines are short. And so I think the ecosystem should worry less about trying to get better funding decisions and more about making opportunities happen at all. (You alluded to this a bit with “Most value comes from finding/creating projects many times your bar”; this is also why I’m running Surplus instead of pushing with Oli on the funding platform).
Possibly we don’t actually disagree; I think a deliberate system with thoughtful accurate impact decisions is great for retrospective evaluation, it just shouldn’t be the only thing. I note that SFF produces pretty good allocations year after year. but the experience of actually being an SFF grantee is awful in many ways, I’m glad at least there are speculation grantors but that’s chump change compared to the actual round, and the actual round annoyingly takes like 3-6mo.
Longview doesn’t publish its grants so it’s hard to argue about them.
Yeah idk, this is the specific thing that bothers me about Longview. Like sure, maybe they’ve made fantastic recommendations in the past, or currently have a lot of good information about where to give. How would I know? What’s stopping them from doing the thing every other EA funder does and just publishing a list of where they moved money?
I do like the Longview grantmakers that I know! I also like all the CG grantmakers I know. (Basically every individual person I meet in AI safety or EA is a person I like very much). But I think the institutional processes and culture at grantmaking orgs shapes what actually happens, perhaps more so than the individual who is nominally “grantmaker”, and so I am happy with Manifund’s work in making grants faster/transparently/decentralized, and excited about Oli’s plan for a revamped S-process.
By my lights, some regrantors are great, but most are substantially worse than the standard professional grantmakers, and it’s apparently nontrivial to tell which are which (and in particular Manifund is not good at this).
Thanks for the criticism; would you be able to say more, either here or privately, about what you see as what is worse about our regrants or regrantors? (In case it helps, we recently made https://manifund.org/about/regranting-data to display where our regrantors have given across the last 3 years.)
I think of our AI safety regrantor program as trying to do a pretty specific thing: make it easy for experts working fulltime in AI safety to seed new opportunities that come across their network. So first, I’d want to check that you’re comparing at similar org stage/check size, between regrantor & professional grantmakers. I think the nature of seeding new opportunities is that a lot of them are going to fail, or go towards things that don’t have an obvious good outcome.
If you’re pretty confident that our regrantor choices are bad, I’d also be interested in hearing you name specific regrantors you think would be good & why; we’re always considering new regrantors from year to year. Also, specific opportunities; while I appreciate your recent writeups on meta considerations around grantmaking, I don’t currently have a sense of where you would literally choose to send money, or would have in past years. (Maybe Bores & other political giving stuff, recently? That’s not crazy but outside of 501c3 scope and I think represents one pretty specific worldview/angle.)
I believe the state of the art is “convince @Eric Neyman to write a blog post in support”
I’m excited for more attempts to map the ecosystem, and especially if we work out ways to keep them automatically up to date (eg with scraping + LLMs) over time!
That whole thing about OpenAI being the largest theft in human history? OpenAI may be the largest, but the model isn’t unique to them, Ought did it two years earlier. Ought started out as a 501c3 non-profit doing AI Alignment Research, with a mission to “scale up good reasoning.” Ought spun off Elicit as an independent public benefit corporation in September 2023, selling its IP and transferring most of its staff to the new entity. Elicit raised a $22 million Series A at a $100 million valuation in early 2025. It’s a for-profit built on research that a non-profit’s donors paid for. I wonder how those donors feel about that.
I’m friends with the cofounders of Elicit, but I have to imagine that Ought’s donors are feeling pretty good about this outcome. My understanding is that the donations helped support the Ought team while they pivoted a bunch of times (including, one time running a prediction market!), eventually landing on Elicit, finding traction, and then spinning out as PBC.
Based on their 990 filings and Claude, it seems like Ought has spent about ~$10m over its lifetime as a nonprofit. (Which, tbf, is somewhat more than I would have guessed before looking this up.) But almost all of that has been recouped/retained in the form of $8m in cash from the sale of its shares of Elicit (which it got from the spinout). Now Ought, with an independent board, can spend that on projects that fulfill its nonprofit mission. For example, Ought recently made a large (~$1m?) donation to Lightcone in support of projects for AI for epistemics.
More so than the cash $, I expect that Ought’s donors (which I’m guessing include Jaan/SFF, but probably you could find out more by asking) feel quite good about Elicit existing. Because Elicit as a product is helping researchers do better science and generally helping people get to better epistemics; and so it doing its thing is good by the lights of its Ought’s donors.
Jungwon expands more on the conversion process here, also making the comparison to the OpenAI spinout. (FWIW, I do also kind of feel good about the OAI Foundation’s endowment and existence as a result of the spinout)
This is the kind of thing Manifund is great at, lmk if you want help with this!
Marcus Abramovitch (who is active on EA Forum) often talks about the bearish case and might be a good candidate to speak with. I believe his position is sth like “I’ll believe it when I see it, but from an outside view, people love saying they’ll donate but rarely follow through”.
Some other considerations for why money might be slower or lower:
A lot of the money might go to funds (eg Longview or CG), which then don’t deploy them quickly to projects, perhaps because lack of grantmaker capacity, or lack of trust, or general lack of incentive to deploy funds fast (in contrast to venture capital, where funders are competing to invest at lower valuations)
The individual donors (esp Anthropic employees) might just be way too busy to evaluate where to give to
The funds might not be spent on specific cause areas that one is expecting (eg maybe more GHD or AW than AIS. maybe not the field of AIS you were excited for)
On balance, I do think a lot of philanthropic giving will happen soon. But I think it would be great to have large liquid prediction markets or perp swaps on this kind of thing, so people and especially charities can hedge against lack of funding, or borrow against future funding.
1) Thanks! I agree this seems like an important problem to solve.
One dumb idea is to make it more “trustless”, eg by recording the phone screen so that the other orgs don’t need rely on word of mouth, they can actually just watch the interview (or get an llm to transcribe and summarize, etc). Obviously you’d want to get opt in from all sides, etc.
2) I hadn’t seen that! It’s cool that exists but I think the ideal version would be better designed and aimed at going viral.
Yeah, I’m not sure exactly how much negotiation would be involved, but given again that Google Books has paved the way for this (and also, they hired one of the Google Books people to run the project?) it might be easier the second time around, maybe relatively cheap way to buy goodwill?
(Other ways of buying goodwill on this particular topic might be to publish the full list of scanned book titles, link to whatever versions are still available for sale, make a statement that they’re supportive of books and authors etc)