EDIT: Tom McGrath, Goodfire co-founder, says similar things in this comment
Oh, I would be surprised if that was their plan, seems like a bad plan for several reasons (though I may be overfitting to how GDM works):
The two hard parts are getting things to work on lab infra, and figuring out the exact recipe that works on the lab’s model, neither of which Goodfire can do
Goodfire might be able to find some general research insights into how to do this well, along with validating the idea. But I back a frontier lab’s ability to figure this out if sufficiently motivated. They might pay Goodfire once for the IP, but then would have built enough in house expertise to do future research themselves, since you need to do that to implement it anyway (if it’s actually a big deal).
More generally it seems hard to get a lab to pay you significantly more than it would cost the lab to rediscover all the work themselves, and I don’t expect that price tag to get above eg $25M (and likely much lower). Maybe the angle would be an aquihire? I’m not sure how hard that would be.
If they wanted to do this at eg GDM an obvious thing would be to talk to my team, since I’m already pretty interested in this topic, and there’s natural safety applications too (and I totally don’t back my ability to make this a sustainable funding source for them)
There’s only like 5 potential customers, which seems kinda risky for something as uncertain as this (though idk maybe this is normal for startups)
More speculatively, if I wanted to make money from this I would probably target companies finetuning open source models, seems like they would be less sophisticated, have more niche needs, and there would be a lot more of them, even if they’re not as rich.
Separately, I’m pretty sure that this isn’t their main plan based on talking to them. And they’ve already publicly talked about various clients they’ve found for their science work, and work using SAEs.
I still don’t really have any model where their investors think their >$1.2B of future profits will come from, if not from somehow helping with frontier model training, so I still currently believe this is the default thing they will do. But I sure feel confused about lots of people saying it’s a bad business model.
I am also somewhat confused about this, but I could buy that if they could eg reliably discover novel scientific insights that’d be pretty lucrative. I also put some credence on investors being willing to make very speculative bets on sexy things like interpretability, without a great business plan, especially since Golden Gate Claude seemed to impress a lot of VCs. Idk at which round this stops working, I’d have guessed Series B is too far but who knows. But I do think it’s plausible there’s some killer app via interpretability with real commercial value, and that this would be easiest to monetise via customers who use open source models/train their own.
Thanks for sharing the thread with Tom, I hadn’t seen that—sounds like he and I are on the same page here.
EDIT: Tom McGrath, Goodfire co-founder, says similar things in this comment
Oh, I would be surprised if that was their plan, seems like a bad plan for several reasons (though I may be overfitting to how GDM works):
The two hard parts are getting things to work on lab infra, and figuring out the exact recipe that works on the lab’s model, neither of which Goodfire can do
Goodfire might be able to find some general research insights into how to do this well, along with validating the idea. But I back a frontier lab’s ability to figure this out if sufficiently motivated. They might pay Goodfire once for the IP, but then would have built enough in house expertise to do future research themselves, since you need to do that to implement it anyway (if it’s actually a big deal).
More generally it seems hard to get a lab to pay you significantly more than it would cost the lab to rediscover all the work themselves, and I don’t expect that price tag to get above eg $25M (and likely much lower). Maybe the angle would be an aquihire? I’m not sure how hard that would be.
If they wanted to do this at eg GDM an obvious thing would be to talk to my team, since I’m already pretty interested in this topic, and there’s natural safety applications too (and I totally don’t back my ability to make this a sustainable funding source for them)
There’s only like 5 potential customers, which seems kinda risky for something as uncertain as this (though idk maybe this is normal for startups)
More speculatively, if I wanted to make money from this I would probably target companies finetuning open source models, seems like they would be less sophisticated, have more niche needs, and there would be a lot more of them, even if they’re not as rich.
Separately, I’m pretty sure that this isn’t their main plan based on talking to them. And they’ve already publicly talked about various clients they’ve found for their science work, and work using SAEs.
See also my other discussion with Tom McGrath (one of the Goodfire cofounders) over here: https://www.lesswrong.com/posts/XzdDypFuffzE4WeP7/themanxloiner-s-shortform?commentId=DcBrTraAcxpyyzgF4
I still don’t really have any model where their investors think their >$1.2B of future profits will come from, if not from somehow helping with frontier model training, so I still currently believe this is the default thing they will do. But I sure feel confused about lots of people saying it’s a bad business model.
I am also somewhat confused about this, but I could buy that if they could eg reliably discover novel scientific insights that’d be pretty lucrative. I also put some credence on investors being willing to make very speculative bets on sexy things like interpretability, without a great business plan, especially since Golden Gate Claude seemed to impress a lot of VCs. Idk at which round this stops working, I’d have guessed Series B is too far but who knows. But I do think it’s plausible there’s some killer app via interpretability with real commercial value, and that this would be easiest to monetise via customers who use open source models/train their own.
Thanks for sharing the thread with Tom, I hadn’t seen that—sounds like he and I are on the same page here.
OK, thanks, that’s a relief then. We shall see who their customers end up being.