I’m extremely keen to hear from people who have used Codex a decent amount (or tried to) and decided it isn’t worth it. Specifically, people who wouldn’t pay $15/mo for a subscription to it. Anyone?
For context, GitHub has 60,000,000 users. If 10% of them buy a $15/mo subscription, that’s a billion dollars a year in annual revenue. A billion dollars is about a thousand times more than the cost to create Codex. (The cost to train the model was negligible since it’s only the 12B param version of GPT-3 fine-tuned. The main cost would be the salaries of the engineers involved, I imagine.)
There is no possible way that 10% of GitHub’s entire user base (mostly free) will pay $15/mo, which is more than GitHub’s standard plan (team, $4/mo), and only slightly less than their most expensive plan (enterprise, $21/mo).
A few tens of thousands of early adopters will probably do so, but tiered pricing will happen long before it becomes popular. I predict there will be some use cases that justify $15/month, but the vast majority will be paid less, and charged by the resulting lines of code, the size/quantity of the prompts used, and/or the time consumed.
What are the main benefits people seek when they buy the more expensive plans? I don’t understand the stuff on the page, but it looks like it’s storage space + more features that make it easier to work in teams. I’m not sure how to compare that stuff to Codex but intuitively I feel like Codex is more valuable, because more people could benefit from Codex than are working in teams. I don’t know what I’m talking about though, which is why I’m asking. :)
If the charge is per token… let me think… suppose Codex gets called up to write something 10 times per programmer work-hour (it would come in clumps probably, not evenly spaced. Sometimes it would not give you what you want and you’d retry a couple times). That’s maybe 1000 tokens per work-hour, which (if it were GPT-3) would cost $0.06, so that’s like $0.50 a day, which comes out to $15.00 a month… I swear I didn’t plan that calculation to come out that way! (But of course it’s just a fermi estimate, could be off by orders of magnitude. Also, the current version of Codex is the 12B param version which probably costs an OOM less than GPT-3)
I’ve seen demos, but have not gotten direct access myself yet (and I’ll gladly pay that to evaluate, and long-term if I end up actually integrating it into my workflow). Agreed that Codex is valuable on different dimensions than GitHub’s current pricing model—for many, it will in fact be more valuable. I mostly pointed out the discrepancy to counter the argument that number of current GitHub users predicts anything about who will pay what amount for Codex.
I think that many many coders have sporadic use, and $0.50/day for days they use it ends up being a lot less than $15/month. My prediction is really that it will provide such widely varying value to different consumers that it’ll be near-impossible to charge the same amount to all of them.
Maybe I’m wrong, but my first reaction to your initial number is that users doesn’t mean active users. I would expect a difference of an order of magnitude, which keeps your conclusion but just with a hundred times more instead of a thousand times more.
That’s reasonable. OTOH if Codex is as useful as some people say it is, it won’t just be 10% of active users buying subscriptions and/or subscriptions might cost more than $15/mo, and/or people who aren’t active on GitHub might also buy subscriptions.
Agreed. Part of the difficulty here is that you want to find who will buy a subscription and keep it. I expect a lot of people to try it, and most of them to drop it (either because they don’t like it or because it doesn’t help them enough for their taste) but no idea how to Fermi estimate that number.
I’m extremely keen to hear from people who have used Codex a decent amount (or tried to) and decided it isn’t worth it. Specifically, people who wouldn’t pay $15/mo for a subscription to it. Anyone?
For context, GitHub has 60,000,000 users. If 10% of them buy a $15/mo subscription, that’s a billion dollars a year in annual revenue. A billion dollars is about a thousand times more than the cost to create Codex. (The cost to train the model was negligible since it’s only the 12B param version of GPT-3 fine-tuned. The main cost would be the salaries of the engineers involved, I imagine.)
There is no possible way that 10% of GitHub’s entire user base (mostly free) will pay $15/mo, which is more than GitHub’s standard plan (team, $4/mo), and only slightly less than their most expensive plan (enterprise, $21/mo).
A few tens of thousands of early adopters will probably do so, but tiered pricing will happen long before it becomes popular. I predict there will be some use cases that justify $15/month, but the vast majority will be paid less, and charged by the resulting lines of code, the size/quantity of the prompts used, and/or the time consumed.
Thanks! Have you used Codex?
What are the main benefits people seek when they buy the more expensive plans? I don’t understand the stuff on the page, but it looks like it’s storage space + more features that make it easier to work in teams. I’m not sure how to compare that stuff to Codex but intuitively I feel like Codex is more valuable, because more people could benefit from Codex than are working in teams. I don’t know what I’m talking about though, which is why I’m asking. :)
If the charge is per token… let me think… suppose Codex gets called up to write something 10 times per programmer work-hour (it would come in clumps probably, not evenly spaced. Sometimes it would not give you what you want and you’d retry a couple times). That’s maybe 1000 tokens per work-hour, which (if it were GPT-3) would cost $0.06, so that’s like $0.50 a day, which comes out to $15.00 a month… I swear I didn’t plan that calculation to come out that way! (But of course it’s just a fermi estimate, could be off by orders of magnitude. Also, the current version of Codex is the 12B param version which probably costs an OOM less than GPT-3)
I’ve seen demos, but have not gotten direct access myself yet (and I’ll gladly pay that to evaluate, and long-term if I end up actually integrating it into my workflow). Agreed that Codex is valuable on different dimensions than GitHub’s current pricing model—for many, it will in fact be more valuable. I mostly pointed out the discrepancy to counter the argument that number of current GitHub users predicts anything about who will pay what amount for Codex.
I think that many many coders have sporadic use, and $0.50/day for days they use it ends up being a lot less than $15/month. My prediction is really that it will provide such widely varying value to different consumers that it’ll be near-impossible to charge the same amount to all of them.
Maybe I’m wrong, but my first reaction to your initial number is that users doesn’t mean active users. I would expect a difference of an order of magnitude, which keeps your conclusion but just with a hundred times more instead of a thousand times more.
That’s reasonable. OTOH if Codex is as useful as some people say it is, it won’t just be 10% of active users buying subscriptions and/or subscriptions might cost more than $15/mo, and/or people who aren’t active on GitHub might also buy subscriptions.
Agreed. Part of the difficulty here is that you want to find who will buy a subscription and keep it. I expect a lot of people to try it, and most of them to drop it (either because they don’t like it or because it doesn’t help them enough for their taste) but no idea how to Fermi estimate that number.