Thinking about it more there’s another, more serious restriction, at least for now: Codex can’t write code that depends on the rest of your codebase. Consider the following lightly-anonymized code from a real-world codebase I contribute to:
def unobserved(self) → Set[str]:
“”″Returns a set of all unobserved random variable names inside this Distribution—that is,
those that are neither observed nor marginalized over.
I don’t think Codex could write that code from just that function signature and that docstring, because a human couldn’t do it: they wouldn’t know how to find the names of the observed and marginalized random variables, and they wouldn’t know that self.components exists or has to be explicitly converted into a set. And if the human didn’t know what random variables are, or what marginalization is, they’d have an even tougher time.
One can imagine a prompt that might elicit this implementation, something like “this is a Python function that returns the elements of set(self.components) that do not appear in self.observed or self.marginalized”, but then we are not really writing docstrings anymore, we are writing programs in a novel language with a stochastic compiler.
This should be fixable with larger context windows, I think. If the prompt for a method could include the whole class definition aside from that method, Codex could at least in principle use other class methods and variables in sensible ways. But this will have to wait on the practical realization of O(n) or O(nlogn) context window scaling.
I created a class initializing the attributes you mentioned, and when adding your docstring to your function signature it gave me exactly the answer you were looking for. Note that it was all in first try, and that I did not think at all about the initialization for components, marginalized or observed—I simply auto-completed.
Ok, but that isn’t what the code actually looks like; the real Distribution.__init__ is much longer. If the idea is that I could rewrite it into a form that fits in Codex’s context window, and then Codex could help me, then to me that’s even less compelling than the “natural language stochastic compiler” approach.
My objection is that usually I want to write code that depends on other code that’s not widely repeated on Github and doesn’t fit in the prompt. Saying “but what if it does fit in the prompt?” isn’t really responsive.
1. if you want a longer init, write a doctring for it
natural language stochastic compiler
I don’t get what you mean here. I’m also not an expert on the Codex’ “Context windows”.
1) in my experience, even if not specified in your prompt, the model still goes or your depency graph (in different files in your repo, not Github) and picks which functions are relevant for the next line 2) if you know which function to. use, then add these function or “API calls” in the docstring;
A “compiler” is anything that translates a program from one representation to another. Usually this translation is from a high-level language (like Java) to a lower-level language (like JVM bytecode), but you can also have e.g. a Python → Javascript compiler that takes in Python code and produces Javascript. A “natural language compiler”, then, is one that takes in ordinary English-or-whatever sentences and emits something executable. I think this is a pretty fair way to talk about Codex: it’s the world’s first natural language compiler that’s any good at all. I call it “stochastic” because its output is not consistent: if you give Codex the same input twice, you won’t necessarily get the same result.
So if I’m writing Python with Codex, let’s say, then whenever I start to write a function I have a choice of implementation languages: Python, or Codex prompt. Codex is helpful when the prompt is easier to write than the Python implementation would have been. The question isn’t just “can I write a prompt that will elicit good-enough Python?”, it’s “is writing the prompt easier, or harder, than writing the Python?” This is why I am not excited by very low-level prompts like “takes the difference between the set s and the set t”; once you understand your problem well enough to write a prompt like that, writing the actual code is only hard if you don’t know Python well. And if you don’t know Python well, using Codex to generate it is a little risky, since then you won’t be able to catch its silly mistakes.
I will write about the context window thing too, since I think it’s close to the heart of our disagreement, but for now I’m out of time.
Thanks for natural language stochastic compiler explanation, makes a lot of sense. I broadly get a sense of what you mean by “context window” since people have been mentioning that quite a lot when talking about GPT-3. As for whether it makes sense to write docstrings for trivial things, I think this is only pointing at the Codex demo examples where people write docstrings and get results, but for most of my use cases, and when it gets really interesting, is when it auto-completes 1) while I’m writing 2) when I’m done writing and it guesses the next line 3) when I start a line by “return ” or “x = ” and wait for his auto-completion. Here, I would have no idea how to formulate it in the docstring, I just generally trust its ability to follow the logic of the code that precedes it (and I find it useful most of the time).
Thinking about it more there’s another, more serious restriction, at least for now: Codex can’t write code that depends on the rest of your codebase. Consider the following lightly-anonymized code from a real-world codebase I contribute to:
I don’t think Codex could write that code from just that function signature and that docstring, because a human couldn’t do it: they wouldn’t know how to find the names of the observed and marginalized random variables, and they wouldn’t know that self.components exists or has to be explicitly converted into a set. And if the human didn’t know what random variables are, or what marginalization is, they’d have an even tougher time.
One can imagine a prompt that might elicit this implementation, something like “this is a Python function that returns the elements of set(self.components) that do not appear in self.observed or self.marginalized”, but then we are not really writing docstrings anymore, we are writing programs in a novel language with a stochastic compiler.
This should be fixable with larger context windows, I think. If the prompt for a method could include the whole class definition aside from that method, Codex could at least in principle use other class methods and variables in sensible ways. But this will have to wait on the practical realization of O(n) or O(nlogn) context window scaling.
I created a class initializing the attributes you mentioned, and when adding your docstring to your function signature it gave me exactly the answer you were looking for. Note that it was all in first try, and that I did not think at all about the initialization for components, marginalized or observed—I simply auto-completed.
Ok, but that isn’t what the code actually looks like; the real Distribution.__init__ is much longer. If the idea is that I could rewrite it into a form that fits in Codex’s context window, and then Codex could help me, then to me that’s even less compelling than the “natural language stochastic compiler” approach.
My objection is that usually I want to write code that depends on other code that’s not widely repeated on Github and doesn’t fit in the prompt. Saying “but what if it does fit in the prompt?” isn’t really responsive.
1. if you want a longer init, write a doctring for it
I don’t get what you mean here. I’m also not an expert on the Codex’ “Context windows”.
1) in my experience, even if not specified in your prompt, the model still goes or your depency graph (in different files in your repo, not Github) and picks which functions are relevant for the next line 2) if you know which function to. use, then add these function or “API calls” in the docstring;
A “compiler” is anything that translates a program from one representation to another. Usually this translation is from a high-level language (like Java) to a lower-level language (like JVM bytecode), but you can also have e.g. a Python → Javascript compiler that takes in Python code and produces Javascript. A “natural language compiler”, then, is one that takes in ordinary English-or-whatever sentences and emits something executable. I think this is a pretty fair way to talk about Codex: it’s the world’s first natural language compiler that’s any good at all. I call it “stochastic” because its output is not consistent: if you give Codex the same input twice, you won’t necessarily get the same result.
So if I’m writing Python with Codex, let’s say, then whenever I start to write a function I have a choice of implementation languages: Python, or Codex prompt. Codex is helpful when the prompt is easier to write than the Python implementation would have been. The question isn’t just “can I write a prompt that will elicit good-enough Python?”, it’s “is writing the prompt easier, or harder, than writing the Python?” This is why I am not excited by very low-level prompts like “takes the difference between the set s and the set t”; once you understand your problem well enough to write a prompt like that, writing the actual code is only hard if you don’t know Python well. And if you don’t know Python well, using Codex to generate it is a little risky, since then you won’t be able to catch its silly mistakes.
I will write about the context window thing too, since I think it’s close to the heart of our disagreement, but for now I’m out of time.
Thanks for natural language stochastic compiler explanation, makes a lot of sense. I broadly get a sense of what you mean by “context window” since people have been mentioning that quite a lot when talking about GPT-3. As for whether it makes sense to write docstrings for trivial things, I think this is only pointing at the Codex demo examples where people write docstrings and get results, but for most of my use cases, and when it gets really interesting, is when it auto-completes 1) while I’m writing 2) when I’m done writing and it guesses the next line 3) when I start a line by “return ” or “x = ” and wait for his auto-completion. Here, I would have no idea how to formulate it in the docstring, I just generally trust its ability to follow the logic of the code that precedes it (and I find it useful most of the time).