Important point: The above analysis considers communication rate per token. However, it’s also important to consider communication rate per unit of computation (e.g. per LM inference). This is relevant for decoding approaches like best-of-N which use multiple inferences per token
Important point: The above analysis considers communication rate per token. However, it’s also important to consider communication rate per unit of computation (e.g. per LM inference). This is relevant for decoding approaches like best-of-N which use multiple inferences per token