The best way to guarantee you’ll know what you did wrong is to isolate a single variable. Start with a process that works. Change exactly one thing. If the new process works better you’ll know exactly why. If the new process fails you’ll know exactly why.
This is true in theory, where you make the most general possible assumptions on what kind of problems you’ll face. Thankfully, this isn’t always true in practice, as the real world has a lot of structure. You can test multiple variables at once when optimizing something.
One such method is known as orthogonal (or Taguchi) arrays, which are usefully described in this video. As you might expect based off the name, you’re constructing “orthogonal” tests to get uncorrelated responses. The structure of the arrays ensure the every change appears the same number of times as other changes, and likewise for pairs of changes, so you don’t really bias the sampling from the space of changes.
Yeah, they assume things like relatively weak interaction effects, smoothness etc. But linearity is very often a good assumption! Linear regression can work shockingly well and shockingly often.
Anyway, orthogonal arrays are cool and you should watch the video. That was the purpose of this comment.
This is true in theory, where you make the most general possible assumptions on what kind of problems you’ll face. Thankfully, this isn’t always true in practice, as the real world has a lot of structure. You can test multiple variables at once when optimizing something.
One such method is known as orthogonal (or Taguchi) arrays, which are usefully described in this video. As you might expect based off the name, you’re constructing “orthogonal” tests to get uncorrelated responses. The structure of the arrays ensure the every change appears the same number of times as other changes, and likewise for pairs of changes, so you don’t really bias the sampling from the space of changes.
Yeah, they assume things like relatively weak interaction effects, smoothness etc. But linearity is very often a good assumption! Linear regression can work shockingly well and shockingly often.
Anyway, orthogonal arrays are cool and you should watch the video. That was the purpose of this comment.