When people say that we shouldn’t base our life decisions on probabilities, they mean (to mean) that we should do max min over some credal set of probability distributions. We don’t have the math to get the details or justification right yet, but we will someday and that heuristic will make sense. The “adversary” that justifies the min operation will be something like your mind / memetically optimized arguments trying to fool you (plus average case = worst case against the universal distribution). Possibly the whole thing will have an elegant, fully Bayesian formulation, but it won’t be easy to discover it in that formulation from scratch. The point is, this type of critique cannot be properly banished on the meta-level alone.
In the context of this specific discussion, it’s valid to point out that a certain amount of hedging against unknown unknowns is wise. But it’s still better to build a probabilistic model (squeezing the credal set along some dimension) then not to build a model. Therefore the AI 2027 authors are much less wrong.
When people say that we shouldn’t base our life decisions on probabilities, they mean (to mean) that we should do max min over some credal set of probability distributions. We don’t have the math to get the details or justification right yet, but we will someday and that heuristic will make sense. The “adversary” that justifies the min operation will be something like your mind / memetically optimized arguments trying to fool you (plus average case = worst case against the universal distribution). Possibly the whole thing will have an elegant, fully Bayesian formulation, but it won’t be easy to discover it in that formulation from scratch. The point is, this type of critique cannot be properly banished on the meta-level alone.
In the context of this specific discussion, it’s valid to point out that a certain amount of hedging against unknown unknowns is wise. But it’s still better to build a probabilistic model (squeezing the credal set along some dimension) then not to build a model. Therefore the AI 2027 authors are much less wrong.