EDIT 3/5/24: In the comments for Counting arguments provide no evidence for AI doom, Evan Hubinger agreed that one cannot validly make counting arguments over functions. However, he also claimed that his counting arguments “always” have been counting parameterizations, and/or actually having to do with the Solomonoff prior over bitstrings.
As one of Evan’s co-authors on the mesa-optimization paper from 2019 I can confirm this. I don’t recall ever thinking seriously about a counting argument over functions.
As one of Evan’s co-authors on the mesa-optimization paper from 2019 I can confirm this. I don’t recall ever thinking seriously about a counting argument over functions.