Trying to stay focused on things I’ve already said, I guess it would just mean adopting any sort of security posture towards dual use concepts, particularly in regards to the attack of entirely surreptitiously replacing the semantics of an existing set of formalisms to produce a bad outcome, and also de-emphasizing cultural norms favoring coordination to focus more on safety. It’s really just like the lemon market for cars → captured mechanics → captured mechanic certification thing, there just needs to be thinking about how things can escalate. Obviously increased openness could still plausibly be a solution to this in some respects, and increased closedness a detriment. My thinking is just that, at some point AI will have the capacity to drown out all other sources of information, and if your defense for this is ‘I’ve read the sequences”, that’s not sufficient, because the AI has read the sequences too, so you need to think ahead to ‘what could AI, either autonomously or in conjunction with human bad actors, do to directly capture my own epistemic formula, overload them with alternate meaning, then deprecate existing meaning’. And you can actually keep going in paranoia from here, because obviously there are also examples of doing this literal thing that are good, like all of science for example, and therefore not just people who will subvert credulity here but people who will subvert paranoia.
I guess the ultra concise warning would be “please perpetually make sure you understand how scientific epistemics fundamentally versus conditionally differ from dark epistemics so that your heuristics don’t end up having you doing Aztec blood magic in the name of induction”
In another post made since this comment, someone did make specific claims and intermingle them with analysis, and it was pointed out that this can also reduce clarity due to heterogenous background assumptions of different readers. I think the project of rendering language itself unexploitable is probably going to be more complicated than I can usefully contribute to. It might not even be solvable at the level I’m focused on, I might literally be making the same mistake.
What would it mean to decelerate Less Wrong?
Trying to stay focused on things I’ve already said, I guess it would just mean adopting any sort of security posture towards dual use concepts, particularly in regards to the attack of entirely surreptitiously replacing the semantics of an existing set of formalisms to produce a bad outcome, and also de-emphasizing cultural norms favoring coordination to focus more on safety. It’s really just like the lemon market for cars → captured mechanics → captured mechanic certification thing, there just needs to be thinking about how things can escalate. Obviously increased openness could still plausibly be a solution to this in some respects, and increased closedness a detriment. My thinking is just that, at some point AI will have the capacity to drown out all other sources of information, and if your defense for this is ‘I’ve read the sequences”, that’s not sufficient, because the AI has read the sequences too, so you need to think ahead to ‘what could AI, either autonomously or in conjunction with human bad actors, do to directly capture my own epistemic formula, overload them with alternate meaning, then deprecate existing meaning’. And you can actually keep going in paranoia from here, because obviously there are also examples of doing this literal thing that are good, like all of science for example, and therefore not just people who will subvert credulity here but people who will subvert paranoia.
I guess the ultra concise warning would be “please perpetually make sure you understand how scientific epistemics fundamentally versus conditionally differ from dark epistemics so that your heuristics don’t end up having you doing Aztec blood magic in the name of induction”
In another post made since this comment, someone did make specific claims and intermingle them with analysis, and it was pointed out that this can also reduce clarity due to heterogenous background assumptions of different readers. I think the project of rendering language itself unexploitable is probably going to be more complicated than I can usefully contribute to. It might not even be solvable at the level I’m focused on, I might literally be making the same mistake.