This is what I think he means:
The object-level facts are not written by or comprehensible to humans, no. What’s comprehensible is the algorithm the AI agent uses to form beliefs and make decisions based on those beliefs. Yudkowsky often compares gradient descent optimizing a model to evolution optimizing brains, so he seems to think that understanding the outer optimization algorithm is separate from understanding the inner algorithms of the neural network’s “mind”.
I think what he imagines as a non-inscrutable AI design is something vaguely like “This module takes in sense data and uses it to generate beliefs about the world which are represented as X and updated with algorithm Y, and algorithm Z generates actions, and they’re graded with a utility function represented as W, and we can prove theorems and do experiments with all these things in order to make confident claims about what the whole system will do.”(The true design would be way more complicated, but still comprehensible.)
I agree that the vocals have gotten a lot better. They’re not free of distortion, but it’s almost imperceptible on some songs, especially without headphones.
The biggest tell for me that these songs are AI is the generic and cringey lyrics, like what you’d get if you asked ChatGPT to write them without much prompting. They often have the name of the genre in the song. Plus the way they’re performed doesn’t always fit with the meaning. You can provide your own lyrics, though, so it’s probably easy to get your AI songs to fly under the radar if you’re a good writer.
Also, while some of the songs on that page sound novel to me, they’re usually more conventional than the prompt suggests. Like, tell me what part of the last song I linked to is afropiano.