I agree. I was making a more narrow claim that post quality is one of the things that correlate pretty well with karma.
Andrii Vasylenko
Karma: 88
the way to win is to do lots of uncorrelated research bets that are individually unlikely to succeed but also do no harm if they fail!
I disagree that alignment will probably get solved by someone pursuing a direction that seems very unlikely even to them. It seems to me that the right way to do things is to figure out what the hard parts of alignment are and then to try to solve them.
that’s the way surprising and novel scientific inventions have always happened in the past.
I think that’s because those fields had reasonably good feedback loops, and so the strategy of “try a bunch of things and see if any of them work” is generally viable. As a counterexample, Einstein’s methodology was on the opposite end of the spectrum.
That seems false to me. If we had a textbook from the future on AI alignment, I think we would certainly be able to build an aligned ASI.
I think that, if an alignment proposal appears to you to have a 40% chance of working (let alone 1%), then in reality, it very likely won’t work.
In my experience, karma correlates rather well with post quality (after accounting for how much time it’s been on the site).
I agree that would happen if a treaty is signed. But I don’t see how that affects your point in the top-level post.
If you are worried about misaligned ASI, you are likely worried about it being developed in the US or China (and thus it does not matter much if other countries are attempting to build ASI).
Are you saying that you don’t think other countries will be able to build ASI even if they have more compute?
So, in practice you just allow them to benefit from AI without really advancing ASI development, and instead potentially causing a slowdown.
As I understand it, you are saying that other countries will have more will to sign a treaty than the US or China. That doesn’t seem like an obvious inference.
If you largely believe that no non-US or non-Chinese company/state will build AGI, then the threat of misaligned ASI is not really an issue
I don’t see how that follows. I don’t think any nation or company currently has the knowledge to build an aligned ASI.
This happens with concepts too.
a python program that always prints “I am conscious”
I think it would be more effective to consider the mental state of whoever wrote that program, not the program itself.
I think that’s mainly because impact is really hard to measure, at least in x-risk reduction.
I think this is relevant here.
I think that’s hard because a) we don’t really know what those “dangerous capability advancements” are, and b) additional complexity is costly towards adoption.
I focus on a scenario where an international agency enforces drastic limits to AI development for two years, starting at the beginning of 2028.
I expect that two years won’t be enough time to solve the alignment problem, especially if it begins over a year into the future (since open-source algorithmic progress would have advanced).
Uh, is this supposed to be labelled “Online” in the sidebar?
I think the main bottleneck to getting a good future is figuring out how to align a superintelligence at all, so I think it is pointless to run an increased risk of misaligned ASI to give some nations a greater fraction of the future lightcone.
But when you ask them why they care about AI safety they don’t provide a particularly coherent answer. So I get more specific: “Why should we think AI is an existential risk?” Again, incoherent answer.
Related: My hobby: running deranged surveys
I don’t think there’s a more effective way to get true object-level beliefs than to go look out there with fresh eyes and figure out what’s actually happening.
Some Fermi estimates:
On the lower end, 4 words/second times 16 bits/word times 4x10^8 seconds adds up to to ~3x10^10.
On the higher end, assuming 10^7 bits/second (about the retina-to-brain bandwidth), it adds up to ~4x10^15.
However, I think the brain is much less data- and compute-efficient than an optimal AGI algorithm would be. So I don’t think it is a good predictor of how much data future AI algorithms will require.