JBlack

Karma: 1,111

JBlack 30 Sep 2022 1:46 UTC
1 point
0 ∶ 0
on: The Path
If they alternated directions, increasing the distance walked each time by a constant factor (e.g. triple), then an exit in either direction would be reached without too much wasted time.

JBlack 30 Sep 2022 0:24 UTC
4 points
in reply to: David Udell’s comment on: David Udell’s Shortform
It’s probably a useful mental technique to consider from both directions, but also consider that choices that appear symmetric at first glance may not actually be symmetric. There are often significant transition costs that may differ in each direction, as well as path dependencies that are not immediately obvious.
As such, I completely disagree with the first paragraph of the post, but agree with the general principle of considering such decisions from both directions and thank you for posting it.

JBlack 27 Sep 2022 1:26 UTC
6 points
3 ∶ 0
on: You are Underestimating The Likelihood That Convergent Instrumental Subgoals Lead to Aligned AGI
Downvoted for clickbait title that makes a claim about every person in the entire world except the author.

JBlack 17 Sep 2022 8:25 UTC
5 points
0 ∶ 0
in reply to: avturchin’s comment on: What’s the longest a sentient observer could survive in the Dark Era?
Well yes, in the sense that the Dark Age is infinitely long and the diagram only shows a finite time span.
The first possibility of Boltzmann brains would be—even in the logarithmic scale—so far off the right side that extending the diagram to fit would mean leaving our solar system to read it. But yes, your statement is correct since in this model there would be still an infinitely long time span beyond that.

JBlack 15 Sep 2022 9:57 UTC
8 points
3 ∶ 0
on: What’s the longest a sentient observer could survive in the Dark Era?
I’m not sure that you can get to 10^60 years even if protons don’t decay. If they do, then you might not have very much more time than whatever the decay timescale is. It could be that free protons decay while bound ones don’t, in which case you might just avoid having any hydrogen or other light elements around.
Assuming you have a galactic mass available to do with as you please and can extract essentially all of its mass-energy, then over 10^60 years you would have a few picowatts available to do all the maintenance that might be required for whatever infrastructure you have. That’s a pretty miserly budget for managing structures at a scale that can convert galactic mass to usable energy! It would have to cover all sensors as well as computation and also any work that might need doing from time to time. I would be very surprised if maintenance costs didn’t scale along with mass, just because the number of particles enormously expands the state space.
Over these sorts of timescales even the most rigidly bound materials diffuse like a gas due to various quantum processes, but maybe there is some clever way to avoid that and still have them able to perform the desired functions when required.
Anything over these sorts of timescales are going to be very speculative, so this is probably about the best “answer” I can give. In reality, we don’t know what happens to matter over 10^30 year timescales even if the cosmological predictions were spot-on.

JBlack 15 Sep 2022 8:30 UTC
2 points
0 ∶ 0
on: Are Speed Superintelligences Feasible for Modern ML Techniques?
“Just like the smartest humans alive only a thousand times faster” is actually presented as a conservative scenario to pump intuition in the right direction. It’s almost certainly achievable by known physics, even if it would be very expensive and difficult for us to achieve directly.
An actual superintelligence will be strictly better than that, because its early iterations will design systems better than we can, and later iterations running on those systems will be able to design systems more effective than we can possibly imagine or even properly comprehend if it were explained to us. They might not be strictly faster, but speed is much easier to extrapolate and communicate than actual superhuman intelligence. People can understand what it means to think much faster in a way that they fundamentally can’t think of actually being smarter.
So in a way, the actual question asked here is irrelevant. A speedup is just an analogy to try to extrapolate to something—anything—that is vastly more capable than our thought processes. The reality would be far more powerful still in ways that we can’t comprehend.

JBlack 14 Sep 2022 3:47 UTC
3 points
0 ∶ 0
on: I’m taking a course on game theory and am faced with this question. What’s the rational decision?
There is no “most rational” solution here. There are too many known unknowns as well as unknown unknowns.
But really, WTF? Neither of those are raises. Option C is to negotiate something that is actually a raise, with a failure on the company providing one meaning that person X goes somewhere else that will actually offer better.

JBlack 14 Sep 2022 2:37 UTC
2 points
0 ∶ 0
on: Find out how utilitarian you are—a mega thread of philosophy polls
It looks like a very specific form of utilitarianism: something like linear state sum hedonic utilitarianism.

JBlack 10 Sep 2022 3:50 UTC
11 points
7 ∶ 2
on: My emotional reaction to the current funding situation
Your work may well be some small part of the most important that humanity has ever conducted in its entire existence. If alignment turns out to matter at all, which seems likely, then the modal outcomes are: incredible advances worth far more than merely trillions of dollars, or total annihilation of everything we care about.
You’ve become part of something very much larger than yourself.
Yes, your prior sacrifices pale in comparison to the resources you’ve been allocated to work on this problem. If anything the difference in scale is not nearly enough. If your work contributes 0.0001% expected payoff in reaching the better future rather than the worse one, then it is worth at least many millions.
The part that makes me more angry is that AI capability work is being funded vastly more, because most of the gains in the medium term are privatized while all of the (immense!) risk is socialized.

JBlack 10 Sep 2022 2:56 UTC
5 points
0 ∶ 0
in reply to: Mientras’s comment on: Should you refrain from having children because of the existential risk posed by artificial intelligence?
It seems likely that for strategic reasons an AGI will not act in a hostile manner until it is essentially certain to permanently win. It also seems likely that any means by which it can permanently win with near-certainty will kill most of the population relatively quickly.
Keep in mind that this should be measured in comparison with the end-of-life scenarios that most people would face otherwise: typically dementia, cancer, chronic lung or cardiovascular disease. It seems unlikely that most of the people alive at the start of an AI doom scenario will suffer much worse than that for much longer.
If it truly is worse than not being alive at all, suicide will be an option in most scenarios.

JBlack 31 Aug 2022 2:29 UTC
2 points
0 ∶ 0
in reply to: Erlja Jkdf.’s comment on: How can I reconcile the two most likely requirements for humanities near-term survival.
Your post—and my comment—are explicitly about necessary requirements for near-term survival. If you want to make another post about indefinite-term existential risks, then we can talk about that.

JBlack 31 Aug 2022 2:24 UTC
2 points
0 ∶ 0
in reply to: LetUsTalk’s comment on: LessWrong’s prediction on apocalypse due to AGI (Aug 2022)
Most of the predicted dates are “soon” on a civilization timescale, not a personal one.
For example, some vague medians around 2030-2070 sorts of ranges for timescale to the first AGI with outliers on both sides, and peak existential risk something like 1-20 year ranges after that. Very few people are predicting high probability of apocalypse in less than a decade.
If that devalues the predictions for you, then I’m not sure what you were expecting. As already stated, the general cloud of predictions seems to have recently moved shorter, rather than continuously longer as you were expecting. It may move longer and/or shorter again in the future as new evidence arises.

JBlack 30 Aug 2022 4:56 UTC
12 points
2 ∶ 0
on: What is the best critique of AI existential risk arguments?
Less than 1%? You need to go to fairly ridiculous extremes to get that level of certainty for even things that we know a lot about. I think that level of certainty for any such question is beyond any rational argument with current evidence.
There are plenty of arguments that would work, if only you held some particular prior belief to that level. A rock-solid belief in a higher power that would prevent it would suffice—for those who believe in such a thing at a 99%+ level. A similarly strong belief that AGI is actually impossible and not just difficult would also suffice.

JBlack 30 Aug 2022 4:43 UTC
2 points
0 ∶ 0
in reply to: Erlja Jkdf.’s comment on: How can I reconcile the two most likely requirements for humanities near-term survival.
No, it is definitely not a strictly necessary requirement for near-term survival. To be “strictly necessary for near-term survival”, such future technologies would have to be guaranteed to kill all of humanity, and soon. That’s ridiculous hyperbole.
There are risks ahead, even existential risks, from other non-AI technologies but not to nearly that extent.

JBlack 30 Aug 2022 1:51 UTC
2 points
0 ∶ 0
on: How can I reconcile the two most likely requirements for humanities near-term survival.
I don’t think either of those are the two most likely.
I see the most likely as related to (2), but nothing to do with the likelihood of AI causing disaster. More likely there will be some other disruption to our society that has nothing to do with AI, but prevents us from making sufficient progress to reach superhuman AGI for the near future. Probably we will recover in the less near future, but that’s out of scope of the question.
Second most likely I see as being some as yet unknown obstacle that makes AGI unexpectedly unlikely with near future technology. The future is, after all, hard to predict. That doesn’t mean that we technologically plateau in general, just that this one problem is much harder than we expect.

JBlack 30 Aug 2022 1:36 UTC
15 points
1 ∶ 0
on: LessWrong’s prediction on apocalypse due to AGI (Aug 2022)
According to a “no fire alarm” model for AI risk, your prediction (every survey shows a later date for doomsday) is exactly what should be expected, right up until doomsday happens and there are no more surveys.
In practice, I think there are some leading indicators and numerous members of the community have shortened their timelines over the past year due to more rapid progress than expected in some respects. I don’t know of anyone who has lengthened their timelines over the same period.

JBlack 29 Aug 2022 4:22 UTC
3 points
1 ∶ 0
in reply to: pseud’s comment on: What’s the Most Impressive Thing That GPT-4 Could Plausibly Do?
In isolation, it’s technically correct.
In the context of being a direct reply to the post, it’s suggesting that “solve alignment” is something that GPT-4 could plausibly do. I certainly disagree with that and voted disagreement accordingly.

JBlack 29 Aug 2022 4:07 UTC
2 points
0 ∶ 0
in reply to: deepthoughtlife’s comment on: Adversarial epistemology
Whether something is hearsay is relative to the proposition in question.
When Charlie testifies that Bob said that he saw Alice at the club, that’s hearsay when trying to establish whether Alice was at the club, or was alive at all, or any other facts about Alice. Charlie is not conveying any direct knowledge about Alice.
It is not hearsay in establishing many facts about Bob at the time of the conversation. E.g. where Bob was at the time of the conversation, whether he was acquainted with Alice, or many other such propositions. It also conveys facts about Charlie. Charlie’s statement conveys direct knowledge of many facts about the conversation that are not dependent upon the veracity of Bob’s statements, and are therefore not hearsay in relation to them.

JBlack 29 Aug 2022 3:37 UTC
2 points
0 ∶ 0
in reply to: jchan’s comment on: Adversarial epistemology
This calculation just used the fact that Y would have been able to give stronger testimony than X, and that lawyers have incentives to present a strong case for their client where possible. In this scenario, the fact that Y was not called is evidence that Y’s testimony would have weakened the case for Z.
The actual objection against hearsay has nothing to do with this calculation at all, as I mentioned in my comment.
You can apply it in ordinary conversation too (to the extent that you apply Bayesian updates in ordinary conversation at all). It’s just that the update is stronger when the equivalent of E|XYZ is more unlikely, and in ordinary conversation it may not be very unlikely resulting in a weaker update.

JBlack 26 Aug 2022 4:12 UTC
5 points
0 ∶ 0
in reply to: Ethan Perez’s comment on: A Test for Language Model Consciousness
Would you update against other humans being conscious at all, if other humans told you they weren’t conscious?
No, I wouldn’t. I’ve read the works of a number of people who do write that they aren’t conscious (usually also with the claim that nobody else is either). I also have plenty of experience of people being simply mistaken about their mental states in other ways.
My model is that humans who say anything are almost certainly conscious, though their state of consciousness may be different from usual in some cases (asleep, hallucinating, drugged, blackout drunk, etc).
It is pretty trivial to write a program that prints “I am conscious” with essentially no possibility that this is true in any meaningful sense, so I don’t have the same expectation of computer programs. I expect that sufficiently complex programs can be conscious, and we should be cautious in claiming that any given program is not, but the statement by itself is meaningless.