Just this guy, you know?
Dagon
Karma: 11,766
First rule of probability and decision theory: no infinities! If you want to postulate very large numbers, go ahead, but be prepared to deal with very tiny probabilities.
Pascal’s wager is a good example—the chance that the wager actually pays off based on this decision is infinitesimal (not zero, but small enough that I can’t really calculate with it), which makes it irrelevant how valuable it is. This gets even easier with the multitude of contradictory wagers on offer—“infinite value” from many different choices, only one of which can you take. Mostly, take the one(s) with lower value but actually believable conditional probability.
There will be no paperclips if planet and maximizer are destroyed.
There might be—some paperclips could survive a comet. More importantly, one paperclip’s worth of resources won’t change the chance of a comet collision by any measurable amount, so the choice is either “completely waste that energy” or “make a paperclip that might survive”.
Only if every other entity’s anti-paperclip stance is known and unchangeable, and if resource->impact is purely linear, can it be assumed that 100% to self-preservation (oh, wait, also to accumulation of power, there’s another balance to be found) is optimal. Neither of these are true, but especially the problem of declining marginal impact.
For any given energy unit decision you could make, there will be a different distribution of future worlds and their number of paperclips. Building one paperclip could EASILY increase the median and average number of future paperclips more than investing one paperclip’s worth of power into comet diversion.
It gets more difficult when coordinating with unaligned agents—one has to decide whether to nudge them toward valuing paperclips, convincing/forcing them to give you more power, or (since they’re unlikely to care as much as you about the glorious clippy future) point THEM at the comet problem so they reduce that risk AND don’t interfere with your paperclips.
If you haven’t played it (it was popular a few years ago in these circles, but I haven’t seen it mentioned recently), it’s worth a run through https://www.decisionproblem.com/paperclips/ . It’s mostly humorous, but based on some very good thinking.
Adversarial action makes this at least an order of magnitude worse. If Carla has to include the chance that Bob could be lying (for attention, for humor, or just pathologically) about his experiences or his history of drug use or hallucinations, she makes an even smaller update. This is especially difficult in group-membership or banning discussions, because LOTS of people lie (or just focus incorrectly on different evidence) for status and for irrelevant-beef reasons.
I don’t think there is a solution, other than to acknowledge that such decisions will always be a balance of false-positive and false-negative, and it’s REQUIRED to ban some innocent (-ish) people in order to protect against the likely-but-not-provably-harmful.
“legal reasons” is pretty vague. With billions of dollars at stake, it seems like public statements can be used against them more than it helps them, should things come down to lawsuits. It’s also the case that board members are people, and want to maintain their ability to work and have influence in future endeavors, so want to be seen as systemic cooperators.
It’s easy to frame Newcomb’s problem such that there’s no opportunity to precommit (and, CDT generally doesn’t see any REASON to precommit there).
Trying to figure out how to update. From the downvotes and comments, I’m clearly considered wrong, but I can’t easily find details on how. Is the statement “We have not and never will take away vested equity” a flat-out lie? I’d expected it was relying heavily on the word “vested”, and what they took away was something non-vested.
Is there a simple link to a specific legal description of what assets a non-signer was entitled to, but lost due to declining to sign?
I think most people have a different conception of what “trying to be clear” means than I do. They can talk abut true love or what’s “really” love, but the very few who actually try to get an operational definition across usually start by tabooing “love” when trying to communicate specific beliefs and experiences (though still using it in more casual, romantic, or poetic settings).
I recommend this as a technique when trying to draw out clarity from someone as well. Don’t make them define terms, most people aren’t great at the activity of generalizing and re-specifying that language is built on. Instead, ask them do describe what they mean in this context, by asking for statements that do not use “love”.
Note: this rarely leads to kissing. If that’s your goal, I’d advise to delay semantic exploration for another time.
I haven’t followed closely—from outside, it seems like pretty standard big-growth-tech behavior. One thing to keep in mind is that “vested equity” is pretty inviolable. These are grants that have been fully earned and delivered to the employee, and are theirs forever. It’s the “unvested” or “semi-vested” equity that’s usually in question—these are shares that are conditionally promised to employees, which will vest at some specified time or event—usually some combination of time in good standing and liquidity events (for a non-public company).
It’s quite possible (and VERY common) that employees who leave are offered “accelerated vesting” on some of their granted-but-not-vested shares in exchange for signing agreements and making things easy for the company. I don’t know if that’s what OpenAI is doing, but I’d be shocked if they somehow took away any vested shares from departing employees.
It would be pretty sketchy to consider unvested grants to be part of one’s net worth—certainly banks won’t lend on it. Vested shared are just shares, they’re yours like any other asset.
Most people aren’t confused, because they’re not trying to be clear and rational. They are definitely confusing to people who prefer specificity and operationally useful descriptions of experiences.
It is used to mean a very wide range of positive feelings, and should generally be taken as poetry rather than communication. Another framing is that the ambiguity makes it very useful for signaling affection and value, without being particularly binding in specific promises.
Which is NOT to say it isn’t “real”. Affection, caring, enjoyment, and willingness-to-sacrifice-for someone are all things that many individuals experience for other individuals. The exact thresholds and mix of qualia that gets described as “love” varies widely, but the underlying feelings seem near-universal.
[no actual direct knowledge]
Yeah, all things like this follow a power distribution—a few users use it A TON and both median and average are far smaller than you’d think. Assuming they throttle the biggest users in some way, they’re making some money on paid use. But it’s probably irrelevant to their overall strategy and profitability.
I’d guess the biggest value for them is in showing growth and “willingness to pay” for this, more than the actual money collected.
The other common optimization process people generally refer to is “markets”. In the same sense that “evolution” is what happens when variation and selection combine, “market” is what happens when multiple traders repeatedly choose how to exchange things.
If it’s possible for super-intelligent AI to be non-sentient, wouldn’t it be possible for insects to evolve non-sentient intelligence as well? I guess I didn’t assume “non-sentient” in the definition of “unaligned”.
It’s always seemed strange to me what preferences people have for things well outside their own individual experiences, or at least outside their sympathized experiences of beings they consider similar to themselves.
Why would one particularly prefer unthinking terrestrial biology (moss, bugs, etc.) over actual thinking being(s) like a super-AI? It’s not like bacteria are any more aligned than this hypothetical destroyer.
Public discourse norms, especially in the twitter age, are funny. Hanania has a lot of options, none of which really change his status much.
he can ignore you. There’s enough volume that he doesn’t respond to most comments, so this isn’t him dodging a particularly harsh criticism, it’s just not worth his notice.
he can answer cheaply, by rephrasing (or just reposting) previous texts.
he can answer a different question that’s vaguely related.
Almost nothing is real. Even non-ai text or images are often false or misleading. Almost everything is real, in that a human decided to cause it to happen.
I lean heavily to #5. On very narrow topics, measurement and rationality tends to win, and there’s not actually much debate about it. There really aren’t many ideological rocket launches, for instance. But once you start talking about human behavior or any other complex or game-theoretic situations, it stops being legible and tractable for logic to really do much good without a whole lot of modeling choices being involved.
This combines well with Robin Hanson’s theory that signaling explains most public choice oddities. Showing that we care about bike riders doesn’t require that we actually make the city work for bike riders.
That’s … a lot. And a lot of extremely diffuse inferred judgments from statistical or imagined others, rather than specific people (including yourself) who will care about it.
You’re absolutely right that the signaling game has enough truth behind it that it’s generally best, for most people, to just follow. You’re forgetting that you’re not most people, and it’s quite possible that structured undergrad education is way harder for you than for most people with your IQ. Some will judge you for that. Most won’t care.
On a practical side, it is a hurdle. I dropped out of college long ago, it slowed my career by a fair bit for the first decade, but stopped mattering at all once I had significant job successes. However, I’ve just changed jobs, and even with significant time with Principal and Distinguished Engineer titles, I had to get my prospective new boss to change their req to say “or equivalent experience” on the job description or HR wouldn’t let me interview. And my background check took extra time because they had trouble verifying my HIGH SCHOOL transcript.
So, I can’t answer for you. The costs (in terms of emotional work) are likely higher than you say, but you may be enough of a different person that they’re not all that high. The benefits are real, but most of them are intangible, and far more about how you think of yourself than how others see you.
I should have specified WHO they want to cooperate with in the future. People with lots of money to spend—businesses. Silence is far preferable to badmouthing former coworkers.