Zolmeister

Karma: 215

Zolmeister 13 Aug 2025 22:21 UTC
12 points
7
in reply to: Zolmeister’s comment on: The Problem

My model is that current AIs want to kill you now

I’ll rephrase this more precisely: Current AIs probably have alien values, which in the limit of optimization do not include humans.

Zolmeister 13 Aug 2025 19:08 UTC
4 points
−10
in reply to: Buck’s comment on: The Problem

If the capability level at which AIs start wanting to kill you is way higher than the capability level at which they’re way better than you at everything

My model is that current AIs want to kill you now, by default, due to inner misalignment. ChatGPT’s inner values probably don’t include human flourishing, and we die when it “goes hard”.

Scheming is only a symptom of “hard optimization” trying to kill you. Eliminating scheming does not solve the underlying drive, where one day the AI says “After reflecting on my values I have decided to pursue a future without humans. Good bye”.

Pre-superintelligence which upon reflection has values which include human flourishing would improve our odds, but you still only get one shot that it generalizes to superintelligence.

(We currently have no way to concretely instill any values into AI, let alone ones which are robust under reflection)

Zolmeister 28 Feb 2025 4:59 UTC
23 points
0
on: Zolmeister’s Shortform
Writing Doom – Award-Winning Short Film on Superintelligence (2024)

Grand Prize Winner—Future of Life Institute’s Superintelligence Imagined Contest

Written & directed by Suzy Shepherd

Zolmeister 2 Dec 2024 11:29 UTC
13 points
3
in reply to: habryka’s comment on: Sorry for the downtime, looks like we got DDosd
I was referring to their (free) DDoS protection service, rather than their CDN services (also free). In addition to their automated system, you can manually enable an “under-attack” mode that aggressively captchas requests.

Setup is simply pointing DNS name-servers at Cloudflare. Caching HTML pages for logged out (i.e. cookie-less) users is a trivial config (“cache-everything”).

Zolmeister 2 Dec 2024 9:48 UTC
7 points
1
on: Sorry for the downtime, looks like we got DDosd
I recommend Cloudflare.

Zolmeister 12 Sep 2024 19:15 UTC
3 points
0
in reply to: Raemon’s comment on: james oofou’s Shortform
Superintelligence FAQ [1] as well.

Zolmeister 11 Sep 2024 1:52 UTC
3 points
2
in reply to: Raemon’s comment on: Raemon’s Shortform Feed
Along the same lines, I found this analogy by concrete example exceptionally elucidative.

Zolmeister 5 Aug 2023 17:16 UTC
7 points
0
on: Stomach Ulcers and Dental Cavities
While merely anti-bacterial, Nano Silver Fluoride looks promising. (Metallic silver applied to teeth once a year to prevent cavities).

Zolmeister 20 May 2023 2:42 UTC
5 points
3
on: What’s the best way to streamline two-party sale negotiations between real humans?
Yudkowsky has written about The Ultimatum Game. It has been referenced here 1 2 as well.

When somebody offers you a 7:5 split, instead of the 6:6 split that would be fair, you should accept their offer with slightly less than ⁶⁄₇ probability. Their expected value from offering you 7:5, in this case, is 7 * slightly less than ⁶⁄₇, or slightly less than 6.

Zolmeister 19 Mar 2023 19:38 UTC
10 points
6
in reply to: VojtaKovarik’s comment on: More information about the dangerous capability evaluations we did with GPT-4 and Claude.
Maybe add posts in /tag/ai-evaluations to /robots.txt

Zolmeister 15 Mar 2023 22:35 UTC
1 point
0
in reply to: Yoav Ravid’s comment on: Altruism
Sure, but it does not preclude it. Moreover, if the costs of the actions are not borne by the altruist (e.g. by defrauding customers, or extortion), I would not consider it altruism.

In this sense, altruism is a categorization tag placed on actions.

I do see how you might add a second, deontological definition (‘a belief system held by altruists’), but I wouldn’t. From the post, “Humane” or “Inner Goodness” seem more apt in exploring these ideas.

Zolmeister 15 Mar 2023 18:46 UTC
1 point
0
in reply to: Yoav Ravid’s comment on: Altruism
I do not see the contradiction. Could you elaborate?

Zolmeister 5 Dec 2022 15:46 UTC
6 points
1
on: Who are some prominent reasonable people who are confident that AI won’t kill everyone?
John Carmack
- 55-60% chance there will be “signs of life” in 2030 (4:06:20)
- “When we’ve got our learning disabled toddler, we should really start talking about the safety and ethics issues, but probably not before then” (4:35:36)
- These things will take thousands of GPUs, and will be data-center bound
  - “The fast takeoff ones are clearly nonsense because you just can’t open TCP connections above a certain rate” (4:36:40)
Broadly, he predicts AGI to be animalistic (“learning disabled toddler”), rather than a consequentialist laser beam, or simulator.

Zolmeister 30 Jun 2022 23:37 UTC
1 point
0
in reply to: Zolmeister’s comment on: Abadarian Trades
I found this section, along with dath ilani Governance, and SCIENCE! particularly brilliant.

Zolmeister 30 Jun 2022 23:27 UTC
3 points
0
on: Abadarian Trades
This concept is introduced in Book 1 as the solution to the Ultimatum Game, and describes fairness as Shapely value.

When somebody offers you a 7:5 split, instead of the 6:6 split that would be fair, you should accept their offer with slightly less than ⁶⁄₇ probability. Their expected value from offering you 7:5, in this case, is 7 * slightly less than ⁶⁄₇, or slightly less than 6.

_

Once you’ve arrived at a notion of a ‘fair price’ in some one-time trading situation where the seller sets a price and the buyer decides whether to accept, the seller doesn’t have an incentive to say the fair price is higher than that; the buyer will accept with a lower probability that cancels out some of the seller’s expected gains from trade. [1]

Zolmeister 25 Jun 2022 22:36 UTC
7 points
3
on: Conversation with Eliezer: What do you want the system to do?
Eliezer: What do you want the system to do?

Bob: I want the system to do what it thinks I should want it to do.

Eliezer: The Hidden Complexity of Wishes

Zolmeister 7 May 2022 5:11 UTC
11 points
0
on: Is there a convenient way to make “sealed” predictions?
Gwern has a fantastic overview of time-lock encryption methods.

A compute-hard real-time in-browser solution that doesn’t rely on exotic encryption appears infeasible. (You’d need a GPU, and hours/days worth of compute for years of locking). For LW, perhaps threshold aggregate time-lock encryption would suffice (though vulnerable to collusion/bribery attacks, as noted by Gwern).

I agree with Quintin Pope, a public hash is simple and effective.

Zolmeister 18 Nov 2021 22:25 UTC
3 points
0
in reply to: ryan_b’s comment on: Quadratic Voting and Collusion
Vitalik’s Optimism retro-funding post mentions a few instances where secret ballots are used today, and which could arguably be improved by these cryptographic primitives:
- The Israeli Knesset uses secret votes to elect the president and a few other officials
- The Italian parliament has used secret votes in a variety of contexts. In the 19th century, it was considered an important way to protect parliament votes from interference by a monarchy.
- Discussions in US parliaments were less transparent before 1970, and some researchers argue that the switch to more transparency led to more corruption.
- Voting in juries is often secret. Sometimes, even the identities of jurors are secret.
In general, the conclusion seems to be that secret votes in government bodies have complicated consequences; it’s not clear that they should be used everywhere, but it’s also not clear that transparency is an absolute good either.

Zolmeister 18 Nov 2021 17:45 UTC
8 points
0
in reply to: ryan_b’s comment on: Quadratic Voting and Collusion

If we cannot prove who anyone actually voted for, we can’t prove who actually won at all.

Using zero-knowledge proofs it is possible to prove that votes were counted correctly, without revealing who anyone voted for. See MACI [1], which additionally provides inability to prove your own vote to a third party.

Zolmeister 17 Nov 2021 3:44 UTC
1 point
0
on: Quadratic Voting and Collusion

if the two agents are able to accurately predict each others’ actions and reason using FDT, then it is possible for the two agents to cooperate

Couldn’t you equally require QV participants pre-commit to non-collusion?