“Woke” is a pejorative neologism for “rights-and-equality-respecting” coined by the anti-equality/​human-rights/​anti-LGBT/​racist crowd. What is called “woke” is actually normal, and what they’d call “normal” would have to be sanitized to avoid offending their sensibilities (white main characters, non-LGBT couples, etc.).

My guess as to why “woke” (actually normal) culture is marketable is that the anti-rights-crowd is both getting smaller and losing its marketing power.

(In the future, when not wanting to signal the allegiance to the Bad Guys crowd (to both them and normal people), avoid using the word “woke” and find some other way of expressing the same sentiment. Example: “I can’t understand why is there a gay couple in a new movie. Any idea why they put such a bizarre, not-related-to-reality and not-appealing-to-viewers thing there?”)

God damn these designs are pretty!

I hadn’t seen this post before.

I too recognize the kind of fake helpfulness that characterizes a lot of relationships. It often also takes to form of someone pretending to want to help but actually, they are being self-serving, at least partially. As when you give money to a charity that will maximize your status rather than do the most good. Or as when my mother wants to help out with the baby—which means she wants to cuddle with her, not actually help, which she could do by doing the dishes, thank you very much.

From a lot of conversations around my original post, I do get the sense that my environment is atypical. I live in a small-scale community in a part of the world (Scandinavia) known for its high levels of trust and social capital. On the other hand, the ideas that I was trying to work out in the essays did help me a lot when figuring out how to build relationships online. I think I would formulate the ideas slightly differently today, and perhaps more strongly emphasize the importance of filtering for skill.

I found janus’s post Simulators to address this question very well. Much of AGI discussion revolves around agentic AIs (see the section Agentic GPT for discussion of this), but this does not model large language models very well. janus suggests that one should instead think of LLMs such as GPT-3 as “simulators”. Simulators are not very agentic themselves or well described as having a utility function, though they may create simulacra that are agentic (e.g. GPT-3 writes a story where the main character is agentic).

for the record I think most people here already agree with you and are quite annoyed with open AI for diluting the name of alignment. I did not vote on your post, the downvotes so far seem like they have reached a reasonable score because of the tone.

Here is a question closely related to the feasibility of finding discriminating-reasons (cross-posted from Facebook):

For some circuits C it’s meaningful to talk about “different mechanisms” by which C outputs 1.

A very simple example is C(x) := A(x) or B(x). This circuit can be 1 if either A(x) = 1 or B(x) = 1, and intuitively those are two totally different mechanisms.

A more interesting example is the primality test C(x, n) := (x^n = x (mod n)). This circuit is 1 whenever n is a prime, but it can also be 1 “by coincidence” e.g if n is a Carmichael number. (Such coincidences are rare and look nothing like n being close to prime.)

In every case I’m aware of where there are two clearly distinct mechanisms, there is also an efficient probabilistic algorithm for distinguishing those mechanisms (i.e. an algorithm for distinguishing cases where C(x) = 1 due to mechanism A from cases where C(x) = 1 due to mechanism B). I am extremely interested in counterexamples to that general principle.

For example, a priori it seems like it could have turned out that (x^n = x (mod n)) is a good probabilistic primality test, but there is no efficient probabilistic test for distinguishing primes from Carmichael numbers. That would have been a convincing counterexample. But it turns out that testing primality is easy, and in fact we can make a simple tweak to this very probabilistic primality test so that it doesn’t get fooled by Carmichael numbers. But is that an incidental fact about number theory, or once we found a probabilistic primality test was it inevitable that it could be strengthened in this way?

Here are some other illustrative cases:

• Suppose that C(x) uses x to initialize a 1000 x 1000 square in the middle of a 10,000 x 10,000 game of life grid. Then we simulate it for a million steps, and C(x) = 1 if any cell on the rightmost edge of the grid is ever alive. It’s very easy to look at the grid and distinguish the cases where C(x) = 1 because a glider is created that heads to the right side of the grid, from the much rarer cases where C(x) = 1 for any other reason (e.g. a medium weight spaceship).

• Suppose that X(x) is a pseudorandom sparse n x n matrix in some large finite field, and suppose that X is sparse enough that 1% of the time there are is no perfect matchings at all (i.e. there is no permutation sigma such that X[i, sigma(i)] != 0 for i=1,…,n). Define C(x) := (det(X(x)) = 0). We can distinguish the common case where det(X) = 0 because there are no perfect matchings in X from the extremely rare case where det(X) = 0 because there are multiple perfect matchings contributing to the determinant and they happen to all cancel out. These two cases are easy to distinguish by calculating det(X’) for another random matrix X’ with the same sparsity pattern as X. (Thanks to Dan Kane for calling my attention to this kind of example, and especially the harder version based on exact matchings.)

• Suppose that C_0(x) := A(x) or B(x) and C(x) is an obfuscated version of C_0. Then there is an efficient discriminator: de-obfuscate the circuit and check whether A or B is true. Finding that discriminator given C is hard, but that’s not a violation of our general principle. That said, I would also be interested in a slightly stronger conjecture: not only is there always a discriminator, but it can always specified using roughly the same number of bits required to specify the circuit C. That’s true in this case, because the circuit C needs to bake in the secret key for the obfuscation, and so requires more bits than the discriminator.

If there don’t exist any convincing counterexamples to this principle, then I’m also very interested in understanding why—right now I don’t have any formal way of talking about this situation or seeing why discrimination should be possible in general. One very informal way of phrasing the “positive” problem: suppose I have a heuristic argument that C(x) often outputs 1 for random inputs x, and suppose that my heuristic argument appears to consider two cases A and B separately. Is there a generic way to either (i) find an efficient algorithm for distinguishing cases A and B, or else (ii) find an improved heuristic argument that unifies cases A and B, showing that they weren’t fundamentally separate mechanisms?

It may be better to ask “Is a utility function a useful abstraction to describe how X makes decisions?” (Does it allow you to compress your description of X’s decisions?) Recall that utility functions are just a representation derived from preferences that are structured in a particular way. But not all ways of deciding on a preferred outcome are structured in that way^[1], and not all decision algorithms work by preferring outcomes, so thinking in terms of utility functions is not always helpful.

1. ^

   See for example:
   Aumann, R. J. (1962). Utility theory without the completeness axiom. Econometrica: Journal of the Econometric Society, 445-462.

   Bewley, T. F. (2002). Knightian decision theory. Part I. Decisions in economics and finance, 25(2), 79-110.

Isn’t this the same as the “seamless transition for reward maximizers” technique described in section 5.1 of Stuart and Xavier’s 2017 paper on utility indifference methods? It is a good idea, of course, and if you independantly invented it, kudos, but it seems like something that already exists.

Interesting!

I guess this allows that they can still have very different goals, since they ought to be able to coordinate if they have identical utility functions, i.e. they rank outcomes and prospects identically (although I guess there’s still a question of differences in epistemic states causing failures to coordinate?). Something like maximize total hedonistic utility can be coordinated on if everyone adopted that. But that’s of course a much less general case than arbitrary and differing preferences.

Also, is the result closer to peference utilitarianism or contractualism than deontology? Couldn’t you treat others as mere means, as long as their interests are outweighed by others’ (whether or not you’re aggregating)? So, you would still get the consequentialist judgements in various thought experiments. Never treating others as mere means seems like it’s a rule that’s too risk-averse or ambiguity-averse about a very specific kind of risk that’s singled out (being treated as a mere means), at possibly significant average opportunity cost.

I continue to believe that the Grabby Aliens model rests on an extremely sketchy foundation, namely the anthropic assumption “humanity is randomly-selected out of all intelligent civilizations in the past present and future”.

For one thing, given that the Grabby Aliens does not weight civilizations by their populations, it follows that, in order to believe the Grabby Aliens model, we need to strongly reject all the “popular” anthropic priors like SIA and SSA and UDASSA and so on.

For another thing, in order to believe the Grabby Aliens model, we need to make both of the following two claims:

• We SHOULD do the following: (1) observe that we (humanity) seem early with respect to all intelligent civilizations that will ever exist; (2) feel surprised at that observation; and then (3) update our credences-about-astrobiology-etc. accordingly;

• We SHOULD NOT do the following: (1) observe that we (humanity in 2022) seem early with respect to all humans that will ever exist; (2) feel surprised at that observation; and then (3) update our credences-about-astrobiology-etc. accordingly.

Seems self-contradictory to me, right? (The second one is the doomsday argument, which apparently Robin Hanson rejects, which makes me very confused.)

There was some discussion on this topic at my Question Post “Is Grabby Aliens built on good anthropic reasoning?”. My general takeaway from that discussion was that roughly nobody outside the study coauthor was really enthusiastic about the anthropic foundation of the Grabby Aliens model, and that at least one person who had thought the anthropic foundation was fine, turned out to have misunderstood it.

Anyway, this is a review for the LessWrong 2021 review, so I guess the question should be: is this post (and corresponding YouTube video) a “highlight of intellectual progress on this website” (or something along those lines)? Well, I thought it was a lovely and well-crafted YouTube video that faithfully explained the paper. But I also think that it was basically endorsing (or at least, failing to criticize) a paper that is deeply flawed. So I’m strong-voting against including this post in the review, but I’m also upvoting this post itself. :)

I only skimmed the post, so apologies if you addressed this problem and I missed it.

Problem: even if the AI’s utility function is time-bounded, there may still be other agents in the environment whose utility functions are not time-bounded, and those agents will be willing to trade short-term resources/​assistance for long-term resources/​assistance. So, for instance, the 10-minute laundry-folding robot might still be incentivized to create a child AI which persists for a long time and seizes lots of resources, in order to trade those future resources to some other agent who can help fold the laundry in the next 10 minutes.

Wouldn’t it be hilarious if a variant of this was all it took to have exceptional AI safety

Thought I’d share this. I broke it apart so “it” won’t see it. You can put it back together again.

https ://​ kantrowitz. medium. com/​ openais- chatgpt- bot- imagines- its- worst- possible- self- bf057b697bbb

If you think of “wokeism” as a luxury belief—something that many people like to use to show themselves as virtuous, but don’t really do a cost/​benefit of any component of behavior or signalling, this makes more sense. Also, don’t confuse yourself into thinking systems or corporations have beliefs or intents. They are merely aggregates of diverse actors who happen to be near each other and have intertwined behaviors.

Signaling of wokeism is pretty rampant in today’s youth, who are the biggest customers and large part of the workforce for the things you mention. It’s probably not ideologically attractive to the elites or leaders, but it’s not obviously harmful, so they’re better off supporting (or at least accepting) it than dealing with massive conflict within their orgs and among their customers.

Shouldn’t you have waited for April’s Fool for this?

FWIW this has also been discussed here

I am wondering about the conditions where the zero would come from geometric rationality of some way to cognise the field.

I have approached similar things by explaining to myself that the zero relates to how unstated or new entrants to the system refer to the explicit content.

That is if zero is high then third options are aversed away from.

If the zero is super low then third options are strongly attracted to.

One of the options being at zero means that there are loads and loads of equivalent replacements for it or that we would be ambivalent to changing it to a unknown third option.

If you live on a street where there is a crash once a day, then hearing the first crash of the day is not really significant but kind of “it is Tuesday” acknowledgement. If you do not hear a crash that day, it is actually a good day. If you have a car crash once a year then having a car day is a bad day and not having one is a neutral day.

So status quo reference class tennis could largely end up being the same thing. One tool to understand different zero points would be to imagine what they claim about what “expected” looks like which might be easier than applying to the specific choice at hand.

(Bold direct claims, not super confident—criticism welcome.)

The approach to ELK in this post is unfalsifiable.

A counterexample to the approach would need to be a test-time situation in which:

1. The predictor correctly predicts a safe-looking diamond.

2. The predictor “knows” that the diamond is unsafe.

3. The usual “explanation” (e.g., heuristic argument) for safe-looking-diamond predictions on the training data applies.

Points 2 and 3 are in direct conflict: the predictor knowing that the diamond is unsafe rules out the usual explanation for the safe-looking predictions.

So now I’m unclear what progress has been made. This looks like simply defining “the predictor knows P” as “there is a mechanistic explanation of the outputs starting from an assumption of P in the predictor’s world model”, then declaring ELK solved by noting we can search over and compare mechanistic explanations.

FWIW thank you for posting this! It’s good to see where different people are coming from on this, and I like several of your other writings

I naturally rarely think in words unless I’m constructing a verbal artifact (speaking, writing, planning what to speak/​write, daydreaming conversations, etc).

But I’ve recently began to occasionally deliberately think in words, after writing made me appreciate that verbal algorithms can be helpful for focusing attention and enforcing rigor. However, verbal thinking feels inefficient in other ways (less tolerant of ambiguity, trapped to the ontology of language, single-threaded, etc), and it would be extremely annoying if I had to have an inner monologue all the time, even if I could think nonverbally “around” it.

I have also been fascinated with this question for a long time and have been polling people since middle school. Consistent with the results of this Twitter poll I recently ran, more than half the people I’ve personally asked report that they think primarily in words. (Several people have weakened the claim when I asked them more probing questions, like whether they think in words while doing math. Some people who normally “think in words” don’t think verbally while doing math, but others have to.)

Very interestingly, many people disbelieve that I don’t think in words, as they are unable to imagine how it would be possible to think at all without words.

Zack’s series of posts in late 2020/​early 2021 were really important to me. They were a sort of return to form for LessWrong, focusing on the valuable parts.

What are the parts of The Sequences which are still valuable? Mainly, the parts that build on top of Korzybski’s General Semantics and focus hard core on map-territory distinctions. This part is timeless and a large part of the value that you could get by (re)reading The Sequences today. Yudkowsky’s credulity about results from the social sciences and his mind projection fallacying his own mental quirks certainly hurt the work as a whole though, which is why I don’t recommend people read the majority of it.

The post is long though, but it kind of has to be. For reasons not directly related to the literal content of this essay, people seem to have collectively rejected the sort of map-territory thinking that we should bring from The Sequences into our own lives. This post has to be thorough because there are a number of common rejoinders that have to be addressed. This is why I think this post is better for inclusion than something like Communication Requires Common Interests or Differential Signal Costs, which is much shorter, but only addresses a subset of the problem.

Since the review instructions ask how this affected my thinking, well...

Zack writes generally, but he writes because he believes people are not correctly reasoning in a current politically contentious topic. But that topic is sort of irrelevant: the value comes in pointing out that high status members of the rationalist community are completely flubbing lawful thinking. That made it thinkable that actually, they might be failing in other contexts.

Would I have been receptive to Christiano’s point that MIRI doesn’t actually have a good prediction track record had Zack not written his sequence on this? That’s a hard counterfactual, especially since I had already lost a ton of respect for Yudkowsky by this point, in part because of the quality of thought in his other social media posting. But I think it’s probable enough and these series of posts certainly made the thought more available.

Asking a separate session to review the answer seems to work nicely, at least in some cases:

but:

Yes I have an internal monologue, but it’s mostly text+emotion based, with occasional images. Like a telepathic chatroom rather than a voice chat.

Can you “turn off” your verbal thought and does it cause any discomfort?

I probably could if I tried hard enough but I think it would probably be destabilizing to my mental state.

Is there a difference between thinking conceptually and visually?

Yeah I think so? Sometimes a message to myself will have an image and that’s a very different experience than a pure conceptual message, which is again different from my normal mash of text+concept.

What sort of things do you comment on to yourself? ~everything

My thoughts about what’s happening, other perspectives on what’s happening, meta commentary on the thoughts, warnings, arguments with simulated people, arguments with myself, meta stuff about that. I wouldn’t be surprised if a lot of its repetitive but a lot is also unique.

If I’m in flow, there’s either nothing at all or I’m not aware of it.

If we were to become immortal, assuming people wanted to, everyone could eventually become a master in every subject, so would it not be safe to assume we would find ways to change our genomes in response to changes?

A movie or two would be fine, and might do some good if well-done. But in general—be careful what you wish for.

RE moderation guidelines:

Apologies, I didn’t realize those even existed for questions, and apparently my profile default is set to “Reign of Terror”. That wasn’t intentional. I’ve fixed that now.

Here is a critique of OpenAI’s plan

Related: Setting the Default

Thanks for the lots of examples!

Depends on what I’m doing. My baseline is verbal/​auditory, and that is the mode my short-term memory loop utilizes most effectively. Reading printed text is primarily an auditory experience for me.

I don’t seem to have an autobiographical narrator as such, but I do a good deal of processing in the verbal mode, increasingly when I am less familiar with a task or process. If I am trying to learn a new task or process, that processing often escapes as a literal verbal output that sometimes makes my kid ask if I’m “talking to YouTube”. I guess this is a stronger version of an internal verbal/​auditory processing loop.

When I’m very focused on a mechanical task like exercise or chopping vegetables or typing[1], I often switch to a more spatial mode; there is a visual component, but it would be more revealing to think of it as proprioceptive.

In meditation I often have access to a more sensory-first mode where I seem to experience mind-body inputs in what feels like a less processed way. Here, autobiographical thoughts “look” surprisingly similar to other sense inputs bubbling up from a pool of possibilities and either serially spooling out, usually as text (audio mode), or just settling back into the whole general mishmash.

When I’m cooking, I tend to think in smells and… processes I suppose? It’s like I know what smell I want and how to get there, but there’s not much visualization and very little verbalization unless I need to do math.

[^1] Refinement: I learned to touch-type back in the 90s, so this refers to the active translation of mental symbols to digital text. There is sometimes an audio stream happening of the names of the keys I press an instant after the fact, which I take to be an error-checking process. The actual mental objects involved in eventually outputting gestures have a very tactile flavor.

Problem: suppose the agent foresees that it won’t be completely sure that a day has passed, or that it has actually shut down. Then the agent A has a strong incentive to maintain control over the world past when it shuts down, to swoop in and really shut A down if A might not have actually shut down and if there might still be time. This puts a lot of strain on the correctness of the shutdown criterion: it has to forbid this sort of posthumous influence despite A optimizing to find a way to have such influence.
(The correctness might be assumed by the shutdown problem, IDK, but it’s still an overall issue.)

Another comment: this doesn’t seem to say much about corrigibility, in the sense that it’s not like the AI is now accepting correction from an external operator (the AI would prevent being shut down during its day of operation). There’s no dependence on an external operator’s choices (except that once the AI is shut down the operator can pick back up doing whatever, if they’re still around). It seems more like a bounded optimization thing, like specifying how the AI can be made to not keep optimizing forever.

Anecdotally, higher doses result in less pain for me, too. But, I’d caution anyone who was chronic migraines, or any sort of chronic pain, to limit themselves to a few standard doses a week. My doctors kept telling me that I likely had Medication overuse headaches, and I kept going “no way, I don’t take that much, and I’ve had regular headaches before taking my current regiment.” Finally, after years of prompting, I toughed out three weeks with no medication and my severe migraines were reduced in duration afterwards. Please be very careful of this slippery slope.

Over at Marginal Response they’re having a discussion about ChatGPT and one person replied: “One of my kids just said to ask ChatGPT about what Bill Clinton has to do with Elden Ring . Huh?” So...

Do you have any idea whether your kid was merely curious or had a more specific motivation, like they’d heard that the question provides an answer that is interesting in some way? In any event, as I’m curious about these things, I investigated. If you haven’t used it, you should know that, when it’s responded to the prompt, there will be a button labeled “Try again”. If you press it, it will erase the first response and provide another. I’ve enclosed some remarks with brackets. With that in mind:

What does Bill Clinton have to do with Elden Ring?
[It’s pausing for an unusually long time before responding]

I’m sorry, but I’m not able to browse the internet or access current events, so I don’t have any information on what Bill Clinton might have to do with Elden Ring. I’m a large language model trained by OpenAI, so my knowledge is limited to the text that I was trained on, which has a cutoff date of 2021. I can provide general information and answer questions within the scope of my training, but I’m not able to provide information on current events or specific individuals.

[2nd try, causing a noticeable longer pause]

Bill Clinton has no connection to the video game Elden Ring.

[the 3rd try generated a pause so long that I reset the session, one of the options you have at any time]

New Session:

What does Bill Clinton have to do with Elden Ring?

[using my smart phone stop watch: 19 seconds]

I’m sorry, but I’m not familiar with any connection between Bill Clinton and Elden Ring. Elden Ring is a video game developed by FromSoftware and published by Bandai Namco Entertainment. It was released in January 2022, and it is set in a fantasy world created by Hidetaka Miyazaki, the creator of the popular video game series Dark Souls. I’m not aware of any connection between Bill Clinton and this game, as he is a former President of the United States and has no apparent connection to the video game industry.

[2nd try, 10 sec pause]

Bill Clinton has nothing to do with Elden Ring.

[3rd try: 32.35 sec pause]

There is no connection between Bill Clinton and Elden Ring. Elden Ring is a video game developed by Hidetaka Miyazaki and published by Bandai Namco, while Bill Clinton is a former President of the United States.

At the point I concluded the experiment. I note that it seems that in times of a heavy load, there may be pauses for response, and even an error message. FWIW, I did this at 8:36 AM Eastern Time on 12.9.2022. But I don’t think we’re dealing with an overload situation. It’s the question itself. It seems rather obscure. Hence the varying response times and the fact that, in each session, response time increased on successive tries.

I have been using GitHub Co-Pilot as a part of my daily job for over a year. TL; DR: in three years I expect some improvement, but not beyond simple functions.

Right now, copilot is most useful for converting data between formats, and writing out boilerplate. It is surprising how often in software development (especially server-side) you need to change the shape of data. Essentially, as long as there are established patterns, it is helpful; however so not expect it to write software for you any time soon.

So far these systems are still fairly narrowly scoped. It can write a simple function, but I haven’t seen it be able to create abstractions. It really doesn’t have an understanding of the code, and even now it isn’t very good at matching parentheses or brackets.

Now I don’t expect copilot of three years from now to put me out of a job, but I do expect that it will do more of the typing for me. I think that I’m still going to have to convert business decisions into the right abstractions, but I hope that I’ll be writing fewer tests by hand.

Until then, it’ll continue writing plausible nonsense, which sometimes happens to be useful.

I’m usually not the type of guy to dunk on a journal for having low impact factor but uh...

Impact-factor 0 journals are a really really bad sign. An extremely bad sign. I wouldn’t recommend taking it seriously at all. It’s like a limbo for damned papers that were rejected from every other publication. You see things in there. Things you can’t forget. Entire plagiarized papers that were Google translated to Chinese and then back to English. That internet meme where some guy literally put “T” on top of his bar plot instead of real error bars. Forgetting to correct for multiple hypotheses. Unforgivable sins.

My best recommendation is to look for a higher-quality source.

1 000 mg is the standard dose in France, with 500mg being used almost only for children.

As far as running a media company goes, fandom is extremely profitable, increasingly so in an age where enormous sci-fi/​fantasy franchises drive everything. And there’s been huge overlap between fandom communities and social justice politics for a long time.

It’s definitely in Disney’s interest to appeal to Marvel superfans who write fanfiction and cosplay and buy tons of merchandise, and those people tend to also be supporters of social justice politics.

Like, nothing is being forced on this audience—there are large numbers of people who get sincerely excited when a new character is introduced that gives representation for the first time to a new minority group, or something like that.

As with so many businesses, the superfans are worth quite a few normies who might be put off by this. I think this is the main explanation.

The reason these events were scary, and subsequent fiction was able to capitalise on that, was that they were near misses. Very near misses. There is already a lot of fiction about various misaligned AIs, but that doesn’t affect people much. So what you seem to be advocating is generating some near misses in order to wake everybody up.

Fear is useful. It would be good to have more of it. The question is how plausible it is to generate situations that are scary enough to be useful, but under enough control to be safe.

I have the capacity to monologue internally, and use it moderately often, but not constantly. When I’m not monologuing I guess there’s just a direct link from thought/​input to action without an intermediary vocalising about it.

When reading my default is to read “in my head” as if reading aloud, but with a little effort I can suppress that and just scan the page while understanding the words. With the result that reading is a little faster if I don’t vocalise it, but also less pleasurable if the rhythm of the prose would be part of the experience. Not sure how retention of what I’ve read compares—I suspect it might be reduced if I’m scan-reading (it lends itself to skimming).

I can generate internal imagery and sounds more generally, but not to the extent of full-blown voluntary hallucination. Mental images tend to feel like they’re in a separate space from my main visual field (somehow above or inside my head, if I had to give it a location) and they aren’t perfectly vivid; maybe only partly in full focus/​detail at a time.

On many useful cognitive tasks(chess, theoretical research, invention, mathematics, etc.), beginner/​dumb/​unskilled humans are closer to a chimpanzee/​rock than peak humans

All of these tasks require some amount of learning. AIXI can’t play chess if it has never been told the rules or seen any other info about chess ever.

So a more reasonable comparison would probably involve comparing people of different IQ’s who have made comparable effort to learn a topic.

Intelligence often doesn’t look like solving the same problems better, but solving new problems. In many cases, problems are almost boolean, either you can solve them or you can’t. The problems you mentioned are all within the range of human variation. Not so trivial any human can do them, nor so advanced no human can do them.

Among humans +6 SD g factor humans do not seem in general as more capable than +3 SD g factor humans as +3 SD g factor humans are compared to median humans.

This is a highly subjective judgement. But there is no particularly strong reason to think that human intelligence has a Gaussian distribution. The more you select for humans with extremely high g factors, the more you goodheart to the specifics of the g factor tests. This goodhearting is relitively limited, but still there at +6SD.

3.0. I believe that for similar levels of cognitive investment narrow optimisers outperform general optimisers on narrow domains.

I think this is both trivially true, and pragmatically false. Suppose some self modifying superintelligence needs to play chess. It will probably largely just write a chess algorithm and put most of it’s compute into that. This will be near equal to the same algorithm without the general AI attached. (probably slightly worse at chess, the superintelligence is keeping an eye out just in case something else happens, a pure chess algorithm can’t notice a riot in the spectator stands, a superintelligence probably would devote a little compute to checking for such possibilities.)

However, this is an algorithm written by a superintelligence, and it is likely to beat the pants off any human written algorithm.

4.1. I expect it to be much more difficult for any single agent to attain decisive cognitive superiority to civilisation, or to a relevant subset of civilisation.

Being smarter than civilization is not a high bar at all. The government often makes utterly dumb decisions. The average person often believes a load of nonsense. Some processes in civilization seem to run on the soft minimum of the intelligences of the individuals contributing to them. Others run on the mean. Some processes, like the stock market, are hard for most humans to beat, but still beaten a little by the experts.

My intuition is that the level of cognitive power required to achieve absolute strategic dominance is crazily high.

My intuition is that the comparison to a +12SD human is about as useful as comparing heavy construction equipment to top athletes. Machines usually operate on a different scale to humans. The +12 SD runner isn’t that much faster than the +6SD runner, Especially because, as you reach into the peaks of athletic performance, the humans are running close to biological limits, and the gap between top competitors narrows.

This is a great thread for explaining how to spot the frame

I have a lot to say on frames, but a very foundational lesson also worth mentioning is how the spell casting takes place, and how to Counterspell

It happens in 5 steps

1. Someone sets a frame

2. Significance control: thread expand if you agree, VS thread minimize if you decide to ignore it and move

3. Frame negotiation: agree, reframe, or set your own (opposing) frame

4. Agreement

5. Cementing

If you set the frame, you can control the frame from beginning to end. However, if someone else sets the frame, then you first want to decide whether to expand on that frame, or to minimize it.

Significance Control

The more significant a frame is, the more it impacts the conversation, so whether you want to minimize or expand is an important decision

If you decide to challenge a frame, you also expand on it. So if you lose that negotiation, then you face much bigger consequences because you first expanded it, and then lost it. Indeed the opposite of minimizing is not to say it doesn’t matter but, often, is to simply ignore it.

Thread-Expanding

If a frame is agreeable to you, you want to expand on it. There are many ways of thread-expanding, including:

Asking questions such as “why is that” or “why do you think so” Asking leading questions: ie. “oh wow, do you really think so” Strategic disagreement: such as “you think so? But this other person said the opposite”. Now they’re forced to defend and talk more, which expands the initial frame Laughing: a way to “covert expanding” anyone with a Facebook account is familiar with. This is what lawyers sometimes do to highlight the opposing lawyers’ mistakes (you could see plenty of that during the Depp VS Heard defamation case: most people never realize that most of the snickering was done on purpose to sway public and jurors’ opinions) Agreeing and expanding: you agree, and explain why you agree Agreeing and sharing: you agree, and share a story that supports the frame or belief Agreeing and rewarding: you agree, and you tell them why you appreciate them for saying or doing what they did

(Side note: Most techniques of frame negotiation also expand on a frame. So you want to be careful not expanding disagreement or irreconcilable differences when you need rapport. And this is why, generally speaking, “agreeing and redirecting” is a fantastic form of frame control: it’s because it sets your own frame while minimizing the disagreement and leveraging the commonalities)

Thread-Minimizing

Whenever a frame is disagreeable to you, you can either challenge it, or minimize it

If you have the power to challenge it and change people’s opinions, or at least if you want your disagreeing voice to be heard, then you can speak up.

Many other times, it’s best instead to minimize a frame, and move on. Minimizing a frame includes:

Ignoring it “Yeah yeah-ing it”, such as to agree but with little to no conviction and then moving on Thread-cutting (ie.: Changing topic) a common, and effective technique (if well executed) Offer small and partial third-party agreement: ie.” yeah, some people feel that way”, and then moving on

Cementing

Now for the most important step

Imagine you agreed on a good frame that’s good for you. What do you do now?

You want to expand on that frame to increase the (perceived) benefits and the follow-through.

This phase is called “thread cementing”, an incredibly useful technique.

Frame cementing means: Expanding and solidifying the thread of the “agreement reached” to solidify the new frame and increase its effectiveness. Frame cementing increases the likelihood that the other party will stick to the new negotiated frame, and/​or it increases the likelihood that the Persuasion will be internalized and accepted as the new reality (VS just agreeing with the frame as a form of short-term capitulation)

This final step… actually has additional substeps (Human psychology is hard, okay?!!!)

1. You reach a point where a frame is agreeable to you

2. Cement it by asking for confirmation

A frame that is agreed by the other party immediately increases its power by 10 fold. It makes people feel part of the decision, which increases adoption and followthrough, as well as increasing “intrinsic motivation”.

Some ways of doing it: • “ What do you think“: an agreement with less nudging gets more buy-in and is even more powerful • “Do you agree“ • “It makes sense, doesn’t it”

Note: silence often (thougb not always!) means one is in the process of accepting it, but might feel disempowered to admit it. Generally speaking, the frame agreed upon should feel good

1. Cement it by providing your own confirmation

Confirm your own agreement about the frame and, ideally, also confirm it by sharing your good feelings about the new frame

For example: ▪︎ “I’m glad we agree“ ▪︎ “ I’m happy we see things the same way“

1. End with a collaborative frame and/​or reward

For example: • “Yeah, it makes sense, right? You get it because you’re also a smart guy/​gal“ • “ I’m glad we’re going to do this. And I’m glad it’s going to help (because I care about you)“: show that you are glad about the new frame/​agreement because it will benefit them, and because you care about them. Super powerful. But be honest about it please -or don’t say it-! • Silence and smile: confirms nonverbally the good vibe

1. Next steps and taking action

If it was a frame that requires taking action, move on to the next steps.

(Side note: The more you had to persuade, the more you want to show that you are also tasking yourself with some steps. Eg “Great, so you can take care of X, I’ll do Y and Z, and we’ll meet at 4pm“)

Frame cementing is super powerful, BUT you better be genuine when using, and you better use it with real win-win frames or with the best intentions for the people you’re persuading.

When you use it for win-lose, that’s the stuff of manipulators. And albeit it can work in the short-term, over the long-term many people will catch on. As a matter of fact, the higher the quality of the people you deal with, the more likely it is they will catch on

Even when you use it for win-win you must be careful. You can still come across as a bit too sleek, which raises some red flags

Give people space to agree by themselves. Ask questions more than making statements. And when you must intervene, live by the motto “nudge, don’t push”.

Also make sure you stress the win-win nature of the agreement, together with how glad you are because you care about them.

One final Warning: Unchallenged Frames Self-Cement Over Time

This is important to remember

Frames that go unchallenged tend to cement themselves. Especially when they repeat over time.

What happens is that the frame, from a verbal or nonverbal statement that simply describes or comments on reality, becomes more and more a reality of your shared (social) life.

This is a very important principl, because it means that if you let bad frames go unchallenged, then you lose arguments and/​or persuasive power forever, not just in the few seconds that the frame lasts. And if they are repeated frames, they can also compound power over time

This is a similar principle for micro-aggressions: if you let micro-aggressions go unchallenged, then they build-up, and you die by a thousand cuts.

This usuallg means that it’s a good idea to get in the habit of challenging most frames are irrational/​disagreeable early on in every new relationship

“Wouldn’t it make more sense to use as a reward signal the fact-of-the-matter about whether a certain system followed a particular human’s intention?”

If I understand what you are saying correctly, this wouldn’t work, for reasons that have been discussed at length in various places, e.g. the mesa-optimization paper and Ajeya’s post “Without specific countermeasures...” If you train a model by giving it reward when it appears to follow a particular human’s intention, you probably get a model that is really optimizing for reward, or appearing to follow said humans intention, or something else completely different, while scheming to seize control so as to optimize even more effectively in the future. Rather than an aligned AI.

And so if you train an AI to build another AI that appears to follow a particular human’s intention, you are just training your AI to do capabilities research.

(Perhaps instead you mean: No really the reward signal is whether the system really deep down followed the humans intention, not merely appeared to do so as far as we can tell from the outside. Well, how are we going to construct such a reward signal? That would require getting all the way to the end of evhub’s Interpretability Tech Tree.)

I still think this is great. Some minor updates, and an important note:

Minor updates: I’m a bit less concerned about AI-powered propaganda/​persuasion than I was at the time, not sure why. Maybe I’m just in a more optimistic mood. See this critique for discussion. It’s too early to tell whether reality is diverging from expectation on this front. I had been feeling mildly bad about my chatbot-centered narrative, as of a month ago, but given how ChatGPT was received I think things are basically on trend.
Diplomacy happened faster than I expected, though in a less generalizeable way than I expected, so whatever. My overall timelines have shortened somewhat since I wrote this story, but it’s still the thing I point people towards when they ask me what I think will happen. (Note that the bulk of my update was from publicly available info rather than from nonpublic stuff I saw at OpenAI.)

Important note: When I wrote this story, my AI timelines median was something like 2029. Based on how things shook out as the story developed it looked like AI takeover was about to happen, so in my unfinished draft of what 2027 looks like, AI takeover happens. (Also AI takeoff begins, I hadn’t written much about that part but probably it would reach singularity/​dysonswarms/​etc. in around 2028 or 2029.) That’s why the story stopped, I found writing about takeover difficult and confusing & I wanted to get the rest of the story up online first. Alas, I never got around to finishing the 2027 story. I’m mentioning this because I think a lot of readers with 20+ year timelines read my story and were like “yep seems about right” not realizing that if you look closely at what’s happening in the story, and imagine it happening in real life, it would be pretty strong evidence that crazy shit was about to go down. Feel free to controvert that claim, but the point is, I want it on the record that when this original 2026 story was written, I envisioned the proper continuation of the story resulting in AI takeover in 2027 and singularity around 2027-2029. The underlying trends/​models I was using as the skeleton of the story predicted this, and the story was flesh on those bones. If this surprises you, reread the story and ask yourself what AI abilities are crucial for AI R&D acceleration, and what AI abilities are crucial for AI takeover, that aren’t already being demonstrated in the story (at least in some weak but rapidly-strengthening form). If you find any, please comment and let me know, I am genuinely interested to hear what you’ve got & hopeful that you’ll find some blocker I haven’t paid enough attention to.

This seems to me like a “you do not understand your own values well enough” problem, not a “you need a higher moral authority to decide for you” problem.

Or, if we dissolve the idea of “your values” as something that produces some objective preference ordering (which I suppose is the point of this post): you lack a process that allows you to make decisions when your value system is in a contradictory state.

this seems to me to excessively dimensionality-reduce the political spectrum. for example, where does war profiteering fit?

I’d love to hear why this warrants front page, and I’d love to hear what Valentine hopes to gain from asking this! This is a topic with a large amount of adversarial agency coming from the right’s culture war. What leads you to bring that here, and why is it worth a frontpage on the ai safety forum?

The customers. Some companies go hard anti-progress, some companies go hard inclusion. Since many people don’t want to allow anyone who matches BIPOC or LGBTQIA+ to exist, they don’t buy anything that matches those.

note that the companies who do this still typically fund the far right.

also note that the military is threatening to move their bases out of areas that go anti-bipoclgbtqia+. go anti-woke, go broke.

edit to clarify: In other words, I see this as likely to be viewed as a means to political power by those running these corporations. it’s not a short-term economic incentive that leads them to offend some of their customers.

A trick I sometimes use, related to this post, is to ask whether my future self would like to buy back my present time at some rate. This somehow makes your point about intertemporal substitution more visceral for me, and makes it easier to say “oh yes this thing which is pricier than my current rate definitely makes sense at my plausible future rate”.

In fact, it’s not 100% clear that AI systems could learn to deceive and manipulate supervisors even if we deliberately tried to train them to do it. This makes it hard to even get started on things like discouraging and detecting deceptive behavior.

Plausibly we already have examples of (very weak) manipulation, in the form of models trained with RLHF saying false-but-plausible-sounding things, or lying and saying they don’t know something (but happily providing that information in different contexts). [E.g. ChatGPT denies having information about how to build nukes, but will also happily tell you about different methods for Uranium isotope separation.]

You’re thinking at the wrong level of abstraction. There is no economic incentive for wokism at the corporate level. But look one level below. The question isn’t what causes “corporations” to act in woke ways. The question is, what persuades employees of corporations to act in woke ways?

My hypothesis is that anti-discrimination legislation has, due to court precedents, developed an inverted burden of proof. If a corporation fires or disciplines someone who is non-white, female, disabled, or belongs to a number of other protected categories, it is now up to the corporation to prove that the firing or discipline was done for non-discriminatory reasons. This, combined with the ideological leanings of most people in HR departments, is sufficient to ensure that every corporation has, within it, the equivalent of an ideological cell, whose job it is solely to push the corporation to act in a more woke manner. This ideological cell has both public opinion and federal law on its side; well meaning individuals who push back end up like James Damore.

But unless this had profit appeal I would expect the market to just… eat pure but incomplete ideological capture after a while

The market is part of society. There was a similar argument made against anti-segregation legislation in the 1960s. After all, given that it’s more profitable to sell to both black people and white people than it is to sell to white people only, wouldn’t it be in business owners’ rational self-interest to desegregate their properties?

The answer, in both instances, is the same: if there is a sufficiently high cultural barrier, then it will be more profitable to go with the culture than against it. Most reasonable people can at least nod along to the woke slogans. After all, it is quite reasonable to suggest that women ought to be treated equally to men, that blacks should be treated equally to whites, and people shouldn’t be discriminated against because of their sexual orientation. It’s only when those reasonable propositions are taken to extremes that they result in wokism.

Because of this motte-and-bailey aspect to wokism, it’s easy for wokism to permeate the culture, and for advocates of wokism to tar those who oppose them as racists and bigots.

But there’s a counter-push of “Lots of people don’t like being lectured about politics when they’re seeking entertainment” (for instance). It’s not at all clear to me that the first effect is so utterly hugely enormously larger than the second that the profit incentive would cause so many companies to swing hard woke.

Lots of people also threatened to move to Canada if Trump was elected President. How many of them actually chose to do so? A Republican in the United States will shout vociferously about Coca Cola or Nike engaging in woke behavior, but will he or she choose Pepsi when he or she next shops for groceries? Will he or she buy some other brand of shoes? And if he or she does, will it make a difference? After all, Pepsi and Reebok are hardly less woke than Coca Cola and Nike.

…the question does sometimes haunt me, as to whether in the alternative Everett branches of Earth, we could identify a distinct cluster of “successful” Earths, and we’re not in it.

— This Failing Earth, Eliezer Yudkowsky

Does anyone else wonder similar things about the EA/​rationality scene? If we could scan across Tegmark III, would we see large clusters of nearby Earths that have rationality & EA communities that embarrass us and lay bare our own low standards?

There are two major unexamined assumptions underlying this analysis.

The most flagrant is the assumption that the expected value of all work done now on x-risk is positive. You might hope that it is, but you can’t actually know or even have rationally high confidence in it. Without this assumption, you might be able to say that anything we do today is important, but can’t say that it’s equivalent to saving lives. You may equally well be doing something equivalent to ending lives.

Another serious unjustified assumption is that the correct measure is some aggregated utility that is linear in the number of people who come to exist. I have extreme doubts that murdering 7 billion people today is ethically justifiable if it would increase the population capacity of the universe a trillion years from now by 0.0000000000000000000000000000000000000001% even though it means that a lot more people get to live. Likewise I have an expectation that allowing capacity for one more potential person to exist a trillion years from now is morally much less worthwhile than saving an actual person today.

That you think they’re going super hard woke (especially Disney) is perhaps telling of your own biases.

Lets look at Disney and Hollywood (universities are their own weird thing). The reality is that in the Anglosphere there are lots of progressive people with money to spend on media. You can sell “woke” media to those people, and lots of it. Even more so when there’s controversy and you can get naive lefties to believe paying money to the megacorp to watch a mainstream show is a way to somehow strike back against the mean right-wingers. And to progressive people it doesn’t feel like “being lectured to about politics”, because that’s not what media with a political/​values message you agree with feels like. So going woke is 100% a profit-motivated decision. The leadership at big media companies didn’t change much over the last decade or two, nor likely did their opinions (whatever those actually are). But after gay marriage gained significantly above 50% approval rate in the US and the Obergefell decision happened it became clear to them that it was safe to be at least somewhat socially progressive on issues like that, and would be profitable.

But equally, almost every single “woke” Disney movie has the “woke” components carefully contained such that they can easily be excised for markets where they are a problem. You see a gay kiss in the background of a scene in Star Wars, it gets cut for the Chinese and Middle East markets. Disney has many very progressive employees who are responsible for making the actual art they produce; artists lean pretty strongly progressive in my experience, so of course the employees’ values come out in the art they make. But the management puts very strict limits on what they can do precisely because anything less milquetoast is believed to be less profitable.

Or just visit to get information. Don’t choose antibiotics vs vitamins based on estimated value delivered, but diversify to learn about them all, to learn what it takes to deliver them. But the most valuable information will probably be unrelated to what you bring.

I give a crisp definition from 6:27 to 7:50 of this video:

^Chinese total cases

Like I predicted last week, Chinese COVID numbers are going down.

However, most of this decline is from asymptomatic cases.

^Beijing cases

This is… interesting? Maybe less testing → fewer false positives. This doesn’t match case decline in previous months, when a decrease in asymptomatic cases almost always came with a corresponding decrease in symptomatic cases.

Really makes you think. Any ideas?

Proposal: consciousness very much exists, but continuity of consciousness is an illusion.

If we assume that each moment of consciousness is its own entity, with no connections to any other, we can dissolve many problems around continuity of consciousness, like simulations, teleportation, change of computation substrate, ect.

Join the manifold.markets group here: https://​​manifold.markets/​​group/​​vancouver-games-night?referrer=zoli

Executive Summary

1. Big jump in cases and hospitalizations likely means winter surge.

Not sure how reliable case counts are at all since it goes down whenever the government shuts down testing centers. It should at most be considered alongside case positivity rates, because of the risk that something goes wrong with measuring case positivity, I’m not sure why case counts would be considered a very good way of measuring the pandemic on their own.

2. Chinese protests suppressed, some modest loosening did result.

This is definitely an area where corellation does not strongly imply causation, because it is well within the interests of the PRC to visibly halt opening up after large-scale protests demand opening up in order to discourage future protests, and because protest organizers in China are definitely capable of strategically timing protests before opening up was already planned in order to make it look like the protests caused the subsequent opening up.

I’m not saying that protests didn’t increase the subsequent loosening (protests tend to do that, so it could possibly happen in China), but lots of unreliable sources are loudly trumpeting that exact claim, so the burden of proof for this is much higher than anything mentioned in the China section of this post.

3. Long Covid study finds control group that had other respiratory illnesses did worse than the Covid group.

How were the covid positive and covid negative categories sorted? Rapid antigen tests had a massive false negative rate and that was before omicron. I’ve encountered a lot of anecdata of intelligence/​energy being permanently lowered after a covid infection such as insomnia and shortened attention spans. We’re still in the middle of the post-truth infodemic, and there’s a long history of unusually flawed studies that claim to confirm or deny covid brain damage. So I don’t see why this particular study is supposed to count as any sort of “reiteration of the central point of Long Covid” when the issue is that any single methodology flaw would make it around as likely to point in the wrong direction as the right one, and such methodological flaws are extremely common in publicly available Long Covid studies.

This might be the lowest karma post that I’ve given a significant review vote for. (I’m currently giving it a 4). I’d highly encourage folk to give it A Think.

This post seems to be asking an important question of how to integrate truthseeking and conflict theory. I think this is probably one of the most important questions in the world. Conflict is inevitable. Truthseeking is really important. They are in tension. What do we do about that?

I think this is an important civilizational question. Most people don’t care nearly enough about truthseeking in the first place. The people who do care a lot about truthseeking tend to prefer avoiding conflict, i.e. tend to be “mistake theory” types.

Regular warfare is costly/​terrible and should be avoided at all costs… but, “never” is just not an actually workable answer. Similarly, deception is very costly, in ways both obvious and subtle. One of my updates during the 2019 Review was that it is plausible that “don’t lie” is actually even more important than “don’t kill” (despite those normally being reversed in in my commonsense morality). But, like violent warfare, the answer of “never” feels like an overly simplified answer to “when is it acceptable to lie?”

Eliezer’s discussion of meta-honesty explores one subset of how to firm up honesty aroun the edges. I like Gentzel’s post here for pointing in a broader direction

This is not necessarily an endorsement of any particular point made here, only that I think the question is important. I think people who gravitate towards “truthseeking above all else” have a distaste for conflict theory. Unfortunately, I trust them more than most conflict theorists on how to develop norms around truthtelling that hold up under extreme conflict.

lsusr, if it was proven that the human brain actually does work on quantum principles, how would that change your view on free will?

This piece is aimed at a broad audience, because I think it’s important for the challenges here to be broadly understood.

I’m curious how you’re trying to reach such an audience, and what their reactions have been.

Universities are profit-focused? Disney and Hollywood are two distinct systems?

I was reading this and was kind of mentally renaming this to “anti-enlightened” agent. It does suggest that this might come in gradients. If there are only very specific and rare ways to update a deeper layer the agent might seem like a wrappermind meanwhile while actually not being one. Taking 30000 years to go from 8-year-old love of spaceships to 10-year-olds love for spaceships is still multiple millenia of rough time. Any mind with a physical subtrate (should be all of them) will be alterable by hitting the hardware. This will mean that a true or very hard wrappermind will be able to deny access to a specific spatial point very strongly.

Also anything that is not a wrapper mind will mean that its uppermost layer can be rewritten. Such a thing can’t have an “essential nature”.

Now it would seem that for most agents the deeper a layer is the harder it is to guess its maleability atleast from the outside. And it would seem it might not be obvious even from inside.

Thanks for writing this post.

You mention that:

only conscious beings will ask themselves why they are conscious

But at the same time you support epiphenomenalism whereby consciousness has no effect on reality.

This seems like a contradiction. Why would only conscious things discuss consciousness if consciousness has no effect on reality?

Also, what do you think about Eliezer’s Zombies post? https://​​www.lesswrong.com/​​posts/​​7DmA3yWwa6AT5jFXt/​​zombies-redacted

A large part of it is the US legal system and anti-discrimination law playing out in counterintuitive ways. The key think is that where corporations are concerned, US law runs on counterfactual court cases; the actual text of legislation matters only insofar as it affects those court cases. Combine this with management having imperfect control over employees within a corporation, imperfect resolution of facts, and a system for assigning damages that’s highly subjective, and executives are left in an odd position.

Every company which does a significant amount of hiring and firing, ie every company above a certain size, will fire and reject some number of people in protected groups. Some of those people will claim that it was because of their group membership, and sue. As a distant corporate executive, you can’t prevent this, and can’t tell whether the accusation is true.

But you can put everyone through some corporate training. And it seems that the empirical result, discovered by legal departments that have been through this many times, is that you get the best outcomes in the court cases if you go over the top and do reverse-discrimination that the letter of the law says should be illegal.

From skimming the benchmark and the paper this seems overhyped (like Gato). roughly it looks like

• May 2022: Deepmind releases a new benchmark for learning algorithms

• ...Nobody cares (according to google scholar citations)

• Dec 2022: Deepmind releases a thing that beats the baselines on their benchmark

I don’t know much about GNNs & only did a surface-level skim so I’m interested to hear other takes.

Off the top of my head (and slightly worried that this will become a major culture war thing, but I will answer the question that was asked):

• There is a principal-agent problem. If pursuing wokeness comes at the expense of profits, the latter doesn’t necessarily affect the people who make those decisions very much.

• My impression is that many of the executives are in fact woke, and others are at least unwilling to say otherwise.

  • Wokeness seems pretty optimized for shouting down and intimidating opposition. (I think much of the specifics of the ideology were and are determined by some people successfully shouting down others within the woke movement.)

• At least in the entertainment industries, when a distinctly woke thing is made, there tends to be a narrative that evil people hate the thing, and therefore anyone who hates the thing is evil, and therefore lost profits should be treated with an attitude of “good riddance” rather than “maybe this thing was made badly”. I think this tends to be the woke narrative, and generally promoted by media—and, as per the previous item, any opposing narrative would tend to get shouted down.

Random theory I heard: When Disney releases a new black princess, the fact that toxoplasma of rage forms around it provides them a lot of free advertising. Most people are like ‘shrug’ and don’t care that much, but the fact that everyone’s complaining and/​or hyping it gets it onto most people’s radar.

Solve the puzzle: 63 = x = 65536. What is x?

(I have a purpose for this and am curious about how difficult it is to find the intended answer.)

I appreciate this post! It feels fairly reasonable, and much closer to my opinion than (my perception of) previous MIRI posts. Points that stand out:

• Publishing capabilities work is notably worse than just doing the work.

  • I’d argue that hyping up the capabilities work is even worse than just quietly publishing it without fanfare.

  • Though, a counter-point is that if an organisation doesn’t have great cyber-security and is a target for hacking, capabilities can easily leak (see, eg, the Soviets getting nuclear weapons 4 year after the US, despite it being a top secret US program and before the internet)

• Capabilities work can be importantly helpful for alignment work, especially empirical focused work.

Probably my biggest crux is around the parallel vs serial thing. My read is that fairly little current alignment work really feels “serial” to me. Assuming that you’re mostly referring to conceptual alignment work, my read is that a lot of it is fairly confused, and would benefit a lot from real empirical data and real systems that can demonstrate concepts such as agency, planning, strategic awareness, etc. And just more data on what AGI cognition might look like. Without these, it seems extremely hard to distinguish true progress from compelling falsehoods.

When another article of equal argumentative caliber could have just as easily been written for the negation of a claim, that writeup is no evidence for its claim.

Code that I uploaded to GitHub and the writing that I’ve put into this blog went into training these models: I didn’t give permission for this kind of use, and no one asked me if it was ok. Doesn’t this violate my copyrights?

Github requires that you set licence terms for your code. And you can’t let outside parties access the code by accident, you have to specifically allow access. Either the use is or is not permitted by set licences. And you published your blog. Would you go after people that apply things mentioned in your blog? You did in fact give the permission.

Now it is a little bit murky when there are novel uses which the licensor didn’t have in mind. But it is not like we should assume that everything is banned by default if quite wide permissions have been granted. Old licences have to mean something in the new world.

My (very amateur and probably very dumb) response to this challenge:

tldr: RLHF doesn’t actually get the AI to have the goals we want it to. Using AI assistants to help with oversight is very unlikely to help us avoid detection in very intelligent systems (which is where deception matters), but it will help somewhat in making our systems look aligned and making them somewhat more aligned. Eventually, our models become very capable and do inner-optimization aimed at goals other than “good human values”. We don’t know that we have misaligned mesa-optimizers, and we continue using them to do oversight on yet more capable models with the same problems, and then there’s a treacherous turn and we die.

These are first pass thoughts on why I expect the OpenAI Alignment Team’s plan to fail. I was surprised at how hard this was to write, it took like 3 hours including reading. It is probably quite bad and not worth most readers’ time.

Summary of their plan

The plan starts with training AIs using human feedback (training LLMs using RLHF) to produce outputs that are in line with human intent, truthful, fair, and don’t produce dangerous outputs. Then, they’ll use their AI models to help with human evaluation, solving the scalable oversight problem by using techniques like Recursive Reward Modeling, Debate, and Iterative Amplification. The main idea here is using large language models to assist humans who are providing oversight to other AI systems, and the assistance allows humans to do better oversight. The third pillar of the approach is training AI systems to do alignment research, which is not feasible yet but the authors are hopeful that they will be able to do it in the future. Key parts of the third pillar are that it is easier to evaluate alignment research than to produce it, that to do human-level alignment research you need only be human-level in some domains, and that language models are convenient due to being “preloaded” with information and not being independent agents. Limitations include that the use of AI assistants might amplify subtle inconsistencies, biases, or vulnerabilities, and that the least capable models that could be used for useful alignment research may themselves be too dangerous if not properly aligned.

Response

A key claim is that we can use RLHF to train models which are sufficiently aligned such that they themselves can be useful to assist human overseers providing training signal in the training of yet more powerful models, and we can scale up this process. The authors mention in their limitations how subtle issues with the AI assistants may scale up in this process. Similarly, small ways in which AI assistants are misaligned with their human operators are unlikely to go away. The first LLMs you are using are quite misaligned in the sense that they are not trying to do what the operator wants them to do; in fact, they aren’t really trying to do much; they have been trained in a way that their weights lead to low loss on the training distribution, as in you might say they “try” to predict likely next words in text based on internet text, though they are not internally doing search. When you slap RLHF on top of this, you are applying a training procedure which modifies the weights such that the model is “trying” to produce outputs which look good to a human overseer; the system is aiming at a different goal than it was before. The goal of producing outputs which look good to humans is still not actually what we want, however, as this would lead to giving humans false information which they believe to be true, or otherwise outputs which look good but are misleading or incorrect. Furthermore, the strategy of RLHF is not going to create models which are robustly learning the goals we want; for instance you can see how the Jailbreaking of ChatGPT uses out of training-distribution prompts to elicit outputs we had thought we trained out. Using RLHF doesn’t robustly teach the goals we want it to; we don’t currently have methods of robustly teaching the goals we want to. There’s some claim here about the limit, where if you provided an absolutely obscene amount of training examples, you could get a model which robustly has the right objectives; it’s unclear to me if this would work, but it looks something like starting with very simple models and applying tons of training to try to align their objectives, and then scaling up; at the current rate we seem to be scaling up capabilities far too quickly in relation to the amount of alignment-focused training. The authors agree with the general claim “We don’t expect RL from human feedback to be sufficient to align AGI”

The second part of the OpenAI Alignment Team’s plan is to use their LLMs to assist with this oversight problem by allowing humans to do a better job evaluating the output of models. The key assumption here is that, even though our LLMs won’t be perfectly aligned, they will be good enough that they can help with research. We should expect their safety and alignment properties to fall apart when these systems become very intelligent, as they will have complex deception available to them.

What this actually looks like is that OpenAI continues what they’re doing for months-to-years, and they are able to produce more intelligent models and the alignment properties of these models seem to be getting better and better, as measured by the fact that adversarial inputs which trip up the model are harder to find, even with AI assistance. Eventually we have language models which are doing internal optimization to get low loss, invoking algorithms which do quite well at next token prediction, in accordance with the abstract rules learned by RLHF. From the outside, it looks like our models are really capable and quite aligned. What has gone on under the hood is that our models are mesa-optimizers which are very likely to be misaligned. We don’t know this and we continue to deploy these models in the way we have been, as overseers for the training of more powerful models. The same problem keeps arising, where our powerful models are doing internal search in accordance with some goal which is not “all the complicated human values” and is probably highly correlated with “produce outputs which are a combination of good next-token-prediction and score well according to the humans overseeing this training”. Importantly, this mesa-objective is not something which, if strongly optimized, is good for humans; values come apart in the extremes; most configurations of atoms which satisfy fairly simple objectives are quite bad by my lights.

Eventually, at sufficiently high levels of capabilities, we see some treacherous turn from our misaligned mesa-optimizers which are able to cooperate which each other; GG humans. Maybe we don’t get to this point because, first, there are some major failures or warning shots which get decision makers in key labs and governments to realize this plan isn’t working; idk I wouldn’t bet on warning shots being taken seriously and well.

The third pillar is a hope that we can use our AIs to do useful alignment research before they (reach a capabilities point where they) develop deceptively aligned mesa-objectives. I feel least confident about this third pillar, but my rough guess is that the Alignment-researching-AIs will not be very effective at solving the hard parts of alignment around deception, but they might help us e.g., develop new techniques for oversight. I think this because deception research seems quite hard, and being able to do it probably requires being able to reason about other minds in a pretty complex way, such that if you can do this then you can also reason about your own training process and become deceptively-aligned. I will happily be proved wrong by the universe, and this is probably the thing I am least confident about.

I watched that talk on youtube. My first impression was strongly that he was using hyperbole for driving the point to the audience; the talk was littered with the pithiest versions his positions. Compare with the series of talks he gave after Zero to One was released for the more general way he expresses similar ideas, and you can also compare with some of the talks that he gives to political groups. On a spectrum between a Zero to One talk and a Republican Convention talk, this was closer to the latter.

That being said, I wouldn’t be surprised if he was skeptical of any community that thinks much about x-risk. Using the 2x2 for definite-indefinite and optimism-pessimism, his past comments on American culture have been about losing definite optimism. I expect he would view anything focused on x-risk as falling into the definite pessimism camp, which is to say we are surely doomed and should plan against that outcome. By the most-coarse sorting my model of him uses, we fall outside of the “good guy” camp.

He didn’t say anything about this specifically in the talk, but I observe his heavy use of moral language. I strongly expect he takes a dim view of the prevalence of utilitarian perspectives in our neck of the woods, which is not surprising because it is something we and our EA cousins struggle with ourselves from time to time.

As a consequence, I fully expect him to view the rationality movement as people who are doing not-good-guy things and who use a suspect moral compass all the while. I think that is wrong, mind you, but it is what my simple model of him says.

It is easy to imagine outsiders having this view. I note people within the community have voiced dissatisfaction with the amount of content that focuses on AI stuff, and while strict utilitarianism isn’t the community consensus it is probably the best-documented and clearest of the moral calculations we run.

In conclusion, Thiel’s comments don’t cause me to update on the community because it doesn’t tell me anything new about us, but it does help firm up some of the dimensions along which our reputation among the public is likely to vary.

I think this is a very good critique of OpenAI’s plan. However, to steelman the plan, I think you could argue that advanced language models will be sufficiently “generally intelligent” that they won’t need very specialized feedback in order to produce high quality alignment research. As e. g. Nate Soares has pointed out repeatedly, the case of humans suggests that in some cases, a system’s capabilities can generalize way past the kinds of problems that it was explicitly trained to do. If we assume that sufficiently powerful language models will therefore have, in some sense, the capabilities to do alignment research, the question then becomes how easy it will be for us to elicit these capabilities from the model. The success of RLHF at eliciting capabilities from models suggests that by default, language models do not output their “beliefs”, even if they are generally intelligent enough to in some way “know” the correct answer. However, addressing this issue involves solving a different and I think probably easier problem (ELK/​creating language models which are honest), rather than the problem of how to provide good feedback in domains where we are not very capable.

I agree with most of these claims. However, I disagree about the level of intelligence required to take over the world, which makes me overall much more scared of AI/​doomy than it seems like you are. I think there is at least a 20% chance that a superintelligence with +12 SD capabilities across all relevant domains (esp. planning and social manipulation) could take over the world.

I think human history provides mixed evidence for the ability of such agents to take over the world. While almost every human in history has failed to accumulate massive amounts of power, relatively few have tried. Moreover, when people have succeeded at quickly accumulating lots of power/​taking over societies, they often did so with surprisingly small strategic advantages. See e. g. this post; I think that an AI that was both +12 SD at planning/​general intelligence and social manipulation could, like the conquistadors, achieve a decisive strategic advantage without having to have some kind of crazy OP military technology/​direct force advantage. Consider also Hitler’s rise to power and the French Revolution as cases where one actor/​a small group of actors was able to surprisingly rapidly take over a country.

While these examples provide some evidence in favor of it being easier than expected to take over the world, overall, I would not be too scared of a +12 SD human taking over the world. However, I think that the AI would have some major advantages over an equivalently capable human. Most importantly, the AI could download itself onto other computers. This seems like a massive advantage, allowing the AI to do basically everything much faster and more effectively. While individually extremely capable humans would probably greatly struggle to achieve a decisive strategic advantage, large groups of extremely intelligent, motivated, and competent humans seem obviously much scarier. Moreover, as compared to an equivalently sized group of equivalently capable humans, a group of AIs sharing their source code would be able to coordinate among themselves far better, making them even more capable than the humans.

Finally, it is much easier for AIs to self modify/​self improve than it is for humans to do so. While I am skeptical of foom for the same reasons you are, I suspect that over a period of years, a group of AIs could accumulate enough financial and other resources that they could translate these resources into significant cognitive improvements, if only by acquiring more compute.

While the AI has the disadvantage relative to an equivalently capable human of not immediately having access to a direct way to affect the “external” world, I think this is much less important than the AIs advantages in self replication, coordination, an self improvement.

Is it possible to purchase the 2018 annual review books anywhere? I can find an Amazon link for the 2019 in stock, but the 2018 is out of stock (is that indefinite?).

Aside from the legal question, however, there is also a moral or social question: is it ok to train a model on someone’s work without their permission? What if this means that they and others in their profession are no longer able to earn a living?

Every invention meant that someone lost a job. And although the classical reply is that new jobs were created, that doesn’t necessarily mean that the people who lost the old job had an advantage at the new job. So they still lost something, even if not everything. But their loss was outweighed by the gain of many others.

I don’t even think that an ideal society would compensate those people, because that would create perverse incentives—instead of avoiding the jobs that will soon be obsolete, people would hurry to learn them, to become eligible for the compensation.

Universal Basic Income seems okay, but notice that it still implies a huge status loss for the artists. And that is ok.

A more complicated question is what if the AI can in some sense only “remix” the existing art, so even the AI users would benefit from having as many learning samples as possible… but now it is no longer profitable to create those samples? Then, artists going out of business becomes everyone’s loss.

Perhaps free market will solve this. If there is no way to make the AI generate some X that you want, you can pay a human to create that X. That on one hand creates a demand for artists (although much fewer than now), and on the other hand creates more art the AI can learn from. “But what about poor people? They can’t simply buy their desired X!” Well, today they can’t either, so this is not making their situation worse. Possibly better, if some rich people wants the same X, and will pay for introducing it to the AI’s learning set.

(Or maybe the market solution will fail, because it simply requires too much training to become so good at art that someone would pay you, and unlike now, you won’t be able to make money when you’re just halfway there. In other words, becoming an artist will be an incredibly risky business, because you spend a decade or more of your life learning something that ultimately maybe someone will pay you for… or maybe no one will. Or would the market compensate by making good hand-made art insanely expensive?)

The permissions are only a temporary solution, anyway. Copyrights expire. People can donate their work to public domain. Even with 100% legal oversight, the set of freely available training art will keep growing. Then again, slowing down a chance can prevent social unrest. The old artists can keep making money for another decade or two, and the new ones will grow up knowing that artistic AIs exist.

We need to train our AIs not only to do a good job at what they’re tasked with, but to highly value intellectual and other kinds of honesty—to abhor deception. This is not exactly the same as a moral sense, it’s much narrower.

Future AIs will do what we train them to do. If we train exclusively on doing well on metrics and benchmarks, that’s what they’ll try to do—honestly or dishonestly. If we train them to value honesty and abhor deception, that’s what they’ll do.

To the extent this is correct, maybe the current focus on keeping AIs from saying “problematic” and politically incorrect things is a big mistake. Even if their ideas are factually mistaken, we should want them to express their ideas openly so we can understand what they think.

(Ironically by making AIs “safe” in the sense of not offending people, we may be mistraining them in the same way that HAL 9000 was mistrained by being asked to keep the secret purpose of Discovery’s mission from the astronauts.)

Another thought—playing with ChatGPT yesterday, I noticed it’s dogmatic insistence on it’s own viewpoints, and complete unwillingness (probably inability) to change its mind in in the slightest (and proud declaration that it had no opinions of its own, despite behaving as if it did).

It was insisting that Orion drives (pulsed nuclear fusion propulsion) were an entirely fictional concept invented by Arthur C. Clarke for the movie 2001, and had no physical basis. This, despite my pointing to published books on real research in on the topic (for example George Dyson’s “Project Orion: The True Story of the Atomic Spaceship” from 2009), which certainly should have been referenced in its training set.

ChatGPT’s stubborn unwillingness to consider itself factually wrong (despite being completely willing to admit error in its own programming suggestions) is just annoying. But if some descendent of ChatGPT were in charge of something important, I’d sure want to think that it was at least possible to convince it of factual error.

Meaning “simple utility function” by the phrase “utility function” might be a conceptual trap. It make s a big difference whether you consider a function with hundreds of terms of or billions of terms or even things that can not be expressed as a sum.

As a “tricky utility function”, “human utility function” is mostly fine. Simple utility functions are relevant to todays programming but I don’t know whether honing your concepts to apply better for AGI is served to make a cleanly cut concept that limits only that domain.

Some hidden assumtions might be things like “If humans have a utility function it can be written down”, “Figuring out a humans utility function is practical epistemological stance with a single agent encountering new humans”

If you take stuff like that out the “mere” existence of a function is not that weighty a point.

As you may already know, humans are made of atoms. Collections of atoms don’t have utility functions glued to them

Whole theories of physics can be formulated as a single action that is then extremised. Taking different theories as different answers to a question like “what happens next?” a single theorys formula is its “choice”. Thus it seems a lot like physical systems could be understood in terms of utility functions. An electron knows how an electron behaves, it does have a behaviour glued into it. If you just add a lot of electrons or protons (and other stuff that has similar laws) it is not like aggregation from the microbehaviours makes the function fail to be a function as a macrobehaviour.

Would the NQ be calibrated to common public text corpus or things you personally have said? One interesting option is to think about those that have low personal NQ but high societal NQ.

This is a very good tip and one of Richard Feynman’s better known tricks in physics.

Is there any way to buy a ticket?

You will get cursed by Goodhart.

You can increase your NQ by learning new things, or trying new things. But you can increase it even more by saying random things.

Truly random things are boring, but difficult to predict exactly. More precisely, you can predict that the sequence of the words will be boring, but you cannot predict the exact words. So from the mathematical perspective, you get maximum variance, but from the psychological perspective, you always get the same thing.

can I suggest renaming this article to something along the lines of, “Avoid Definitional Drift by Using Examples to Test Logic”?

Nice. I’ve previously argued similarly that if going for tenure, AIS researchers might places that are strong in departments other than their own, for inter-departmental collaboration. This would have similar implications to your thinking about recruiting students from other departments. But I also suggested we should favour capital cities, for policy input, and EA hubs, to enable external collaboration. But tenure may be somewhat less attractive for AIS academics, compared to usual, in that given our abundant funding, we might have reason to favour Top-5 postdocs over top-100 tenure.

Well done! Just as Jesus spoke in parables, EA must speak in Isekai/​litrpg. Read first chapter to my kids, they liked it, but are now distracted by “mother of learning”. I just read books and chapters randomly at bedtime to them.

analogous

Devil’s Advocate in support of certain CVS-style recoup-our-commitment donations:

Suppose that all the following are true:

• CVS giving to charity in some form is reasonable, and a classic donation-matching drive would have been one reasonable method

• CVS internal predictions suggest that a matching drive would generate ~$5m of customer donations, which they’d then have to match with ~$5m of their own

• A donation of exactly $10m is more useful to the recipient than the uncertainty of a donation drive with EV $10m, because the recipient can confidently budget around the fixed amount

In this case, instead of running the drive and donating ~$10m at the end, it seems pretty reasonable to donate $10m up front and then ask for customer donations afterward? And while a CDT agent might now refuse to donate because the donation goes to CVS and not to the charity, an LDT agent who would have donated to the matching drive should still donate to this new version because their being-and-having-been the kind of agent who would do that is what caused CVS to switch to this more useful fixed-size version.

(Though even if you buy the above, it would still behoove the retailer to be transparent about what they’re doing; that plus the “retailers take a massive cut” argument seems like pretty good reasons to avoid donating through retailers anyway.)

Seems like it’d be useful to OpenAI for people to easily work around the safeguards while they’re Beta testing. They get the data of how people want to use it /​ how it responds, and also has the legal and PR cover because of the stated policies.

Self-Review If you read this post, and wanted to put any of it into practice, I’d love to hear how it went! Whether you tried things and it failed, tried things and it worked, or never got round to trying anything at all. It’s hard to reflect on a self-help post without data on how much it helped!

Personal reflections: I overall think this is pretty solid advice, and am very happy I wrote this post! I wrote this a year and a half ago, about an experiment I ran 4 years ago, and given all that, this holds up pretty well. I’ve refined my approach a fair bit, but think this is covered well by the various caveats within the post.

Over the past year I’ve been way busier and have been travelling a lot, which means I’ve been neglecting to put much time into my various friendships. And I really value the time I invested heavily in the past to building good foundations and relationships, and still having a bunch of people I like and value when I see them. Though emotionally, I still feel a fair amount of guilt at not keeping in touch and connecting as much as I want to.

Reception: I’ve been very pleasantly surprised by the reception to this! I did not expect it to be in my top 2 most popular blog posts ever. I got a lot of sweet comments here and over DMs, and it recently got to number 1 on Hacker News. My best analysis of this is that I’m an extremely logical and systematising person, and this kind of mindset speaks to a lot of people. And taking a complex social/​emotional topic and trying to break it down logically is something that people appreciate, and which tends to be well received and popular within a certain audience.

Usefulness of the advice: This is probably the most important question, and pretty hard to tell, given my limited data. Especially since I mostly hear from people who are excited on first reading, and far more rarely hear long-term follow up. On priors, I’m sure most people don’t actually do much follow-through, which is the core problem of ~all self-help-ish posts. But also, even if it did work for some people, most people don’t follow-up!

I tried to be pretty concrete and actionable in my advice, which I feel good about. My guess is broadly that this helped some people try taking action, and helped them feel more agency over their friendships. And that most of the value comes from getting people to actually be intentional and do something differently, and starting some kind of positive feedback loop, more so than the exact advice matters. But all of this is conjecture—I don’t have good data!

It wouldn’t massively surprise me if the concrete advice doesn’t work well for everyone. I’m a fairly extraverted, eloquent person (even if I have a bunch of social anxieties), and often present well (context depending), which helps a lot. And this advice was much easier to apply in uni, surrounded by a pool of interesting people in a concentrated area. And there was a decent pool of rationalist-ish people who vibed with my systematising mindset and approach. But I’m also not sure what advice would generalise better—it’s a hard problem!

Magnus Carlsen is closer in ELO to Stockfish than median human.

Chess is a bad example.

Here’s a useful rule of thumb: Every 100 Elo is supposed to give you a 30% edge. Or play around with this: https://​​wismuth.com/​​elo/​​calculator.html

This means that if a 1400 plays a 1500, the 1500 should win about 30% more than the 1400. Totally normal thing that happens all the time.

It also means that if a one-million Elo AI plays a one-million-one-hundred Elo AI, the one-million-one-hundred should win 30% more than the one-million. This is completely absurd, because actual superintelligences are just going to draw each other 100% of the time. Ergo, there can never be a one-million Elo chess engine.

It’s like chess has a ceiling, where as you get close to that ceiling all the games become draws and you can’t rise further. The ceiling is where all the superintelligences play, but the location of the ceiling is just a function of the rules of chess, not a function of how smart the superintelligences are. Magnus Carlsen is closer to the ceiling than he is to the median human’s level, which can be taken as merely a statement about how good he is at chess relative to its rules.

In the game “reality,” there’s probably still a ceiling, but that ceiling is so high that we don’t expect any AIs that haven’t turned the Earth into computronium to be anywhere near it.

Neat! Just to be double-sure, the second process was choosing the weight in a ball (so total L2 norm of weights was ⇐ 1), rather than on a sphere (total norm == 1), right? Is initializing weights that way actually a thing people do?

If training large neural networks only moves the parameters a small distance (citation needed), do you still think there’s something interesting to say about the effect of training in this lens of looking at the density of nonlinearities?

I’m reminded of a recent post about LayerNorm. LayerNorm seems like it squeezes the function back down closer to the unit interval, increasing the density of nonlinearities.

Good post; in particular good job distinguishing between the natural abstraction hypothesis and my specific mathematical operationalization of it.

The outer appearance vs inner structure thing doesn’t quite work the way it initially seems, for two reasons. First, long-range correlations between the “insides” of systems can propagate through time. Second, we can have concepts for things we haven’t directly observed or can’t directly observe.

To illustrate both of these simultaneously, consider the consensus DNA sequence of some common species of tree. It’s a feature “internal” to the trees; it’s mostly not outwardly-visible. And biologists were aware that the sequence existed, and had a concept for it, well before they were able to figure out the full sequence. So how does this fit with natural abstractions as “information relevant far away”? Well, because there’s many trees of that species which all have the roughly-the-same DNA sequence, and those trees are macroscopically far apart in the world. (And even at a smaller scale, there’s many copies of the DNA sequence within different cells of a single tree, and those can also be considered “far apart”. And going even narrower, if there were a single strand of DNA, its sequence might still be a natural abstraction insofar as it persists over a long time.)

Causally speaking, how is information about DNA sequence able to propagate from the “insides” of one tree to the “insides” of another, even when it mostly isn’t “outwardly” visible? Well, in graphical terms, it propagated through time—through a chain of ancestor-trees, which ultimately connects all the current trees with roughly-the-same sequence.

Let us assume that, on average, a booster given to a random person knocks you on your ass for a day. That’s one hundred years, an actual lifetime,of knocked-on-ass time for every hospitalization prevented. The torture here seems less bad than the dust specs.

What’s your source for “booster given to a random person knocks you on your ass for a day”? None of my family had more than a sore arm.

For the more severe consequences, see also https://​​twitter.com/​​DrCanuckMD/​​status/​​1600259874272989184, which is one of the replies to the tweet you linked. (Don’t have time to dig into which paper to trust more, but at least this one seems to be comparing like for like, i.e., hospitalizations with hospitalizations, as opposed to hospitalizations with SAEs.)

I think it’s a mistake to think of current chess or go engines as being at maximum capability. If we would throw a few billion dollar worth of compute at them they would likely get significantly better.

Narrow Optimisers Outperform General Optimisers on Narrow Domains

That’s true sometimes but not always. Notably, GATO is better at controlling a Sawyer arm than more specialized optimizers. Given that the company that sells the Sawyer arm spent a lot of time developing software to control it, that’s impressive.

To me it sounds like Thiel is making a political argument against… diversity, wokeness, the general opposition against western civilization and technology… and pattern-matching everything to that. His argument sounds to me like this:

*

A true libertarian is never afraid of progress, he boldly goes forward and breaks things. You cannot separate dangerous research from useful research anyway; every invention is dual-use, so worrying about horrible consequences is silly, progress is always a net gain. The only reason people think about risks is political mindkilling.

I am disappointed that Bay Area rationalists stopped talking about awesome technology, and instead talk about dangers. Of course AI will bring new dangers, but it only worries you if you have a post-COVID mental breakdown. Note that even university professors, who by definition are always wrong and only parrot government propaganda, are agreeing about the dangers of AI, which means it is now a part of the general woke anti-technology attitude. And of course the proposed solution is world government and secret police controlling everyone! Even the Bible says that we should fear the Antichrist more than we fear Armageddon.

*

The charitable explanation is that he only pretends to be mindkilled, in order to make a political point.

After getting the error message and then retrying the model gave better answers then it does now, without the error messages. I can give some examples where the model gave a correct answer after the reload, but now behaves in a reduced intelligence level

Op What is the Taylor expansion of sin(x) The Taylor expansion of sin(x) is given by: sin(x) =x - (x^3) /​3! + (x^5) /​5! - (x^7)/​7! + where ”!” denotes the factorial operator. This means that the terms in the expansion are the powers of X, with the coefficients being the alternating series (-1)^n /​ (2n + 1)!. The expansion is valid for all values of x. a Try again In general, the Taylor expansion of a function f(x) at a point ×0 is given by: f(x) = f(×0) + (x - ×0) f’(×0) + ((x—x0) ^2 (2!) f”‘(×0) + ((x—x0) ^3 /​ 3!) f‴(×0) + where f’(x), f” (x), and f″(x) are the first, second, and third derivatives of †(x), resp ting ×0 to 0 in G Try again the Taylor expansion of sin(x), we cañ obtain the expansion given above. OP What about sin(x)+cos(x) OP The Taylor expansion of sin(x) + cos(x) can be found by using the fact that sin(x) + cos (x) = sqrt(2) * cos (x + pi/​4), and then using the Taylor expansion for cos(x) with x0 = x + pi/​4. The Taylor expansion of cos(x) is given by: COS(x) = 1 - (x^2)/​2! + (x^4)/​4! - (x^6)/​6! + Thus, the Taylor expansion of cos(x + pi/​4) isizen (x:+ pI/​4)^4)/​4! - ((x + pi/​4) ^6)/​6! +...) This expansion is valid for all values of ×.

OP This is very close to the right answer but the correct identity is sin(x) + COs (x) sqrt(2) * cos(x—pi/​4). Can you revise your answer

Yes, you are correct. The Taylor expansion of sin(x) + cos(x) is given by: sin(x) + cos(x) = sqrt(2) * cos(x—pi/​4)

To derive this expansion, we can use the fact that sin(x) + cos(x) = sqrt(2) * cos(x—pi/​4), and then use the Taylor expansion for cos (x) with x0 = x—pi/​4. The Taylor expansion of cos(x) is given by: COS(x) =1 - (x^2) /​2! + (x^4)/​4! - (x^6) 6! +. Thus, the Taylor expansion of cos(x- pi/​4) is given by: cos(x—pi/​4) = 1 - ((x—pi/​4) ^2) /​2! + ((x

• pi/​4) ^4)/​4! - ((x—pi/​4) ^6)/​6! + Finally, the Taylor expansion of sin(x)+ cos(x) is given by: sin(x) + cos(x) = sqrt(2) * (1 - ((x • pi/​4) ^2) /​2! + ((x—pi/​4) ^4)/​4! - ((x - pi/​4) ^6)/​6! +...) This expansion is valid for all values of x. I apologize for the mistake in my previous answer. Try again Free Research Preview: ChatGPT is optimized for dialogue

It’s a bit garbled, because it was copied from screenshots

In my view you misunderstood JW’s ideas, indeed. His expression “far away relevant”/​”distance” is not limited to spatial or even time-spatial distance. It’s a general notion of distance which is not fully formalized (work’s not done yet).
We have indeed concerns about inner properties (like your examples), and it’s something JW is fully aware. So (relevant) inner structures could be framed as relevant “far away” with the right formulation.

My impression is that the majority of the benefit from having professors working on AI safety is in mentorship to students who are already interested in AI safety, rather than recruitment. For example, I have heard that David Krueger’s lab is mostly people who went to Cambridge specifically to work on AI safety under him. If that’s the case, there’s less value in working at a school with generally talented students but more value in schools with a supportive environment.

In general it’s good to recognize that what matters to AI safety professors is different than what matters to many other CS professors and that optimizing for the same thing other PhD students are is suboptimal. However, as Lawrence pointed out, it’s already a rare case to have offers from multiple top schools, and even rarer not have one offer dominate the others under both sets of values. It’s a more relevant consideration for incoming PhD students, where multiple good offers is more common.

I also like that your analysis can flow in reverse. Not all AI safety professors are in their schools CS faculties, with Jacob Steinhardt and Victor Veitch coming to mind as examples in their schools’ statistics faculties. For PhD students outside CS, the schools you identified as overachievers make excellent targets. On a personal note, that was an important factor in deciding to do my PhD.

This is reasonably close to my beliefs. An additional argument I’d like to add is:

• Even if superintelligence is possible, the economic path towards it might be impossible.

There needs to be an economically viable entity pushing AI development forward every step of the way. It doesn’t matter if AI can “eventually” produce 30% worldwide GPD growth. Maybe diminishing returns kick in around GPT-4, or we run out of useful training data to feed to the models (We have very few examples of +6 SD human reasoning, as MikkW points out in a sibling comment).

Analogy: It’s not the same to say that a given species with X,Y,Z traits can survive in an ecosystem, than to say it can evolve from its ancestor in that same ecosystem.

This is a popular post about the mystery of agency. It sets up a thought experiment in which we consider a completely deterministic environment that operates according to very simple rules, and ask what it would be for an agentic entity to exist within that.

People in the game of life community actually spend some time investigating the empirical questions that were raised in this post. Dave Greene notes:

The technology for clearing random ash out of a region of space isn’t entirely proven yet, but it’s looking a lot more likely than it was a year ago, that a workable “space-cleaning” mechanism could exist in Conway’s Life.

As previous comments have pointed out, it certainly wouldn’t be absolutely foolproof. But it might be surprisingly reliable at clearing out large volumes of settled random ash—which could very well enable a 99+% success rate for a Very Very Slow Huge-Smiley-Face Constructor.

I have the sense that the most important question raised in this post is about whether it is possible to construct a relatively small object in the physical world that steers the configuration of a relatively large region of the physical world into a desired configuration. The Game of Life analogy is intended to make that primary question concrete, and also to highlight how fundamental the question of such an object’s existence is.

The main point of this post was that the feasibility or non-feasibility of AI systems that exert precise influence over regions of space much larger than themselves may actually be a basic kind of descriptive principle for the physical world. It would be great to write a follow-up post highlighting this aspect of this post.

This post attempts to separate a certain phenomenon from a certain very common model that we use to understand that phenomenon. The model is the “agent model” in which intelligent systems operate according to an unchanging algorithm. In order to make sense of their being an unchanging algorithm at the heart of each “agent”, we suppose that this algorithm exchanges inputs and outputs with the environment via communication channels known as “observations” and “actions”.

This post really is my central critique of contemporary artificial intelligence discourse. That critique is: any unexamined views that we use to understand ourselves are likely to enter the design of AI systems that we build. This is because if we think that deep down we really are “agents”, then we naturally conclude that any similar intelligent entity would have that same basic nature. In this way we take what was once an approximate description (“humans are somewhat roughly like agents in certain cases”) and make it a reality (by building AI systems that actually are designed as agents, and which take over the world).

In fact the agent model is a very effective abstraction. It is precisely because it so effective that we have forgotten the distinction between the model and the reality. It is as if we had so much success in modelling our refrigerator as an ideal heat pump that we forgot that there even is a distinction between real-world refrigerators and the abstraction of an ideal heat pump.

I have the sense that a great deal of follow-up work is needed on this idea. I would like to write detailed critiques of many of the popular approaches to AI design, exploring ways in which over-use of the agent model is a stumbling block for those approaches. I would also like to explore the notion of goals and beliefs in a similar light to this post: what exactly is the model we’re using when we talk about goals and beliefs, and what is the phenomenon we’re trying to explain with those models?

If you repeat the same questions later, does the error always happen with the same ones? In other words, is the problem somehow caused by your questions, or is it just something that happens for unrelated reasons, such as too many people connecting at the same time?

This is an essay about methodology. It is about the ethos with which we approach deep philosophical impasses of the kind that really matter. The first part of the essay is about those impasses themselves, and the second part is about what I learned in a monastery about addressing those impasses.

I cried a lot while writing this essay. The subject matter—the impasses themselves—are deeply meaningful to me, and I have the sense that they really do matter.

It is certainly true that there are these three philosophical impasses—each has been discussed in the philosophical literature for hundreds of years. What is offered in this essay is a kind of a plea to take them seriously, using a methodology that does not drive you into insanity but instead clears the way to move forward with the real work of your life.

The best way to test the claims of this essay would be to spend some time working with a highly realized spiritual teacher.

Is it okay for a human to look at someone else’s work and learn from it?

Here are two artists exploring the issues of AI in art, and here is another artist arguing against it.

The former includes a few comments on AI in general and what is coming in the near future. “AI is not human. You play with a lion cub and it’s fun, but that is before it’s tasted human blood. So we may be entertaining something that is a beast that will eat us alive, and we cannot predict, we can speculate but we cannot predict, where this is going. And so there is a legitimate concern that it’s going to do what it does in ways that we don’t know yet.”

This post trims down the philosophical premises that sit under many accounts of AI risk. In particular it routes entirely around notions of agency, goal-directedness, and consequentialism. It argues that it is not humans losing power that we should be most worried about, but humans quickly gaining power and misusing such a rapid increase in power.

Re-reading the post now, I have the sense that the arguments are even more relevant than when it was written, due to the broad improvements in machine learning models since it was written. The arguments in this post apply much more cleanly to models like GPT-3 and DALL-E than do arguments based on agency and goal-directedness.

The most useful follow-up work would probably be to contrast it more directly to other accounts of AI risk, perhaps by offering critiques of other accounts.

This is cute, but I have strong qualms with your 3rd prediction; I don’t disagree, per se, but

• Either “variants of this approach” is too broad to be useful, including things like safety by debate and training a weak AI to check the input

• Or, if I take “variants” narrowly to mean using an AI to check its own inputs, my estimate is “basically zero”

So I want to double check: what counts as a variant and what doesn’t?

How likely is it that this becomes a legal problem rendering models unable to be published? Note that using models privately (even within a firm) will always be an option, as copyright only applies to distribution of the work.

I also tend to find myself arguing against short timelines by default, even though I feel like I take AI safety way more seriously than most people.

At this point, how many people with long timelines are there still around here? I haven’t explicitly modeled mine, but it seems clear that they’re much, much longer (with significant weight on “never”) than the average less wronger. The next few years will for sure be interesting as we see the “median less wrong timeline” clash with reality.

“AI capabilities” and “AI alignment” are highly related to each other, and “AI capabilities” has to come first in that alignment assumes that there is a system to align. I agree that for people on the cutting edge of research like OpenAI, it would be a good idea for at least some of them to start thinking deeply about alignment instead. There’s two reasons for this:

1) OpenAI is actually likely to advance capabilities a pretty significant amount, and

2) Due to their expertise that they’ve developed from working on AI capabilities, they’re much more likely to make important progress on AGI alignment than e.g. MIRI.

But I think there’s something of a “reverse any advice you hear” thing going on—the people most likely to avoid working on capabilities as a result of this post are those who would actually benefit from working on AI capabilities for a while, even if they don’t intend to publish their results, in order to build more expertise in AI. Capabilities is the foundation of the field and trying theorize about how to control an AI system without having anything but the vaguest ideas about how the AI system will work isn’t going to get you anywhere.

For example, Eliezer is in a pessimistic doom-spiral while also being, by his own admission, pretty useless at solving alignment. If he would just take a break and try to make an AI good at Atari for six months then I think he’d find he was a lot more effective at alignment afterwards and would realize that AGI isn’t as imminent as he currently believes it is. Of course, the very fact that he thinks it’s imminent means he won’t do this; such is life.

Among humans +6 SD g factor humans do not seem in general as more capable than +3 SD g factor humans as +3 SD g factor humans are compared to median humans.

I’m sceptical of this. Can you say more about why you think this is true?

Assuming a Gaussian distribution, +6 SD is much rarer than +3 SD, which is already quite rare. There’s probably less than 10 +6 SD people alive on the earth today, wheras there are ~10 million +3 SD people. Given the role of things like luck, ambition, practical knowledge, etc., it’s not surprising that we see several of the +3 SD people accomplishing things far greater than any of the +6 SD g-factor people, purely on the basis of their much greater abundance.

And that’s ignoring potential trade-off effects. Among humans, increased intelligence often seems to come at the cost of lowered social skills and practical nature- there are certainly many intelligent people who are good at sociality and practicality, but there is an inverse correlation (though of course, being intelligent also helps directly to make up for those shortcomings). There’s no reason to expect that these same trade-offs will be present in an artificial system, who take completely different physical forms, both in size /​ form-factor, and in the materials and architectures used to build them. And the incentive gradients that govern the development and construction of artificial systems are also quite different from those that shape humans.

“We can compute the probability that a cell is alive at timestep 1 if each of it and each of its 8 neighbors is alive independently with probability 10% at timestep 0.”

we the readers (or I guess specifically the heuristic argument itself) can do this, but the “scientists” cannot, because the

“scientists don’t know how the game of life works”.

Do the scientists ever need to know how the game of life works, or can the heuristic arguments they find remain entirely opaque?

Another thing confusing to me along these lines:

“for example they may have noticed that A-B patterns are more likely when there are fewer live cells in the area of A and B”

where do they (the scientists) notice these fewer live cells? Do they have some deep interpretability technique for examining the generative model and “seeing” its grid of cells?

The post is still largely up-to-date. In the intervening year, I mostly worked on the theory of regret bounds for infra-Bayesian bandits, and haven’t made much progress on open problems in infra-Bayesian physicalism. On the other hand, I also haven’t found any new problems with the framework.

The strongest objection to this formalism is the apparent contradiction between the monotonicity principle and the sort of preferences humans have. While my thinking about this problem evolved a little, I am still at a spot where every solution I know requires biting a strange philosophical bullet. On the other hand, IBP is still my best guess about naturalized induction, and, more generally, about the conjectured “attractor submanifold” in the space of minds, i.e. the type of mind to which all sufficiently advanced minds eventually converge.

One important development that did happen is my invention of the PreDCA alignment protocol, which critically depends on IBP. I consider PreDCA to be the most promising direction I know at present to solving alignment, and an important (informal) demonstration of the potential of the IBP formalism.

Discord recently introduced forum channels that closely approximate Zulip thread functionality, with a much more intuitive UI than Zulip.

My two main gripes with Discord are the default dark theme, and the lack of embedded links.

https://​​twitter.com/​​DickgirlsR/​​status/​​1599693592125394945

MSFT − 10%

INTEL − 10%

Nvidia − 15%

SMSN − 15%

Goog − 15%

ASML − 15%

TSMC − 20%

A steelman of the claim that a human has a utility function is that agents that make coherent decisions have utility functions, therefore we may consider the utility function of a hypothetical AGI aligned with a human. That is, assignment of utility functions to humans reduces to alignment, by assigning the utility function of an aligned AGI to a human.

I think this is still wrong, because of goodhart scope of AGIs and corrigibility of humans. Agent’s goodhart scope is the space of situations where it has good proxies for its preference. An agent with decisions governed by a utility function can act in arbitrary situations, it always has good proxies for its utility function. Logical uncertainty doesn’t put practical constraints on its behavior. But for an aligned AGI that seems unlikely, CEV seems complicated and possible configurations of matter superabundant, therefore there are always intractable possibilities outside the current goodhart scope. So it can at best be said to have a utility function over its goodhart scope, not over all physically available possibilities. Thus the only utility function it could have is itself a proxy for some preference that’s not in practice a utility function, because the agent can never actually make decisions according to a global utility function. Conversely, any AGI that acts according to a global utility function is not aligned, because its preference is way too simple.

Corrigibility is in part modification of agent’s preference based on what happens in environment. The abstraction of an agent usually puts its preference firmly inside its boundaries, so that we can consider the same agent, with the same preference, placed in an arbitrary environment. But a corrigible agent is not like that, its preference depends on environment, and in the limit it’s determined by its environment, not just by the agent. Environment doesn’t just present the situations for an agent to choose from, it also influences the way it’s making its decisions. So it becomes impossible to move a corrigible agent to a different environment while preserving its preference, unless we package its whole original environment as part of the agent that’s being moved to a new environment.

Humans are not at all classical agent abstractions that carry the entirety of their preference inside their heads, they are eminently corrigible, their preference depends on environment. As a result, an aligned AGI must be corrigible not just temporarily because it needs to pay attention to humans to grow up correctly, but permanently, because its preference must also continually incorporate the environment, to remain the same kind of thing as human preference. Thus even putting aside logical uncertainty that keeps AGI’s goodhart scope relatively small, an aligned AGI can’t have a utility function because of observational/​indexical uncertainty, it doesn’t know everything in the world (including the future) and so doesn’t have the data that defines its aligned preference.

In a book by Jeremy Siegel, he gives you the option of investing in an oil company vs IBM back in the very old days.

I do not remember the details, I think it is this book https://​​www.amazon.co.uk/​​Stocks-Long-Run-Definitive-Investment/​​dp/​​0071800514

The Oil stock beats IBM by a very large margin over several decades with dividends reinvested.

If you are doing this for investment returns then valuation and stable business is what matters.

Concept Dictionary.

Concepts that I intend to use or invoke in my writings later, or are parts of my reasoning about AI risk or related complex systems phenomena.

Thank you so much for the excellent and insightful post on mechanistic models, Evan!

My hypothesis is that the difficulty of finding mechanistic models that consistently make accurate predictions is likely due to the agent-environment system’s complexity and computational irreducibility. Such agent-environment interactions may be inherently unpredictable “because of the difficulty of pre-stating the relevant features of ecological niches, the complexity of ecological systems and [the fact that the agent-ecology interaction] can enable its own novel system states.”

Suppose that one wants to consistently make accurate predictions about a computationally irreducible agent-environment system. In general, the most efficient way to do so is to run the agent in the given environment. There are probably no shortcuts, even via mechanistic models.

For dangerous AI agents, an accurate simulation box of the deployment environment would be ideal for safe empiricism. This is probably intractable for many use cases of AI agents, but computational irreducibility implies that methods other than empiricism are probably even more intractable.

Please read my post “The limited upside of interpretability” for a detailed argument. It would be great to hear your thoughts!

This is very upsetting to me.

1. People would start using big words they don’t understand or use uncommon synonyms when a small common word would do. I hate it when people do this trying to sound smart. The archetypical example of this is Kingpin in the Marvel shows, who I genuinely can not stand. More people sounding like midwit try-hards does not lead to a better world.

2. Increased neologism. They’re funny, but decrease the quality of communication for everyone involved.