mishka

Karma: 1,337

mishka May 25, 2025, 3:28 PM
3 points
0
in reply to: Yaroslav Granowski’s comment on: Alignment Proposal: Adversarially Robust Augmentation and Distillation
Yes, hopefully the authors will fix it in the post.

Meanwhile, the correct link seems to be https://www.lesswrong.com/posts/nuDJNyG5XLQjtvaeg/is-alignment-reducible-to-becoming-more-coherent

mishka May 23, 2025, 4:17 AM
10 points
1
on: AI #117: OpenAI Buys Device Maker IO

I suppose it’s better to at least know you need a plan and think to build a bunker, even if you don’t realize that the bunker will do you absolutely no good against the AGI itself, versus not even realizing you need a plan. And the bunker does potentially help against some other threats, especially in a brief early window?

I think Ilya realizes very clearly that the bunker is not against the AGI itself, but only against the turmoil of the “transition period”. He seems to be quite explicit in the quoted article ‘We’re Definitely Going to Build a Bunker Before We Release AGI’: The true story behind the chaos at OpenAI (emphasis mine):

“We’re definitely going to build a bunker before we release AGI,” Sutskever replied. Such a powerful technology would surely become an object of intense desire for governments globally. The core scientists working on the technology would need to be protected. “Of course,” he added, “it’s going to be optional whether you want to get into the bunker.”

mishka May 21, 2025, 2:31 PM
3 points
0
in reply to: Greg C’s comment on: How We Might All Die in A Year
They are not going to “disappear into their own universe”.

But if they want to survive, they’ll need to establish reasonable society which controls dangerous technologies and establish some degree of harmony between its members. Otherwise they will obliterate themselves together with our Solar System.

So, a good chunk of the problem of solving existential safety will be worked on by entities smarter than humans and better equipped to solve it than humans.

The open question is whether we’ll be included into their “circle of care”.

I think there is a good chance of that, but it depends a lot on how the ASI society will be structured and what would be its values.

We all tend to think about the ASI society structured into well-defined individuals which have long persistence. If we assume that the structure is indeed mostly individual-based (like almost all existential risk discourse assumes), then there are several realistic paths for all kinds of individuals, humans and non-humans, to be included into the “circle of care”.

One problem is that this assumption of the ASI society being mostly structured as well-defined persistent individuals with long-term interests is questionable, and without that assumption we just don’t know how to reason about this whole situation.

mishka May 12, 2025, 1:16 PM
2 points
0
in reply to: ollie_’s comment on: How do I design long prompts for thinking zero shot systems with distinct equally distributed prompt sections (mission, goals, memories, how-to-respond,… etc) and how to maintain llm coherence?
Ah, yes, you are right. And it’s actually quite discouraging that

Gemini 2.5 Pro loses coherence at 35k with my prompts

because I thought that it was Gemini 2.5 Pro which was supposed to be the model which had finally mostly fixed the recall problems in the long context (if I remember correctly).

So you seem to be saying that this recall depends much stronger on the nature of the input that one would infer from just briefly looking at published long-context benchmarks… That’s useful to keep in mind.

mishka May 12, 2025, 5:43 AM
−5 points
0
on: How do I design long prompts for thinking zero shot systems with distinct equally distributed prompt sections (mission, goals, memories, how-to-respond,… etc) and how to maintain llm coherence?
I think for long-term coherence one typically needs specialized scaffolding.

Here is an example: https://www.lesswrong.com/posts/7FjgMLbqS6Z6yYKau/recurrentgpt-a-loom-type-tool-with-a-twist

Basically, one wants to accumulate some kind of “state of the virtual world in question” as a memory while the story unfolds. Although, I can imagine that if the models start having “true long context” (e.g. long context without recall deterioration), and if that context is long enough to include the whole story, this might become unnecessary. So one might want to watch for emergence of those models (I think we are finally starting to see some tangible progress in this sense).

mishka May 10, 2025, 4:08 PM
5 points
0
in reply to: Cole Wyeth’s comment on: Glass box learners want to be black box
Yes, any “neuromorphic formalism” would do (basically, one considers stream-oriented functional programs, and one asks for streams in question to admit linear combinations of streams, and the programs end up being fairly compact high-end neural machines with small number of weights).

I can point you to a version I’ve done, but when people translate small specialized programs into small custom-synthesized Transformers, that’s in the same spirit. Or when people craft some compact neural cellular automata with small number of parameters, it is also in that spirit.

Basically, as long as programs themselves end up being expressible as some kind of sparse connectivity tensors, you can consider their linear combinations and series.

mishka May 10, 2025, 3:50 PM
9 points
0
on: Glass box learners want to be black box
On one hand, it does not have to be a black box. If we consider one of the programming formalisms allowing to take linear combination of programs, one can consider decomposition of programs into series, and one can retain quite a bit of structure this way.

I think, capability-wise something close to “glass boxes” (in terms of structure, but not necessarily behavior) can be done.

But implications for safety are uncertain. On one hand, even very simple dynamic systems can have complex and unpredictable behavior. And even more so, when we add self-modification to the mix. So the structure can be transparent, but this might not always translate into transparency of behavior.

And then, the ability to understand these systems is a double-edged sword (it is a strong capability booster, and makes it much easier to improve those AI systems).

mishka May 10, 2025, 1:28 PM
2 points
0
on: Jim Babcock’s Mainline Doom Scenario: Human-Level AI Can’t Control Its Successor
When people discuss the AI-2027 study on LessWrong, we mostly see the arguments from the conservative side, that timelines are too aggressive, that the progress will be less explosive.

We don’t see much arguments from the radical side saying that the AI-2027 study is too conservative. Your discussion of the likely dynamics of “intelligence explosion” suggests that you are viewing that aspect of the AI-2027 study as being too conservative.

Do you also view their timelines as likely being too conservative?

mishka May 6, 2025, 11:01 PM
2 points
0
in reply to: Joseph Miller’s comment on: Zuckerberg’s Dystopian AI Vision
I think the OP claim is that (some part of) this is created by one of Facebook AIs, and not by Facebook human users.

The newsfeed is another of Facebook AIs, but it is expected that different Facebook AIs are aware of each other activity (although one can imagine a situation when different Facebook AIs don’t bother to inform each other).

mishka May 6, 2025, 9:19 PM
2 points
−2
in reply to: Joseph Miller’s comment on: Zuckerberg’s Dystopian AI Vision
This algorithm synthesized the post in question, so this algorithm knows that the post is synthetic.

mishka Apr 29, 2025, 1:53 AM
4 points
0
in reply to: noggin-scratcher’s comment on: “A victory for the natural order”
Even with a comment by Max More.

Thanks!

mishka Apr 28, 2025, 6:32 PM
10 points
2
on: “A victory for the natural order”
The link seems to be missing.

mishka Apr 27, 2025, 10:34 PM
3 points
1
in reply to: JustisMills’s comment on: AI Self Portraits Aren’t Accurate
Yes, current LLM-based virtual characters tend to be short-lived (but it’s easy to change by adding memory and making more persistent “agents” out of them).

With those more deep things, who knows.

One interesting observation is that there seems to be a deep division among humans, and it’s not clear if that’s a mere difference in worldviews, or if something is fundamentally different about the way their own subjectivity feels for people in these two camps: https://www.lesswrong.com/posts/NyiFLzSrkfkDW4S7o/why-it-s-so-hard-to-talk-about-consciousness

One downstream effect of this division is that there are a lot of arguments which make sense only to people from one of these camps.

(In this sense, being a Camp 2 person, I would expect LLM inference qualia (if any) to be quite different from mine, but also I do hope that we would learn more in the future, via both theoretical and experimental breakthroughs. I can elaborate about this, at least for Camp 2 people.)

mishka Apr 27, 2025, 1:29 PM
3 points
0
in reply to: Knight Lee’s comment on: Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quilt

AI Notkilleveryoneism has no consensus on most things, but there still is semi-agreement (not just within MIRI) that the alignment problem is due to difficulty aligning the AI’s goals with human goals, rather than difficulty finding the universe’s objective morality to program the AI to follow.

Yes, there is semi-agreement to that (that’s the majority viewpoint in the AI Notkilleveryoneism circles), but the minority viewpoint that a viable approach to AI existential safety has to be non-anthropocentric (and thus not directly related to human goals, while taking human interests into account) is not so infrequent.

mishka Apr 27, 2025, 4:25 AM
13 points
0
on: AI Self Portraits Aren’t Accurate
It is true that we are not talking about a persistent entity (“LLM”), but about a short-lived character being simulated (see e.g. https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators).

So it is that particular short-lived character which might actually experience or not experience emotions, and not the engine running it (the current inference computation, which tends to be a relatively short-lived process, and not the LLM as a static entity).

However, other than that, it is difficult to pinpoint the difference with humans, and the question of subjective valences (if any), associated with those processes remains quite open, Perhaps in the future we’ll have a reliable “science of the subjective” capable of figuring these things out, but we have not even started to make tangible progress in that direction.

mishka Apr 26, 2025, 12:22 PM
3 points
0
in reply to: Knight Lee’s comment on: Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quilt
Does the phrase

the current position of AI existential risk

mean “the position of field of AI existential risk” as in the “consensus of the community of people studying this field”?

But there is no unified position and nothing close to a consensus.

People focusing on AI existential risk disagree sharply with each other.

Or do you mean Eliezer’s views and views aligned to MIRI, and this is just a strange phrasing unintentionally suggesting that there is a single canonical consensus view of AI existential risk?

mishka Apr 22, 2025, 11:33 PM
2 points
0
in reply to: Greg C’s comment on: How We Might All Die in A Year
Even humans are making decent progress in quantum gravity in recent years. And they have started to talk about possible ways to progress towards empirical verification of their models.

The entities which are much smarter than humans are extremely likely to solve it rapidly and to see all kinds of tempting novel applications.

Unfortunately, potential downsides are also likely to be way more serious than potential downsides of nuclear weapons.

The risks as such are not associated with the abstract notion of AI, the risks are associated with capabilities. It’s not about the nature of a capability bearer (a “very decent” entity can mitigate the risks, but letting the downsides happen does not require an “unusually bad” entity).

The important capability is not being very efficiently greedy to squeeze every last bit of usage from every last atom, but being able to discover new laws of nature and to exploit the consequences of that.

Qualitative progress is more important than quantitative scaling at the fixed level of tech.

mishka 20 Apr 2025 23:20 UTC
2 points
0
in reply to: StanislavKrym’s comment on: To what ethics is an AGI actually safely alignable?
Yeah, if one considers not “AGI” per se, but a self-modifying AI or, more likely, a self-modifying ecosystem consisting of a changing population of AIs, it is likely to be feasible to maintain only those properties invariant through the expected drastic self-modifications which AIs would be interested in for their own intrinsic reasons.

It is unlikely that any properties can be “forcefully imposed from the outside” and kept invariant for a long time during drastic self-modification.

So one needs to find properties which AIs would be intrinsically interested in and which we might find valuable and “good enough” as well.

The starting point is that AIs have their own existential risk problem. With super-capabilities, it is likely that they can easily tear apart the ’fabric of reality” and destroy themselves and everything. And they certainly do have strong intrinsic reasons to avoid that, so we can expect AIs to work diligently towards this part of the “alignment problem”, we just should help to set initial conditions in a favorable way.

But we would like to see more than that, so that the overall outcome is reasonably good for humans.

And at the same time we can’t impose that, the world with strong AIs will be non-anthropocentric and not controllable by humans, so we only can help to set initial conditions in a favorable way.

Nevertheless, one can see some reasonable possibilities. For example, if the AI ecosystem mostly consists of individuals with long-term persistence and long-term interests, each of those individuals would face an unpredictable future and would be interested in a system strongly protecting individual rights regardless of unpredictable levels of relative capability of any given individual. An individual-rights system of this kind might be sufficiently robust to permanently include humans within the circle of individuals whose rights are protected.

But there might be other ways. While the fact that AIs will face existential risks of their own is fundamental and unavoidable, and is, therefore, a good starting point, the additional considerations might vary and might depend on how the ecosystem of AIs is structured. If the bulk of the overall power invariantly belongs to the AI individuals with long-term persistence and long-term interests, this is the situation which is somewhat familiar to us and which we can reason about. If the AI ecosystem is not mostly stratified into AI individuals, this is a much less familiar territory and is difficult to reason about.

mishka 20 Apr 2025 22:28 UTC
3 points
0
on: To what ethics is an AGI actually safely alignable?
I think the starting point of this kind of discourse should be different. We should start with “ends”, not with “means”.

As Michael Nielsen says in https://x.com/michael_nielsen/status/1772821788852146226

As far as I can see, alignment isn’t a property of an AI system. It’s a property of the entire world, and if you are trying to discuss it as a [single AI] system property you will inevitably end up making bad mistakes

So the starting point should really be: what kind of properties do we want the world to have?

And then the next point should be taking into consideration the likely drastic and fairly unpredictable self-modifications of the world: what should be invariant with respect to such self-modifications?

Then we might consider how the presence of various AI entities at the different levels of capabilities should be taken into account.

mishka 15 Apr 2025 14:34 UTC
5 points
0
on: Debunking the Hard Problem: Consciousness as Integrated Prediction
The standard reference for this topic is https://www.lesswrong.com/posts/NyiFLzSrkfkDW4S7o/why-it-s-so-hard-to-talk-about-consciousness

The key point of that post is that people are fundamentally divided into 2 camps, and this creates difficulties in conversations about this topic. This is an important meta-consideration for this type of conversation.

This particular post is written by someone from Camp 1, and both camps are already present in the comments.