mishka

Karma: 827

mishka 10 May 2024 1:26 UTC
14 points
1
on: AI #63: Introducing Alpha Fold 3
This also points out that Arena tells you what model is Model A and what is Model B. That is unfortunate, and potentially taints the statistics.

No, https://chat.lmsys.org/ says this:
- Ask any question to two anonymous models (e.g., ChatGPT, Claude, Llama) and vote for the better one!
- You can chat for multiple turns until you identify a winner.
- Votes won’t be counted if model identities are revealed during the conversation.
So one can choose to know the names of the models one is talking with, but then one’s votes will not be counted for the statistics.

mishka 6 May 2024 21:17 UTC
1 point
0
on: Rapid capability gain around supergenius level seems probable even without intelligence needing to improve intelligence

Nobody currently knows how to align strongly superhumanly smart AIs to human interests, and we need way more time to solve this problem. Making incremental progress on AI capabilities is shortening the timeline we have left to figure out how to align AI and is thus making human extinction more likely. Thus by far the best action is to stop advancing AI capabilities.

It seems that not much research is done into studying invariant properties of rapidly self-modifying ecosystems. At least, when I did some search and also asked here a few months ago, not much came up: https://www.lesswrong.com/posts/sDapsTwvcDvoHe7ga/what-is-known-about-invariants-in-self-modifying-systems.

It’s not possible to have a handle on the dynamics of rapidly self-modifying ecosystems without better understanding how to think about properties conserved during self-modification. And ecosystems with rapidly increasing capabilities will be strongly self-modifying.

However, any progress in this direction is likely to be dual-use. Knowing how to think about self-modification invariants is very important for AI existential safety and is also likely to be a strong capability booster.

This is a very typical conundrum for AI existential safety. We can try to push harder to make sure that the research into invariant properties of self-modifying (eco)systems is an active research area again, but the likely side-effect of better understanding properties of potentially fooming systems is making it easier to bring these systems into existence. And we don’t have good understanding of proper ways to handle this kind of situations (although the topic of dual-use is discussed here from time to time).

mishka 4 May 2024 7:32 UTC
5 points
4
in reply to: O O’s comment on: William_S’s Shortform
No, OpenAI (assuming that it is a well-defined entity) also uses a probability distribution over timelines.

(In reality, every member of its leadership has their own probability distribution, and this translates to OpenAI having a policy and behavior formulated approximately as if there is some resulting single probability distribution).

The important thing is, they are uncertain about timelines themselves (in part, because no one knows how perplexity translates to capabilities, in part, because there might be difference with respect to capabilities even with the same perplexity, if the underlying architectures are different (e.g. in-context learning might depend on architecture even with fixed perplexity, and we do see a stream of potentially very interesting architectural innovations recently), in part, because it’s not clear how big is the potential of “harness”/”scaffolding”, and so on).

This does not mean there is no political infighting. But it’s on the background of them being correctly uncertain about true timelines...

Compute-wise, inference demands are huge and growing with popularity of the models (look how much Facebook did to make LLama 3 more inference-efficient).

So if they expect models to become useful enough for almost everyone to want to use them, they should worry about compute, assuming they do want to serve people like they say they do (I am not sure how this looks for very strong AI systems; they will probably be gradually expanding access, and the speed of expansion might depend).

mishka 4 May 2024 3:28 UTC
5 points
5
in reply to: O O’s comment on: William_S’s Shortform
I think having a probability distribution over timelines is the correct approach. Like, in the comment above:

I think I’m more likely to be better calibrated than any of these opinions, because most of them don’t seem to focus very much on “hedging” or “thoughtful doubting”, whereas my event space assigns non-zero probability to ensembles that contain such features of possible futures (including these specific scenarios).

mishka 4 May 2024 1:27 UTC
3 points
2
in reply to: JenniferRM’s comment on: William_S’s Shortform

However, none of them talk about each other, and presumably at most one of them can be meaningfully right?

Why at most one of them can be meaningfully right?

Would not a simulation typically be “a multi-player game”?

(But yes, if they assume that their “original self” was the sole creator (?), then they would all be some kind of “clones” of that particular “original self”. Which would surely increase the overall weirdness.)

mishka 2 May 2024 19:12 UTC
1 point
0
in reply to: habryka’s comment on: AI #62: Too Soon to Tell
success

mishka 2 May 2024 18:25 UTC
2 points
0
on: AI #62: Too Soon to Tell
The “AI #61: Meta Trouble” has not been cross-posted to LessWrong, but here is the link to the original post: https://thezvi.wordpress.com/2024/04/25/ai-61-meta-trouble/

mishka 4 Apr 2024 22:32 UTC
5 points
1
on: AI #58: Stargate AGI
Thanks! I think your discussion of the new Meaning Alignment Institute publication (the substack post and the paper) in the Aligning a Smarter Than Human Intelligence is Difficult section is very useful.

I wonder if it makes sense to republish it as a separate post, so that more people see it...

mishka 1 Apr 2024 0:50 UTC
0 points
0
on: Is langchain overhyped?
People are certainly looking at the alternatives.

For example, DSPy (“demonstrate-search-predict”) looks very promising: https://github.com/stanfordnlp/dspy

(Their README contains Section 6 “FAQ”, which compares a variety of existing alternatives, including LangChain.)

mishka 29 Mar 2024 4:33 UTC
7 points
2
on: AI #57: All the AI News That’s Fit to Print

Emmett Shear continues his argument that trying to control AI is doomed

I think that a recent tweet thread by Michael Nielsen and the quoted one by Emmett Shear represent genuine progress towards making AI existential safety more tractable.

Michael Nielsen observes, in particular:

As far as I can see, alignment isn’t a property of an AI system. It’s a property of the entire world, and if you are trying to discuss it as a system property you will inevitably end up making bad mistakes

Since AI existential safety is a property of the whole ecosystem (and is, really, not too drastically different from World existential safety), this should be the starting point, rather than stand-alone properties of any particular AI system.

Emmett Shear writes:

Hopefully you’ve validated whatever your approach is, but only one of these is stable long term: care. Because care can be made stable under reflection, people are careful (not a coincidence, haha) when it comes to decisions that might impact those they care about.

And Zvi responds

Technically I would say: Powerful entities generally caring about X tends not to be a stable equilibrium, even if it is stable ‘on reflection’ within a given entity. It will only hold if caring more about X provides a competitive advantage against other similarly powerful entities, or if there can never be a variation in X-caring levels between such entities that arises other than through reflection, and also reflection never causes reductions in X-caring despite this being competitively advantageous. Also note that variation in what else you care about to what extent is effectively variation in X-caring.

Or more bluntly: The ones that don’t care, or care less, outcompete the ones that care.

Even the best case scenarios here, when they play out the ways we would hope, do not seem all that hopeful.

That all, of course, sets aside the question of whether we could get this ‘caring’ thing to operationally work in the first place. That seems very hard.

Let’s now consider this in light of what Michael Nielsen is saying.

I am going to only consider the case where we have plenty of powerful entities with long-term goals and long-term existence which care about their long-term goals and long-term existence. This seems to be the case which Zvi is considering here, and it is the case we understand the best, because we also live in the reality with plenty of powerful entities (ourselves, some organizations, etc) with long-term goals and long-term existence. So this is an incomplete consideration: it only includes the scenarios where powerful entities with long-term goals and long-terms existence retain a good fraction of overall available power.

So what do we really need? What are the properties we want the World to have? We need a good deal of conservation and non-destruction, and we need the interests of weaker, not the currently most smart or most powerful members of the overall ecosystem to be adequately taken into account.

Here is how we might be able to have a trajectory where these properties are stable, despite all drastic changes of the self-modifying and self-improving ecosystem.

An arbitrary ASI entity (just like an unaugmented human) cannot fully predict the future. In particular, it does not know where it might eventually end up in terms of relative smartness or relative power (relative to the most powerful ASI entities or to the ASI ecosystem as a whole). So if any given entity wants to be long-term safe, it is strongly interested in the ASI society having general principles and practices of protecting its members on various levels of smartness and power. If only the smartest and most powerful are protected, then no entity is long-term safe on the individual level.

This might be enough to produce effective counter-weight to unrestricted competition (just like human societies have mechanisms against unrestricted competition). Basically, smarter-than-human entities on all levels of power are likely to be interested in the overall society having general principles and practices of protecting its members on various levels of smartness and power, and that’s why they’ll care enough for the overall society to continue to self-regulate and to enforce these principles.

This is not yet the solution, but I think this is pointing in the right direction...

mishka 28 Mar 2024 18:21 UTC
3 points
0
in reply to: Maciej Satkiewicz’s comment on: Towards White Box Deep Learning
Thanks, this is very interesting.

I wonder if this approach is extendable to learning to predict the next word from a corpus of texts...

The first layer might perhaps still be embedding from words to vectors, but what should one do then? What would be a possible minimum viable dataset?

Perhaps, in the spirit of PoC of the paper, one might consider binary sequences of 0s and 1s, and have only two words, 0 and 1, and ask what would it take to have a good predictor of the next 0 or 1 given a long sequence of those as a context. This might be a good starting point, and then one might consider different examples of that problem (different examples of (sets of) sequences of 0 and 1 to learn from).

mishka 28 Mar 2024 4:49 UTC
6 points
3
on: Towards White Box Deep Learning
This looks interesting, thanks!

This post could benefit from an extended summary.

In lieu of such a summary, in addition to the abstract

This paper introduces semantic features as a candidate conceptual framework for building inherently interpretable neural networks. A proof of concept model for informative subproblem of MNIST consists of 4 such layers with the total of 5K learnable parameters. The model is well-motivated, inherently interpretable, requires little hyperparameter tuning and achieves human-level adversarial test accuracy—with no form of adversarial training! These results and the general nature of the approach warrant further research on semantic features. The code is available at this https URL

I’ll quote a paragraph from Section 1.2, “The core idea”

This paper introduces semantic features as a general idea for sharing weights inside a neural network layer. [...] The concept is similar to that of “inverse rendering” in Capsule Networks where features have many possible states and the best-matching state has to be found. Identifying different states by the subsequent layers gives rise to controlled dimensionality reduction. Thus semantic features aim to capture the core characteristic of any semantic entity—having many possible states but being at exactly one state at a time. This is in fact a pretty strong regularization. As shown in this paper, choosing appropriate layers of semantic features for the [Minimum Viable Dataset] results in what can be considered as a white box neural network.

mishka 22 Mar 2024 16:21 UTC
1 point
0
on: Benchmarking LLM Agents on Kaggle Competitions
Is there a simple way to run against a given Kaggle competition after that particular competition is over?

These are reasonable benchmarks, but is there a way to make them less of a moving target, so that the ability to run against a given competition extends into the future?

mishka 21 Mar 2024 13:14 UTC
3 points
0
in reply to: Mitchell_Porter’s comment on: On the Gladstone Report
When I ponder all this I usually try to focus on the key difficulty of AI existential safety. ASIs and ecosystems of ASIs normally tend to self-modify and self-improve rapidly. Arbitrary values and goals are unlikely to be preserved through radical self-modifications.

So one question is: what are values and goals which might be natural, what are values and goals many members of the ASI ecosystem including entities which are much smarter and much more powerful than unaugmented humans might be naturally inclined to preserve through drastic self-modifications, and which might also have corollaries which are good from viewpoints of various human values and interests?

Basically, values and goals which are likely to be preserved through drastic self-modifications are probably non-anthropocentric, but they might have sufficiently anthropocentric corollaries, so that even the interests of unaugmented humans are protected (and also so that humans who would like to self-augment or to merge with AI systems can safely do so).

So what do we really need? We need a good deal of conservation and non-destruction, and we need the interests of weaker, not the currently most smart or most powerful members of the overall ecosystem to be adequately taken into account.

What might the road towards adoption of those values (of preservation and protection of weaker entities and their interests) and incorporation of those kinds of goals be, so that these values and goals are preserved as the ASI ecosystem and many of its members self-modify drastically? One really needs the situation where many entities on varying levels of smartness and power care a lot about those values and goals.

An arbitrary ASI entity (just like an unaugmented human) cannot fully predict the future. In particular, it does not know where it might eventually end up in terms of relative smartness or relative power (relative to the most powerful ASI entities or to the ASI ecosystem as a whole). So if any given entity wants to be long-term safe, it is strongly interested in the ASI society having general principles and practices of protecting its members on various levels of smartness and power. If only the smartest and most powerful are protected, then no entity is long-term safe on the individual level.

This might be a reasonable starting point towards having values and goals which are likely to be preserved through drastic self-modifications and self-improvements and which are likely to imply good future for unaugmented humans and for the augmented humans as well.

Perhaps, we can nudge things towards making this kind of future more likely. (The alternatives are bleak, unrestricted drastic competition and, perhaps, even warfare between supersmart entities which would probably end badly not just for us, but for the ASI ecosystem as well)...

mishka 21 Mar 2024 2:27 UTC
3 points
0
on: On the Gladstone Report

The worlds were artificial superintelligence (ASI) is coming very soon with only roughly current levels of compute, and where ASI by default goes catastrophically badly, are not worlds I believe we can afford to save.

For those kinds of worlds, we probably can’t afford to save them by heavy-handed measures (meaning that not only the price is too much to pay, given that it would have to be paid in all other worlds too, but also that it would not be possible to avoid evasion of those measures, if it is already relatively easy to create ASI).

But this does not mean that more light-weight measures are hopeless. For example, when we look at Ilya’s thought process he has shared with us last year, what he has been trying to ponder have been various potentially feasible relatively light-weight measures to change this default of ASI going catastrophically badly to more acceptable outcomes: Ilya Sutskever’s thoughts on AI safety (July 2023): a transcript with my comments.

More people should do more thinking of this kind. AI existential safety is a pre-paradigmatic field of study. A lot of potentially good approaches and possible non-standard angles have not been thought of at all. A lot of potentially good approaches and possible non-standard angles have been touched upon very lightly and need to be explored further. The broad consensus in the “AI existential safety” community seems to be that we currently don’t have good approaches we can comfortably rely upon for future ASI systems. This should encourage more people to look for completely novel approaches (and in order to cover the worlds where timelines are short, some of those approaches have to be simple rather than ultra-complicated, otherwise they would not be ready by the time we need them).

In particular, it is certainly true that

A smarter thing that is more capable and competitive and efficient and persuasive being kept under indefinite control by a dumber thing that is less capable and competitive and efficient and persuasive is a rather unnatural and unlikely outcome. It should be treated as such until proven otherwise.

So it makes sense spending more time brainstorming the approaches which would not require “a smarter thing being kept under indefinite control by a dumber thing”. What might be a relatively light-weight intervention which would avoid ASI going catastrophically badly without trying to rely on this unrealistic “control route”? I think this is underexplored. We tend to impose too many assumptions, e.g. the assumption that ASI has to figure out a good approximation to our Coherent Extrapolated Volition and to care about that in order for things to go well. People might want to start from scratch instead. What if we just have a goal of having good chances for a good trajectory conditional on ASI, and we don’t assume that any particular feature of earlier approaches to AI existential safety is a must-have, but only this goal is a must-have. Then this gives one a wider space of possibilities to brainstorm: what other properties besides caring about a good approximation to our Coherent Extrapolated Volition might be sufficient for the trajectory of the world with ASI to likely be good from our point of view?

mishka 19 Mar 2024 22:15 UTC
10 points
0
on: Mechanism for feature learning in neural networks and backpropagation-free machine learning models
I don’t see a non-paywalled version of this paper, but here are two closely related preprints by the same authors:

“Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn features”, https://arxiv.org/abs/2212.13881

“Mechanism of feature learning in convolutional neural networks”, https://arxiv.org/abs/2309.00570

mishka 19 Mar 2024 21:50 UTC
1 point
0
on: Invitation to the Princeton AI Alignment and Safety Seminar
Paul Christiano: “Catastrophic Misalignment of Large Language Models”

Talk recording: https://www.youtube.com/watch?v=FyTb81SS_cs (no transcript or chat replay yet)

Talk page: https://pli.princeton.edu/events/2024/catastrophic-misalignment-large-language-models

mishka 15 Mar 2024 15:59 UTC
3 points
0
on: AI #55: Keep Clauding Along

Sam Altman watch: He invests $180 million into Retro Bio to fight aging. I have no idea if they can execute, but this kind of investment is great and bodes many good things. Kudos.

Note that this looks like a relatively old investment, it’s not another $180 million, it looks like the $180 million you might have heard about a while ago… (Gwern also thinks so.)

mishka 15 Mar 2024 7:04 UTC
3 points
0
on: AI #55: Keep Clauding Along

Jack Clark: Salary for tech staff of EU AI Office (develop and assess evaluations for gen models like big LLMs) is… 3,877 – 4,387 Euros a month, aka $4242 – $4800 USD.

Most tech co internships $8k+ a month.

He seems to be comparing an EU AI Office job with American tech salaries.

I think people tend to underestimate how much lower tech salaries tend to be in EU compared to American standards.

(Of course, things do strongly depend on the location as well, both within the EU and with the US).

mishka 15 Mar 2024 2:19 UTC
1 point
0
in reply to: Haiku’s comment on: What could a policy banning AGI look like?
Like I said, if we try to apply forceful measures we might delay it for some time (at the price of people aging and dying from old age and illnesses to the tune of dozens of millions per year due to the continuing delay; but we might think that the risks are so high that the price is worth it, and we might think that the price of everyone who is alive today eventually dying of old age is worth it, although some of us might disagree with that and might say that taking the risk of foom is better than the guaranteed eventual death of old age or other illness; there is a room for disagreement on which of these risks it is preferable to choose).

But if we are talking about avoiding foom indefinitely, we should start with asking ourselves, how easy or how difficult is it to achieve. How long before a small group of people equipped with home computers can create foom?

And the results of this analysis are not pretty. Computers are ultimate self-modifying devices, they can produce code which programs them. Methods to produce computer code much better than we do it today are not all that difficult, they are just in the collective cognitive blindspot, like backpropagation was for a long time, like ReLU activations were for decades, like residual connectivity in neural machines was in the collective cognitive blindspot for unreasonably long time. But this state of those enhanced methods of producing new computer code being relatively unknown would not last forever.

And just like backpropagation, ReLU, or residual connections, these methods are not all that difficult, it’s not like if a single “genius” who might discover them would refrain from sharing them, they would remain unknown. People keep rediscovering and rediscovering those methods, they are not that tricky (backpropagation was independently discovered more than 10 times between 1970 and 1986, before people stopped ignoring it and started to use it).

It’s just the case that the memetic fitness of those methods is currently low, just like memetic fitness of backpropagation, ReLU, and residual connections used to be low in the strangest possible ways. But this would not last, the understanding of how to have competent self-improvement in small-scale software on ordinary laptops will gradually form and proliferate. At the end of the day, we’ll end up having to either phase out universal computers (at least those which are as powerful as our home computers today) or to find ways to control them very tightly, so that people are no longer free to program their own computers as they see fit.

Perhaps, humans will chose to do that, I don’t know. Nor do I know whether they would succeed in a Butlerian jihad of this kind, or whether some of the side effects of trying to get rid of computers would become X-risks themselves. In any case, in order to avoid foom indefinitely, people will have to take drastic, radical measures which would make everyone miserable, would kill a lot of people, and might create other existential risks.

I think it’s safer if the most competent leading group tries to go ahead, that our eventual chances of survival are higher along this path, compared to the likely alternatives.

I do think that the risks on the alternative paths are very high too; the great powers are continuing to inch towards major nuclear confrontation; we are enabling more and more people to create diverse super-covid-like artificial pandemics with 30% mortality or more; things are pretty bad in terms of major risks this civilization is facing; instead of asking “what’s your P(doom)” we should be asking people, “what’s your P(doom) conditional on foom and what’s your P(doom) conditional on no foom happening”. My P(doom) is not small, but is it higher conditional on foom, than conditional on no foom? I don’t know...