Aprillion (Peter Hozák)
Sounds like an AI would be searching for Pareto optimality to satisfy multiple (types of) objectives in such a case—https://en.wikipedia.org/wiki/Multi-objective_optimization ..
The `nearcasting` links point to an edit version of the page. So for us mortals who don’t have edit access, the page is https://www.lesswrong.com/posts/Qo2EkG3dEMv8GnX8d/ai-strategy-nearcasting
“Can’t we test whether the code works without knowing anything about programming?”
Knowing what to test to reliably decrease uncertainty about “whether the code works” includes knowing “a fair bit” about software engineering.
I agree with the distinction that being a programmer is not the only way how to know about programming, many hiring managers are not programmers themselves, they just have to know a fair bit about software engineering.
These google drive links don’t work for me .. I get redirected to Bin (when logged in to google) or HTTP 404 not found in private window. Could you please generate a link for sharing?
For a programmer who is not into symbolic math, would you say that following summary is accurate enough or did I miss some intuition here:
if an overparametrized network has linear dependency between paramaters, it can perform as if it was an underparametrized network
but the trick is that a flat basin is easier to reach by SGD or similar optimizations processes than if we had to search small targets
hm, I gave it some time, but still confused .. can you name some types of reward that humans have?
I see. These are implemented differently in humans, but my intuition about the implementation details is that “reward signal” as a mathematically abstract object can be modeled by single value even if individual components are physically implemented by different mechanisms, e.g. an animal could be modeled as if was optimizing for a pareto optimum between a bunch of normalized criteria.
reward = S(hugs) + S(food) + S(finishing tasks) + S(free time) - S(pain) ...
People spend their time cooking, risk cutting fingers, in order to have better food and build relationships. But no one would want to get cancer to obtain more hugs, presumably not even to increase number of hugs from 0 to 1, so I don’t feel human rewards are completely independent magisteria, there must be some biological mechanism to integrate the different expected rewards and pains into decisions.
Spending energy on computation of expected value can be included in the model, we might decide that we would get lower reward if we overthink the current decision and that would be possible to model as included in the one “reward signal” in theory, even though it would complicate predictability of humans in practice (however, it turns out that humans can be, in fact, hard to predict, so I would say this is a complication of reality, not a useless complication in the model).
Allostasis is a more biologically plausible explanation of “what a brain does” than homeostasis, but to your point: I do think optimizing for happiness and doing kinda-homeostasis are “just the same somehow”.
I have a slightly circular view that the extension of happiness exists as an output of a network with 86 billion neurons and 60 trillion connections, and that it is a thing that the brain can optimize for. Even if the intension of happiness as defined by a few English sentences is not the thing, and even if optimization for slightly different things would be very fragile, the attractor of happiness might be very small and surrounded by dystopian tar pits, I do think it is something that exists in the real world and is worth searching for.
Though if we cannot find any intension that is useful, perhaps other approaches to AI Alignment and not the “search for human happiness” will be more practical.
[EDITED]: good point, no idea what they meant with “uniform” distribution, the realization for me was about the connection that I can often assume errors are normally distributed, thus L2 is often the obvious choice
see your β there? you assume that people remember to “control for bias” before they apply tools that assume Gaussian error
that is indeed what I should have remembered about the implications of “we can often assume approximately normal distribution” from my statistics course ~15 years ago, but then I saw people complaining about sensitivity to outliers in 1 direction and I failed to make a connection until I dug deeper into my reasoning
I mostly fixed the page by removing quotes from links (edited as markdown in VS Code, 42 links were like [](”...”) and 64 quotes were double-escaped \”) … feel free to double check (I also sent feedback to moderators, maybe they want to check for similar problems on other pages on DB level)
TypeError: Comparing different “solutions”.
How do I know that I generated a program that halts?
a) I can prove to myself that my program halts ⇒ the solution consists of both the program and the proof ⇒ the verification problem is a sub-problem of the generation problem.
b) I followed a trusted process that is guaranteed to produce valid solutions ⇒ the solution consists of both the program and the history that generated the proof of the process ⇒ if the history is not shared between the 2 problems, then you redefined “verification problem” to include generation of all of the necessary history, and that seems to me like a particularly useless point of view (for the discussion of P vs NP, not useless in general).
In the latter point of view, you could say:
Predicate: given a set of numbers, is the first the sum of the latter 2?
Generation problem: provide an example true solution: “30 and prime factors of 221”
Verification problem: verify that 30 is the sum of prime factors of 221
WTF does that have to say about P vs NP? ¯\_(ツ)_/¯
See the
Humor
tag ¯\_(ツ)_/¯
To be continued in the form of a science fiction story Unnatural Abstractions.
No idea about original reasons, but I can imagine a projected chain of reasoning:
there is a finite number of conjunctive obstacles
if a single person can only think of a subset of obstacles, they will try to solve those obstacles first, making slow(-ish) progress as they discover more obstacles over time
if a group shares their lists, each individual will become aware of more obstacles and will be able to solve more of them at once, potentially making faster progress
Staying in meta-level, if AGI weren’t going to be created “by the ML field”, would you still believe problems on your list cannot possibly be solved within 6-ish months if companies would throw $1b at each of those problems?
Even if competing groups of humans augmented by AI capabilities existing “soon” were trying to solve those problems with combined tools from inside and outside ML field, the foreseeable optimization pressure is not enough for those foreseeable collective agents to solve those known-known and known-unknown problems that you can imagine?
Building a tunnel from 2 sides is the same thing even if those 2 sides don’t see each other initially. I believe some, but not all, approaches will end up seeing each other, that it’s not a bad sign if we are not there yet.
Since we don’t seem to have time to build 2 “tunnels” (independent solutions to alignment), a bad sign would be if we could prove all of the approaches are incompatible with each other, which I hope is not the case.
I agree with the explicitly presented evidence and reasoning steps, but one implied prior/assumption seems to me so obscenely wrong (compared to my understanding about social reality) that I have to explain myself before making a recommendation. The following statement:
“stacking” means something like, quadrupling the size of your team of highly skilled alignment researchers lets you finish the job in ~1/4 of the time
implies a possibility that approximately neg-linear correlation between number of people and time could exist (in multidisciplinary software project management in particular and/or in general for most collective human endeavors). The model of Nate that I have in my mind believes that reasonable readers ought to believe that:
as a prior, it’s reasonable to expect more people will finish a complex task in less time than fewer people would, unless we have explicit reasons to predict otherwise
Brooks’s law is a funny way to describe delayed projects with hindsight, not a powerful predictor based on literally every single software project humankind ever pursued
I am making a claim about the social norm that it’s socially OK to assume other people can believe in linear scalability, not a belief whether other people actually believe that 4x the people will actually finish in 1⁄4 time by default.
Individually, we are well calibrated to throw a TypeError at the cliche counterexamples to the linear scalability assumption like “a pregnant woman delivers one baby in 9 months, how many …”.
And professional managers tend to have an accurate model of applicability of this assumption, individually they all know how to create the kind of work environment that may open the possibility for time improvements (blindly quadrupling the size of your team can bring the project to a halt or even reverse the original objective, more usually it will increase the expected time because you need to lower other risks, and you have to work very hard for a hope of 50% decrease in time—they are paid to believe in the correct model of scalability, even in cases when they are incentivized to say more optimistic professions of belief in public).
Let’s say 1000 people can build a nuclear power plant within some time unit. Literally no one will believe that one person will build it a thousand times slower or that a million people will build it a thousand times faster.
I think it should not be socially acceptable to say things that imply that other people can assume that others might believe in linear scalability for unprecedented large complex software projects. No one should believe that only one person can build Aligned AGI or that a million people can build it thousand times faster than a 1000 people. Einstein and Newton were not working “together”, even if one needed the other to make any progress whatsoever—the nonlinearity of “solving gravity” is so qualitatively obvious, no one would even think about it in terms of doubling team size or halving time. That should be the default, a TypeError law of scalability.
If there is no linear scalability by default, Alignment is not an exception to other scalability laws. Building unaligned AGI, designing faster GPUs, physically constructing server farms, or building web apps … none of those are linearly scalable, it’s always hard management work to make a collective human task faster when adding people to a project.
Why is this a crux for me? I believe the incorrect assumption leads to rationally-wrong emotions in situations like these:
Also, I’ve tried a few different ways of getting researchers to “stack” (i.e., of getting multiple people capable of leading research, all leading research in the same direction, in a way that significantly shortens the amount of serial time required), and have failed at this.
Let me talk to you (the centeroid of my models of various AI researchers, but not any one person in particular). You are a good AI researcher and statistically speaking, you should not expect you to also be an equally good project manager. You understand maths and statistically speaking, you should not expect you to also be equally good at social skills needed to coordinate groups of people. Failling at a lot of initial attempts to coordinate teams should be the default expectation—not one or two attempts and then you will nail it. You should expect to fail more ofthen than the people who are getting the best money in the world for aligning groups of people towards a common goal. If those people who made themselves successful in management initially failed 10 times before they became billionaires, you should expect to fail more times than that.
Recommendation
You can either dilute your time by learning both technical and social / management skills or you can find other experts to help you and delegate the coordination task. You cannot solve Alignment alone, you cannot solve Alignment without learning, and you cannot learn more than one skill at a time.
The surviving worlds look like 1000 independent alignment ideas, each pursued by 100 different small teams. Some of the teams figured out how to share knowledge between some of the other teams and connect one or two ideas and merge teams iff they figure out explicit steps how to shorten time by merging teams.
We don’t need to “stack”, we need to increase the odds of a positive black swan.
Yudkowsky, Christiano, and the person who has the skills to start figuring out the missing piece to unify their ideas are at least 10,000 different people.
if everyone followed the argmax approach I laid out here. Are there any ways they might do something you think is predictably wrong?
While teamwork seems to be assumed in the article, I believe it’s worth spelling out explicitly that argmaxing for a plan with highest marginal impact might mean joining and/or building a team where the team effort will make the most impact, not optimizing for highest individual contribution.
Spending time to explain why a previous research failed might help 100 other groups to learn from our mistake, so it could be more impactful than pursuing the next shiny idea.
We don’t want to optimize for the naive feeling of individual marginal impact, we want to keep in mind the actual goal is to make an Aligned AGI.
oh, I didn’t realize there was this event yesterday, I wrote an ai-safety inspired short story independently 😅 if anyone would wish to comment, feel free to leave me a github issue
https://peter.hozak.info/fiction/heat_death/prologue