awenonian

Karma: 91

Things roll downhill

awenonian6 Dec 2022 15:27 UTC

19 points

0 comments1 min readLW link

awenonian 31 Mar 2022 21:50 UTC
9 points
in reply to: KatWoods’s comment on: Why Agent Foundations? An Overly Abstract Explanation
I doubt my ability to be entertaining, but perhaps I can be informative. The need for mathematical formulation is because, due to Goodhart’s law, imperfect proxies break down. Mathematics is a tool which is rigorous enough to get us from “that sounds like a pretty good definition” (like “zero correlation” in the radio signals example), to “I’ve proven this is the definition” (like “zero mutual information”).
The proof can get you from “I really hope this works” to “As long as this system satisfies the proof’s assumptions, this will work”, because the proof states it’s assumptions clearly, while “this has worked previously” could, and likely does, rely on a great number of unspecified commonalities previous instances had.
It gets precise and pedantic because it turns out that the things we often want to define for this endeavor are based on other things. “Mutual information” isn’t a useful formulation without a formulation for “information”. Similarly, in trying to define morality, it’s difficult to define what an agent should do in the world (or even what it means for an agent to do things in the world), without ideas of agency and doing, and the world. Every undefined term you use brings you further from a formulation you could actually use to create a proof.
In all, mathematical formulation isn’t the goal, it’s the prerequisite. “Zero correlation” was mathematically formalized, but that was not sufficient.

awenonian 7 Jul 2021 22:25 UTC
9 points
in reply to: Daniel V’s comment on: The Point of Trade
I feel like I might be being a little coy stating this, but I feel like “heterogeneous preferences” may not be as inadequate as it seems. At least, if you allow that those heterogeneous preferences are not only innate like taste preference for apples over oranges.
If I have a comparative advantage in making apples, I’m going to have a lot of apples, and value the marginal apple less than the marginal orange. I don’t think this is a different kind of “preference” than liking the taste of oranges better: Both base out in me preferring an orange to an apple. And so we engage in trade for exactly that reason. In fact, I predict we stop trading once I value the marginal apple more than the marginal orange (or you, vice versa), regardless of the state of my comparative advantage. (That is, in a causal sense, the comparative advantage is covered by my marginal value assignments. My comparative advantage may inform my value assignments, but once you know those, you don’t need to know if I still have a comparative advantage or not for the question “will I trade apples for oranges?”)
Comparative advantage is why trade is useful, but I don’t know if it’s really accurate to say that heterogeneous preferences are an “inadequate answer to why we trade.”

awenonian 28 Mar 2022 16:21 UTC
8 points
in reply to: johnswentworth’s comment on: Why Agent Foundations? An Overly Abstract Explanation
I understand the point of your dialog, but I also feel like I could model someone saying “This Alignment Researcher is really being pedantic and getting caught in the weeds.” (especially someone who wasn’t sure why these questions should collapse into world models and correspondence.)
(After all, the Philosopher’s question probably didn’t depend on actual apples, and was just using an apple as a stand-in for something with positive utility. So, the inputs of the utility functions could easily be “apples” (where an apple is an object with 1 property, “owner”. Alice prefers apple.owner=”alice” (utility(a): return int(a.owner==‘alice’)), and Bob prefers apple.owner=”bob”) To sidestep the entire question of world models, and correspondence.)
I suspect you did this because the half formed question about apples was easier to come up with than a fully formed question that would necessarily require engagement with world models, and I’m not even sure that’s the wrong choice. But this was the impression I got reading it.

awenonian 31 Mar 2021 20:23 UTC
8 points
in reply to: johnswentworth’s comment on: Core Pathways of Aging
I’m still confused. My biology knowledge is probably lacking, so maybe that’s why, but I had a similar thought to dkirmani after reading this: “Why are children born young?” Given that sperm cells are active cells (which should give transposons opportunity to divide), why do they not produce children with larger transposon counts? I would expect whatever sperm divide from to have the same accumulation of transposons that causes problems in the divisions off stem cells.
Unless piRNA and siRNA are 100% at their jobs, and nothing is explicitly removing transposons in sperm/eggs better than in the rest of the body, then surely there should be at least a small amount of accumulation of transposons across generations. Is this something we see?
I vaguely remember that women are born with all the egg cells they’ll have, so, if that’s true, then maybe that offers a partial explanation (only half the child genome should be as infected with transposons?). I’m not sure it holds water, because since egg cells are still alive, even if they aren’t dividing more, they should present opportunities for transposons to multiply.
Another possible explanation I thought of was that, in order to be as close to 100% as possible, piRNA and siRNA work more than normal in the gonads, which does hurt the efficacy of sperm, but because you only need 1 to work, that’s ok. Still, unless it is actually 100%, there should be that generational accumulation.
This isn’t even just about transposons. It feels like any theory of aging would have to contend with why sperm and eggs aren’t old when they make a child, so I’m not sure what I’m missing.

awenonian 11 Jun 2022 19:17 UTC
5 points
5
in reply to: LVSN’s comment on: AGI Safety FAQ / all-dumb-questions-allowed thread
No, they really don’t. I’m not trying to be insulting. I’m just not sure how to express the base idea.
The issue isn’t exactly that computers can’t understand this, specifically. It’s that no one understands what those words mean enough. Define reason. You’ll notice that your definition contains other words. Define all of those words. You’ll notice that those are made of words as well. Where does it bottom out? When have you actually, rigorously, objectively defined these things? Computers only understand that language, but the fact that a computer wouldn’t understand your plan is just illustrative of the fact that it is not well defined. It just seems like it is, because you have a human brain that fills in all the gaps seamlessly. So seamlessly you don’t even notice that there were gaps that need filling.
This is why there’s an emphasis on thinking about the problem like a computer programmer. Misalignment thrives in those gaps, and if you gloss over them, they stay dangerous. The only way to be sure you’re not glossing over them is to define things with something as rigorous as Math. English is not that rigorous.

awenonian 25 Nov 2021 19:20 UTC
5 points
in reply to: gwern’s comment on: Yudkowsky and Christiano discuss “Takeoff Speeds”
Can I try to parse out what you’re saying about stacked sigmoids? Because it seems weird to me. Like, in that view, it still seems like showing a trendline is some evidence that it’s not “interesting”. I feel like this because I expect the asymptote of the AlphaGo sigmoid to be independent of MCTS bots, so surely you should see some trends where AlphaGo (or equivalent) was invented first, and jumped the trendline up really fast. So not seeing jumps should indicate that it is more a gradual progression, because otherwise, if they were independent, about half the time the more powerful technique should come first.
The “what counter argument can I come up with” part of me says, tho, that how quickly the sigmoid grows likely depends on lots of external factors (like compute available or something). So instead of sometimes seeing a sigmoid that grows twice as fast as the previous ones, you should expect one that’s not just twice as tall, but twice as wide, too. And if you have that case, you should expect the “AlphaGo was invented first” sigmoid to be under the MCTS bots sigmoid for some parts of the graph, where it then reaches the same asymptote as AlphaGo in the mainline. So, if we’re in the world where AlphaGo is invented first, you can make gains by inventing MCTS bots, which will also set the trendline. And so, seeing a jump would be less “AlphaGo was invented first” and more “MCTS bots were never invented during the long time when they would’ve outcompeted AlphaGo version −1″
Does that seem accurate, or am I still missing something?

awenonian 20 Sep 2021 21:15 UTC
5 points
in reply to: DanArmak’s comment on: This Can’t Go On
I still think the argument holds in this case, because even computer software isn’t atom-less. It needs to be stored, or run, or something somewhere.
I don’t doubt that you could drastically reduce the number of atoms required for many products today. For example, you could in future get a chip in your brain that makes typing without a keyboard possible. That chip is smaller than a keyboard, so represents lots of atoms saved. You could go further, and have that chip be an entire futuristic computer suite, by reading and writing your brain inputs and outputs directly it could replace the keyboard, mouse, monitors, speakers, and entire desktop, plus some extra stuff, like also acting as a VR Headset, or video game console, or whatever. Lets say you manage to squeeze all that into a single atom. Cool. That’s not enough. For this growth to go on for those ~8000 years, you’d need to have that single-atom brain chip be as valuable as everything on Earth today. Along with every other atom in the galaxy.
I think at some point, unless the hottest thing in the economy becomes editing humans to value specific atoms arbitrary amounts (which sounds bad, even if it would work), you can’t get infinite value out of things. I’m not even sure human minds have the capability of valuing things infinitely. I think even with today’s economy, you’d start to hit some asymptotes (i.e. if one person had everything in the world, I’m not sure what they’d do with it all. I’m also not sure they’d actually value it any more than if they just had 90% of everything, except maybe the value on saying “I have it all”, which wouldn’t be represented in our future economy)
And still, the path to value per atom has to come from somewhere, and in general it’s going to be making stuff more useful, or smaller, but there’s only so useful a single atom can be, and there’s only so small a useful thing can be. (I imagine some math on the number of ways you could arrange a set of particles, multiplied by the number of ways a particular arrangement could be used, as an estimate. But a quick guess says that neither of those values are infinite, and, I expect that number to be dominated by ways of arranging particles, not by number of uses, considering that even software on a computer is actually different arrangements of the electrons.)
So I guess that’s the heart of it to me, there’s certainly a lot more value we can squeeze out of things, but if there’s not literally infinite, it will run out at some point, and that ~8000 year estimate is looking pretty close to whatever the limit is, if it’s not already over it.

awenonian 9 Jun 2022 15:03 UTC
4 points
2
in reply to: Evan R. Murphy’s comment on: AGI Safety FAQ / all-dumb-questions-allowed thread
In addition to the mentions in the post about Facebook AI being rather hostile to the AI safety issue in general, convincing them and top people at OpenAI and Deepmind might still not be enough. You need to prevent every company who talks to some venture capitalists and can convince them how profitable AGI could be. Hell, depending on how easy the solution ends up being, you might even have to prevent anyone with a 3080 and access to arXiv from putting something together in their home office.
This really is “uproot the entire AI research field” and not “tell Deepmind to cool it.”

awenonian 19 Dec 2021 20:59 UTC
4 points
on: The Plan
“Real search systems (like gradient descent or evolution) don’t find just any optima. They find optima which are [broad and robust]”
I understand why you think that broad is true. But I’m not sure I get robust. In fact, robust seems to make intuitive dis-sense to me. Your examples are gradient descent and evolution, neither of which have memory, so, how would they be able to know how “robust” an optima is? Part of me thinks that the idea comes from how, if a system optimized for a non-robust optima, it wouldn’t internally be doing anything different, but we probably would say it failed to optimize, so it looks like successful optimizers optimize for robust optima. Plus that broad optima are more likely to be robust. I’m not sure, but I do notice my confused on the inclusion of “robust”. My current intuition is kinda like “Broadness and robustness of optima are very coupled. But, given that, optimization for robust optima only happens insofar as it is really optimization for broad optima. Optimization for robust but not broad optima does not happen, and optimization for statically broad but more robust optima does not happen better.”

awenonian 18 Jan 2023 4:12 UTC
3 points
0
on: Models Don’t “Get Reward”
Back in Reward is not the optimization target, I wrote a comment, which received a (small I guess) amount of disagreement.
I intended the important part of that comment to be the link to Adaptation-Executers, not Fitness-Maximizers. (And more precisely the concept named in that title, and less about things like superstimuli that are mentioned in the article) But the disagreement is making me wonder if I’ve misunderstood both of these posts more than I thought. Is there not actually much relation between those concepts?
There was, obviously, other content to the comment, and that could be the source of disagreement. But I only have that there was disagreement to go on, and I think it would be bad for my understanding of the issue to assume that’s where the disagreement was, if it wasn’t.

awenonian 26 Jul 2022 22:07 UTC
3 points
−5
in reply to: paulfchristiano’s comment on: Reward is not the optimization target
I interpret OP (though this is colored by the fact that I was thinking this before I read this) as saying Adaptation-Executers, not Fitness-Maximizers, but about ML. At which point you can open the reference category to all organisms.
Gradient descent isn’t really different from what evolution does. It’s just a bit faster, and takes a slightly more direct line. Importantly, it’s not more capable of avoiding local maxima (per se, at least).
What links here?
- awenonian's comment on Models Don’t “Get Reward” by Sam Ringer (18 Jan 2023 4:12 UTC; 3 points)

awenonian 4 Jul 2022 19:31 UTC
3 points
in reply to: anonymousaisafety’s comment on: I No Longer Believe Intelligence to be “Magical”
So, I want to note a few things. The original Eliezer post was intended to argue against this line of reasoning:
I occasionally run into people who say something like, “There’s a theoretical limit on how much you can deduce about the outside world, given a finite amount of sensory data.”
He didn’t worry about compute, because that’s not a barrier on the theoretical limit. And in his story, the entire human civilization had decades to work on this problem.
But you’re right, in a practical world, compute is important.
I feel like you’re trying to make this take as much compute as possible.
Since you talked about headers, I feel like I need to reiterate that, when we are talking to a neural network, we do not add the extra data. The goal is to communicate with the neural network, so we intentionally put it in easier to understand formats.
In the practical cases for this to come up (e.g. a nascent superintelligence figuring out physics faster than we expect), we probably will also be inputting data in an easy to understand format.
Similarly, I expect you don’t need to check every possible esoteric format. The likelihood of the image using an encoding like 61 bits per pixel, with 2 for red, 54 for green and 5 for blue is just, very low, a priori. I do admit I’m not sure if only using “reasonable” formats would cut down the possibilities into the computable realm (obviously depends on definitions of reasonable, though part of me feels like you could (with a lot of work) actually have an objective likeliness score of various encodings). But certainly it’s a lot harder to say that it isn’t than just saying “f(x) = (63 pick x), grows very fast.”
Though, since I don’t have a good sense for whether “reasonable” ones would be a more computable number, I should update in your direction. (I tried to look into something sort of analogous, and the most common 200 passwords cover a little over 4% of all used passwords, which, isn’t large enough for me to feel comfortable expecting that the most “likely” 1,000 formats would cover a significant quantity of the probability space, or anything.)
(Also potentially important. Modern neural nets don’t really receive things as a string of bits, but instead as a string of numbers, nicely split up into separate nodes. (yes, those numbers are made of bits, but they’re floating point numbers, and the way neural nets interact with them is through all the floating point operations, so I don’t think the neural net actually touches the bit representation of the number in any meaningful way.)

awenonian 30 Jun 2022 16:57 UTC
3 points
in reply to: anonymousaisafety’s comment on: I No Longer Believe Intelligence to be “Magical”
“you’re jumping to the conclusion that you can reliably differentiate between...”
I think you absolutely can, and the idea was already described earlier.
You pay attention to regularities in the data. In most non-random images, pixels near to each other are similar. In an MxN image, the pixel below is a[i+M], whereas in an NxM image, it’s a[i+N]. If, across the whole image, the difference between a[i+M] is less than the difference between a[i+N], it’s more likely an MxN image. I expect you could find the resolution by searching all possible resolutions from 1x<length> to <length>x1, and finding which minimizes average distance of “adjacent” pixels.
Similarly (though you’d likely do this first), you can tell the difference between RGB and RGBA. If you have (255, 0, 0, 255, 0, 0, 255, 0, 0, 255, 0, 0), this is probably 4 red pixels in RGB, and not a fully opaque red pixel, followed by a fully transparent green pixel, followed by a fully transparent blue pixel in RGBA. It could be 2 pixels that are mostly red and slightly green in 16 bit RGB, though. Not sure how you could piece that out.
Aliens would probably do a different encoding. We don’t know what the rods and cones in their eye-equivalents are, and maybe they respond to different colors. Maybe it’s not Red Green Blue, but instead Purple Chartreuse Infrared. I’m not sure this matters. It just means your eggplants look red.
I think, even if it had 5 (or 6, or 20) channels, this regularity would be born out, between bit i and bit i+5 being less than bit i and i+1, 2, 3, or 4.
Now, there’s still a lot that that doesn’t get you yet. But given that there are ways to figure out those, I kinda think I should have decent expectations there’s ways to figure out other things, too, even if I don’t know them.
I do also think it’s important to zoom out to the original point. Eliezer posed this as an idea about AGI. We currently sometimes feed images to our AIs, and when we do, we feed them as raw RGB data, not encoded, because we know that would make it harder for the AI to figure out. I think it would be very weird, if we were trying to train an AI, to send it compressed video, and much more likely that we do, in fact, send it raw RGB values frame by frame.
I will also say that the original claim (by Eliezer, not the top of this thread), was not physics from one frame, but physics from like, 3 frames, so you get motion, and acceleration. 4 frames gets you to third derivatives, which, in our world, don’t matter that much. Having multiple frames also aids in ideas like the 3d → 2d projection, since motion and occlusion are hints at that.
And I think the whole question is “does this image look reasonable”, which you’re right, is not a rigorous mathematical concept. But “‘looks reasonable’ is not a rigorous concept” doesn’t get followed by “and therefore is impossible” Above are some of the mathematical descriptions of what ‘reasonable’ means in certain contexts. Rendering a 100x75 image as 75x100 will not “look reasonable”. But it’s not beyond thinking and math to determine what you mean by that.

awenonian 15 Jun 2022 20:01 UTC
3 points
0
in reply to: LVSN’s comment on: AGI Safety FAQ / all-dumb-questions-allowed thread
Let me ask you this. Why is “Have the AI do good things, and not do bad things” a bad plan?

awenonian 15 Jun 2022 13:48 UTC
3 points
0
in reply to: LVSN’s comment on: AGI Safety FAQ / all-dumb-questions-allowed thread
I think you missed the point. I’d trust an aligned superintelligence to solve the objections. I would not trust a misaligned one. If we already have an aligned superintelligence, your plan is unnecessary. If we do not, your plan is unworkable. Thus, the problem.
If you still don’t see that, I don’t think I can make you see it. I’m sorry.

awenonian 10 Jun 2022 15:28 UTC
3 points
0
in reply to: LVSN’s comment on: AGI Safety FAQ / all-dumb-questions-allowed thread
Again, what is a “reason”? More concretely, what is the typeof a “reason”? You can’t program an AI in English, it needs to be programmed in code. And code doesn’t know what “reason” means.
It’s not exactly that your plan “fails” anywhere particularly. It’s that it’s not really a plan. CEV says “Do what humans would want if they were more the people they want to be.” Cool, but not a plan. The question is “How?” Your answer to that is still under specified. You can tell by the fact you said things like “the AI could just...” and didn’t follow it with “add two numbers” or something simple (we use the word “primitive”), or by the fact you said “etc.” in a place where it’s not fully obvious what the rest actually would be. If you want to make this work, you need to ask “How?” to every single part of it, until all the instructions are binary math. Or at least something a python library implements.

awenonian 1 Apr 2022 16:01 UTC
3 points
on: Using Prediction Markets to Guide Government Policy
Space pirates can profit by buying shares in the prediction market that pay money if Ceres shifts to a pro-Earth stance and then invading Ceres.
Has this line got a typo, or am I misunderstanding? Don’t the pirates profit by buying pro-Mars shares, then invading to make Ceres pro-Mars (because Ceres is already pro-Earth)?
Mars bought pro-Earth to make pro-Mars more profitable, in the hope that pirates would buy pro-Mars and then invade.

awenonian 13 Mar 2022 3:07 UTC
3 points
in reply to: Steven Byrnes’s comment on: Late 2021 MIRI Conversations: AMA / Discussion
In my post I wrote:
Am I correct after reading this that this post is heavily related to embedded agency? I may have misunderstood the general attitudes, but I thought of “future states” as “future to now” not “future to my action.” It seems like you couldn’t possibly create a thing that works on the last one, unless you intend it to set everything in motion and then terminate. In the embedded agency sequence, they point out that embedded agents don’t have well defined i/o channels. One way is that “action” is not a well defined term, and is often not atomic.
It also sounds like you’re trying to suggest that we should be judging trajectories, not states? I just want to note that this is, as far as I can tell, the plan: https://www.lesswrong.com/posts/K4aGvLnHvYgX9pZHS/the-fun-theory-sequence
Life’s utility function is over 4D trajectories, not just 3D outcomes.
From the synopsis of High Challenge
instead of making a corrigible AGI.
I’m not sure I interpret corrigibility as exactly the same as “preferring the humans remain in control” (I see you suggest this yourself in Objection 1, I wrote this before I reread that, but I’m going to leave it as is) and if you programmed that preference into a non-corrigible AI, it would still seize the future into states where the humans have to remain in control. Better than doom, but not ideal if we can avoid it with actual corrigibility.
But I think I miscommunicated, because, besides the above, I agree with everything else in those two paragraphs.
See discussion under “Objection 1” in my post.
I think I maintain that this feels like it doesn’t solve much. Much of the discussion in the Yudkowsky conversations was that there’s a concern on how to point powerful systems in any direction. Your response to objection 1 admits you don’t claim this solves that, but that’s most of the problem. If we do solve the problem of how to point a system at some abstract concept, why would we choose “the humans remain in control” and not “pursue humanity’s CEV”? Do you expect “the humans remain in control” (or the combination of concepts you propose as an alternative) to be easier to define? Easier enough to define that it’s worth pursuing even if we might choose the wrong combination of concepts? Do you expect something else?

awenonian 17 Nov 2021 1:18 UTC
3 points
on: Ngo and Yudkowsky on alignment difficulty
So, I’m not sure if I’m further down the ladder and misunderstanding Richard, but I found this line of reasoning objectionable (maybe not the right word):
“Consider an AI that, given a hypothetical scenario, tells us what the best plan to achieve a certain goal in that scenario is. Of course it needs to do consequentialist reasoning to figure out how to achieve the goal. But that’s different from an AI which chooses what to say as a means of achieving its goals.”
My initial (perhaps uncharitable) response is something like “Yeah, you could build a safe system that just prints out plans that no one reads or executes, but that just sounds like a complicated way to waste paper. And if something is going to execute them, then what difference is it whether that’s humans or the system itself?”
This, with various mention of manipulating humans, seems to me to like it would most easily arise from an imagined scenario of AI “turning” on us. Like that we’d accidentally build a Paperclip Maximizer, and it would manipulate people by saying things like “Performing [action X which will actually lead to the world being turned into paperclips] will end all human suffering, you should definitely do it.” And that this could be avoided by using an Oracle AI that just will tell us “If you perform action X, it will turn the world into paperclips.” And then we can just say “oh, that’s dumb, let’s not do that.”
And I think that this misunderstands alignment. An Oracle that tells you only effective and correct plans for achieving your goals, and doesn’t attempt to manipulate you into achieving its own goals, because it doesn’t have its own goals besides providing you with effective and correct plans, is still super dangerous. Because you’ll ask it for a plan to get a really nice lemon poppy seed muffin, and it will spit out a plan, and when you execute the plan, your grandma will die. Not because the system was trying to kill your grandma, but because that was the most efficient way to get a muffin, and you didn’t specify that you wanted your grandma to be alive.
(And you won’t know the plan will kill your grandma, because if you understood the plan and all its consequences, it wouldn’t be superintelligent)
Alignment isn’t about guarding against an AI that has cross purposes to you. It’s about building something that understands that when you ask for a muffin, you want your grandma to still be alive, without you having to say that (because there’s a lot of things you forgot to specify, and it needs to avoid all of them). And so even an Oracle thing that just gives you plans is dangerous unless it knows those plans need to avoid all the things you forgot to specify. This was what I got out of the Outcome Pump story, and so maybe I’m just saying things everyone already knows…