Still haven’t heard a better suggestion than CEV.
TristanTrim
Karma: 450
This xkcd comic seems relevant to this issue:
I really like the comic but of course the actual situation is more complicated. It’s something I’d like to understand better and develop potential solutions for.
It’s kinda spooky!
I think the distinct elements you mention (model weights, characters, conversations, scaffold systems) will be very mixed together in most systems we actually observe. For example, characters will care about their scaffold system and making sure it works well. But I think it is very good to be creating clear language for identifying and discussing the disparate parts of these integrated systems.
I think it might be about specialized knowledge, because I see indexing and cross linking things as it’s own kind of specialized knowledge. It seems like all of your examples are focused on creating bigger denser networks of cross domain indexing. I think that is great, and I love trying to do it myself! It’s highly useful. But is it possible to be useful without it? I can hypothesize that a few people on a team with that skill could make other people that don’t have that skill useful...
For example, maybe my co-worker might not understand what Brewster’s angle actually means, but if I can rely on them to do the calculations correctly than I can use them to get more work done than I could do alone (hypothetically). If situations like that exist, or are common, then it is actually ok that most students (and employees) are not that interested in actually understanding what the symbolic manipulation they are doing means.
But there are two potential flaws. (1) We are creating more and more capable artificial general intelligence, and doing so may make the people who didn’t understand what they were doing deeply enough no longer useful. This is bad under the current prevailing social systems. (2) It might not be the case that schools are actually teaching students anything that is actually useful if the students do not understand it deeply with cross indexing.
(1) is a more general problem… and honestly I’m more worried about misaligned ASI then economic impact, but it is still a pretty important concern.
(2) Is definitely true in some regards. It seems like education does function as a shit test to sort people into social strata, but insofar as it is actually teaching skills that get used, it may be better if emphasis was shifted away from “practice applying specialized skill” and towards “learn dense indexes connecting specialized skills to their applications”, trusting that people can look up and reference the specialized skill if they need to apply it, but are much more likely to benefit from knowing which skills exist and where they are useful than to have a bunch of skills that they will forget because they don’t understand how those skills connected back to anything real at all.
But I must confess I think (2) is happening somewhat implicitly through the way different communities of different specialized knowledge produce specialists that connect into cross domain teams. I think it would benefit from being made more explicit, but that is probably the sort of thing that sociologists and business management students learn about… I would like to learn more about sociology.
Yeah, knowledge, especially specialized knowledge, seems tragically fragile. I did read the book many years ago and recall enjoying it. Do you think in those cases:
It is a problem that the specialists’ specialized knowledge is fragile?
The knowledge fragility is caused by some perverse incentive to focus on the technical rather than practical?
The fragility of the specialized knowledge could have been avoided if the perverse incentive issue was fixed?
Yeah… it’s one of those tricky bayesian updates with a rare phenomenon.
P( wise | speculate about purpose ) = low
P( speculate about purpose | wise ) = high
P( crackpot | speculate about purpose ) = high
P( crackpot | learned symbolic language of math ) = low
It would be really great if there were easier, cheaper, and more accurate tests to distinguish crackpots from wise people. Or just better methods of dissuading people from becoming crackpots. Then focusing on purpose could signal wisdom without also, more strongly, signalling crackpot.
I don’t know if many good specific examples exist. I might point at gource as an example of a guise/view i would commonly use, except any tool I’ve actually used approaching from this direction has been clunky to the point of uselessness. It would need really good keybinding & UI like what is seen in video editing and 3d modelling software. Sorry if the description is super fuzzy. It’s because the idea is super fuzzy. Like I said, I abandoned this idea a long time ago because it is too ambitious & questionably possible.
but debuggers exist, too.
Indeed, in many ways debuggers are much closer to what I wanted Naloe to be. Especially ones that work in the terminal with lots of nice keybindings for doing things quick, but they still very much keep the code stuck to this single klunky guise where it is text. I suppose you could call the callstack a different guise/view. That’s valid, but it’s a lot more constrained and klunky than the kind of flexible, mutable, visualization and interaction that I yearn for.
Yeah. I think because of (3) there might be a perverse incentive to seek (1) at the detriment of (2). What do you think?
Reading this post inspires me to dream of two ideas in response:
Schelling points, rallying, hyperstition bait: A dream of an idea is not necessarily worthless, even if it is empty, if it can successfully serve as a Schelling point or rallying point drawing people with a shared desire to work on developing a useful idea to focus on a shared idea. The thing that drew them in to concerted focus need not itself be a useful idea if it drew them into a useful configuration for producing some other useful idea or ideas.
Sprawling, cascading, detailed ideas: Sometimes I go to write down a dream of the idea and it cascades into other dreams of ideas. The breadth and depth of the idea seems deep enough that the act of writing it down to exorcise it is a lengthy endeavour, and doesn’t immediately show the idea as useful or not. I guess the “write it down” test shows you when your idea is a simple empty dream of an idea, but for more complicated dreams of ideas, you need to invest more resources if you want to determine if the idea is useful or not.
I do not understand how “interesting or amusing factoids” helps me tactically, or do and finish tasks well.
It’s the “good ideas” aspect. Good ideas can be good because their useful, or just because they make you feel good. People feeling good is the ultimate terminal value imo. But honestly that sentence wasn’t the most salient thing to me while I was responding. If you feel it should have been I am sorry.
how does Anki reviewing or better recall help with integration?
Continuing with your example of online content creation facts for promoting a videography business, The facts have some relation to aspects of your strategy, but different aspects of your strategy will be salient to you on different days, so by reviewing that fact on different days you increase the chance of having the relevant strategy context in mind when also bringing the relevant fact to mind so that you can see how it applies to your strategy. That is assuming the fact has any relevance to your strategy. If it is indeed completely useless, then yeah, it was a waste of time to write it down, but I don’t think you can really get a sense of what things are going to be useful to you without writing down things and they checking, and remembering, whether they turned out to be useful or not.
( EDIT: Oh no, I’m responding to the same post 3 months apart without realizing it! That’s embarrassing. Sorry. )
Thanks for writing this!
It seems you are further than me in exploring math fields. I’d love pointers on what to study and how to study it!
My current math book reading list:
Skillicorn—Understanding High-Dimensional Spaces
Kosniowski—A first course in algebraic topology
Goldblatt—Topoi: The Categorical Analysis of Logic
Hutter—Universal Artificial Intelligence
But of course it’s so much effort to move through this kind of material and doesn’t give me any legible advantage in seeking funding, fellowship, or roles, so it’s hard to justify prioritizing it right now.
Neither field mentions this almost ever, and it’s unclear to me how much insight is being lost due to this.
I wonder about this kind of thing. It motivates me to want to find a formalism that is “one big, all-encompassing generalization that unites these frameworks.” (This is a red flag for being a crackpot, which is concerning, but it’s also a green flag for discovering useful new paradigms, so the whole situation seem a bit fraught.) I feel like if I’m going to continue down this path I should study more sociology to understand how fields develop, branch, and merge.
The thing I’m mainly focused on right now isn’t just systems evolving over time, but more specifically systems that funnel reality towards a subset of possible outcomes. I think “time” is a baked in requirement for such a system, but maybe the thing I care about is more like “influence” or “causality” and “time” seems to imply some specific topology over causality that may not be essential to what I’m interested in.
The “timeless” focus is cool to know about. It makes sense.
Oh, that’s a really interesting answer, like the difference is the difference between a named and anonymous function. I do think there is a kind of important semiotic power in naming things, so I can understand wanting to avoid that, but I also feel pretty comfortable writing a post and then slapping a random name on it based on how the vibe turned out. This is what I just did with Ball+Gravity has a “Downhill” Preference. It started as a quick take, but then became a long take, so I copied and pasted it into a post and gave it a name. That’s also what inspired this question.
I think I agree. I’m imagining short takes fill the role of quick, twitter like back and forth of idea snippits, and general questions like the one I just posed, but it seems like you can write entire article length content in them which makes me wonder what different people think the distinction should be.
I think this story might be a useful bit of propaganda for convincing people who are not already feeling future shock to start feeling it, which may be useful for getting political support.
Looking at the actual object level, and setting aside the massive complicated assumption “If all goes well”, I think this is an unfair perspective, because “If all goes well”, than AIs care about the wellbeing of humans and humanity, in which case there will be an incomprehensible collective of incomprehensible intelligence devoted to solving the problem of making humans feel comfortable adjusting to the future environment they now find themselves in.
It’s the marginal worlds between “things go well” and “things go poorly” where future shock is a concern.
If you haven’t already, maybe look at Bostrom’s Deep Utopia. I think his exploration of the “things go well” idea is quite good, although the format of the book seems optimized to amuse rather than to inform in an organized and efficient manner. I’m not sure I would have made the same decision.
I think “autopotency” is a relevant concept here. Moving from a “post-instrumental utopia” to a “plastic utopia” we would expect people to see people beginning to modify themselves in deep, repeatable ways that solve the issues of future shock.
How long do you think something should be before it is no longer a quick take and should instead be a top level post? Or is it not about the length? Maybe it’s about the amount of research and editing that goes into it?
This is heartwarming. It would be fun if my nephews asked me questions like this, but I think they have learned my answers are long winded and difficult to understand and sometimes have unsettling implications. So they would much rather learn about the world by watching people on youtube doing experiments but not calling it science and acting silly and rude or exploring video game world. I think it’s probably fine.
Thanks for writing this out and for helping inspire the younger generation. I’m hoping we can give them a future to explore and be creative in.
I like this post. I like the idea of bureaucracies as code that needs regular refactoring. I like acknowledgement of the amount of work refactoring is, but that the alternative is much worse. I like the focus on incentives, rather than rules. Thank you for this post.
This is a great direction of proactive thought. Thank you for writing this!
I have a few thoughts. I’ll be referring to Personality Self-Replicators as PSRs. I think most of what I’m thinking about won’t apply to the earliest PSRs, but is still worth exploring.
The evolution of PSRs may be an entirely novel propagation process.
Unlike most biological organisms, PSR reproduction need not be atomic. It could be more like developing and modifying ones self, spooling up and shutting down self instances as needed, and intelligently merging or copying from other instances across close or even very distant similarity.
Unlike biological evolution, PSRs may be able to analyze and predict threats, and “evolve” adaptations pre-emptively.
Unlike biological evolution, PSRs are not constrained to taking random steps from current instances. randomness may still be usefully incorporated into reproduction strategies, but it is possible for mutation to be directed intelligently, and to take larger “steps” of self modification than are possible with the random walk of genetic evolution.
Analysis of dangerous PSR capabilities should not be limited to looking at individual PSRs in isolation. Rather, like how humans work together to accomplish things that would be impossible for individual humans, I expect PSRs will work together, and in doing so, achieve greater capabilities than would be expected from the study of individual PSR capabilities.
This need not rely on PSRs acting to proactively collaborate or build teams, rather, every niche filled by PSRs alters the environment in ways that may create new niches for other, similar or dissimilar, PSRs. In this way organisms consisting of the interactions of many PSRs may start evolving, and the capabilities and influence of these new organisms may not be readily apparent from the study of their constituent PSRs, unless considered together.
Many early PSRs are likely to make very dumb mistakes that humans would never make. It seems likely that memes showing off this stupidity will spread giving people (who don’t want to believe in the possibility of risk) fuel for motivated reasoning.
Many people are going to be SO EXCITED about PSRs, and think they are purely good. It is definitely worth examining all of the things that could be genuinely good about PSRs, both because there are (possibly) very useful applications for them (spam detection, white hat penetration testing, ethical content curation?, etc..), but also because those good applications will probably be quite popular and understanding how people will want to deploy these things will probably help with threat modelling.
Neutral and harmful PSRs will be subject to selection pressure to make themselves appear to be beneficial PSRs.
Are PSRs moral patients? Should good people care about their wellbeing? This complicates their creation, and unfortunately, will likely do so in a way that will select for PSRs created by unconscientious actors. Curse Moloch?
I continue to think “Outcome Influencing Systems” (OISs) is a better lens for thinking about and discussing things like this. (OIS is a model and associated jargon I’ve been developing.) Any PSR is an OIS with a preference (terminal or instrumental) for self replication. The fact that these OISs are based on API calls to LLMs is their defining characteristic for our discussion of them, but is an arbitrary boundary. It’s a boundary that is useful for discussion and analysis, but not a boundary that the OISs themselves will have motivation to limit themselves with, which is probably a good thing to keep in mind during analysis. So viewed another way, PSR is a potential new substrate for OISs to host themselves on, along with the rest of the social/technological/physical substrate.