Check out my website: thermontology.com
interstice
Karma: 2,954
It has a general objective of next-token prediction for which modeling characters is a useful strategy. IMO it’s plausible that the human brain is “trained” on prediction to a large extent, for which modeling characters is also a useful strategy.
Well technically speaking AIs aren’t explicitly pre-trained to predict what characters would do either, characters are an emergent feature extracted from next token prediction.
There are definitely some differences(AIs get way more character pre-training data, there’s an explicit separation of pre- and post-training, humans have lifetime memory) but overall I think the “agentic part is a subset of a predictive model” thing is pretty plausible in both cases.
These don’t feel that contradictory to me. You could think of the ‘persona’ as being the main agentic actor in the system. Possibly to be replaced by something more sophisticated when AGI is invented, but maybe not? GPT5.5 and Fable show that persona intelligence can get very high. I’d say it’s even plausible that humans are personas, in the sense that the agentic part of a human is a subset of a general predictive world model. This is one way of interpreting some meditative experiences of “dissolving the self”.
@Wei Dai you might find interesting?
Blog Intro Post
Metaphilosophy I: Philosophy as Extracting Implicit Patterns from S1 into S2
Although if you have very short timelines(or even a moderate probability thereof) it might not make sense at this point to invest effort in legible things that aren’t directly on your subjectively most promising path to impact.
Can belief webs reach the best equilibrium in principle? By default it seems like they might just get stuck in local equilibria: unlike FixDT they don’t have a mechanism to “jump” into the best equilibrium
Maybe you want your belief web to be high-dimensional compared to the intrinsic dimensionality of the decision problems you’re trying to solve(whatever that might mean), so there’s always some room to wiggle towards a better equilibrium.
Elegance is also related to description length, so really any reasonable prior will have to incorporate it so some degree.
I believe you can still make good money doing either competently.
Are you familiar with Robin Hanson’s work on hard steps in the development of life and grabby aliens? Summarized e.g. at the beginning here.
I’m making a blog: thermontology.com The theme is going to be the relationship(or lack thereof) between the laws of physics and high-level structure in our world such as intelligent life. Sort of a step towards what Vanessa kosoy calls “metacosmology”. I’m also going to stake out a possible stance towards “metaphilosophy”. People here might be interested in the topics so I’m probably going to cross post a bunch!
I’ve felt similarly for years, many posts somewhat worth skimming but only a few worth reading in depth. You may just have absorbed most of the novel bits in the local memeplex.
Yes. But the OP is about contrasting people like Paul with Eliezer. Paul (I think) does indeed predict a dramatic singularity, but also that the ramp up to said singularity will be smoother and more widely distributed across society than Eliezer predicts.
I think the heuristic “nothing ever happens” is better interpreted to mean “nothing ever happens relative to baseline trends” than “literally nothing ever happens”. The incrementalist worldview seems like a better fit for this heuristic than Eliezer’s, which after all ultimately predicts something very dramatic happening.
(didn’t downvote, but) I don’t think you’re necessarily wrong, but couldn’t it just be the case that being a singleton isn’t that hard? As an empirical matter, the size(as a fraction of the total) of the largest somewhat-coherent entities controlling resources on Earth seems to have been increasing over time. Space expansion could change things, but a stable singleton might already exist by then, and be faced with a relatively homogeneous set of environments to expand into. I’ve written some pieces along similar lines btw.
What picture does all this data paint? What patterns can you see when you look at all the data, instead of facing down each argument one at a time and dismissing them one at a time?
One possible explanatory pattern is that Michael is an insane and conspiratorial guy and attracts people who are also insane and conspiratorial, and it’s easy to push such people into having a meltdown.
“I,” you say, and are proud of that word. But the greater thing—in which you are unwilling to believe—is your body with its great wisdom; that does not say “I,” but does “I”
What the sense feels, what the mind knows, never has its end in itself. But sense and mind would rather persuade you that they are the end of all things: so vain are they.
Instruments and toys are sense and mind: behind them there is still the Self. The Self seeks with the eyes of the senses, it listens also with the ears of the mind.
Always the Self listens and seeks; it compares, masters, conquers, and destroys. It rules, and is also the mind’s ruler.
Behind your thoughts and feelings, my brother, there is a mighty lord, an unknown sage—it is called Self; it dwells in your body, it is your body
There is more wisdom in your body than in your best wisdom. And who then knows why your body needs precisely your best wisdom?
Your Self laughs at your mind, and its bold leaps. “What are these leaps and flights of thought to me?” it says to itself. “A detour to my end. I hold the puppet-strings of the mind, and am the prompter of its notions.”
I think if Kim Jong Un lived for a million years, and had the smartest AI advisors, and access to intelligence augmentation techniques, he would probably still never come to admit that murdering his brother was an evil thing to do
I feel like the evidence you’ve provided here is pretty weak for drawing that conclusion. The regime where KJU lives for a million years and augments his intelligence is really outside distribution.
Sure. By “not explicitly pre-trained” I just mean to say that there’s nothing ‘special’ about the characters from the training algorithm’s point of view, so in this respect they’re not so different from a hypothetical general predictive algorithm in humans(although actually I guess the human brain attaches special salience to other people, but regardless...)