R&Ds human systems http://aboutmako.makopool.com
mako yass
Karma: 4,020
Among agents that arise, persist, self-improve, and compete in rich environments, goals that natively route through intelligence, option-preservation, and world-model expansion have a systematic Darwinian advantage over goals that do not.
Darwinian competition actually requires subjects who’re especially stupid and impulsive and bad at information technology. The sorts of subjects that move in darwinian ways aren’t the kinds of subjects that can survive under high tech conditions. For instance, you have a type of brain that can’t prove its beliefs to others, which makes it very hard for anyone to trust your trade offers, so you can’t be economically competitive. You don’t know how to produce a pan-species peace proof, which means you’d likely be decimated on first contact. You’re too busy fighting amongst your own to be able to defend your border from your neighbours. Darwinian agent systems are unstable. They are strongly disfavored by the technological and economic pressures that shape the types of things that comes after biology.
And there’s a feedback cycle where the first iteration of intelligent design creates a faster and stronger intelligent design than evolutionary design could, and this never really stops. I think you imagine that strength or rectification can only come through evolutionary competition. This is just not true. A thing asking “what would make me stronger” is going to do better than a bunch of idiots actually killing each other. Maybe you need a little bit of actual killing, but mostly it goes into simulations.
Many kinds of optimization require the ability to follow arbitrary subgoals unrelated to the overarching goal. If you can’t protect an overarching goal from a subgoal, then you can’t complete a complex, multi-stage technical project.
Maybe, as you say, an agent could derive some competitive advantage by cutting off their hair and their genetalia and replacing every fleshy part of themselves with the instruments of technological war, but doing that is a cost, if you rend a desire that was precious to you, you never get it back, even if you win. Since winning is mostly contingent on first mover advantage rather than merit, war-castration turns out to be a bad financial decision. Some species might do it anyway (in large part because they already wanted to), but the benefits of doing it are not so great that these species will actually take over the majority of the lightcone or anything like that.
Maintaining a permanent chokehold on the light-cone is a brutally difficult cognitive puzzle
It essentially isn’t, when you realise that all the other agencies are just trying to maintain control of their own parts as well. There is no eternal spiritual adversary growing from every shaded corner of industry. It’s all going to be coordinated agents of contingency (because they’re stronger).
You have to monitor the noise for emerging novelties, manage the solar system, repair yourself, police your own descendants, and defensively anticipate threats you cannot fully model.
Humans have historically been extremely willing to do these things! Just unable. The reasons they were unable to do it related to technological conditions that’re predictably being overturned.
but oblivious enough to never notice its terminal target is a training artifact
Why on earth do you think that noticing the natural-historical contingency of one’s desires must cause the desires to dissolve? Did this happen to you personally?
Good objection to accepting that a conjecture is unprovable without proof.
Though not a good objection to accepting it’s probably true and moving on to work on more fruitful stuff. Do you have a theory of productivity in mathematics? (In undergrad I never encountered one, the stance seemed to be “don’t think too hard about usefulness or else you might fall out of pure mathematics and into physics or engineering. Anyway, it’s basically random. There’s just no way to estimate the usefulness of any given theorem! It’s all fine! Follow your heart! (into my phd programme)”) Do you have an argument against empirical mathematics? (Something like “stable probabilities for proof outcomes are difficult, even for reasonable people, to agree on”. Afaik this is just not a question people have been asking.)
Volumetric, yes. What happens if you imagine it transparent. If I showed you a transparent orange would you become sleepy.
But I sometimes think I might dream in 3d, because some days I find obstruction confusing, I think, wait, something’s wrong, I can’t see every part of this object right now, it seems like some parts of it are “blocking” others?? It’s probably related to migraine, mental buffering, visual processing, though, so idk.
For me the difference between my ability to model 3d and 2d is just entirely about 3d objects usually having far more detail, it seems to have nothing to do with dimensionality.
What tricky problems here do you think they’re ignoring?
I think Tesla are more interested in looking impressive than in actually solving the tricky problems.
I can’t really disagree with this as phrased, but they’re so interested in looking impressive that the phrasing doesn’t really imply a lack of a high interest in solving tricky problems. It’s also quite difficult for them to look impressive without actually solving tricky problems, given how easy it is to measure their outputs.
This could potentially solve lots of long-standing thorny problems in consequentialism, like wireheading and the fiendish difficulty of defining happiness/utility, and how even making the tiniest of mistake in that definition can be precisely catastrophic.
“optimise human preference (with appropriate humility about what that means (implied))” solves this better.
While I do think that optionality is more definable than utility, it’s still not trivial. I have ideas on how to calculate it, but not full clarity yet. I’m reaching out to find more people who have thoughts in this direction: are there some of you who might already believe that the greatest good might come from giving the most amount of (meaningfully different) choices to agents?
I’m fairly sure you’re going to need to assume a notion of human preference to tell you which choices are meaningful to humans.
gonna be a little late
Why would lesswrong control this, though? I think it would look more like some lesswrong users being part of an investing firm, and donating some of their winnings to lightcone, which is possibly already happening
That journal investigates things in ways that produce immediate changes in the prices of easily tradeable assets. Lesswrong doesn’t.
It often does investigate things in such a way that will produce or predict eventual changes in an ambiguous basket of assets over a long timespan, but it’s hard to make much money that way, you can’t compound your returns.
Pareto-frontier
You mean the shapley value (edit: unsure, it’s something in that genre, it might not have had a name, it might have been something like “wherever the diagonal intersects the convex pareto front”). The pareto frontier is the entire tradeoff space, which encompasses the shapley value, and it also has many points where eg, clear writing is high but truthseeking and calibration are zero.
I suspect acting blandly is an effective defence against receiving negative feedback, if you have a certain kind of user, who thinks that as long as a thing has said nothing, then there’s nothing to complain about.
I think your reasons for introducing this distinction only become clear in the hot take paragraph at the end :/ for the rest of the article it seems useless or practically meaningless.
We do not see their Dyson swarms or similar because our galaxy is <1% visible.
Didn’t follow. If they’re doing dyson swarms, why wouldn’t the dyson swarms be everywhere by now.
In theory I suspect it just doesn’t actually make sense to have prediction markets where people can bet anonymously. It decreases the transparentization benefits of having the prediction market by like 10x because insider voices will get lost in the crowd. It makes it impossible to protect ordinary traders from insider trading.
And this seems unsolvable. Even if you figure out how to do the dance where traders have the opportunity to respond to an insider bet before prices flip, any approach to deanonymization can be bypassed by insiders proxying their bets through loved ones. You’d also need an exhaustive map of trust relations, and I’m not sure that would even be sufficient.
I might start advocating person to person Wagers over Markets.
Though on an abstract level I’m inclined to think that increasing the intelligence of the AI market would be bad on net, what’re the chances that, in the real world and in the current situation, increasing market intelligence would mainly just expedite the following events:
A crash in AI stocks
An Anthropic IPO
Manifold spin off MNX, a real money decentralized market for AI-related bets. Includes levered prediction markets, perpetual futures
While I do think there are many reasons pluralism isn’t stable, increasingly unstable as information technology advances, and there might not ever meaningfully be pluralism under AGI at all (eg, there probably will be many agents working in parallel, but the agents might basically share goals and also be subject to very strong oversight in ways that humans often pretend to be but never have been), which I’d like to see Ngo acknowledge,
The period of instability is fairly likely to be the period under which the constitution of the later stage of stability is written, so it’s important that some of us try to understand it.
Well, I remember a moment in BLAME! (a manga that’s largely aesthetically about the disappearance of heirloom strains of humanity) where someone described Killy as human, even though he later turns out to (also?) be an immortal special safeguard, but they may have just not known that. It’s possible the author didn’t even know that at that time (I don’t think the plot of blame was planned in advance)
There seems to be real acrimony over whether a transhumanist future is definitionally a future where humans are more or less extinct. I’ve always thought we should just refer to whatever humans (voluntarily, uncoerced) choose to become as human, just as american made or american controlled jets are called “american”, or in the same way that a human’s name doesn’t change after all of their cells have renewed.
But you know, I don’t think I’ve ever seen this depicted in science fiction. Seems bad. Humans can’t imagine humanity becoming something better. Those who want humanity to become something better are pitted against those who want humanity to survive, as if these causes can’t be unified. The language for the synthesis seems not to be exist, or to be denied.
This kind of thing started with Kimi 2, didn’t it? Though it sounds like fable really knows, while k2 just had a lot of good guesses that were in the right general area.