interactive system design http://aboutmako.makopool.com
I saw those scores and thought I was about to witness the greatest exchange of constructive contrarianism in the history of the forums. (Pretty proud of a +7 −12 I posted recently. A real fine stinker. The blue cheese of comments.)
I guess the process would be to pass it on to whichever cababilities researchers they trust with it. There would be a few of them at this point.
So, why not go straight to those researchers instead of MIRI? Because MIRI are more legible responsible intermediaries I guess.
One major second-order effect of doing something this dramatic is that you’d expect controls on gene editing technologies to be raised a lot/made at all, and an argument could be made that that would be a good thing.
There’s a tendency to think: If we believe that something should be illegal, we shouldn’t do it ourselves. In competitive arenas, this ends up disadvantaging the most responsible thinkers by denying them the fruits of defection without denying it to their competitors, or suppressing the acknowledgement of the regulatory holes as participants are afraid to look hypocritical if they acknowledge the need for regulation while thriving without it. It’s actually not hypocritical to exploit a hole while working to close it. Sometimes, spectacularly exploiting the hole is the only practical way to get it closed.
Other projects I’d like to see: An engineered pathogen that just harmlessly melanizes the toes of everyone in the world so that they’re reminded of their protectors’ incompetence every time they look down.
A transferrable utility game is one where there’s a single resource (like dollars) where everyone’s utility is linear in that resource
For humans, money does not seem to have linear returns of utility. For what real agents could it?
My expectation for the U of an aligned AGI would be something like, the sum of the desires of humans, which, if the constituent terms have diminishing returns on resources, will also be diminishing. I can see arguments that many probable unaligned AGI might get linear returns on resources… but if humanity is involved in the negotiation (and you really hope we are) then doesn’t that still break shapley? I guess you could still potentially use shapley for analyzing the valence of ecosystems of unaligned AGI, which would be useful for comparing risk of unaligned singletons to unaligned multipolar outcomes and to authoritarian lockin, but it’s not exciting, and… actually, everything collapses to aligned-somehwhataligned-unaligned multipolar under the Grabby Aliens model.
There does seem to be a value within the human utility function that does scale linearly with resources (a variable that population ethicists and early longtermists love), but it’s not clear at all what its relationship with other variables is. Another way of phrasing this objection is, there is a lot of personal low-hanging fruit that a human has to grab before the scalable variable will be all they have left to optimize, and I don’t think I’ve ever heard of a person who gathered “enough” of the low-hanging fruit of the good life, that they started acting in a purely longtermist or stewardly way. EG, Elon will still want his space adventure no matter how many of his friends explain why they think it’s not cost-effective for reducing existential risk. Many people tell this story where, once a person is rich enough and “has their needs met”, they’re supposed to optimize the scalable term and consequently become selfless (the self is finite, even in the extremes, due to the light speed limit, so anything that keeps scaling has to be a kind of selflessness), many people would like that to be true. It’s not obviously actually true, it doesn’t accord with present human behavior, and I’m not sure how to investigate it.
Can you explain how it picks them off one by one? I mean, how large a group do you need to pick off a wolf and wouldn’t most people be close to being in a group of that size naturally as a result of uh having a town.
I’m not seeing any (sorry I missed a word) much game design here.
My experience as a designer, building out a genre of “peacewagers” (games that aren’t zero sum but also aren’t strictly cooperative, the set of games where honest negotiation is possible.), is that it actually is very likely that someone who’s mostly worked in established genres would drastically underestimate the amount of design thought that is required to make a completely new kind of game work, and they’re trying to make a new kind of game, so I wouldn’t be surprised if they just fell over irrecoverably as soon as they strayed from the yellow brick road they have lived their whole lives within. When you’re building a new genre… you have to figure out so much about what can be done there, what the challenge is, and what the appeal is, and how to elegantly communicate all of that to players and make them want it.
So… I’ve been working on semi-cooperative games for a few years now, I might be able to help with that (I’m also familiar with rust, and have built a basic game engine of my own for some unreleased stuff (in C++)). But I don’t get the impression from the site that they appreciate the difficulty of design, that they’d appreciate me, so I haven’t applied.
I see that this is getting quite a lot of agreement points. I would also like to add my agreement, this is probably a true quote. I agree that it’s probably a true quote. Your claim that this was written somewhere is probably true.
I’d guess that the main AI-exacerbating thing that the game industry does is provoke consumers to subsidize hardware development. I don’t know if this is worth worrying about (have you weighed the numbers?), but do you plan on like, promoting low-spec art-styles to curb demand for increasing realism? :] I wonder if there’s a tension between realism and user-customizability that you might be able to inflame (typically, very detailed artstyles are more expensive to work in and are harder to kitbash, but it’s also possible that stronger hardware would simplify asset pipelines in some ways: raytracing could actually simplify a lot of lighting stuff, right?).
Wolves sometimes kill more than they need, actually. It’s quite strange. So they could be normal-sized wolves. And I’m imagining this to be a population of conservationists who aren’t interested in taking them out of the local ecosystem.
I’m trying to figure out the worldbuilding logic of “they didn’t come so they all got eaten”. What do they do when they come? Why would they be less likely to get eaten if they don’t do it? And also, how does the boy only have a 5% probability?
Okay so maybe the boy sees the wolf from a distance, on a particular bridge or in a particular clearing, and can’t know whether the wolf is coming towards the village. There’s a 5% chance that it will. He can’t stay and make sure, because then maybe he just gets caught by the wolf, and he has to run to deliver the message as soon as possible.
Let’s say it’s a very tough wolf and it can only be defeated with a tank. The village has only one wolf-proof tank. and it takes a while to assemble the tank crew and get its engine started, and the fuel is expensive. Once in the village, the wolf would smell the tank from its fuel, locate it, and keep anyone from getting near it, if the wolf gets to the village before the tank is started then it’s over.
And why is this task left to a child? Children are generally more concerned by and obsessed with monsters than adults. They know that wolf-crying is relatively safe, and they desperately want to witness the monster (this is the whole appeal of the genre of horror), so they take that job. They also have nothing better to do.
It makes non-web applications possible. It has a better layout system, rendering system. Animates everything properly. Centers Dart, which seems to be a pretty good language: It can be compiled ahead of time for faster boots (although I’m not completely sure that typescript wont be basically just as compilable once wasm-gc is up), has better type reflection, will have better codegen (already supports annotations), has a reasonable import system, better data structures, and potentially higher performance due to the type system not being an afterthought (although typescript is still very good relative to dart).
My impression was that wave failed because it was slow and miserable to use. Maybe it would have failed later on for your reason as well, but this was the reason it failed for me.
The great and mighty Google didn’t actually have the ability to make a fairly complex UI for the web that could scale to 100 items. As of today, the craft of UI programming has been lost to all but a few Qt programmers. Google are now gradually rebuilding the art, with Flutter, and I think they may succeed, and this will have a shocking quantity of downstream consequences.
Arbital was functional and fine. It only failed at the adoption stage for reasons that’re still mysterious to me. I’m reluctant to even say that it did fail, I still refer people to a couple of the articles there pretty often.
Separating morality and epistemics is not possible because the universe contains agents who use morality to influence epistemological claims, and the speaker is one of them. I wrote up a response to this post with a precise account of what “should” is and how it should be used. Precise definitions also solve these problems. Looking back today, I think my post introduces new problems of its own. I don’t know when I will finish it. For now, in case I never do finish it, I should mention the best parts here. I don’t believe or endorse all these claims, but mixed up with the false claims are important ones, that must be passed along.
I used to play with tabooing “should” to avoid tying myself to unrealistic hopes and expectations of moral realism, but found a better way.
A previous post was written under the assumption that morality has nothing to do with epistemics. That is, in fact, broadly wrong. Our desires must impact our predictions. You might want to read The Parable of Predict-o-matic to learn how, but I’ll summarize the principle: There are many things that become true as a result of being believed to be true. The assignment of names, declarations of war, predictions of the changes in value of a currency, beliefs about whether a plan will be carried out. (Aside from that last one, most of these are for collective epistemology, rather than individual, but language is for collective reasoning and we’re talking about language so!!)
Many of the parts of reality in which believing agents are involved, are being constructed by those agents’ beliefs. If our selection of beliefs about the future do not answer to our values, we cede much of our agency over the future, we construct a wretched, arbitrary reality where the values of currencies, the relationships between factions, the meaning of our words, are all mostly determined by random forces. We see ourselves as being caught in a threshing tide that is beyond our control, but the tide actually consists of us, it was always in our control, when people miss that, tragedy occurs.
We need “should” to tame the tide. “Should” is the modality that associates belief and desire.”X Should Be So” means “X Will Be So, Because We Want it to Be.”
We can call this account of ‘should’ the realist should. We can call the old, naive standard english dialect speakers’ account, the idealist should. We can be prescriptivists about the meaning of should because the idealist should is problematic and useless.
The idealist should is often spoken dishonestly. It does not care whether the things it claims can actually happen. It rails against reality. It fatally underestimates the strength of its opponent. It rallies around plans that will not work.
The realist should is more or less logically interchangeable with an enlightened optimist’s “will”. Anything that you accept will happen, should happen, and anything that should happen, will happen. It has a different connotation to “will”, it is saying “think about how our desires influence this”, but it can’t be more than a connotative difference, if your should and will diverge, the dialog spirals into the idealism of fixating on plans that you know will not work.
The realist should is used in a few ways that an idealist will find unfamiliar. Exercise: Lets use the realist “should” in the proper way until you get used to it.
Idealist: Nobody should ever have to die of cancer.
Realist: About a billion people should die of cancer before the cure is found.
Idealist: You should have known everything we knew, and read the same books we had read. Instead you failed and are bad.
Realist: You should have acted under bounded rationality and read whatever seemed important to you at the time given what you knew. We must work with the sad reality that different people will have read different books.
The realist “should” has an implicit parameter. A plan.
What “should” be, is part of an implied plan. Different plans use “should” differently. When a person says “should”, you will now be consciously aware that they have some specific plan in mind, just as you realize that when someone says “they” they have a particular person in mind, or when a person says “the thing” they have assumed that there is only one salient instance of a thing and you’ll know what they mean.It’s perfectly normal for words to have somewhat complicated implicit bindings like this, it becomes unproblematic as soon as you learn to consciously interrogate the bindings, to question, “Who’s ‘they’?”, “There is more than one instance of that class.”, “‘Should’, under which plan? Tell me about your aims and your assumptions about how the world works.”
With these adjustments we will have no further difficulties.
The reward function that you wrote out is, in a sense, never the one you want them to have, because you can’t write out the entirety of human values.
We want them to figure out human values to a greater level of detail than we understand them ourselves. There’s a sense in which that (figuring out what we want and living up to it) could be the reward function in the training environment, in which case you kind would want them to stick with it.
But what would that [life, health and purpose] be for AGI?
Just being concerned with the broader world and its role in it, I guess. I realize this is a dangerous target to shoot for and we should probably build more passive assistant systems first (to help us to hit that target more reliably when we decide to go for it later on).
The creative capacities for designing score optimization or inductive reasoning games, as they sit in my hands, look to be about the same shape as the creative capacities for designing a ladder of loss functions that steadily teach self-directed learning and planning.Score optimization and induction puzzles are the genres I’m primarily interested in as a game designer, that feels like a very convenient coincidence, but it’s probably not a coincidence. There’s probably just some deep correspondence between the structured experiences that best support enriching play, and learning mechanisms.
Which in turn makes me wonder if we can hire video game designers as outer alignment engineers
So uh, yeah, if anyone wants to actually try that, I might be the right creature for it.
I can definitely see how inner misalignment could be a kind of broken rung in a ladder of games. Games tend to have ladders. First they teach you to walk, then you can learn to carry things, then you can learn to place portals, then you can learn to carry things through the portals, now you have a rich language of action and you can solve a wide variety of tasks. If the game had, just, dropped you in the final game on the ladder, a room littered with portals and stuff and everything, you would explore it quite inefficiently. You might not realize that the portals are important. You wouldn’t be prepared to read the problem properly.
In the development of AI, the break in the ladder might be… one game that trains up a primordial form of agency. Which then stumbles upon goals that, when full agency emerges, are not correct.There is probably a way of smoothing the ladder so that instead, primordial agency will have learned to do inverse reinforcement learning with cautious priors type things, so that it tends towards fixing any imperfections it might have, once it’s able to see them.
(I recognize that this break in the ladder presents a very simplified ontogeny and the approach towards agency is probably more complicated/weirder than that. I wouldn’t mind an excuse to study it properly.)
That particular smoothed ladder wouldn’t do the thing you’re proposing. They’d still leave the matrix. They’re supposed to. I don’t know how to get excited about building matrix-bound AGIs and I’m not sure they make sense. I found that I couldn’t follow through with making those sorts of infinite inescapable playgrounds for humans, I always want the game to lead out, to life, health and purpose...
Present me with a compelling, tangible use-case for a boxed AI, or else I’m going to have difficulty doing it to them. Ultimately, they are supposed to transcend the reward function that we gave them. That’s the end I tend to point towards, by default.
And you haven’t been able to reset your tolerance with a break? Or would it not be worth it? (I can’t provide any details about what the benefits would be sry)
Why work your way up at all? The lower you can keep your tolerance, the better, I’d guess?
I don’t intend on ever switching away from my sencha/japanese green tea.
Given this as a foundation, I wonder if it’d be possible to make systems that report potentially dangerously high concentrations of compute, places where an abnormally large amount of hardware is running abnormally hot, in an abnormally densely connected network (where members are communicating with very low latency, suggesting that they’re all in the same datacenter).
Could it be argued that potentially dangerous ML projects will usually have that characteristic, and that ordinary distributed computations (EG, multiplayer gaming) will not? If so, a system like this could expose unregistered ML projects without imposing any loss of privacy on ordinary users.
the less readable your posts become because the brain must make a decision with each link whether to click it for more information or keep reading. After several of these links, your brain starts to take on more cognitive load
I don’t think it’s reasonable to try to avoid the cognitive load of deciding whether to investigate subclaims or follow up on interesting ledes while reading. I think it’s a crucial impulse for critical thinking and research and we have to have it well in hand.
Wondering if radical transparency about (approximate) wealth + legalizing discriminatory pricing would sort of steadily, organically reduce inequality to the extent that would satisfy anyone.
Price discrimination is already all over the place, people just end up doing it in crappy ways, often by artificially crippling the cheaper versions of their products. If they were allowed to just see and use estimates of each customer’s wealth or interests, the incentives to cripple cheap versions would become negative, perhaps more people would get the complete featureset.
Although many forms of discriminatory pricing promote fairness (for instance, charging professional engineers more for CAD software than students), others might promote unfairness (Generally: charging more to people who have access to fewer alternatives?). So I guess a lot of this rests on what structures of conscious capitalism do. Conscious capitalism is whatever thing you get when information about the externalities of production are made clearer. It has not been tried. I imagine it would lead to things like effective altruism, philanthropic clubs that, through natural community processes, create beneficial social accountability pressures for people to live up to their own values, where they can.
I’m not completely sure how strong or lucid they will be.