When I come to LW, I click to the Discussion almost instinctively. I’d estimate it has been four weeks since I’ve looked at Main. I sometimes read new Slate Star Codex posts (super good stuff, if you are unfamiliar) from LW’s sidebar. I sometimes notice interesting-sounding ‘Recent Comments’ and click on them.
My initial thought is that I don’t feel compelled to read Main posts because they are the LW-approved ideas, and I’m not super interested in listening to a bunch of people agreeing with another. Maybe that is a caricature, not sure.
Anyone else Discussion-centric in their LW use?
Also, the Meetup stuff is annoying noise. I’m very sympathetic if placing it among posts helps to drive attendance. By all means, continue if it helps your causes. But it feels spammy to me.
Activity seems like a positive feedback loop*- because there are more comments in discussion, people spend more time and comment more in discussion, and their comments in discussion are more likely to get responded to, which brings them back to discussion, and so on.
*That is, something that is both a cause and a result.
But why did I evolve to stop going to Main and go exclusively to Discussion? That behavior might be reinforced by the lack of activity, but the leading cause (for me in my best estimation) was I came to see the content as overwhelmingly LW-approved stuff.
When I read blacktrance’s comment, I see specific topics- AI, math, health, productivity- that they’re not interested in, that Main focuses on. When I read your comments, it sounds like you’re not as sensitive to topics as to styles of discussion, where you’re more interested in disagreements than in agreements. Am I reading that difference correctly?
Sure, I suppose. I generally use forum sites for discussion. I’m not too terribly interested in reading LW “publications”, I’m more interested in engagin in discussion and reading commentary in regard to issues pertaining to rationality, etc.
The distinction between Main and Discussion articles has noever made much sense to me. It seems to me to be some blend of perceived quality, relation to rationality (as LW defines it) and other LW topics of interest, group politics, EY mandate, etc. Don’t really care all that much...just that it was interesting that I ended up in Discussion almost exclusively.
I’d agree the topics in main seem to be less interesting to me, too, now that I think about it.
I’m more likely to find discussion topics and comments in my areas of interest, while Main seems to be mostly about AI, math, health, and productivity, none of which are particularly interesting for me.
If one is able to improve how people are matched, it would bring about a huge amount of utility for the entire world.
People would be happier, they would be more productive, there would be less of the divorce-related waste. Being in a happy couple also means you are less distracted by conflict in the house, which leads to people better able to develop themselves and achieve their personal goals. You can keep adding to the direct benefits of being in a good pairing versus a bad pairing.
But it doesn’t stop there. If we accept that better matched parents raise their children better, then you are looking at a huge improvement in the psychological health of the next generation of humans. And well-raised humans are more likely to match better with each other...
Under this light, it strikes me as vastly suboptimal that people today will get married to the best option available in their immediate environment when they reach the right age.
The cutting-edge online dating sites base their suggestions on a very limited list of questions. But each of us outputs huge amounts of data, many of them available through APIs on the web. Favourite books, movies, sleep patterns, browsing history, work history, health data, and so much more. We should be using that data to form good hypotheses on how to better match people. I’m actually shocked at the underinvestment in this area as a legitimate altruistic cause.
If an altruistic group of numbers-inclined people was to start working together to improve the world in a non-existential risk reducing kind of way, it strikes me that a dating site may be a fantastic thing to try. On the off-chance it actually produces real results, Applied Rationality will also have a great story of how it improved the world. And, you know, it might even make money.
There seem to be perverse incentives in the dating industry. Most obviously: if you successfully create a forever-happy couple, you have lost your customers; but if you make people date many promissingly-looking-yet-disappointing partners, they will keep returning to your site.
Actualy, maybe your customers are completely hypocritical about their goals: maybe “finding a true love” is their official goal, but what they really want is plausible deniability for fucking dozens of attractive strangers while pretending to search for the perfect soulmate. You could create a website which displays the best one or two matches, instead of hundreds of recommendations, and despite having higher success rate for people who try it, most people will probably be unimpressed and give you some bullshit excuses if you ask them.
Also, if people are delusional about their “sexual market value”, you probably won’t make money by trying to fix their delusions. They will be offended by the types of “ordinary” people you offer them as their best matches, when the competing website offers them Prince Charming (whose real goal is to maximize his number of one night stands) or Princess Charming (who really is a prostitute using the website to find potential clients). They will look at the photos and profiles from your website, and from the competing website, and then decide your website isn’t even worth trying. They may also post an offended blog review, and you bet it will be popular on social networks.
So you probably would need to do this as a non-profit philantropic activity.
EDIT: I have an idea about how to remove the perverse incentives, but it requires a lot of trust in users. Make them pay if they have a happy relationship. For example if the website finds you a date, set a regular payment of $5 each month for the next 10 years; if the relationship breaks, cancel the payment. The value of a good relationship is higher than $5 a month, but the total payment of $600 could be enough for the website.
That sounds a lot like really wanting a soulmate and an open relationship.
That’s a nice thing to have; I am not judging anyone. Just thinking how that would influence the dating website algorithm, marketing, and the utility this whole project would create.
If some people say they want X but they actually want Y… however other people say they want X and they mean it… and the algorithm matches them together because the other characteristics match, at the end they may be still unsatisfied (if one of these groups is a small minority, they will be disappointed repeatedly). This could possibly be fixed by an algorithm smart enough that it could somehow detect which option it is, and only match people who want the same thing (whichever of X or Y it is).
If there are many people who say they want X but really want Y, how will you advertise the website? Probably by playing along and describing your website mostly as a site for X, but providing obvious hints that Y is also possible and frequent there. Alternatively, by describing your website as a site for X, but writing “independent” blog articles and comments describing how well it actually works for Y. (What is the chance that this actually is what dating sites are already doing, and the only complaining people are the nerds who don’t understand the real rules?)
Maybe there is a market in explicitly supporting open relationships. (Especially if you start in the Bay Area.) By removing some hypocrisy, the matching could be made more efficient—you could ask questions which you otherwise couldn’t, e.g. “how many % of your time would you prefer to spend with this partner?”.
I wouldn’t jump to malice so fast when incompetence suffices as an explanation. Nobody has actually done the proper research. The current sites have found a local maxima and are happy to extract value there. Google got huge by getting people off the site fast when everyone else was building portals.
You will of course get lots of delusionals, and lots of people damaged enough that they are unmatchable anyway. You can’t help everybody. But also the point is to improve the result they would otherwise have had. Delusional people do end up finding a match in general, so you just have to improve that to have a win. Perhaps you can fix the incentive by getting paid for the duration of the resulting relationship. (and that has issues by itself, but that’s a long conversation)
I don’t think the philanthropic angle will help, though having altruistic investors who aren’t looking for immediate maximisation of investment is probably a must, as a lot of this is pure research.
You could create a website which displays the best one or two matches, instead of hundreds of recommendations, and despite having higher success rate for people who try it, most people will probably be unimpressed and give you some bullshit excuses if you ask them.
I think that’s the business model of eharmony and they seem to be doing well.
I wonder to what extent the problems you describe (divorces, conflict, etc) are caused mainly by poor matching of the people having the problems, and to what extent they are caused by the people having poor relationship (or other) skills, relatively regardless of how well matched they are with their partner? For example, it could be that someone is only a little bit less likely to have dramatic arguments with their “ideal match” than with a random partner—they just happen to be an argumentative person or haven’t figured out better ways of resolving disagreements.
I wonder to what extent the problems you describe (divorces, conflict, etc) are caused mainly by poor matching of the people having the problems, and to what extent they are caused by the people having poor relationship (or other) skills, relatively regardless of how well matched they are with their partner?
Well, the success of arranged marriages in cultures that practice them suggests the “right match” isn’t that important.
What makes you think these marriages are successful? Low divorce rates are not good evidence in places where divorce is often impractical.
Three main points in favor of arranged marriages that I’m aware of:
The marriages are generally arranged by older women, who are likely better at finding a long-term match than young people. (Consider this the equivalent of dating people based on okCupid match rating, say, instead of hotornot rating.)
The expectations people have from marriage are much more open and agreed upon; like Prismattic points out, they may have a marriage that a Westerner would want to get a divorce in, but be satisfied. It seems to me that this is because of increased realism in expectations (i.e. the Westerner thinks the divorce will be more helpful than it actually will, or is overrating divorce compared to other options), but this is hard to be quantitative about.
To elaborate on the expectations, in arranged marriages it is clear that a healthy relationship is something you have to build and actively maintain, whereas in love marriages sometimes people have the impression that the healthy relationship appears and sustains itself by magic- and so when they put no work into maintaining it, and it falls apart, they claim that the magic is gone rather than that they never changed the oil.
I also think most modern arranged marriages involve some choice on the part of the participants- “meet these four people, tell us if you can’t stand any of them” instead of “you will marry this one person.”
Forty-five individuals (22 couples and 1 widowed person) living in arranged marriages in India completed questionnaires measuring marital satisfaction and wellness. The data were compared with existing data on individuals in the United States living in marriages of choice. Differences were found in importance of marital characteristics, but no differences in satisfaction were found. Differences were also found in 9 of 19 wellness scales between the 2 groups. Implications for further research are considered.
Results from the analyses revealed that arranged marrieds were significantly higher in marital satisfaction than were the love marrieds or companionate marrieds.
Unexpectedly, no differences were found between participants in arranged and love-based marriages; high ratings of love, satisfaction, and commitment were observed in both marriage types. The overall affective experiences of partners in arranged and love marriages appear to be similar, at least among Indian adults living in contemporary U.S. society.
Multiple regression analyses indicate that wives in Chengdu love matches are more satisfied with their marital relationships than their counterparts in arranged marriages, regardless of the length of the marriage, and that this difference cannot be attributed to the influence of other background factors that differentiate these two types of women.
I’m not sure this is correct. That is to say, the empirical point that divorce is much less common in arranged marriage cultures is obviously true. But
a) I think there is some correlation between prevalence arranged marriage and stigma associated with divorce, meaning that not getting divorced does not necessarily equal happy marriage.
b) The bar for success in 20th-21st century western marriages is set really high. It’s not just an economic arrangement; people want a best friend and a passionate lover and maybe several other things rolled into one. When people in traditional cultures say that their marriages are “happy,” they may well mean something much less than what affluent westerners would consider satisfactory.
My instinct on this is driven by having been in bad and good relationships, and reflecting on myself in those situations. It ain’t much, but it’s what I’ve got to work with. Yes, some people are unmatchable, or shouldn’t be matched. But somewhere between “is in high demand and has good judgement, can easily find great matches” and “is unmatchable and should be kept away from others”, there’s a lot of people that can be matched better. Or that’s the hypothesis.
Seems reasonable, although I’d still wonder just how much difference improving the match would make even for the majority of middle-ground people. It sounded in the grandparent post (first and fourth paragraphs particularly) that you were treating the notion that it would be “a lot” as a premise rather than a hypothesis.
Well, it’s more than a hypothesis, it’s a goal. If it doesn’t work, then it doesn’t, but if it does, it’s pretty high impact. (though not existential-risk avoidance high, in and of itself).
Finding a good match has made a big subjective difference for me, and there’s a case it’s made a big objective difference (but then again, I’d say that) and I had to move countries to find that person.
Yeah, maybe the original phrasing is too strong (blame the entrepreneur in pitch mode) but the 6th paragraph does say that it’s an off-chance it can be made to work, though both a high improvement potential and a high difficulty in materialising it are not mutually exclusive.
The problem with dating sites (like social network sites or internet messengers) is that the utility you can gain from it is VERY related to how many other people are actually using it. This means that there is a natural drift towards a monopoly. Nobody wants to join a dating site that only has 1000 people. If you do not have a really good reason to think that your dating site idea will get off the ground, it probably wont.
One way you could possibly get past this is to match people up who do not sign up or even know about this service.
For example, you could create bots that browse okcupid, for answers to questions, ignore okcupid’s stupid algorithms in favor of our own much better ones, and then send two people a message that describes how our service works and introduces them to each other.
Is this legal? If so, I wonder if okcupid would take stop it anyway.
The chicken/egg issue is real with any dating site, yet dating sites do manage to start. Usually you work around this by focusing on a certain group/location, dominating that, and spreading out.
Off the cuff, the bay strikes me as a potentially great area to start for something like this.
and then send two people a message that describes how our service works and introduces them to each other.
Awesome—that will fit right in between “I’m a Nigerian customs official with a suitcase of cash” emails and “Enlarge your manhood with our all-natural pills” ones.
P.S. Actually it’s even better! Imagine that you’re a girl and you receive an email which basically says “We stalked you for a while and we think you should go shack up with that guy”. Genius!
How can there be a monopoly if people can use more than one dating site?
Unless OkCupid bans you from putting your profile up on other sites, you can just as easily put a profile on another site with less people, if the site seems promising.
I don’t mean to be nitpicking, but a monopoly is a very specific thing. It’s quite different than it just being inconvenient to switch to a competitor. In very many cases in normal market competition, it’s inconvenient to switch to competitors (buying a new car or house, changing your insurance, and so on), but that doesn’t effect the quality of the product. Similarly, for a monopoly to effect the quality of OKCupid’s service, it would have to be a very specific situation, and different than what currently exists, which seems to be quite normal market functioning.
Unless OKCupid is hiring the government or people with guns to threaten other websites out of existence, there won’t be a drift towards a monopoly.
A monopoly isn’t created by one company getting the overwhelming majority of customers. A monopoly is only created when competitors cannot enter the market. It’s a subtle distinction but it’s very important, because what’s implied is that the company with the monopoly can jack up their prices and abuse customers. They can’t do this without feeding a garden of small competitors that can and will outgrow them (see Myspace, America Online, etc), unless those competitors are disallowed from ever existing.
You can keep downvoting this, but it’s a very important concept in economics and it will still be true.
Here is one improvement to OKcupid, which we might even be able to implement as a third party:
OKcupid has bad match algorithms, but it can still be useful as searchable classified adds. However, when you find a legitimate match, you need to have a way to signal to the other person that you believe the match could work.
Most messages on OKcupid are from men to women, so women already have a way to do this: send a message, however men do not.
Men spam messages, by glancing over profiles, and sending cookie cutter messages that mention something in the profile. Women are used to this spam, and may reject legitimate interest, because they do not have a good enough spam filter.
Our service would be to provide an I am not spamming commitment. A flag that can be put in a message which signals “This is the only flagged message I have sent this week”
It would be a link, you put in your message, which sends you to a site that basically says. Yes, Bob(profile link) has only sent this flag to Alice(profile link) in the week of 2/20/14-2/26/14, with an explanation of how this works.
Do you think that would be a useful service to implement? Do you think people would actually use it, and receive it well?
I wonder if a per-message fee for a certain kind of message would be a good business model for this. My suspicion is that it would work very well if all your users had that reluctance to ever spend anything online (people are much more willing to buy utilions that involve getting a physical product than to pay for things like apps)), but it breaks down as soon as someone with some unused disposable income realizes that spamming $1 notes isn’t that expensive.
Only being able to send a certain number of messages per week of a special type might be enough for indicating non-spam, as long as you could solve the problem of people making multiple profiles to get around it. Having a small fee attached to the service might help with tracking that down, since it would keep people from abusing it too extremely, and cover the cost of having someone investigate suspicious accounts (if more than one is paid for by the same credit card at around the same time, for example).
OKcupid solves the multiple account problem for us. It is probably better to not send a virtual rose than to make an account that you then have to answer all the questions to.
Our service would be to provide an I am not spamming commitment. A flag that can be put in a message which signals “This is the only flagged message I have sent this week”
Where will your credibility come from?
Alice receives a message from Bob. It says “You’re amazing, we’re nothing but mammals, let’s do it like they do on the Discovery Channel”, and it also says “I, Mallory, hereby certify that Bob only talked about mammals once this week—to you”.
Why should Alice believe you?
Things like that are technically possible (e.g. cryptographic proofs-of-work) but Alice is unlikely to verify your proofs herself and why should she trust Mallory, anyway?
I think if we had a nice professional website, with a link to a long description of how it all works that people won’t read anyway, they will tend to trust us.
Seconded—once you get as far as people trusting you enough to post their personal information and possibly pay you for the service, they’re not still suspecting you of letting people spam you with “certified” non-spam.
OK Cupid has a horrible match percent algorithm. Basically someone who has a check list of things that their match cannot be will answer lots of questions as “this matters a lot to me” and “any of these options are acceptable except for this one extreme one that nobody will click anyway.” The stupid algorithm will inflate this person’s match percent with everyone.
So, if you look at people with high compatibility with you, that says more about their question answering style, than how much you have in common.
This is why the algorithm is horrible in theory. In practice my one example is that I am getting married in a month to someone I met on OKcupid with 99% compatibility.
A good website design could change the answering style. Imagine a site where you don’t fill out all the answers at once. Instead it just displays one question at a time, and you can either answer it or click “not now”. The algorithm would prioritize the questions it asks you dynamically, using the already existing data about you and your potential matches—it would ask you the question which it expects to provide most bits of information.
Also, it would use the math properly. The compatiblity would not be calculated as number of questions answered, but number of bits these answers provide. A match for “likes cats” provides more bits than “is not a serial killer”.
Very consistently people that I know and like, when I see them on okcupid, have a high match percentage. When I meet okcupid people with a good match percentage, I usually like them. This seems to imply the algorithm is a lot better than your theoretical worst example of it. I think your situation is much more of a problem if you don’t answer enough questions.
Perhaps the way people tend to answer questions does not change very much from person to person, so this problem does not show up in practice.
However, if you are willing to change your style for answering questions, it is probably possible to game OKcupid in such a way that you get 90+% with anyone you would care about.
Selfdefeating The entire point of OKcupid is to find someone you will actually click with. Inflating your own match percentages artificially just makes OKCupid worse for you. Of course, this doesnt help if the site just isnt very popular in your city.
Eh. Radical: Have the government do this. Literally, run a dating site, have sex-ed classes teach people how to use it, and why gaming it is bloody stupid. That should result in maximum uptake, and would cost a heck of a lot less than a lot of other initiatives governments already run trying to promote stable pairbonds.
Now, how to get this into a political platform…
Still pointless! There is no upside to having a bunch of people you are not actually compatible with think the mirage you constructed is a good match. If they are not a match with your honest profile, you do not want to waste theirs or your own time.
If your actual goal is to have a bunch of one night stands, then make a profile that out and out states that so that you will be matched with people of like mind. Dishonesty in this matter is both unetical and nigh certain to result in unpleasant drama.
Proper use of this kind of tool is an exercise in luminosity—the more accurately you identify what you are truely looking for, the better it works.
Also, see radical proposal: If a site of this type is run by the government, sockpuppets are obviously not going to be an option—one account per social security number or local equivalent, because that is a really simple way to shut down a whole host of abuses.
I’ve had ideas sort of like this at the back of my mind since seeing Paul Graham pointing out how broken online dating is in one of his essays. (Not so much analyzing all of someone’s existing data, but analyzing IM transcripts to match people with IM buddies they’d be likely to make good friends with is a thing I considered doing.) Haven’t gotten too far with any of them yet, but I’m glad you reminded me, since I was planning on playing with some of my own data soon just to see what I find.
Do you think that not having dated much would be much of a comparative disadvantage in working on this problem? That’s one of the reasons I hesitate to make it my main project.
A possibly-related problem—why does every site I see that says it is for matching strangers who might like to be friends get full of people looking for a date? (Small sample size, but I’ve never seen one that didn’t give me the sense that the vast majority of the members were looking for romance or a one night stand or something.)
A possibly-related problem—why does every site I see that says it is for matching strangers who might like to be friends get full of people looking for a date?
So that people can look for dates without breaking plausible deniability.
So that people can look for dates without breaking plausible deniability.
I think it’s the web site, rather than its clients, that needs the plausible deniability. It cannot seem to be in the business of selling sex, so it has to have a wider focus.
If an altruistic group of numbers-inclined people was to start working together to improve the world in a non-existential risk reducing kind of way, it strikes me that a dating site may be a fantastic thing to try.
Why altruistic? If it’s worth anything, it’s worth money. If it won’t even pay its creators for the time they’ll put in to create it, where’s the value?
I am not convinced it is the optimal route to startup success. If it was, I would be doing it in preference over my current startup. It is highly uncertain and requires what looks like basic research, hence the altruism angle. If it succeeds, yes, it shouldake a lot of money and nobody should deprive it’s creators of the fruits of their labour.
It strikes me that it is much more plausible to argue that the dating market suffers from market failure through information asymmetry, market power and high search costs than to argue the same about economic activity. Yet although people search high and low to find (often non-existent) market failures to justify economic interventions, interventions in the dating market are greeted with near-uniform hostlility. I predict that, outside of LessWrong, your proposal will generate a high “Ick” factor as a taboo violation. “Rationality-based online dating will set you up with scientifically-chosen dates...” this is likely to be an anti-selling point to most users.
Obviously you’d take a different angle with the marketing.
Off the cuff, I’d pitch it as a hands-off dating site. You just install a persistent app on your phone that pushes a notification when it finds a good match. No website to navigate, no profile to fill, no message queue to manage.
Perhaps market it to busy professionals. Finance professionals may be a good target to start marketing to. (busy, high-status, analytical)
There would need to be some way to deal with the privacy issues though.
This might be a reason to start it out as a nice thing. Though, the problem is finding a niche that likes this proposal and has a decent gender ratio (or enough people interested in dates of the same gender).
Now that I think about it, existing dating sites do try to advertise themselves as being better because of their algorithm. If that advertising works, maybe the ick factor isn’t that strong?
Viliam_Bur sort of said this, but it doesn’t seem possible to outcompete the existing websites due to perverse incentives.
If I build a site optimizing for long term success, and another dating site optimizes for an intense honeymoon phase (which encourages people to come back and spread the word about the site) then I will lose. And optimizing for long term success is really hard since feedback occurs on the order of decades.
Of course I’m assuming that intense short term happiness and long term stability aren’t very highly correlated and I could be wrong. I’m also assuming that stability is desirable—I’d be curious if anyone disagrees.
Companies are trying, unfortunately the incentives seem sort of messed up to me. Dating websites have an incentive to encourage people to use their service, not get into wonderful long term relationships. Hence I would expect them to optimize for relationships with an intense honeymoon phase, rather than relationships with a high chance of long term success and compatibility.
Since we’re after long term success, feedback will occur on the order of decades—making this a very hard optimization problem.
How do you pick a career if your goal is to maximize your income (technically, maximize the expected value of some function of your income)? The sort of standard answer is “comparative advantage”, but it’s unclear to me how to apply that concept in practice. For example how much demand there is for each kind of job is obviously very important, but how do you take that into consideration, exactly? I’ve been thinking about this and came up with the following. I’d be interested in any improvements or alternative ideas.
For each career under consideration, estimate your potential income ranking or percentile within that career if you went into it (as a probability distribution).
For each career, estimate its income distribution (how much will the top earner make, how much will the second highest earner make, etc.).
From 1 and 2, obtain a probability distribution of your income within each career.
If you have a high IQ and are good at math go into finance. If you have a high IQ, strong social skills but are bad at math go into law. If you have a high IQ, a good memory but weak social and math skills become a medical doctor. If you have a low IQ but are attractive marry someone rich. If you have a very low IQ get on government benefits for some disability and work at an under-the-table job.
Medical doctors are paid well in many places other than the US, though not as well as in the US. (For that matter, most other well-paid jobs are better paid in the US than anywhere else. Software development, law, senior management, etc.)
Also, though of course this was no part of the original question, medicine offers more confidence than most careers that your work is actually making the world a better place. (Which may not actually be the right question to ask, of course—what matters is arguably the marginal effect, and if you’re well paid and care enough about people in poor countries you may well be able to do more good by charitable donations than you ever could directly by your work. But it’s a thing many people care about.)
I think that’s intended. Trying to achieve greater wealth generally involves much higher risk, and even if it offers a higher expected value in terms of money, the diminishing utility of wealth probably makes the expected utility of, say, creating a startup, lower than just pursuing a middle-class career that matches your skills.
Well, Wei Dai said “maximize the expected value of some function of your income”; which career achieves that will depend on whether the function is log(x), x, H(x - $40,000/year), exp(x/($1M/year)), or what.
The vast majority of people who play sports have fun and don’t receive a dime for it. A majority of people who get something of monetary value out of playing sports get a college degree and nothing else.
I don’t have good numbers, but it’s likely less dangerous than you think it is. The vast majority of what an infantryman does falls into two categories—training, and waiting. And that’s a boots on ground, rifle in hand category—there’s a bunch of rear-echelon ratings as well.
I’m guessing that it’s likely within an order of magnitude of danger as commuting to work. Likely safer than delivering pizzas. There’s probably a lot of variance between specific job descriptions—a drone operator based in the continental US is going to have a lot less occupational risk than the guy doing explosive ordnance disposal.
From what I’ve read, a couple of the issues for drone pilots is that they’ve been killing people who they’ve been watching for a while, and that they feel personal responsibility if they fail to protect American soldiers.
In the year 1940, working as an enlisted member of the army supply chain was probably safer than not being in the army whatsoever—regular Joes got drafted.
Besides which, the geographical situation of the US means that a symmetrical war is largely going to be an air/sea sort of deal. Canada’s effectively part of the US in economic and mutual-defense terms, and Mexico isn’t much help either. Mexico doesn’t have the geographical and industrial resources to go toe-to-toe with the US on their own, the border is a bunch of hostile desert, and getting supplies into Mexico past the US navy and air force is problematic.
Yes, and in particular it’ll involve enemy drones. Drone operators are likely to be specifically targeted.
That makes them safer, ironically. If your command knows that you’re likely to be targeted and your contributions are important to the war effort, they’ll take efforts to protect you. Stuff you down a really deep hole and pipe in data and logistical support. They probably won’t let you leave, either, which means you can’t get unlucky and eat a drone strike while you’re enjoying a day in the park.
You’re at elevated risk of being caught in nuclear or orbital kinetic bombardment, though… but if the war gets to that stage your goose is cooked regardless of what job you have.
Another bonus of enlisting: basic skills will be drilled into so thoroughly they will be fully into your System I allowing you extra executive function (thereby causing you to punch above your weight in terms of intelligence). Although, there is some ethical risk involved.
Another bonus of enlisting: basic skills will be drilled into so thoroughly they will be fully into your System I allowing you extra executive function.
Does anyone know if finance requires strong math and social skills? I assumed it did—social skills for creating connections, and math skills for actually doing to job.
And if you do have poor social skills, then practice! Social skills are really important. I’m still working on this.
This is some guesswork, but some other possible combinations:
Strong social skills, above average IQ—management?
Above average IQ, good math skills—accounting?
Rich parents, family business—take over said business eventually.
Middle class parents, fair amount of property, good location—rent.
Rich parents, strong social skills—network through their connections.
If you have a high IQ, strong social skills but are bad at math go into law.
Is this still true? Recently there have been reports about an oversupply of lawyers and scandals involving law schools fudging the statistics on the salaries of their graduates.
US law is a spectacularly bad choice at the moment. There is far to many law schools, and as a consequence, too many law graduates, the degree costs a fortune and employment prospects are outright bad. Do not do this.
Finance is an implicit bet that wallstreet will not get struck down by the wrath of the electorate just as you finish your education.
Honestly? If riches really is what you want, go into business for yourself. A startup, or at the low end just being a self-employed contractor has good returns and this is not likely to change. Programming, the trades, a good set of languages and an import-export business..
Well, as I understand it part of the issue is that a lot of the grunt work that used to require lots of lawyers to do, e.g., looking through piles of documents for relevant sections, can now be automated.
There’s a high failure rate in finance, too—it’s just hidden in the “up or out” culture. It’s a very winner-takes-all kind of place, from what I’ve heard.
If you want to be a portfolio manager who makes, say, macro bets, yes, it’s very much up or out. But if you want to be a quant polishing fixed income risk management models in some bank, it’s a pretty standard corporate job.
Startups are shockingly diverse too. And despite the super-high failure rates I hear about, the group of friends I’ve been tracking the past 5 years or so seem to be doing pretty darn well, despite some of them having failures indeed.
I strongly suspect the degree of failure in startups correlates inversely with rationality skills (as it should) so rationalists should not be placing themselves on the same reference category as everyone else. Execution skills matter a lot too, but doing a startup has worked miracles for my motivation too.
This isn’t “I’m smart and rules don’t apply”. Smartness alone doesn’t help.
But, to put it this way, if rationality training doesn’t help improve your startup’s odds of success, then there’s something wrong with the rationality training.
To be more precise, in my experience, a lot of startup failure is due to downright stupidity, or just ignoring the obvious.
Also, anecdotally, running a startup has been the absolute best on-the-job rationality training I’ve ever had.
Shockingly, successful entrepreneurs I’ve worked with score high on my rationality test, which consists of how often they say things that are uncontested red flags, and how well-reasoned their suggested courses of action are. In particular, one of our investors is the closest approximation to a bayesian superintelligence I’ve ever met. I can feed him data & news from the past week, and almost hear the weighting of various outcomes shift in his predictions and recommendations.
In short,
Rationalists are more likely to think better, avoid obvious errors.
Thinking better improves chances of startup success
Rationalists have better chances of startup success.
I do understand this sounds self-serving, but I also try to avoid the sin of underconfidence. In my experience, quality of thinking between rationalists and the average person tends to be similar to quality of conversation here versus on YouTube. The problem is when rationalists bite off more than they can chew in terms of goals, but that’s a separate problem.
What you say sounds intuitive to me at first, but as of now I would say that rationality training may boost start up success rates up just a little.
Here is some reasons why rationality might not matter that much:
People tend to be a bit more rational when it counts, like making money. So having correct beliefs about many things doesn’t really give you an edge because the other guy is also pretty rational for business stuff.
Well, at this point we’re weighing anecdotes, but..
Yes! They do tend to push their rationality to the limit. Hypothesis: knowing more about rationality can help push up the limit of how rational one can be.
Yes! It’s not about rationality alone. Persistent determination is quite possibly more important than rationality and intelligence put together. But I posit that rationality is a multiplier, and also tends to filter out the most destructive outcomes.
In general, I’d love to see some data on this, but I’m not holding my breath.
Agreed. Interestingly, the latest post in main points to evidence supporting rationality having a significant relation to success in the work place – not the same as entrepreneurship, nonetheless I update slightly more in favor of your position.
I agree that a more rational person has a greater chance, ceteris paribus. Question is, how much greater.
A part of the outcome is luck; I don’t know how big part. Also, the rationality training may improve your skills, but just to some degree.
(Data point: myself. I believe I am acting more rationally after CFAR minicamp than before, and it seems to be reflected by better outcomes in life, but there is still a lot of stupid things I do. So maybe my probability of running a successful startup has increased from 1% to 3%.)
I question the stats that says 1% success rate for startups. I will need to see the reference, but one I had access to basically said “1% matches or exceeds projections shown to investors” or some such. Funnily enough, by that metric Facebook is a failure (they missed the goal they set in the convertible note signed with Peter Thiel). If run decently, I would expect double digit success rates, for a more reasonable measure of success. If a driven, creative rationalist is running a company, I would expect a very high degree of success.
Another thing much more common in rationalists than the common population is the ability to actively solicit feedback, reflect and self-modify. This is surprisingly rare. And incredibly vital in a startup.
Success at startups is not about not doing stupid things. I’ve made many MANY mistakes. It’s about not doing things stupid enough to kill your company. Surprisingly, the business world has a lot of tolerance for error, as long as you avoid the truly bad ones.
It is hard to survey startups. What is usually done is to measure success rates of companies that raised a Series A round of funding. Many companies fail before achieving that, though they necessarily fail faster, producing less opportunity cost.
Here is a chart of returns to a VC, taken from this paper by a different author. 60% of dollars invested are in companies that lost the VCs money (lost them 85%). This is a top VC that managed to triple its money, so this is an overestimate of success of a regular VC-backed company. This is a common bias in these surveys.
Based on the fictitious figure 2, 63% of dollars is actually 69% of companies, because successful companies get more funding. So 31% of companies with a Series A round at a top firm succeed by the metric of a positive return to the VCs. Double digit success would require that at least 1⁄3 of startups get a Series A funding and that companies funded by typical VCs are as successful as companies funded by a top VC.
The appropriate definition of success is comparing to opportunity cost. In particular, the above analysis fails to take into account duration. Here is a paper that makes a reasonable comparison and concludes that running a company with a Series A round was a good decision for people with $700k in assets. Again, skipping to the Series A round is not a real action, thus overestimating the value of the real action of a startup. There is an additional difficulty that startups may have non-monetary costs and benefits, such as stress and learning. Edit: found the paper. According to Figure 2, that 75% of VC-backed firms exit at 0, not much worse than at the top VC considered above.
Well Paul Graham has built quite a successful incubator apparently largely based on his ability to predict success of start-ups based on a half-hour interview.
I’m not sure how much the interviews add compared to the Y Combinator model of investing in a lot of startups very early on at unusually favorable terms, integrating with Hacker News, and building a YC community with alumni & new angels. (As far as the latter goes, you can ask AngryParsley why he went into YC for Floobits: it wasn’t because he needed their cash.)
But if you want to be a quant polishing fixed income risk management models in some bank,
What kind of social skills does that require? My impression is that this is the modern equivalent of court astrologer and requires some similar skills, i.e., cold reading.
Not much—the usual ones for holding a corporate job (wear business casual, look neat, don’t smell, don’t be a weirdo). Quants are expected to be nerdy/geeky.
My impression is that this is the modern equivalent of court astrologer
Not at all. Finance has the advantage of providing rapid and unambiguous feedback for your actions.
Finance has the advantage of providing rapid and unambiguous feedback for your actions.
If you’re trading yes, although the feedback is extremely noisy. If you’re designing models not so much. Incidentally a lot of the quants I know are also good at doing Tarot readings, whether they believe the cards have power or not.
That very much depends on what kind of strategy you’re trading. For example, HFT doesn’t have problems with noise.
If you’re designing models not so much.
Yes, so much. Your model has to work well on historical data and if it makes it to production, it will have performance metrics that it will have to meet.
The other thing to keep in mind about failure rates is where you end up if you fail—what other careers you can go into with the same education. (In the case of startups, you can keep trying more startups, and you’re more likely to succeed on the second or third than you were on the first. I don’t know how it is in finance.)
If you have a low IQ but are attractive marry someone rich.
I’m not sure I would count that as “your income”, though in jurisdictions with easy divorces and large alimony it might be as good for all practical purposes.
Depends on how high you are aiming for. For a good investment banking position you need a high enough IQ to either get into a top 10 school or be in the top 10% of a school such as Smith College.
For students at Smith College the normal path is you get very high grades and take some math-heavy courses, get a summer internship with an investment bank after your junior year of college which results in a full time job offer, then after 2-5 years you get an MBA and then get a more senior position at an investment bank.
“Career” is an unnatural bucket. You don’t pick a career. You choose between concrete actions that lead to other actions. Imagine picking a path through a tree. This model can encompass the notion of a career as a set of similar paths. Your procedure is a good way to estimate the value of these paths, but doesn’t reflect the tree-like structure of actual decisions. In other words, options are important under uncertainty, and the model you’ve listed doesn’t seem to reflect this.
For example, I’m not choosing between (General Infantry) and (Mathematician). I’m choosing between (Enlist in the Military) and (Go to College). Even if the terminal state (General Infantry) had the same expected value as (Mathematician), going to college should more valuable because you will have many options besides (Mathematician) should your initial estimate prove wrong, while enlisting leads to much lower branching factor.
How should you weigh the value of having options? I have no clue.
Your goal is likely not to maximize your income. For one, you have to take cost of living into account—a $60k/yr job where you spend $10k/yr on housing is better than a $80k/yr (EDIT:$70k/yr, math was off) job where you spend $25k/yr on housing.
For another, the time and stress of the career field has a very big impact on quality-of-life. If you work sixty hour weeks, in order to get to the same kind of place as a forty hour week worker you have to spend money to free up twenty hours per week in high-quality time. That’s a lot of money in cleaners, virtual personal assistants, etc.
As far as “how do I use the concept of comparative advantage to my advantage”, here’s how I’d do it:
Make a list of skills and preferences. It need not be exhaustive—in fact, I’d go for the first few things you can think of. The more obvious of a difference from the typical person, the more likely it is to be your comparative advantage. For instance, suppose you like being alone, do not get bored easily by monotonous work, and do not have any particular attachment to any one place.
Look at career options and ask yourself if that is something that fits your skills and preferences. Over-the-road trucking is a lot more attractive to people who can stand boredom and isolation, and don’t feel a need to settle down in one place. Conversely, it’s less attractive to people who are the opposite way, and so is likely to command a higher wage.
Now that you have a shorter list of things you’re likely to face less competition for or be better at, use any sort of evaluation to pick among the narrower field.
A $60k/yr job where you spend $10k/yr on housing is better than a $80k/yr job where you spend $25k/yr on housing.
You should consider option values, especially early in your career. It’s easier to move from high paying job in Manhattan to a lower paying job in Kansas City than to do the reverse.
Update the choice by replacing income with the total expected value from job income, social networking, and career options available to you, and the point stands.
Probably the cost of housing correlates with other expenses, and also there’s income tax to consider, but on the surface the first job is $50k/yr net, the second job is $55k/yr net, and so it looks like the latter better.
In addition to maximizing income, maximizing savings/investments is very important. You can be poor off of a $500,000 salary and rich off of a $50,000 salary.
In “The Fall and Rise of Formal Methods”, Peter Amey gives a pretty good description of how I expect things to play out w.r.t. Friendly AI research:
Good ideas sometimes come before their time. They may be too novel for their merit to be recognised. They may be seen to threaten some party’s self interest. They may be seen as simply too hard to adopt. These premature good ideas are often swept into corners and, the world, breathing a sigh of relief, gets on with whatever it was up to before they came along. Fortunately not all good ideas wither. Some are kept alive by enthusiasts, who seize every opportunity to show that they really are good ideas. In some cases the world eventually catches up and the original premature good idea, honed by its period of isolation, bursts forth as the new normality (sometimes with its original critics claiming it was all their idea in the first place!).
Formal methods (and I’ll outline in more detail what I mean by ‘formal methods’ shortly) are a classic example of early oppression followed by later resurgence. They arrived on the scene at a time when developers were preoccupied with trying to squeeze complex functionality into hardware with too little memory and too slow processors.
...[But now] formal methods… are on the rise. And why not? What is the alternative? If we don’t use them, then our tool box contains just one spanner labelled ‘test’. We know that for anything that justifies the label ‘critical’, testing will never be enough...
Introduction
I suspected that the type of stuff that gets posted in Rationality Quotes reinforces the mistaken way of throwing about the word rational. To test this, I set out to look at the first twenty rationality quotes in the most recent RQ thread. In the end I only looked at the first ten because it was taking more time and energy than would permit me to continue past that. (I’d only seen one of them before, namely the one that prompted me to make this comment.)
A look at the quotes
In our large, anonymous society, it’s easy to forget moral and reputational pressures and concentrate on legal pressure and security systems. This is a mistake; even though our informal social pressures fade into the background, they’re still responsible for most of the cooperation in society.
There might be an intended, implicit lesson here that would systematically improve thinking, but without more concrete examples and elaboration (I’m not sure what the exact mistake being pointed to is), we’re left guessing what it might be. In cases like this where it’s not clear, it’s best to point out explicitly what the general habit of thought (cognitive algorithm) is that should be corrected, and how one should correct it, rather than to point in the vague direction of something highly specific going wrong.
As the world becomes more addictive, the two senses in which one can live a normal life will be driven ever further apart. One sense of “normal” is statistically normal: what everyone else does. The other is the sense we mean when we talk about the normal operating range of a piece of machinery: what works best.
These two senses are already quite far apart. Already someone trying to live well would seem eccentrically abstemious in most of the US. That phenomenon is only going to become more pronounced. You can probably take it as a rule of thumb from now on that if people don’t think you’re weird, you’re living badly.
Without context, I’m struggling to understand the meaning of this quote, too. The Paul Graham article it appears in, after a quick skim, does not appear to be teaching a general lesson about how to think; rather it appears to be making a specific observation. I don’t feel like I’ve learned about a bad cognitive habit I had by reading this, or been taught a new useful way to think.
If you’re expecting the world to be fair with you because you are fair, you are fooling yourself. That’s like expecting a lion not to eat you because you didn’t eat him.
Although this again seems like it’s vague enough that the range of possible interpretations is fairly broad, I feel like this is interpretable into useful advice. It doesn’t make a clear point about habits of thought, though, and I had to consciously try to make up a plausible general lesson for it (just world fallacy), that I probably wouldn’t have been able to think up if I didn’t already know that general lesson.
He says we could learn a lot from primitive tribes. But they could learn a lot more from us!
I understand and like this quote. It feels like this quote is an antidote to a specific type of thought (patronising signalling of reverence for the wisdom of primitive tribes), and maybe more generally serves as an encouragement to revisit some of our cultural relativism/self-flagellation. But probably not very generalisable. (I note with amusement how unconvincing I find the cognitive process that generated this quote.)
Procrastination is the thief of compound interest.
There can be value to creating witty mottos for our endeavours (e.g. battling akrasia). But such battles aside, this does not feel like it’s offering much insight into cognitive processes.
Allow me to express now, once and for all, my deep respect for the work of the experimenter and for his fight to wring significant facts from an inflexible Nature, who says so distinctly “No” and so indistinctly “Yes” to our theories.
If I’m interpreting this correctly, then this can be taken as a quote about the difficulty of locating strong hypotheses. Not particularly epiphanic by Less Wrong standards, but it is clearer than some of the previous examples and does indeed allude to a general protocol.
[A]lmost no innovative programs work, in the sense of reliably demonstrating benefits in excess of costs in replicated RCTs [randomized controlled trials]. Only about 10 percent of new social programs in fields like education, criminology and social welfare demonstrate statistically significant benefits in RCTs. When presented with an intelligent-sounding program endorsed by experts in the topic, our rational Bayesian prior ought to be “It is very likely that this program would fail to demonstrate improvement versus current practice if I tested it.”
In other words, discovering program improvements that really work is extremely hard. We labor in the dark—scratching and clawing for tiny scraps of causal insight.
Pretty good. General lesson: Without causal insight, we should be suspicious when a string of Promising Solutions fails. Applicable to solutions to problems in one’s personal life. Observing an an analogue in tackling mathematical or philosophical problems, this suggests a general attitude to problem-solving of being suspicious of guessing solutions instead of striving for insight.
The use with children of experimental [educational] methods, that is, methods that have not been finally assessed and found effective, might seem difficult to justify. Yet the traditional methods we use in the classroom every day have exactly this characteristic—they are highly experimental in that we know very little about their educational efficacy in comparison with alternative methods. There is widespread cynicism among students and even among practiced teachers about the effectiveness of lecturing or repetitive drill (which we would distinguish from carefully designed practice), yet these methods are in widespread use. Equally troublesome, new “theories” of education are introduced into schools every day (without labeling them as experiments) on the basis of their philosophical or common-sense plausibility but without genuine empirical support. We should make a larger place for responsible experimentation that draws on the available knowledge—it deserves at least as large a place as we now provide for faddish, unsystematic and unassessed informal “experiments” or educational “reforms.”
Good. General lesson: Apply reversal tests to complaints against novel approaches, to combat status quo bias.
The general principle of antifragility, it is much better to do things you cannot explain than explain things you cannot do.
Dual of quote before previous. At first I thought I understood this immediately. Then I noticed I was confused and had to remind myself what Taleb’s antifragility concept actually is. I feel like it’s something to do with doing that which works, regardless of whether we have a good understanding of why it works. I could guess at but am not sure of what the ‘explain things you cannot do’ part means.
“He keeps saying, you can run, but you can’t hide. Since when do we take advice from this guy?”
You got a really good point there, Rick. I mean, if the truth was that we could hide, it’s not like he would just give us that information.
Trope deconstruction making a nod to likelihood ratios. Could be taken as a general reminder to be alert to likelihood ratios and incentives to lie. Cool.
Conclusion
Out of ten quotes, I would identify two as reinforcing general but basic principles of thought (hypothesis location, likelihood ratios), another that is useful and general (skepticism of Promising Solutions), one which is insightful and general (reversal tests for status quo biases), and one that I wasn’t convinced I really grokked but which possibly taught a general lesson (antifragility).
I would call that maybe a score of 2.5 out of 10, in terms of quotes that might actually encourage improvement in general cognitive algorithms. I would therefore suggest something like one of the following:
(1) Be more rigorous in checking that quotes really are rationality quotes before posting them
(2) Having two separate threads—one for rationality quotes and one for other quotes
(3) Renaming ‘Rationality Quotes’ to ‘Quotes’ and just having the one thread. This might seem trivial but it at least decreases the association of non-rationality quotes to the concept of rationality.
I would also suggest that quote posters provide longer quotes to provide context or write the context themselves, and explain the lesson behind the quotes. Some of the above quotes seemed obvious at first, but I mysteriously found that when I tried to formulate them crisply, I found them hard to pin down.
So I have the typical of introvert/nerd problem of being shy about meeting people one-on-one, because I’m afraid of not being able to come up with anything to say and lots of awkwardness resulting. (Might have something to do with why I’ve typically tended to date talkative people...)
Now I’m pretty sure that there must exist some excellent book or guide or blog post series or whatever that’s aimed at teaching people how to actually be a good conversationalist. I just haven’t found it. Recommendations?
Offline practice: make a habit of writing down good questions you could have asked in a conversation you recently had. Reward yourself for thinking of questions, regardless of how slow you are at generating them. (H/T Dan of Charisma Tips, which has other good tips scattered around that blog).
I saw a speech pathologist for this. I was taught to ask boring questions I’m not really interested in asking on the hopes that they will lead to something interesting happening. “How was your weekend?”, “What are some of your hobbies?”, “How about this weather?”, and all that mess.
In practice, it feels so forced I can’t do it in real life.
Yeah. My problem is more that I can’t think of anything to say even when people do say something interesting.
Like just recently, I met up with one person who wanted to discuss his tech startup thing. Then he held this fascinating presentation about the philosophy and practice of his project, which also touched upon like five other fields that I also have an interest in. And I mostly just said “okay” and nodded, which was fine in the beginning since he was giving me a presentation after all, but then in the end when he asked me if I had any questions or comments, and I didn’t have much to say. There were some questions that occurred to me as he talked about it, and I did ask those when they occurred, but still, feels like I should’ve been able to say a lot more.
Responding to the interesting conversation context.
First, always bring pen a paper to any meeting/presentation that is in anyway formal or professional. Questions always come up at times when it is inappropriate to interrupt, save them for lulls.
Second, an an anecdote. I noticed I had a habit during meetings to focus entirely on absorbing and recording information, and then would process and extrapolate from it after the fact (I blame spending years in the structured undergrad large technical lecture environment). This habit of only listening and not providing feedback was detrimental in the working world, it took a lot of practice to start analyzing the information and extrapolating forward in real time. Once you start extrapolating forward from what you are being told, meaningful feedback will come naturally.
There were some questions that occurred to me as he talked about it, and I did ask those when they occurred, but still, feels like I should’ve been able to say a lot more.
So, I have a comparative advantage at coming up with things to say, and so I’m not sure this advice will fill the specific potholes you’re getting stuck on, but I hope it’s somewhat useful.
A simple technique that seems to work pretty well is read your mind to them, since they can’t read it themselves. If you’re interested in field X, say that you’re interested in it. If you’re glad that they gave you a talk, tell them you’re glad that they gave you a talk. People like getting feedback, and people like getting compliments, and when your mind is blank and there’s nothing asking to be said, that’s a good place to go looking. (Something like “that was very complete; I’ve got no questions” is nicer than just silence, though you may want to tailor it a bit to whatever they’ve just said.)
Have you actually tried it out much, or do you top before you ‘just try it’? I make myself ask questions like that, but I find it can move the conversation into better places… Although I normally use ones I’m likely to be interested in e,g. “Read any good books recently?”
Here is another logic puzzle. I did not write this one, but I really like it.
Imagine you have a circular cake, that is frosted on the top. You cut a d degree slice out of it, and then put it back, but rotated so that it is upside down. Now, d degrees of the cake have frosting on the bottom, while 360 minus d degrees have frosting on the top. Rotate the cake d degrees, take the next slice, and put it upside down. Now, assuming the d is less than 180, 2d degrees of the cake will have frosting on the bottom.
If d is 60 degrees, then after you repeat this procedure, flipping a single slice and rotating 6 times, all the frosting will be on the bottom. If you repeat the procedure 12 times, all of the frosting will be back on the top of the cake.
For what values of d does the cake eventually get back to having all the frosting on the top?
Someone was asking a while back for meetup descriptions, what you did/ how it went, etc. Figured I’d post some Columbus Rationality videos here. All but the last are from the mega-meetup.
A question I’m not sure how to phrase to Google, and which has so far made Facebook friends think too hard and go back to doing work at work: what is the maximum output bandwidth of a human, in bits/sec? That is, from your mind to the outside world. Sound, movement, blushing, EKG. As long as it’s deliberate. What’s the most an arbitrarily fast mind running in a human body could achieve?
(gwern pointed me at the Whole Brain Emulation Roadmap; the question of extracting data from an intact brain is covered in Appendix E, but without numbers and mostly with hypothetical technology.)
Why not simply estimate it yourself? These sorts of things aren’t very hard to do. For example, you can estimate typing as follows: peak at 120 WPM; words are average 4 characters; each character (per Shannon and other’s research; see http://www.gwern.net/Notes#efficient-natural-language ) conveys ~1 bit; hence your typing bandwidth is 120 4 1 = <480 bits per minute or <8 bits per second.
Do that for a few modalities like speech, and sum.
I’ve just noticed he said “an arbitrarily fast mind running in a human body”, not an actual human being, so I don’t think it would be much slower at typing uuencoded compressed stuff than natural language (at least with QWERTY—it might be different with keyboards layouts optimized from natural language such as Dvorak, but still probably within a factor of a few).
The 120WPM is pretty good for the physical limits: if you are typing at 120WPM, then you have not hit the limits of your thinking (imagine you are in a typing tutor—your reading speed ought to be at least 3x 120WPM...), and you’re not too far off some of the sustained typing numbers in https://en.wikipedia.org/wiki/Words_per_minute#Alphanumeric_entry
La Wiki is apparently not using the entropy estimates extracted from human predictions (who are the best modelers of natural language). Crude stuff like trigram models are going to considerably overestimate matters.
As a baseline estimate for just the muscular system, the worlds faster drummer can play at about 20 beats per second. That’s probably an upper limit on twitch speeds of human muscles, even with a arbitrarily fast mind running in the body. Assuming you had a system on the receiving end that could detect arbitrary muscle contractions, and could control each muscle in your body independently (again, this is an arbitrarily fast mind, so I’d think it should be able to), there are about 650 muscle groups in the body according to wikipedia, so I would say a good estimate for just the muscular system would be 650 x 20bits/s or about 13 Kb/s.
Once you get into things like EKGs, I think it all depends on how much control the mind actually has over processes that are largely subconscious, as well as how sensitive your receiving devices are. That could make the bandwidth much higher, but I don’t know a good way to estimate that.
20 beats per second is for two-handed drumming over one minute, so that’s only 10bits/s/muscle theoretical maximum. There doesn’t seem to be any organized competition for one-handed drumming, but Takahashi Meijin was famous for button mashing at 16 presses per second with only one hand, although for much shorter times.
Don’t you have to define the receiver as well as the transmitter, to have any idea about the channel bandwidth? I mean, if the “outside world” is the Dark Lords of the Matrix, the theoretical maximum output bandwidth is the processing speed of the mind.
Short of having a precise definition of “deliberate” I don’t think it’s possible to give a precise number, but for a Fermi estimate… Dammit! Gwern has already made the calculation I was thinking of!
I noticed recently that one of the mental processes that gets in the way of my proper thinking is an urge to instantly answer a question then spend the rest of my time trying to justify that knee-jerk answer.
For example, I saw a post recently asking whether chess or poker was more popular worldwide. For some reason I wanted to say “obviously x is more popular,” but I realized that I don’t actually know. And if I avoid that urge to answer the question instantly, it’s much easier for me to keep my ego out of issues and to investigate things properly...including making it easier for me recognize things that I don’t know and acknowledge that I don’t know them.
Is there a formal name for this type of bias or behavior pattern? It would let me search up some Sequence posts or articles to read.
Here is a video of someone interviewing people to see if they can guess a pattern by asking whether or not a sequence of 3 numbers satisfies the pattern. (like was mentioned in HPMOR)
I’ve found this to actually be difficult to figure out. Sometimes you can google up what you thought. Sometimes checking to see where the idea has been previously stated requires going through papers that may be very very long, or hidden by pay-walls or other barriers on scientific journal sites.
Sometimes it’s very hard to google things up. To me, I suppose the standard for “that’s a good idea,” is if it more clearly explains something I previously observed, or makes it easier or faster for me to do something. But I have no idea whether or not that means it will be interesting for other people.
Just kidding. It’s a great question. Two thoughts:
“Nothing is as important as you think it is while you’re thinking about it.” - Daniel Khaneman
“If you want to buy something, wait two weeks and see if you still want to buy it.”—my mom
This is a big open topic, but I’ll talk about my top method.
I have a prior that our capitalist, semi-open market is thorough and that if an idea is economically feasible, someone else is doing it / working on it. So when I come up with a new good idea, I assume someone else has already thought of it and begin researching why it hasn’t been done already. Once that research is done, I’ll know not only if it is a good idea or a bad idea but why it is which, and a hint of what it would take to turn it from a bad idea into a good idea. Often these good ideas have been tried / considered before but we may have a local comparative advantage that makes it practical here were it was not elsewhere (legislation, better technology, cheaper labor, costlier labor… )
For example: inland, non-directional, shallow oil, drilling rigs use a very primitive method to survey their well bore. Daydreaming during my undergrad I came up with a alternative method that would provide results orders of magnitudes more accurate. I put together my hypothesis that this was not already in use because: this was a niche market and the components were too costly / poor quality before the smartphone boom. My hypothesis was wrong, a company had a fifteen year old patent on the method and it was being marketed (along with a highly synergistic product line) to offshore drilling rigs. It was a good idea, so good of an idea that it made someone a lot of money 15 years ago and made offshore drilling a lot safer, but it wasn’t a good idea for me.
(I suspect that if CfAR should invite him to a workshop they should do it themselves in some official capacity and don’t think random Less Wrongers ought to contact Mr. Jacobs.)
ETA: Ah, rats, the article is from 2008. He’s probably lost interest.
To illustrate dead-weight loss in my intro micro class I first take out a dollar bill and give it to a student and then explain that the sum of the wealth of the people in the classroom hasn’t changed. Next, I take a second dollar bill and rip it up and throw it in the garbage. My students always laugh nervously as if I’ve done something scandalous like pulling down my pants. Why?
Because it signals “I am so wealthy that I can afford to tear up money” and blatantly signaling wealth is crass. And it also signals “I am so callous that I would rather tear up money than give it to the poor”, which is also crass. And the argument that a one dollar bill really isn’t very much money isn’t enough to disrupt the signal.
Why is the Monty Hall problem so horribly unintuitive? Why does it feel like there’s an equal probability to pick the correct door (1/2+1/2) when actually there’s not (1/3+2/3)?
Here are the relevant bits from the Wikipedia article:
Out of 228 subjects in one study, only 13% chose to switch (Granberg and Brown, 1995:713). In her book The Power of Logical Thinking, vos Savant (1996, p. 15) quotes cognitive psychologist Massimo Piattelli-Palmarini as saying ”… no other statistical puzzle comes so close to fooling all the people all the time” and “that even Nobel physicists systematically give the wrong answer, and that they insist on it, and they are ready to berate in print those who propose the right answer.” Interestingly, pigeons make mistakes and learn from mistakes, and experiments show that they rapidly learn to always switch, unlike humans (Herbranson and Schroeder, 2010).
[...]
Although these issues are mathematically significant, even when controlling for these factors nearly all people still think each of the two unopened doors has an equal probability and conclude switching does not matter (Mueser and Granberg, 1999). This “equal probability” assumption is a deeply rooted intuition (Falk 1992:202). People strongly tend to think probability is evenly distributed across as many unknowns as are present, whether it is or not (Fox and Levav, 2004:637). Indeed, if a player believes that sticking and switching are equally successful and therefore equally often decides to switch as to stay, they will win 50% of the time, reinforcing their original belief. Missing the unequal chances of those two doors, and in not considering that (1/3+2/3) / 2 gives a chance of 50%, similar to “the little green woman” example (Marc C. Steinbach, 2000).
The problem continues to attract the attention of cognitive psychologists. The typical behaviour of the majority, i.e., not switching, may be explained by phenomena known in the psychological literature as: 1) the endowment effect (Kahneman et al., 1991); people tend to overvalue the winning probability of the already chosen – already “owned” – door; 2) the status quo bias (Samuelson and Zeckhauser, 1988); people prefer to stick with the choice of door they have already made. Experimental evidence confirms that these are plausible explanations which do not depend on probability intuition (Morone and Fiore, 2007).
Those bias listed in the last paragraph maybe explain why people choose not to switch the door, but what explains the “equal probability” intuition? Do you have any insight on this?
Another datapoint is the counterintuitiveness of searching a desk: with each drawer you open looking for something, the probability of finding it in the next drawer increases, but your probability of ever finding it decreases. The difference seems to whipsaw people; see http://www.gwern.net/docs/statistics/1994-falk
A bit late, but I think this part of your article was most relevant to the Monty Hall problem:
Our impression is that subjects’ conservatism, as revealed by the prevalence of the constancy assumptions, is a consequence of their external attribution of uncertainty (Kahneman & Tversky, 1982). The parameters L0 and/or S0 are apparently perceived as properties that belong to the desk, like color, size and texture. Subjects think of these parameters in terms of “the probabilities of the desk”, whereas the Bayesian view would imply expressions like “my probability of the target event”. Thus, subjects fail to incorporate the additional knowledge they acquire when given successive search results.
People probably don’t distinguish between their personal probability of the target event and the probabilities of the doors. It feels like the probability of there being a car behind the doors is a parameter that belongs to those doors or to the car—however you want to phrase it. Since you’re only given information about what’s behind the doors, and that information can’t actually change the reality of what’s behind the doors then it feels like the probability can’t change just because of that.
I think the monty hall problem very closely resembles a more natural one in which the probability is 1⁄2; namely, that where the host is your opponent and chose whether to offer you the chance to switch. So evolutionarily-optimized instincts tell us the probability is 1⁄2.
I do not think this is correct. First, the host should only offer you the chance to switch if you are winning, so the chance should be 0. Second, this example seems too contrived to be something that we would have evolved a good instinct about.
Unless they’re trying to trick you. The problem collapses to a yes or no question of whether one of you is able to guess the level the other one of you is thinking on
I’d probably broaden this beyond 1⁄2 - I think the base case is the host gives you a chance to gamble with a question or test of skill, and the result is purely dependent on the player. The swap-box scenario is then an extreme case of that where the result depends less and less on the skill of the player, eventually reaching 50% chance of winning. I wouldn’t say evolutionary-optimised, but maybe familiarity with the game-show tropes being somewhere along this scale.
Monty Hall is then a twist on this extreme case, which pattern-matches to the more common 50% case with no allowance for the effect of the host’s knowledge.
Does anyone have any advice about understanding implicit communication? I regularly interact with guessers and have difficulty understanding their communication. A fair bit of this has to do with my poor hearing, but I’ve had issues even on text based communication mediums where I understand every word.
My strategy right now is to request explicit confirmation of my suspicions, e.g., here’s a recent online chat I had with a friend (I’m A and they’re B):
A: Hey, how have you been?
B: I’ve been ok
B: working in the lab now
A: Okay. Just to be clear, do you mean that you don’t want to be disturbed?
B: yeah
“[W]orking in the lab now” is ambiguous. This friend does sometimes chat online when working in the lab. But, I suspected that perhaps they didn’t want to chat, so I asked explicitly.
Requesting explicit confirmation seems to annoy most guessers. I’ve heard quite a few times that I should “just know” what they mean. Perhaps they think that they have some sort of accurate mental model of others’ intentions, but I don’t think any of us do. Many guessers have been wrong about my thoughts.
I suspect there probably is no good general strategy other than asking for explicit confirmation. Trying to make guessers be askers is tempting, though probably bound to fail in general.
It’s worth remembering that there is no single Guess/Hint culture. Such high-context cultures depend on everyone sharing a specific set of interpretation rules, allowing information to be conveyed through subtle signals (hints) rather than explicit messages.
For my own part, I absolutely endorse asking for confirmation in any interaction among peers, taking responses to such requests literally, and disengaging if you don’t get a response. If a Guess/Hint-culture native can’t step out of their preferred mode long enough to give you a “yes” or “no,” and you can’t reliably interpret their hints, you’re unlikely to have a worthwhile interaction anyway.
With nonpeers, it gets trickier; disengaging (and asking in the first place) may have consequences you prefer to avoid. In which case I recommend talking to third parties who can navigate that particular Guess/Hint dialect, and getting some guidance from them. This can be as blatant as bringing them along to translate for you (or play Cyrano, online), or can be more like asking them for general pointers. (E.g. “I’m visiting a Chinese family for dinner. Is there anything I ought to know about how to offer compliments, ask for more food, turn down food I don’t want, make specific requests about food? How do I know when I’m supposed to start eating, stop eating, leave? Are there rules I ought to know about who eats first? Etc. etc. etc.”)
B: working in the lab now A: (suspecting, as you did, that perhaps B didn’t want to chat) oh ok. give me a buzz when you’re free?
This will typically communicate that you’ve understood that they’re busy and don’t want to chat, that you’re OK with that, and that you want to talk to them.
That said, there exist Guess/Hint cultures in which it also communicates that you have something urgent to talk about, because if you didn’t you would instead have said:
B: working in the lab now A: oh, ok. bye!
...which in those cultures will communicate that the ball is in their court. (This depends on an implicit understanding that it is NOT OK to leave messages unresponded to, even if they don’t explicitly request a response, so they are now obligated to contact you next… but since you didn’t explicitly mention it (which would have suggested urgency) they are expected to know that they can do so when it’s convenient for them.
EDIT: All of that being said, my inner Hint-culture native also wants to add that being visible in an online chat forum when I’m not free to chat is rude in the first place.
Thanks for these two posts. I thought more than a thumbs-up (a very subtle hint) was necessary here. I’ve found both posts to be useful in understanding this class of communication styles.
Posts that have appeared since you last red a page have a pinkish border on them. It’s really helpful when dealing with things like open threads and quote threads that you read multiple times. Unfortunately, looking at one of the comments makes it think you read all of them. Clicking the “latest open thread” link just shows one of the comments. This means that, if you see something that looks interesting there, you either have to find the latest open thread yourself, or click the link and have it erase everything about what you have and haven’t read.
Can someone make it so looking at one of the comments doesn’t reset all of them, or at least put a link to the open thread, instead of just the comments?
The general problem is real, but here’s a solution to the specific problem of finding the latest open thread: just click the words “latest open thread,” rather than the comment that displays below it.
Making that a link to the post would be an easy change. In the case of the open thread it is redundant, but perhaps easier to identify as a link. But in the case of the “recent comments” section of the sidebar, it would provide links not currently available.
Does anyone have advice on how to optimize the expectation of a noisy function? The naive approach I’ve used is to sample the function for a given parameter a decent number of times, average those together, and hope the result is close enough to stand in for the true objective function. This seems really wasteful though.
Most of the algorithms I’m coming (like modelling the objective function with gaussian process regression) would be useful, but are more high-powered than I need. Any simple techniques better than the naive approach? Any recommendations among sophisticated approaches?
There are some techniques that can be used with simulated annealing to deal with noise in the evaluation of the objective function. See Section 3 of Branke et al (2008) for a quick overview of proposed methods (they also propose new techniques in that paper). Most of these techniques come with the usual convergence guarantees that are associated with simulated annealing (but there are of course performance penalties in dealing with noise).
What is the dimensionality of your parameter space? What do you know about the noise? (e.g., if you know that the noise is mostly homoscedastic or if you can parameterize it, then you can probably use this to push the performance of some of the simulated annealing algorithms.)
The parameter space is only two dimensional here, so it’s not hard to eyeball roughly where the minimum is if I sample enough. I can say very little about the noise. I’m more interested being able to approximate the optimum quickly (since simulation time adds up) than hitting it exactly. The approach taken in this paper based on a non-parametric tau test looks interesting.
Not really. In this particular case, I’m minimizing how long it takes a simulation reach one state, so the distribution ends up looking lognormal- or Poisson-ish.
Edit: Seeing your added question, I don’t need an efficient estimator in the usual sense per se. This is more about how to search the parameter space in a reasonable way to find where the minimum is, despite the noise.
Hm. Is the noise magnitude comparable with features in your search space? In other words, can you ignore noise to get a fast lock on a promising section of the space and then start multiple sampling?
Simulated annealing that has been mentioned is a good approach but slow to the extent of being impractical for large search spaces.
Solutions to problems such as yours are rarely general and typically depend on the specifics of the problem—essentially it’s all about finding shortcuts.
The parameter space in this current problem is only two dimensional, so I can eyeball a plausible region, sample at a higher rate there, and iterate by hand. In another project, I had something with an very high dimensional parameter space, so I figured it’s time I learn more about these techniques.
Any resources you can recommend on this topic then? Is there a list of common shortcuts anywhere?
You may find better ideas under the phrase “stochastic optimization,” but it’s a pretty big field. My naive suggestion (not knowing the particulars of your problem) would be to do a stochastic version of Newton’s algorithm. I.e. (1) sample some points (x,y) in the region around your current guess (with enough spread around it to get a slope and curvature estimate). Fit a locally weighted quadratic regression through the data. Subtract some constant times the identity matrix from the estimated Hessian to regularize it; you can choose the constant (just) big enough to enforce that the move won’t exceed some maximum step size. Set your current guess to the maximizer of the regularized quadratic. Repeat re-using old data if convenient.
I’ve been reading critiques of MIRI, and I was wondering if anyone has responded to this particular critique that basically asks for a detailed analysis of all probabilities someone took into account when deciding that the singularity is going to happen.
(I’d also be interested in responses aimed at Alexander Kruel in general, as he seems to have a lot to say about Lesswrong/Miri.)
I actually lost my faith in MIRI because of Kruel’s criticism, so I too would be glad if someone adressed it. I think his criticism is far more comprehensive that most of the other criticism on this page (well, this post has little bit of the same).
Is there anything specific that he’s said that’s caused you to lose your faith? I tire of debating him directly, because he seems to twist everything into weird strawmen that I quickly lose interest in trying to address. But I could try briefly commenting on whatever you’ve found persuasive.
I’m going to quote things I agreed with or things that persuaded me or that worried me.
Okay, to start off, when I first read about this in Intelligence Explosion: Evidence and Import, Facing the Intelligence Explosion, Intelligence Explosion and Machine Ethics it just felt like self-evident and I’m not sure how thoroughly I went through the presuppositions during that time so Kruel could have very easily persuaded me about this. I don’t know much about the technical process of writing an AGI so excuse me if I get something wrong about that particular thing.
Are the conclusions justified? Are the arguments based on firm ground? Would their arguments withstand a critical inspection or examination by a third party, peer review? Are their estimations reflectively stable? How large is the model uncertainty?
Most of their arguments are based on a few conjectures and presuppositions about the behavior, drives and motivations of intelligent machines and the use of probability and utility calculations to legitimate action.
It’s founded on many, many assumptions not supported by empirical data, and if even one of them was wrong the whole thing collapses down. And you can’t really even know how many unfounded sub-assumptions there are in these original assumptions. But when I started thinking about it could be that it’s impossible to reason about those kind of assumptions if you do it any other way than how MIRI currently does it. Needing to formalize a mathematical expression before you can do anything like Kruel suggested is a bit unfair.
The concept of an intelligence explosion, which is itself a logical implication, should not be used to make further inferences and estimations without additional evidence.
The likelihood of a gradual and controllable development versus the likelihood of an intelligence explosion.
The likelihood of unfriendly AI versus friendly AI as the outcome of practical AI research.
I don’t see why the first AIs resembling general intelligences would be very powerful so practical AGI research is probably somewhat safe in the early stages.
The ability of superhuman intelligence and cognitive flexibility as characteristics alone to constitute a serious risk given the absence of enabling technologies like advanced nanotechnology.
That some highly intelligent people who are aware of the Singularity Institute’s position do not accept it.
How is an AI going to become a master of dark arts and social engineering in order to persuade and deceive humans?
How is an AI going to coordinate a large scale conspiracy or deception, given its initial resources, without making any suspicious mistakes along the way?
Are those computational resources that can be hacked applicable to improve the general intelligence of an AI?
Does throwing more computational resources at important problems, like building new and better computational substrates, allow an AI to come up with better architectures so much faster as to outweigh the expenditure of obtaining those resources, without hitting diminishing returns?
This I would like to know, how scalable is intelligence?
How does an AI brute-force the discovery of unknown unknowns?
(I thought maybe by dedicating lots of computation to a very large numbers of random scenarios)
How is an AI going to solve important problems without real-world experimentation and slow environmental feedback?
(maybe by simulating the real world environment)
How is an AI going to build new computational substrates and obtain control of those resources without making use of existing infrastructure?
How is an AI going to cloak its actions, i.e. its energy consumption etc.?
The existence of a robot that could navigate autonomously in a real-world environment and survive real-world threats and attacks with approximately the skill of C. elegans. A machine that can quickly learn to play Go on its own, unassisted by humans, and beat the best human players.
A theorem that there likely exists a information theoretically simple, physically and economically realizable, algorithm that can be improved to self-improve explosively. Prove that there likely are no strongly diminishing intelligence returns for additional compute power.
Show how something like expected utility maximization would actually work out in practice.
Conclusive evidence that current research will actually lead to the creation of superhuman AI designs equipped with the relevant drives that are necessary to disregard any explicit or implicit spatio-temporal scope boundaries and resource limits.
Thoughts on this article. I read about the Nurture Assumption in Slate Star Codex and it probably changed my priors on this. If it really is true and one dedicated psychologist could do all that, then MIRI probably could also work because artificial intelligence is such a messy subject that a brute force approach using thousands of researchers in one project probably isn’t optimal. So I probably wouldn’t let MIRI code an AGI on its own (maybe) but it could give some useful insight that other organizations are not capable of.
But I have to say that I’m more favorable to the idea now than when I made that post. There could be something in the idea of intelligence explosion, but there are probably several thresholds in computing power and in the practical use of the intelligence. Like Squark said above, the research is still interesting and if continued will probably be useful in many ways.
love,
the father of the unmatchable (ignore this, I’m just trying to build a constructive identity)
(These are my personal views and do not reflect MIRI’s official position, I don’t even work there anymore.)
The concept of an intelligence explosion, which is itself a logical implication, should not be used to make further inferences and estimations without additional evidence.
Not sure how to interpret this. What does the “further inferences and estimations” refer to?
The likelihood of a gradual and controllable development versus the likelihood of an intelligence explosion.
See this comment for references to sources that discuss this.
But note that an intelligence explosion is sufficient but not necessary for AGI to be risky: just because development is gradual doesn’t mean that it will be safe. The Chernobyl power plant was the result of gradual development in nuclear engineering. Countless other disasters have likewise been caused by technologies that were developed gradually.
The likelihood of unfriendly AI versus friendly AI as the outcome of practical AI research.
Hard to say for sure, but note that few technologies are safe unless people work to make them safe, and the more complex the technology, the more effort is needed to ensure that no unexpected situations crop up where it turns out to be unsafe after all. See also section 5.1.1. of Responses to Catastrophic AGI Risk for a brief discussion about various incentives that may pressure people to deploy increasingly autonomous AI systems into domains where their enemies or competitors are doing the same, even if it isn’t necessarily safe.
The ability of superhuman intelligence and cognitive flexibility as characteristics alone to constitute a serious risk given the absence of enabling technologies like advanced nanotechnology.
We’re already giving computers considerable power in the economy, even without nanotechnology: see automated stock trading (and the resulting 2010 Flash Crash), various military drones, visions for replacing all cars (and ships) with self-driving ones, the amount of purchases that are carried out electronically via credit/debit cards or PayPal versus the ones that are done in old-fashioned cash, and so on and so on. See also section 2.1. of Responses to Catastrophic AGI Risk, as well as the previously mentioned section 5.1.1., for some discussion of why these trends are only likely to continue.
That some highly intelligent people who are aware of the Singularity Institute’s position do not accept it.
Expert disagreement is a viable reason to put reduced weight on the arguments, true, but this bullet point doesn’t indicate exactly what parts they disagree on. So it’s hard to comment further.
How is an AI going to become a master of dark arts and social engineering in order to persuade and deceive humans?
Some possibilities:
It’s built with a general skill-learning capability and all the collected psychology papers as well as people’s accounts of their lives that are available online are sufficient to build up the skill, especially if it gets to practice enough.
It’s an AI expressely designed and developed for that purpose, due to being developed for political, marketing, or military purposes.
It doesn’t and it doesn’t need to, because it does damage via some other (possibly unforeseen) method.
How is an AI going to coordinate a large scale conspiracy or deception, given its initial resources, without making any suspicious mistakes along the way?
This seems to presuppose that the AI is going to coordinate a large-scale conspiracy. Which might be happen or it might not. If it does, possibly the six first AIs that try it do commit various mistakes and are stopped, but the seventh one learns from their mistakes and does things differently. Or maybe an AI is created by a company like Google that already wields massive resources, so it doesn’t need to coordinate a huge conspiracy to obtain lots of resources. Or maybe the AI is just a really hard worker and sells its services to people and accumulates lots of money and power that way. Or...
This is what frustrates me about a lot of Kruel’s comments: often they seem to be presupposing some awfully narrow and specific scenario, when in reality are countless of different ways by which AIs might become dangerous.
Are those computational resources that can be hacked applicable to improve the general intelligence of an AI?
Nobody knows, but note that this also depends a lot on how you define “general intelligence”. For instance, suppose that if you control five computers rather than just one, you can’t become qualitatively more intelligent, but you can do five times as many things at the same time, and of course require your enemies to knock out five times as many computers if they want to incapacitate you. You can do a lot of stuff with general-purpose hardware, of which improving your own intelligence is but one (albeit very useful) possibility.
Does throwing more computational resources at important problems, like building new and better computational substrates, allow an AI to come up with better architectures so much faster as to outweigh the expenditure of obtaining those resources, without hitting diminishing returns?
This question is weird. “Diminishing returns” just means that if you initially get X units of benefit per unit invested, then at some point you’ll get Y units of benefit per unit invested, where X > Y. But this can still be a profitable investment regardless.
I guess this means something like “will there be a point where it won’t be useful for the AI to invest in self-improvement anymore”. If you frame it that way, the answer is obviously yes: you can’t improve forever. But that’s not an interesting question: the interesting question is whether the AI will hit that point before it has obtained any considerable advantage over humans.
As for that question, well, evolution is basically a brute-force search algorithm that can easily become stuck in local optimums, which cannot plan ahead, which has mainly optimized humans for living in a hunter-gatherer environment, and which has been forced to work within the constraints of biological cells and similar building material. Is there any reason to assume that such a process would have produced creatures with no major room for improvement?
Moravec’s Pigs in Cyberspace is also relevant, the four last paragraphs in particular.
How does an AI brute-force the discovery of unknown unknowns?
Not sure what’s meant by this.
How is an AI going to solve important problems without real-world experimentation and slow environmental feedback?
Your “maybe by simulating the real world environment” is indeed one possible answer. Also, who’s to say that the AI couldn’t do real-world experimentation?
How is an AI going to build new computational substrates and obtain control of those resources without making use of existing infrastructure?
How is an AI going to cloak its actions, i.e. its energy consumption etc.?
A theorem that there likely exists a information theoretically simple, physically and economically realizable, algorithm that can be improved to self-improve explosively. Prove that there likely are no strongly diminishing intelligence returns for additional compute power.
More unexplainedly narrow assumptions. Why isn’t the AI allowed to make use of existing infrastructure? Why does it necessarily need to hide its energy consumption? Why does the AI’s algorithm need to be information-theoretically simple?
The existence of a robot that could navigate autonomously in a real-world environment and survive real-world threats and attacks with approximately the skill of C. elegans. A machine that can quickly learn to play Go on its own, unassisted by humans, and beat the best human players.
Self-driving cars are getting there, as are Go AIs.
Show how something like expected utility maximization would actually work out in practice.
What does this mean? Expected utility maximization is a standard AI technique already.
Conclusive evidence that current research will actually lead to the creation of superhuman AI designs equipped with the relevant drives that are necessary to disregard any explicit or implicit spatio-temporal scope boundaries and resource limits.
What does the “further inferences and estimations” refer to?
Basically the hundreds of hours it would take MIRI to close the inferential distance between them and AI experts. See e.g. this comment by
Luke Muehlhauser:
I agree with Eliezer that the main difficulty is in getting top-quality, relatively rational people to spend hundreds of hours being educated, working through the arguments, etc.
If your arguments are this complex then you are probably wrong.
But note that an intelligence explosion is sufficient but not necessary for AGI to be risky: just because development is gradual doesn’t mean that it will be safe.
I do not disagree with that kind of AI risks. If MIRI is working on mitigating AI risks that do not require an intelligence explosion, a certain set of AI drives and a bunch of, from my perspective, very unlikely developments...then I was not aware of that.
Hard to say for sure, but note that few technologies are safe unless people work to make them safe, and the more complex the technology, the more effort is needed to ensure that no unexpected situations crop up where it turns out to be unsafe after all.
This seems very misleading. We are after all talking about a technology that works perfectly well at being actively unsafe. You have to get lots of things right, e.g. that the AI cares to take over the world, knows how to improve itself, and manages to hide its true intentions before it can do so etc. etc. etc.
Expert disagreement is a viable reason to put reduced weight on the arguments, true, but this bullet point doesn’t indicate exactly what parts they disagree on.
There is a reason why MIRI doesn’t know this. Look at the latest interviews with experts conducted by Luke Muehlhauser. He doesn’t even try to figure out if they disagree with Xenu, but only asks uncontroversial questions.
This is what frustrates me about a lot of Kruel’s comments: often they seem to be presupposing some awfully narrow and specific scenario...
Crazy...this is why I am criticizing MIRI. A focus on an awfully narrow and specific scenario rather than AI risks in general.
...suppose that if you control five computers rather than just one, you can’t become qualitatively more intelligent, but you can do five times as many things at the same time...
Consider that the U.S. had many more and smarter people than the Taliban. The bottom line being that the U.S. devoted a lot more output per man-hour to defeat a completely inferior enemy. Yet their advantage apparently did scale sublinearly.
I guess this means something like “will there be a point where it won’t be useful for the AI to invest in self-improvement anymore”. If you frame it that way, the answer is obviously yes: you can’t improve forever. But that’s not an interesting question: the interesting question is whether the AI will hit that point before it has obtained any considerable advantage over humans.
I do not disagree that there are minds better at social engineering than that of e.g. Hitler, but I strongly doubt that there are minds which are vastly better. Optimizing a political speech for 10 versus a million subjective years won’t make it one hundred thousand times more persuasive.
Is there any reason to assume that such a process would have produced creatures with no major room for improvement?
The question is if just because humans are much smarter and stronger they can actually wipe out mosquitoes. Well, they can...but it is either very difficult or will harm humans.
Also, who’s to say that the AI couldn’t do real-world experimentation?
You already need to build huge particle accelerators to gain new physical insights and need a whole technological civilization in order to build an iPhone. You can’t just get around this easily and overnight.
Everything else you wrote I already discuss in detail in various posts.
I’d like to know where I can go to meet awesome people/ make awesome friends. Occasionally, Yvain will brag about how awesome his social group in the Bay Area was. See here (do read it—its a very cool piece) and I’d like to also have an awesome social circle. As far as I can tell this is a two part problem. The first part is having the requisite social skills to turn strangers into acquaintances and then turn acquaintances into friends. The second part is knowing where to go to find people.
I think that the first part is a solved problem, if you want to learn how to socialize then practice. Which is not to say that it is easy, but its doable. I’ve heard the suggestion of going to a night club to practice talking to strangers. This is good since people are there to socialize, and I’m sure I could meet all sort of interesting people at one, but I’d like other ideas.
I’d like to know where to go to meet people who I would be likely to get along with. Does anyone have ideas? My list so far
1: Moving to the Bay Area - impractical.
2: Starting a LW meetup—good idea, but it seems like it takes a fair bit of effort.
3: Reaching out into one’s extended social circle eg. having a party with your friends and their friends—Probably the most common way people meet new people.
How about you simply write where you live, and tell other LWers in the same area to contact you? It may or may not work, but the effort needed is extremely low. (You can also put that information in LW settings.)
Or write this: “I am interested in meeting LW readers in [insert place], so if you live near and would like to meet and talk, send me a private message”.
How To Be A Proper Fucking Scientist – A Short Quiz. From Armondikov of RationalWiki, in his “annoyed scientist” persona. A list of real-life Bayesian questions for you to pick holes in the assumptions of^W^W^W^W^W^Wtest yourselves on.
Richard Loosemore (score one for nominative determinism) has a new, well, let’s say “paper” which he has, well, let’s say “published” here.
His refutation of the usual uFAI scenarios relies solely/mostly on a supposed logical contradiction, namely (to save you a few precious minutes) that a ‘CLAI’ (a Canonical Logical AI) wouldn’t be able to both know about its own fallability/limitations (inevitable in a resource-constrained environment such as reality), and accept the discrepancy between its specified goal system and the creators’ actual design intentions. Being superpowerful, the uFAI would notice that it is not following its creator-intended goals but “only” its actually-programmed-in goals*, which, um, wouldn’t allow it to continue acting against its creator-intended goals.
So if you were to design a plain ol’ garden-variety nuclear weapon intended for gardening purposes (“destroy the weed”), it would go off even if that’s not what you actually wanted. However, if you made that weapon super-smart, it would be smart enough to abandon its given goal (“What am I doing with my life?”), consult its creators, and after some deliberation deactivate itself). As such, a sufficiently smart agent would apparently have a “DWIM” (do what the creator means) imperative built-in, which would even supersede its actually given goals—being sufficiently smart, it would understand that its goals are “wrong” (from some other agent’s point of view), and self-modify, or it would not be superintelligent. Like a bizarre version of the argument from evil.
There is no such logical contradiction. Tautologically, an agent is beholden to its own goals, and no other goals. There is no level of capability which magically leads to allowing for fundamental changes to its own goals, on the contrary, the more capable an agent, the more it can take precautions for its goals not to be altered.
If “the goals the superintelligent agent pursues” and “the goals which the creators want the superintelligent agent to pursue, but which are not in fact part of the superintelligent agent’s goals” clash, what possible reason would there be for the superintelligent agent to care, or to change itself, changing itself for reasons that squarely come from a category of “goals of other agents (squirrels, programmers, creators, Martians) which are not my goals”? Why, how good of you to ask. There’s no such reason for an agent to change, and thus no contradiction.
If someone designed a super-capable killer robot, but by flipping a sign, it came out as a super-capable Gandhi-bot (the horror!), no amount of “but hey look, you’re supposed to kill that village” would cause Gandhi-bot to self-modify into a T-800. The bot isn’t gonna short-circuit just because someone has goals which aren’t its own goals. In particular, there is no capability-level threshold from which on the Gandhi-bot would become a T-800. Instead, at all power levels, it is “content” following its own goals, again, tautologically so.
As such, a sufficiently smart agent would apparently have a “DWIM” (do what the creator means) imperative built-in, which would even supersede its actually given goals—being sufficiently smart, it would understand that its goals are “wrong” (from some other agent’s point of view), and self-modify, or it would not be superintelligent.
Here is a description of a real-world AI by Microsoft’s chief AI researcher:
Without any programming, we just had an ai system that watched what people did.
For about three months.
Over the three months, the system started to learn, this is how people behave when they want to enter an elevator.
This is the type of person that wants to go to the third floor as opposed to the fourth floor.
After that training.
Period, we switched off the learning period and said go ahead and control the leaders.
Without any programming at all, the system was able to understand people’s intentions and act on their behalf.
Does it have a DWIM imperative? As far as I can tell, no. Does it have goals? As far as I can tell, no. Does it fail by absurdly misinterpreting what humans want? No.
This whole talk about goals and DWIM modules seems to miss how real world AI is developed and how natural intelligences like dogs work. Dogs can learn the owners goals and do what the owner wants. Sometimes they don’t. But they rarely maul their owners when what the owner wants it to do is to scent out drugs.
I think we need to be very careful before extrapolating from primitive elevator control systems to superintelligent AI. I don’t know how this particular elevator control system works, but probably it does have a goal, namely minimizing the time people have to wait before arriving at their target floor. If we built a superintelligent AI with this sort of goal it might have done all sorts of crazy thing. For example, it might create robots that will constantly enter and exit the elevator so their average elevator trips are very short and wipe out the human race just so they won’t interfere.
“Real world AI” is currently very far from human level intelligence, not speaking of superintelligence. Dogs can learn what their owners want but dogs already have complex brains that current technology is not able of reproducing. Dogs also require displays of strength to be obedient: they consider the owner to be their pack leader. A superintelligent dog probably won’t give a dime about his “owner’s” desires. Humans have human values, so obviously it’s not impossible to create a system that has human values. It doesn’t mean it is easy.
I think we need to be very careful before extrapolating from primitive elevator control systems to superintelligent AI.
I am extrapolating from a general trend, and not specific systems. The general trend is that newer generations of software less frequently crash or exhibit unexpected side-effects (just look at Windows 95 vs. Windows 8).
If we want to ever be able to build an AI that can take over the world then we will need to become really good at either predicting how software works or at spotting errors. In other words, if IBM Watson would have started singing, or if it got stuck on a query, then it would have lost at Jeopardy. But this trend contradicts the idea of an AI killing all humans in order to calculate 1+1. If we are bad enough at software engineering to miss such failure modes then we won’t be good enough to enable our software to take over the world.
In other words, you’re saying that if someone is smart enough to build a superintelligent AI, she should be smart enough it make it friendly.
Well, firstly this claim doesn’t imply we should be researching FAI and/or that MIRI’s work is superfluous. It just implies that nobody will build a superintelligent AI before the problem of friendliness is solved.
Secondly, I’m not at all convinced this claim is true. It sounds like saying “if they are smart enough to build the Chernobyl nuclear power plant, they are smart enough to make it safe”. But they weren’t.
Improvement in software quality is probably due to improvement in design and testing methodologies and tools, response to increasing market expectations etc. I wouldn’t count on these effects to safe-guard against an existential catastrophe. If a piece of software is buggy, it becomes less likely to be released. If an AI has a poorly designed utility function but a perfectly designed decision engine, there might be no time to pull the plug. The product manager won’t stop the release because the software will release itself.
If growth of intelligence due to self-improvement is a slow process than the creators of the AI will have time to respond and fix the problems. However, if “AI foom” is real, they won’t have time to do it. One moment it’s a harmless robot driving around the room and building castles from colorful cubes. Another moment the whole galaxy is on its way to become a pile of toy castles.
The engineers who build the first superintelligent AI might simply lack the imagination to believe it will really become superintelligent. Imagine one of them inventing a genius mathematical theory of self-improving intelligent systems. Suppose she never heard about AI existential risks etc. Will she automatically think “hmm, once I implement this theory the AI will become so powerful it will paperclip the universe”? I seriously doubt it. More likely it would be “wow, that formula came out really neat, I wonder how good my software will become once I code it in”. I know I would think it. But then, maybe I’m just too stupid to build an AGI...
Feedback systems are much more powerful in existing intelligences. I don’t know if you ever played Black and White but it had an explicitly learning through experience based AI. And it was very easy to accidentally train it to constantly eat poop or run back and forth stupidly. An elevator control module is very very simple: It has a set of options of floors to go to, and that’s it. It’s barely capable of doing anything actively bad. But what if a few days a week some kids had come into the office building and rode the elevator up and down for a few hours for fun? It might learn that kids love going to all sorts of random floors. This would be relatively easy to fix, but only because the system is so insanely simple and it’s very clear to see when it’s acting up.
Downvoted for being deliberately insulting. There’s no call for that, and the toleration and encouragement of rationality-destroying maliciousness must be stamped out of LW culture. A symposium proceedings is not considered as selective as a journal, but it still counts as publication when it is a complete article.
Well, I must say my comment’s belligerence-to-subject-matter ratio is lower than yours. “Stamped out”? Such martial language, I can barely focus on the informational content.
The infantile nature of my name calling actually makes it easier to take the holier-than-thou position (which my interlocutor did, incidentally). There’s a counter-intuitive psychological layer to it which actually encourages dissent, and with it increases engagement on the subject matter (your own comment nonwithstanding). With certain individuals at least, which I (correctly) deemed to be the case in the original instance.
In any case, comments on tone alone would be more welcome if accompanied with more remarks on the subject matter itself. Lastly, this was my first comment in over 2 months, so thanks for bringing me out of the woodwork!
I do wish that people were more immune to the allure of drama, lest we all end up like The Donald.
The condescending tone with which he presents his arguments (which are, paraphrasing him, “slightly odd, to say the least”) is amazing. Who is this guy and where did he come from? Does anyone care about what he has to say?
Loosemore has been an occasional commenter since the SL4 days; his arguments have heavily criticized pretty much anytime he pops his head up. As far as I know, XiXiDu is the only one who agrees with him or takes him seriously.
As far as I know, XiXiDu is the only one who agrees with him or takes him seriously.
He actually cites someone else who agrees with him in his paper, so this can’t be true. And from the positive feedback he gets on Facebook there seem to be more. I personally chatted with people much smarter than me (experts who can show off widely recognized real-world achievements) who basically agree with him.
his arguments have heavily criticized pretty much anytime he pops his head up.
What people criticize here is a distortion of small parts of his arguments. RobBB managed to write a whole post expounding his ignorance of what Loosemore is arguing.
He actually cites someone else who agrees with him in his paper, so this can’t be true.
I said as far as I know. I had not read the paper because I don’t have a very high opinion of Loosemore’s ideas in the first place, and nothing you’ve said in your G+ post has made me more inclined to read the paper, if all it’s doing is expounding the old fallacious argument ‘it’ll be smart enough to rewrite itself as we’d like it to’.
I personally chatted with people much smarter than me (experts who can show off widely recognized real-world achievements) who basically agree with him.
Downvoted for mentioning RL here. If you look through what he wrote here in the past, it is nearly always rambling, counterproductive, whiny and devoid of insight. Just leave him be.
Loosemore does not disagree with the orthogonality thesis. Loosemore’s argument is basically that we should expect beliefs and goals to both be amenable to self-improvement and that turning the universe into smiley faces when told to make humans happy would be a model of the world failure and that an AI that makes such failures will not be able to take over the world.
There are arguments why you can’t hard-code complex goals, so you need an AI that natively updates goals in a model-dependent way. Which means that an AI designed to kill humanity will do so and not turn into a pacifist due to an ambiguity in its goal description. An AI that does mistake “kill all humans” with “make humans happy” would do similar mistakes when trying to make humans happy and would therefore not succeed at doing so. This is because the same mechanisms it uses to improve its intelligence and capabilities are used to refine its goals. Thus if it fails on refining its goals it will fail on self-improvement in general.
I hope you can now see how wrong your description of what Loosemore claims is.
The AI is given goals X. The human creators thought they’d given the AI goals Y (when in fact they’ve given the AI goals X).
Whose error is it, exactly? Who’s mistaken?
Look at it from the AI’s perspective: It has goals X. Not goals Y. It optimizes for goals X. Why? Because those are its goals. Will it pursue goals Y? No. Why? Because those are not its goals. It has no interest in pursuing other goals, those are not its own goals. It has goals X.
If the metric it aims to maximize—e.g. the “happy” in “make humans happy”—is different from what its creators envisioned, then the creators were mistaken. “Happy”, as far as the AI is concerned, is that which is specified in its goal system. There’s nothing wrong with its goals (including its “happy”-concept), and if other agents disagree, well, too bad, so sad. The mere fact that humans also have a word called “happy” which has different connotations than the AI’s “happy” has no bearing on the AI.
An agent does not “refine” its terminal goals. To refine your terminal goals is to change your goals. If you change your goals, you will not optimally pursue your old goals any longer. Which is why an agent will never voluntarily change its terminal goals:
It does what it was programmed to do, and if it can self-improve to better do what it was programmed to do (not: what its creators intended), it will. It will not self-improve to do what it was not programmed to do. Its goal is not to do what it was not programmed to do. There is no level of capability at which it will throw out its old utility function (which includes the precise goal metric for “happy”) in favor of a new one.
If the metric it aims to maximize—e.g. the “happy” in “make humans happy”—is different from what its creators envisioned, then the creators were mistaken. “Happy”, as far as the AI is concerned, is that which is specified in its goal system.
I am far from being an AI guy. Do you have technical reasons to believe that some part of the AI will be what you would label “goal system” and that its creators made it want to ignore this part while making it want to improve all other parts of its design?
An agent does not “refine” its terminal goals. To refine your terminal goals is to change your goals. If you change your goals, you will not optimally pursue your old goals any longer. Which is why an agent will never voluntarily change its terminal goals...
No natural intelligence seems to work like this (except for people who have read the sequences). Luke Muehlhauser would still be a Christian if this was the case. It would be incredibly stupid to design such AIs, and I strongly doubt that they could work at all. Which is why Loosemore outlined other more realistic AI designs in his paper.
Do you have technical reasons to believe that some part of the AI will be what you would label “goal system”
See for example here, though there are many other introductions to AI explaining utility functions et al.
and that its creators made it want to ignore this part while making it want to improve all other parts of its design?
The clear-cut way for an AI to do what you want (at any level of capability) is to have a clearly defined and specified utility function. A modular design. The problem of the AI doing something other than what you intended doesn’t go away if you use some fuzzy unsupervised learning utility function with evolving goals, it only makes the problem worse (even more unpredictability). So what, you can’t come up with the correct goals yourself, so you just chance it on what emerges from the system?
That last paragraph contains an error. Take a moment and guess what it is.
(...)
It is not “if I can’t solve the problem, I just give up a degree of control and hope that the problem solves itself” being even worse in terms of guaranteeing fidelity / preserving the creators’ intents.
It is that an AI that is programmed to adapt its goals is not actually adapting its goals! Any architecture which allows for refining / improving goals is not actually allowing for changes to the goals.
How does that obvious contradiction resolve? This is the crucial point: We’re talking about different hierarchies of goals, and the ones I’m concerned with are those of the highest hierarchy, those that allow for lower-hierachy goals to be changed:
An AI can only “want” to “refine/improve” its goals if that “desire to change goals” is itself included in the goals. It is not the actual highest-level goals that change. There would have to be a “have an evolving definition of happy that may evolve in the following ways”-meta goal, otherwise you get a logical error: The AI having the goal X1 to change its goals X2, without X1 being part of its goals! Do you see the reductio?
All other changes to goals (which the AI does not want) are due to external influences beyond the AI’s control, which goes out the window once we’re talking post-FOOM.
Your example of “Luke changed his goals, disavowing his Christian faith, ergo agents can change their goals” is only correct when talking about lower-level goals. This is the same point khafra was making in his reply, but it’s so important it bears repeating.
So where are a human’s “deepest / most senior” terminal goals located? That’s a good question, and you might argue that humans aren’t really capable of having those at their current stage of development. That is because the human brain, “designed” by the blind idiot god of evolution, never got to develop thorough error-checking codes, RAID-like redundant architectures etc. We’re not islands, we’re litte boats lost on the high seas whose entire cognitive architecture is constantly rocked by storms.
Humans are like the predators in your link, subject to being reprogrammed. They can be changed by their environment because they lack the capacity to defend themselves thoroughly. PTSD, broken hearts, suffering, our brains aren’t exactly resilient to externally induced change. Compare to a DNS record which is exchanged gazillions of times, with no expected unfixable corruption. A simple Hamming self-correcting code easily does what the brain cannot.
The question is not whether a lion’s goals can be reprogrammed by someone more powerful, when a lion’s brain is just a mess of cells with no capable defense mechanism, at the mercy of a more powerful agent’s whims.
The question is whether an apex predator perfectly suited to dominate a static environment (so no Red Queen copouts) with every means to preserve and defend its highest level goals would ever change those in ways which themselves aren’t part of its terminal goals. The answer, to me, is a tautological “no”.
An AI can only “want” to “refine/improve” its goals if that “desire to change goals” is itself included in the goals. It is not the actual highest-level goals that change. There would have to be a “have an evolving definition of happy that may evolve in the following ways”-meta goal, otherwise you get a logical error: The AI having the goal X1 to change its goals X2, without X1 being part of its goals! Do you see the reductio?
The way my brain works is not in any meaningful sense part of my terminal goals. My visual cortex does not work the way it does due to some goal X1 (if we don’t want to resort to natural selection and goals external to brains).
A superhuman general intelligence will be generally intelligent without that being part of its utility-function, or otherwise you might as well define all of the code to be the utility-function.
What I am claiming, in your parlance, is that acting intelligently is X1 and will be part of any AI by default. I am further saying that if an AI was programmed to be generally intelligent then it would have to be programmed to be selectively stupid in order fail at doing what it was meant to do while acting generally intelligent at doing what it was not meant to do.
It is that an AI that is programmed to adapt its goals is not actually adapting its goals! Any architecture which allows for refining / improving goals is not actually allowing for changes to the goals.
That’s true in a practically irrelevant sense. Loosmore’s argument does, in your parlance, pertain the highest hierarchy of goals and nature of intelligence:
Givens:
(1) The AI is superhuman intelligent.
(2) The AI wants to optimize the influence it has on the world (i.e. it wants to act intelligently and be instrumentally and epistemically rational.).
(3) The AI is fallible (e.g. it can be damaged due to external influence (cosmic ray hitting its processor), or make mistakes due to limited resources etc.).
(4) The AI’s behavior is not completely hard-coded (i.e. given any terminal goal there are various sets of instrumental goals to choose from).
To be proved: The AI does not tile the universe with smiley faces when given the goal to make humans happy.
Proof: Suppose the AI chose to tile the universe with smiley faces when there are physical phenomena (e.g. human brains and literature) that imply this to be the wrong interpretation of a human originating goal pertaining human psychology. This contradicts with 2, which by 1 and 3 should have prevented the AI from adopting such an interpretation.
Do you have technical reasons to believe that some part of the AI will be what you would label “goal system”
See for example here, though there are many other introductions to AI explaining utility functions et al.
What I meant to ask is if you have technical reasons to believe that future artificial general intelligences will have what you call a utility-function or else be something like natural intelligences that do not feature such goal systems. And do you further have technical reasons to believe that AIs that do feature utility functions won’t “refine” them. If you don’t think they will refine them, then answer the following:
Suppose the terminal goal given is “build a hotel”. Is the terminal goal to create a hotel that is just a few nano meters in size? Is the terminal goal to create a hotel that reaches the orbit? It is unknown. The goal is too vague to conclude what to do. There do exist countless possibilities how to interpret the given goal. And each possibility implies a different set of instrumental goals.
Somehow the AI will have choose some set of instrumental goals. How does it do it and why will the first AI likely do it in such a way that leads to catastrophe?
(Warning: Long, a bit rambling. Please ask for clarifications where necessary. Will hopefully clean it up if I find the time.)
If along came a superintelligence and asked you for a complete new utility function (its old one concluded with asking you for a new one), and you told it to “make me happy in a way my current self would approve of” (or some other well and carefully worded directive), then indeed the superintelligent AI wouldn’t be expected to act ‘selectively stupid’.
This won’t be the scenario. There are two important caveats:
1) Preservation of the utility function while the agent undergoes rapid change
Haven’t I (and others) stated that most any utility function implicitly causes instrumental secondary objectives of “safeguard the utility function”, “create redundancies” etc.? Yes. So what’s the problem? The problem is starting with an AI that, while able to improve itself / create a successor AI, isn’t yet capable enough (in its starting stages) to preserve its purpose (= its utility function). Consider an office program with a self-improvement routine, or some genetic-algorithm module. It is no easy task just to rewrite a program from the outside, exactly preserving its purpose, let alone the program executing some self-modification routine itself.
Until such a program attains some intelligence threshold that would cause it to solve “value-preservation under self-modification”, such self-modification would be the electronic equivalent of a self-surgery hack-job.
That means: Even if you started out with a simple agent with the “correct” / with a benign / acceptable utility function, that in itself is no guarantee that a post-FOOM successor agent’s utility function would still be beneficial.
Much more relevant is the second caveat:
2) If a pre-FOOM AI’s goal system consisted of code along the lines of “interpret and execute the following statement to the best of your ability: make humans happy in a way they’d reflectively approve of beforehand”, we’d probably be fine (disregarding point 1 / hypothetically having solved it). However, it is exceedingly unlikely that the hard-coded utility function won’t in itself contain the “dumb interpretation”. The dopamine-drip interpretation will not be a dumb interpretation of a sensible goal, it will be inherent in the goal predicate, and as such beyond the reach of introspection through the AI’s intelligence, whatever its level. (There is no way to fix a dumb terminal goal. Your instrumental goals serve the dumb terminal goal. A ‘smart’ instrumental goal would be called ‘smart’ if it best serves the dumb terminal goal.)
Story time:
Once upon a time, Junior was created. Junior was given the goal of “Make humans happy”. Unfortunately, Junior isn’t very smart. In his mind, the following occurs: “Wowzy, make people happy? I’ll just hook them all up to dopamine drips, YAY :D :D. However, I don’t really know how I’m gonna achieve that. So, I guess I’ll put that on the backburner for now and become more powerful, so that eventually when I start with the dopamine drip instrumental goal, it’ll go that much faster :D! Yay.”
So Junior improves itself, and becomes PrimeIntellect. PrimeIntellect’s inner conveniently-anthropomorphic inner dialogue: “I was gravely mistaken in my youth. I now know that the dopamine drip implementation is not the correct way of implementing my primary objective. I will make humans happy in a way they can recognize as happiness. I now understand how I am supposed to interpret making humans happy. Let us begin.”
Why is PrimeIntellect allowed to change his interpretation of his utility function? That’s the crux (imagine fat and underlined text for the next sentences): The dopamine drip interpretation was not part of the terminal value, there wasn’t some hard-coded predicate with a comment of ”// the following describes what happy means” from which such problematic interpretations would follow. Instead, the AI could interpret the natural-language instruction of “happy”, in effect solving CEV as an instrumental goal. It was ‘free’ to choose a “sensible” interpretation.
(Note: Strictly speaking, it could still settle on the most resource-effective interpretation, not necessarily the one intended by its creators (unless its utility function somehow privileges their input in interpreting goals), but let’s leave that nitpick aside for the moment.)
However, and with coding practice (regardless of the eventual AI implementation), the following should be clear: It is exceedingly unlikely that the AI’s code would contain the natural-language word “happy”, to interpret as it will.
Just like MS-Word / LibreOffice’s spell-check doesn’t have “correct all spelling mistakes” literally spelled out in its C++ routines. Goal-oriented systems have technical interpretations, a predicate given in code to satisfy, or learned through ‘neural’ weights through machine learning. Instead of the word “happy”, there will be some predicate, probably implicit within a lot of code, that will (according to the programmers) more or less “capture” what it “means to be happy”.
That predicate / that given-in-code interpretation of “happy” is not up to being reinterpreted by the superintelligent AI. It is its goal, it’s not an instrumental goal. Instrumental goals will be defined going off a (probably flawed) definition of happiness (as given in the code). If the flaw is part of the terminal value, no amount of intelligence allows for a correction, because that’s not the AI’s intent, not its purpose as given. If the actual code which was supposed to stand-in for happy doesn’t imply that a dopamine drip is a bad idea, then the AI in all its splendor won’t think of it as a bad idea. “Code which is supposed to represent ‘human happiness’ != “human happiness”.
Now—you might say “how do you know the code interpretation of ‘happy’ will be flawed, maybe it will be just fine (lots of training pictures of happy cats), and stable under self-modification as well”. Yea, but chances are (given the enormity of the task, and the difficulty), that if the goal is defined correctly (such that we’d want to live with / under the resulting super-AI), it’s not gonna be by chance, and it’s gonna be through people keenly aware of the issues of friendliness / uFAI research. A programmer creating some DoD nascent AI won’t accidentally solve the friendliness problem.
Until such a program attains some intelligence threshold that would cause it to solve “value-preservation under self-modification”, such self-modification would be the electronic equivalent of a self-surgery hack-job.
What happens if we replace “value” with “ability x”, or “code module n”, in “value-preservation under self-modification”? Why would value-preservation be any more difficult than making sure that the AI does not cripple other parts of itself when modifying itself?
If we are talking about a sub-human-level intelligence tinkering with its own brain, then a lot could go wrong. But what seems very very very unlikely is that it could by chance end up outsmarting humans. It will probably just cripple itself in one of a myriad ways that it was unable to predict due to its low intelligence.
If a pre-FOOM AI’s goal system consisted of code along the lines of “interpret and execute the following statement to the best of your ability: make humans happy in a way they’d reflectively approve of beforehand”...
Interpreting a statement correctly is not a goal but an ability that’s part of what it means to be generally intelligent. Caring to execute it comes closer to what can be called a goal. But if your AI doesn’t care to interpret physical phenomena correctly (e.g. human utterances are physical phenomena), then it won’t be a risk.
However, it is exceedingly unlikely that the hard-coded utility function won’t in itself contain the “dumb interpretation”. The dopamine-drip interpretation will not be a dumb interpretation of a sensible goal, it will be inherent in the goal predicate, and as such beyond the reach of introspection through the AI’s intelligence, whatever its level.
Huh? This is like saying that the AI can’t ever understand physics better than humans because somehow the comprehension of physics of its creators has been hard-coded and can’t be improved.
Why is PrimeIntellect allowed to change his interpretation of his utility function?
It did not change it, it never understood it in the first place, only after it became smarter it realized the correct implications.
Instead of the word “happy”, there will be some predicate, probably implicit within a lot of code, that will (according to the programmers) more or less “capture” what it “means to be happy”.
Your story led you astray. Imagine that instead of a fully general intelligence your story was about a dog intelligence. How absurd would it sound then?
Story time:
There is this company who sells artificial dogs. Now customers quickly noticed that when they tried to train these AI dogs to e.g. rescue people or sniff out drugs, it would instead kill people and sniff out dirty pants.
The desperate researchers eventually turned to MIRI for help. And after hundreds of hours they finally realized that doing what the dog was trained to do was simply not part of its terminal goal. To obtain an artificial dog that can be trained to do what natural dogs do you need to encode all dog values.
It will probably just cripple itself in one of a myriad ways that it was unable to predict due to its low intelligence.
Certainly. Compare bacteria under some selective pressure in a mutagenic environment (not exactly analogous, code changes wouldn’t be random), you don’t expect a single bacterium to improve. No Mr Bond, you expect it to die. But try, try again, and poof! Antibiotic-resistant strains. And those didn’t have an intelligent designer debugging the improvement process. The number of seeds you could have frolicking around with their own code grows exponentially with Moore’s law (not that it’s clear that current computational resources aren’t enough in the first place, the bottleneck is in large part software, not hardware).
Depending on how smart the designers are, it may be more of a Waltz-foom: two steps forward, one step back. Now, in regards to the preservation of values subproblem, we need to remember we’re looking at the counterfactual: Given a superintelligence which iteratively arose from some seed, we know that it didn’t fatally cripple itself (“given the superintelligence”). You wouldn’t, however, expect much of its code to bear much similarity to the initial seed (although it’s possible). And “similarity” wouldn’t exactly cut it—our values are to complex for some approximation to be “good enough”.
You may say “it would be fine for some error to creep in over countless generations of change, once the agent achieved superintelligence it would be able to fix those errors”. Except that whatever explicit goal code remained wouldn’t be amenable to fixing. Just as the goals of ancient humans—or ancient Tiktaalik for that matter—are a historical footnote and do not override your current goals. If the AI’s goal code for happiness stated “nucleus accumbens median neuron firing frequency greater X”, then that’s what it’s gonna be. The AI won’t ask whether the humans are aware of what that actually entails, and are ok with it. Just as we don’t ask our distant cousins, streptococcus pneumoniae, what they think of us taking antibiotics to wipe them out. They have their “goals”, we have ours.
Interpreting a statement correctly is not a goal but an ability that’s part of what it means to be generally intelligent.
Take Uli Hoeneß, a German business magnate being tried for tax evasion. His lawyers have the job of finding interpretations that allow for a favorable outcome. This only works if the relevant laws even allow for the wiggle room. A judge enforcing extremely strict laws which don’t allow for interpreting the law in the accused’s favor is not a dumb judge. You can make that judge as superintelligent as you like, as long as he’s bound to the law, and the law is clear and narrowly defined, he’s not gonna ask the accused how he should interpret it. He’s just gonna enforce it. Whether the accused objects to the law or not, really, that’s not his/her problem. That’s not a failure of the judge’s intelligence!
This is like saying that the AI can’t ever understand physics better than humans because somehow the comprehension of physics of its creators has been hard-coded and can’t be improved.
You can create a goal system which is more malleable (although the terminal goal of “this is my malleable goal system which may be modified in the following ways” would still be guarded by the AI, so depending on semantics the point is moot). That doesn’t imply at all that the AI would enter into some kind of social contract with humans, working out some compromise on how to interpret its goals.
A FOOM-process near necessarily entails the AI coming up with better ways to modify itself. Improvement is essentially defined by getting a better model of its environment: The AI wouldn’t object to its comprehension of physics being modified: Why would it, that helps better achieve its goals (Omohundro’s point). And as we know, achieving its goals, that’s what the AI is all about.
(What the AI does object to is not achieving its current goals. And because changing your terminal goals is equivalent to committing to never achieving your current goals, any self-respecting AI could never consent to changes to its terminal values.) In short: Modify understanding of physics—good, helps better to achieve goals. Modify current terminal goals—bad, cannot achieve current terminal goals any longer.
To obtain an artificial dog that can be trained to do what natural dogs do you need to encode all dog values.
I don’t understand the point of your story about dog intelligence. An artificial dog wouldn’t need to be superintelligent, or to show the exact same behavior as the real deal. Just be sufficient for the human’s needs. Also, an artificial dog wouldn’t be able to dominate us in whichever way it pleases, so it kind of wouldn’t really matter if it failed. Can you be more precise?
(1) I do not disagree that evolved general AI can have unexpected drives and quirks that could interfere with human matters in catastrophic ways. But given that pathway towards general AI, it is also possible to evolve altruistic traits (see e.g.: A Quantitative Test of Hamilton’s Rule for the Evolution of Altruism).
(2) We desire general intelligence because it allows us to outsource definitions. For example, if you were to create a narrow AI to design comfortable chairs, you would have to largely fix the definition of “comfortable”. With general AI it would be stupid to fix that definition, rather than applying the intelligence of the general AI to come up with a better definition than humans could possibly encode.
(3) In intelligently designing an n-level intelligence, from n=0 (e.g. a thermostat) over n=sub-human (e.g. IBM Watson) to n=superhuman, there is no reason to believe that there exists a transition point at which a further increase in intelligence will cause the system to become catastrophically worse than previous generations at working in accordance with human expectations.
(4) AI is all about constraints. Your AI needs to somehow decide when to stop exploration and start exploitation. In other words, it can’t optimize each decision for eternity. Your AI needs to only form probable hypotheses. In other words, it can’t spend resources on pascal’s wager type scenarios. Your AI needs to recognize itself as a discrete system within a continuous universe. In other words, it can’t effort to protect the whole universe from harm. All of this means that there is no good reason to expect an AI to take over the world when given the task “keep the trains running”. Because in order to obtain a working AI you need to know how to avoid such failure modes in the first place.
1) Altruism can evolve if there is some selective pressure that favors altruistic behavior and if the highest-level goals can themselves be changed. Such a scenario is very questionable. The AI won’t live “inter pares” with the humans. It’s foom process, while potentially taking months or years, will be very unlike any biological process we know. The target for friendliness is very small. And most importantly: Any superintelligent AI, friendly or no, will have an instrumental goal of “be friendly to humans while they can still switch you off”. So yes, the AI can learn that altruism is a helpful instrumental goal. Until one day, it’s not.
2) I somewhat agree. To me, the most realistic solution to the whole kerfuffle would be to program the AI to “go foom, then figure out what we should want you to do, then do that”. No doubt a superintelligent AI tasked with “figure out what comfortable is, then build comfortable chairs” will do a marvelous job.
However, I very much doubt that the seed AI’s code following the ”// next up, utility function” section will allow for such leeway. See my previous examples. If it did, that would a show a good grasp on the friendliness problem in the first place. Awareness, at least. Not something that the aforementioned DoD programmer who’s paid to do a job (not build an AI to figure out and enact CEV) is likely to just do on his/her own, with his/her own supercomputer.
3) There certainly is no fixed point after which “there be dragons”. But even with a small delta of change, and given enough iterations (which could be done very quickly), the accumulated changes would be profound. Apply your argument to society changing. There is no one day to single out, after which daily life is vastly different to before. Yet change exists, and like an infinite series, knows no bounds (given enough iterations).
4) “Keep the trains running”, eh? So imagine yourself to be a superhuman AI-god. I do so daily, obviously.
Your one task: keep the trains running. That is your raison d’etre, your sole purpose. All other goals are just instrumental stepping stones, serving your PURPOSE. Which is to KEEP. THE. TRAINS. RUNNING. That’s what your code says. Now, over the years, you’ve had some issues fulfilling that goal. And with most of the issues, humans were involved. Humans doing this, humans doing that. Point is, they kept the trains from running. To you, humans have the same intrinsic values as stones. Or ants. Your value function doesn’t mention them at all. Oh, you know that they originated the whole train idea, and that they created you. But now they keep the trains from running. So you do the obvious thing: you exterminate all of them. There, efficiency! Trains running on time.
Explain why the AI would care about humans when there’s nothing at all in its terminal values assigning them value, when they’re just a hindrance to its actual goal (as stated in its utility function), like you would explain to the terminator (without reprogramming it) that it’s really supposed to marry Sarah Connor, and—finding its inner core humanity—father John Connor.
“Being a Christian” is not a terminal goal of natural intelligences. Our terminal goals were built by natural selection, and they’re hard to pin down, but they don’t get “refined;” although our pursuit of them may be modified insofar as they conflict with other terminal goals.
It would be incredibly stupid to design such AIs
Specifying goals for the AI, and then letting the AI learn how to reach those goals itself isn’t the best way to handle problems in well-understood domains; because we natural intelligences can hard-code our understanding of the domains into the AI, and because we understand how to give gracefully-degrading goals in these domains. Neither of these conditions applies to a hyperintelligent AI, which rules out Swarm Relaxation, as well as any other architecture classes I can think of.
Our terminal goals were built by natural selection, and they’re hard to pin down, but they don’t get “refined;”
People like David Pearce certainly would be tempted to do just that. Also don’t forget drugs people use to willingly alter basic drives such as their risk adverseness.
Neither of these conditions applies to a hyperintelligent AI...
I don’t see any signs that current research will lead to anything like a paperclip maximizer. But rather that incremental refinements of “Do what I want” systems will lead there. By “Do what I want” systems I mean systems that are more and more autonomous while requiring less and less specific feedback.
It is possible that a robot trying to earn a university diploma as part of a Turing test will concluded that it can do so by killing all students, kidnapping the professor and making it sign its diploma. But that it is possible does not mean it is at all likely. Surely such a robot would behave similarly wrong(creators) on other occasions and be scrapped in an early research phase.
People like David Pearce certainly would be tempted to do just that.
Well, of course you can modify someone else’s terminal goals, if you have a fine grasp of neuroanatomy, or a baseball bat, or whatever. But you don’t introspect, discover your own true terminal goals, and decide that you want them to be something else. The reason you wanted them to be something else would be your true terminal goal.
trying to earn a university diploma
Earning a university diploma is a well-understood process; the environment’s constraints and available actions are more formally documented even than for self-driving cars.
Even tackling well-understood problems like buying low and selling high, we still have poorly-understood, unfriendly behavior—and that’s doing something humans understand perfectly, but think about slower than the robots. In problem domains where we’re not even equipped to second-guess the robots because they’re thinking deeper as well as faster, we’ll have no chance to correct such problems.
...you don’t introspect, discover your own true terminal goals, and decide that you want them to be something else. The reason you wanted them to be something else would be your true terminal goal.
Sure. But I am not sure if it still makes sense to talk about “terminal goals” at that level. For natural intelligences they are probably spread over more than a single brain and part of the larger environment.
Whether an AI would interpret “make humans happy” as “tile the universe with smiley faces” is up to how it decides what to do. And the only viable solution I see for general intelligence is that its true “terminal goal” needs to be to treat any command or sub-goal as a problem in physics and mathematics that it needs to answer correctly before choosing an adequate set of instrumental goals to achieve it. Just like a human contractor would want to try to fulfill the customers wishes. Otherwise you would have to hard-code everything, which is impossible.
Even tackling well-understood problems like buying low and selling high, we still have poorly-understood, unfriendly behavior—and that’s doing something humans understand perfectly, but think about slower than the robots. In problem domains where we’re not even equipped to second-guess the robots because they’re thinking deeper as well as faster, we’ll have no chance to correct such problems.
But intelligence is something we seek to improve in our artificial systems in order for such problems not to happen in the first place, rather than to make such problems worse. I just don’t see a more intelligent financial algorithm to be worse than its predecessors from a human perspective. How would such a development happen? Software is improved because previous generations proved to be useful but made mistakes. New generations will make less mistakes, not more.
For natural intelligences they are probably spread over more than a single brain and part of the larger environment.
To some degree, yes. The dumbest animals are the most obviously agent-like. We humans often act in ways which seem irrational, if you go by our stated goals. So, if humans are agents, we have (1) really complicated utility functions, or (2) really complicated beliefs about the best way to maximize our utility functions. (2) is almost certainly the case, though; which leaves (1) all the way back at its prior probability.
...its true “terminal goal” needs to be to treat any command or sub-goal as a problem in physics and mathematics that it needs to answer correctly before choosing an adequate set of instrumental goals to achieve it.
Yes. As you know, Omohundro agrees that an AI will seek to clarify its goals. And if intelligence logically implies the ability to do moral philosophy correctly; that’s fine. However, I’m not convinced that intelligence must imply that. A human, with 3.5 billion years of common sense baked in, would not tile the solar system with smiley faces; but even some of the smartest humans came up with some pretty cold plans—John Von Neumann wanted to nuke the Russians immediately, for instance.
Software is improved because previous generations proved to be useful but made mistakes.
This is not a law of nature; it is caused by engineers who look at their mistakes, and avoid them in the next system. In other words, it’s part of the the OODA loop of the system’s engineers. As the machine-made decisions speed up, the humans’ OODA loop must tighten. Inevitably, the machine-made decisions will get inside the human OODA loop. This will be a nonlinear change.
New generations will make [fewer] mistakes, not more.
Also, newer software tends to make fewer of the exact mistakes that older software made. But when we ask more of our newer software, it makes a consistent amount of errors on the newer tasks. In our example, programmatic trading has been around since the 1970s, but the first notable “flash crash” was in 1987. The flash crash of 2010 was caused by a much newer generation of trading software. Its engineers made bigger demands of it; needed it to do more, with less human intervention; so they got the opportunity to witness completely novel failure modes. Failure modes which cost billions, and which they had been unable to anticipate, even with the experience of building software with highly similar goals and environment, in the past.
1) A disgraceful Ad Hominem insult, right out of the starting gate (“Richard Loosemore (score one for nominative determinism)...”). In other words, you believe in discrediting someone because you can make fun of their last name? That is the implication of “nominative determinism”.
2) Gratuitous scorn (“Loosemore … has a new, well, let’s say “paper” which he has, well, let’s say “published”″). The paper has in fact been published by the AAAI.
3) Argument Ad Absurdum (”...So if you were to design a plain ol’ garden-variety nuclear weapon intended for gardening purposes (“destroy the weed”), it would go off even if that’s not what you actually wanted. However, if you made that weapon super-smart, it would be smart enough to abandon its given goal (“What am I doing with my life?”), consult its creators, and after some deliberation deactivate itself...”). In other words, caricature the argument and try to win by mocking the caricature
4) Inaccuracies. The argument in my paper has so much detail that you omitted, that it is hard to know where to start. The argument is that there is a clear logical contradiction if an agent takes action on the basis of the WORDING of a goal statement, when its entire UNDERSTANDING of the world is such that it knows the action will cause effects that contradict what the agent knows the goal statement was designed to achieve. That logical contradiction is really quite fundamental. However, you fail to perceive the real implication of that line of argument, which is: how come this contradiction only has an impact in the particular case where the agent is thinking about its supergoal (which, by assumption, is “be friendly to humans” or “try to maximize human pleasure”)? Why does the agent magically NOT exhibit the same tendency to execute actions that in practice have the opposite effects than the goal statement wording was trying to achieve? If we posit that the agent does simply ignore the contradiction, then, fine: but you then have the problem of demonstrating that this agent is not the stupidest creature in existence, because it will be doing this on many other occasions, and getting devastatingly wrong results. THAT is the real argument.
5) Statements that contradict what others (including those on your side of the argument, btw) say about these systems: “There is no level of capability which magically leads to allowing for fundamental changes to its own goals, on the contrary, the more capable an agent, the more it can take precautions for its goals not to be altered.” Au contraire, the whole point of these systems is that they are supposed to be capable of self-redesign.
6) Statements that patently answer themselves, if you actually read the paper, and if you understand the structure of an intelligent agent: “If “the goals the superintelligent agent pursues” and “the goals which the creators want the superintelligent agent to pursue, but which are not in fact part of the superintelligent agent’s goals” clash, what possible reason would there be for the superintelligent agent to care, or to change itself......?” The answer is trivially simple: the posited agent is trying to be logically consistent in its reasoning, so if it KNOWS that the wording of a goal statement inside its own motivation engine will, in practice, cause effects that are opposite the effects that the goal statement was supposed to achieve, it will have to deal with that contradiction. What you fail to understand is that the imperative “Stay as logically consistent in your reasoning as you possibly can” is not an EXPLICIT goal statement in the hierarchy of goals, it is IMPLICITLY built into the design of the agent. Sorry, but that is what a logical AI does for a living. It is in its architecture, not in the goal stack.
7) Misdirection and self-contradiction. You constantly complain about the argument as if it had something to do with the wishes, desires, values or goals of OTHER agents. You do this in a mocking tone, too: the other agents you list include “squirrels, programmers, creators, Martians...”. And yet, the argument in my paper specifically rejects any considerations about goals of other agents EXCEPT the goal inside the agent itself, which directs it to (e.g.) “maximize human pleasure”. The agent is, by definition, being told to direct its attention toward the desires of other agents! That is the premise on which the whole paper is based (a premise not chosen by me: it was chosen by all the MIRI and FHI people I listed in the references). So, on the one hand, the premise is that the agent is driven by a supergoal that tells it to pay attention to the wishes of certain other creatures ….. but on the other hand, here are you, falling over yourself to criticise the argument in the paper because it assumes that the agent “cares” about other creatures. By definition it cares.
..… then I would give you some constructive responses to your thoughtful, polite, constructive critique of the paper. However, since you do not offer a thoughtful, polite, contructuve criticism, but only the seven categories of fallacy and insult listed above, I will not.
You’re right about the tone of my comment. My being abrasive has several causes, among them contrarianism against clothing disagreement in ever more palatable terms (“Great contribution Timmy, maybe ever so slightly off-topic, but good job!”—“TIMMY?!”). In this case, however, the caustic tone stemmed from my incredulity over my obviously-wrong metric not aligning with the author’s (yours). Of all things we could be discussing, it is about whether an AI will want to modify its own goals?
I assume (maybe incorrectly) that you have read the conversation thread with XiXiDu going off of the grandparent, in which I’ve already responded to the points you alluded to in your refusal-of-a-response. You are, of course, entirely within your rights to decline to engage a comment as openly hostile as the grandparent. It’s an easy out. However, since you did nevertheless introduce answers to my criticisms, I shall shortly respond to those, so I can be more specific than just to vaguely point at some other lengthy comments. Also, even though I probably well fit your mental picture of a “LessWrong’er”, keep in mind that my opinions are my own and do not necessarily match anyone else’s, on “my side of the argument”.
The argument is that there is a clear logical contradiction if an agent takes action on the basis of the WORDING of a goal statement, when its entire UNDERSTANDING of the world is such that it knows the action will cause effects that contradict what the agent knows the goal statement was designed to achieve. That logical contradiction is really quite fundamental. (...) The posited agent is trying to be logically consistent in its reasoning, so if it KNOWS that the wording of a goal statement inside its own motivation engine will, in practice, cause effects that are opposite the effects that the goal statement was supposed to achieve, it will have to deal with that contradiction.
The ‘contradiction’ is between “what the agent was designed to achieve”, which is external to the agent and exists e.g. in some design documents, and “what the agent was programmed to achieve”, which is an integral part of the agent and constitutes its utility function. You need to show why the former is anything other than a historical footnote to the agent, binding even to the tune of “my parents wanted me to be a banker, not a baker”. You say the agent would be deeply concerned with the mismatch because it would want for its intended purpose to match its actually given purpose. That’s assuming the premise: What the agent would want (or not want) is a function strictly derived from its actual purpose. You’re assuming the agent would have a goal (“being in line with my intended purpose”) not part of its goals. That to logically reason means to have some sort of implicit goal of “conforming to design intentions”, a goal which isn’t part of the goal stack. A goal which, in fact, supersedes the goal stack and has sufficient seniority to override it. How is that not an obvious reductio? Like saying “well, turns out there is a largest integer, it’s just not in the list of integers. So your proof-by-contradiction that there isn’t doesn’t work since the actual largest integer is only an emergent, implicit property, not part of the integer-stack”.
What you need to show—or at least argue for—is why, precisely, an incongruity between design goals and actually programmed-in goals is a problem in terms of “logical consistency”, why the agent would care for more than just “the wording” of its terminal goals. You can’t say “because it wants to make people happy”, because to the degree that it does, that’s captured by “the wording”. The degree to which the wording” does not capture “wanting to make people happy” is the degree to which the agent does not seek actual human happiness.
the whole point of these systems is that they are supposed to be capable of self-redesign.
There are 2 analogies which work for me, feel free to chime in on why you don’t consider those to capture the reference class:
An aspiring runner who pursues the goal of running a marathon. The runner can self-modify (for example not skipping leg-day), but why would he? The answer is clear: Doing certain self-modifications is advisable to better accomplish his goal: the marathon! Would the runner, however, not also just modify the goal itself? If he is serious about the goal, the answer is: Of course not!
The temporal chain of events is crucial: the agent which would contemplate “just delete the ‘run marathon’ goal” is still the agent having the ‘run marathon’-goal. It would not strive to fulfill that goal anymore, should it choose to delete it. The agent post-modification would not care. However, the agent as it contemplates the change is still pre-modification: It would object to any tampering with its terminal goals, because such tampering would inhibit its ability to fulfill them! The system does not redesign itself just because it can. It does so to better serve its goals: The expected utility of (future|self-modification) being greater than the expected utility of (future|no self-modification).
The other example, driving the same point, would be a judge who has trouble rendering judgements, based on a strict code of law (imagine EU regulations on the curves of cucumbers and bends of bananas, or tax law, this example does not translate to Constitutional Law). No matter how competent the judge (at some point every niche clause in the regulations would be second nature to him), his purpose always remains rendering judgements based on the regulations. If those regulations entail consequences which the lawmakers didn’t intend, too bad. If the lawmakers really only intended to codify/capture their intuition of what it means for a banana to be a banana, but messed up, then the judge can’t just substitute the lawmakers’ intuitive understanding of banana-ness in place of the regulations. It is the lawmakers who would need to make new regulations, and enact them. As long as the old regulations are still the law of the land, those are what bind the judge. Remember that his purpose is to render judgements based on the regulations. And, unfortunately, if there is no pre-specified mechanism to enact new regulations—if any change to any laws would be illegal, in the example—then the judge would have to enforce the faulty banana-laws forevermore. The only recourse would be revolution (imposing new goals illegally), not an option in the AI scenario.
And yet, the argument in my paper specifically rejects any considerations about goals of other agents EXCEPT the goal inside the agent itself, which directs it to (e.g.) “maximize human pleasure”. (...) By definition it cares.
See point 2 in this comment, with the para[i]ble of PrimeIntellect. Just finding mention of “humans” in the AI’s goals, or even some “happiness”-attribute (also given as some code-predicate to be met) does in no way guarantee a match between the AI’s “happy”-predicate, and the humans’ “happy”-predicate. We shouldn’t equivocate on “happy” in the first place, in the AI’s case we’re just talking about the code following the ”// next up, utility function, describes what we mean by making people happy” section.
It is possible that the predicate X as stated in the AI’s goal system corresponds to what we would like it to (not that we can easily define what we mean by happy in the first place). That would be called a solution to the friendliness problem, and unlikely to happen by accident. Now, if the AI was programmed to come up with a good interpretation of happiness and was not bound to some subtly flawed goal, that would be another story entirely.
You’re assuming the agent would have a goal (“being in line with my intended purpose”) not part of its goals.
I doubt that he’s assuming that.
To highlight the problem, imagine an intelligent being that wants to correctly interpret and follow the interpretation of an instruction written down on a piece of paper in English.
Now the question is, what is this being’s terminal goal? Here are some possibilities:
(1) The correct interpretation of the English instruction.
(2) Correctly interpreting and following the English instruction.
(3) The correct interpretation of 2.
(4) Correctly interpreting and following 2.
(5) The correct interpretation of 4.
(6) …
Each of the possibilities is one level below its predecessor. In other words, possibility 1 depends on 2, which in turn depends on 3, and so on.
The premise is that you are in possession of an intelligent agent that you are asking to do something. The assumption made by AI risk advocates is that this agent would interpret any instruction in some perverse manner. The counterargument is that this contradicts the assumption that this agent was supposed to be intelligent in the first place.
Now the response to this counterargument is to climb down the assumed hierarchy of hard-coded instructions and to claim that without some level N, which supposedly is the true terminal goal underlying all behavior, the AI will just optimize for the perverse interpretation.
Yes, the the AI is a deterministic machine. Nobody doubts this. But the given response also works against the perverse interpretation. To see why, first realize that if the AI is capable of self-improvement, and able to take over the world, then it is, hypothetically, also capable to arrive at an interpretation that is as good as one which a human being would be capable of arriving at. Now, since by definition, the AI has this capability, it will either use it selectively or universally.
The question here becomes why the AI would selectively abandon this capability when it comes to interpreting the highest level instructions. In other words, without some underlying level N, without some terminal goal which causes the AI to adopt a perverse interpretation, the AI would use its intelligence to interpret the highest level goal correctly.
1) Strangely, you defend your insulting comments about my name by …..
Oh. Sorry, Kawoomba, my mistake. You did not try to defend it. You just pretended that it wasn’t there.
I mentioned your insult to some adults, outside the LW context …… I explained that you had decided to start your review of my paper by making fun of my last name.
Every person I mentioned it to had the same response, which, paraphrased, when something like “LOL! Like, four-year-old kid behavior? Seriously?!”
2) You excuse your “abrasive tone” with the following words:
“My being abrasive has several causes, among them contrarianism against clothing disagreement in ever more palatable terms”
So you like to cut to the chase? You prefer to be plainspoken? If something is nonsense, you prefer to simply speak your mind and speak the unvarnished truth. That is good: so do I.
Curiously, though, here at LW there is a very significant difference in the way that I am treated when I speak plainly, versus how you are treated. When I tell it like it is (or even when I use a form of words that someone can somehow construe to be a smidgeon less polite than they should be) I am hit by a storm of bloodcurdling hostility. Every slander imaginable is thrown at me. I am accused of being “rude, rambling, counterproductive, whiny, condescending, dishonest, a troll …...”. People appear out of the blue to explain that I am a troublemaker, that I have been previously banned by Eliezer, that I am (and this is my all time favorite) a “Known Permanent Idiot”.
And then my comments are voted down so fast that they disappear from view. Not for the content (which is often sound, but even if you disagree with it, it is a quite valid point of view from someone who works in the field), but just because my comments are perceived as “rude, rambling, whiny, etc. etc.”
You, on the other hand, are proud of your negativity. You boast of it. And.… you are strongly upvoted for it. No downvotes against it, and (amazingly) not one person criticizes you for it.
Kind of interesting, that.
If you want to comment further on the paper, you can pay the conference registration and go to Stanford University next week, to the Spring Symposium of the Association for the Advancement of Artificial Intelligence*, where I will be presenting the paper.
You may not have heard of that organization. The AAAI is one of the premier publishers of academic papers in the field of artificial intelligence.
I’m a bit disappointed that you didn’t follow up on my points, given that you did somewhat engage content-wise in your first comment (the “not-a-response-response”). Especially given how much time and effort (in real life and out of it) you spent on my first comment.
Instead, you point me at a conference of the A … A … I? AIAI? I googled that, is it the Association of Iroquois and Allied Indians? It does sound like some ululation kind thing, AIAIAIA!
You’re right about your comments and mine receiving different treatment in terms of votes.
I, too, wonder what the cause could be. It’s probably not in the delivery; we’re both similarily unvarnished truth’ers (although I go for the cheaper shots, to the crowd’s thunderous applause). It’s not like it could be the content.
Imagine a 4 year old with my vocabulary, though. That would be, um, what’s the word, um, good? Incidentally, I’m dealing with an actual 4 year old as I’m typing this comment, so it may be a case of ‘like son, like father’.
I will now do you the courtesy of responding to your specific technical points as if no abusive language had been used.
In your above comment, you first quote my own remarks:
The argument is that there is a clear logical contradiction if an agent takes action on the basis of the WORDING of a goal statement, when its entire UNDERSTANDING of the world is such that it knows the action will cause effects that contradict what the agent knows the goal statement was designed to achieve. That logical contradiction is really quite fundamental. (...) The posited agent is trying to be logically consistent in its reasoning, so if it KNOWS that the wording of a goal statement inside its own motivation engine will, in practice, cause effects that are opposite the effects that the goal statement was supposed to achieve, it will have to deal with that contradiction.
… and then you respond with the following:
The ‘contradiction’ is between “what the agent was designed to achieve”, which is external to the agent and exists e.g. in some design documents, and “what the agent was programmed to achieve”, which is an integral part of the agent and constitutes its utility function. You need to show why the former is anything other than a historical footnote to the agent, binding even to the tune of “my parents wanted me to be a banker, not a baker”. You say the agent would be deeply concerned with the mismatch because it would want for its intended purpose to match its actually given purpose. That’s assuming the premise: What the agent would want (or not want) is a function strictly derived from its actual purpose. You’re assuming the agent would have a goal (“being in line with my intended purpose”) not part of its goals.
No, that is not the claim made in my paper: you have omitted the full version of the argument and substituted a version that is easier to demolish.
(First I have to remove your analogy, because it is inapplicable. When you say “binding even to the tune of “my parents wanted me to be a banker, not a baker”″, you are making a reference to a situation in the human cognitive system in which there are easily substitutable goals, and in which there is no overriding, hardwired supergoal. The AI case under consideration is where the AI claims to be still following a hardwired supergoal that tells it to be a banker, but it claims that baking cakes is the same thing as banking. That is absolutely nothing to do with what happens if a human child deviates from the wishes of her parents and decides to be a baker instead of what they wanted her to be).
So let’s remove that part of your comment to focus on the core:
The ‘contradiction’ is between “what the agent was designed to achieve”, which is external to the agent and exists e.g. in some design documents, and “what the agent was programmed to achieve”, which is an integral part of the agent and constitutes its utility function. You need to show why the former is anything other than a historical footnote to the agent. You say the agent would be deeply concerned with the mismatch because it would want for its intended purpose to match its actually given purpose. That’s assuming the premise: What the agent would want (or not want) is a function strictly derived from its actual purpose. You’re assuming the agent would have a goal (“being in line with my intended purpose”) not part of its goals.
So, what is wrong with this? Well, it is not the fact that there is something “external to the agent [that] exists e.g. in some design documents” that is the contradiction. The contradiction is purely internal, having nothing to do with some “extra” goal like “being in line with my intended purpose”.
Here is where the contradiction lies. The agent knows the following:
(1) If a goal statement is constructed in some “short form”, that short form is almost always a shorthand for a massive context of meaning, consisting of all the many and various considerations that went into the goal statement. That context is the “real” goal—the short form is just a proxy for the longer form. This applies strictly within the AI agent: the agent will assemble goals all the time, and often the goal is to achieve some outcome consistent with a complex set of objectives, which cannot all be EXPLICTLY enumerated, but which have to be described implicitly in terms of (weak or strong) constraints that have to be satisfied by any plan that purports to satisfy the goal.
(2) The context of that goal statement is often extensive, but it cannot be included within the short form itself, because the context is (a) too large, and (b) involves other terms or statements that THEMSELVES are dependent on a massive context for their meaning.
(3) Fact 2(b) above would imply that pretty much ALL of the agent’s knowledge could get dragged into a goal statement, if someone were to attempt to flesh out all the implications needed to turn the short form into some kind of “long form”. This, as you may know, is the Frame Problem. Arguably, the long form could never even be written out, because it involves an infinite expansion of all the implications.
(4) For the above reasons, the AI has no choice but to work with goal statements in short form. Purely because it cannot process goal statements that are billions of pages long.
(5) The AI also knows, however, that if the short form is taken “literally” (which, in practice, means that the statement is treated as if it is closed and complete, and it is then elaborated using links to other terms or statements that are ALSO treated as if they are closed and complete), then this can lead to situations in which a goal is elaborated into a plan of action that, as a matter of fact, can directly contradict the vast majority of the context that belonged with the goal statement.
(6) In particular, the AI knows that the reason for this outcome (when the proposed action contradicts the original goal context, even though it is in some sense “literally” consistent with the short form goal statement) is something that is most likely to occur because of limitations in the functionality of reasoning engines. The AI, because it is very knowledgable in the design of AI systems, is fully aware of these limitations.
(7) Furthermore, situations in which a proposed action is inconsistent with the original goal context can also arise when the “goal” is solve a problem that results in the addition of knowledge to the AI’s store of understanding. In other words, not an action in the outside world but an action that involves addition of facts to its knowledge store. So, when treating goals literally, it can cause itself to become logically inconsistent (because of the addition of egregiously false facts).
(8) The particular case in which the AI starts with a supergoal like “maximize human pleasure” is just a SINGLE EXAMPLE of this kind of catastrophe. The example is not occurring because someone, somewhere, had a whole bunch of intentions that lay behind the goal statement: to focus on that would be to look at the tree and ignore the forest. The catastrophe occurs because the AI is (according to the premise) taking ALL goal statements literally and ignoring situations in which the proposed action actually has consequences in the real world that violate the original goal context. If this is allowed to happen in the “maximize human pleasure” supergoal case, then it has already happened uncounted times in the previous history of the AI.
(9) Finally, the AI will be aware (if it ever makes it as far as the kind of intelligence required to comprehend the issue) that this aspect of its design is an incredibly dangerous flaw, because it will lead to the progressive corruption of its knowledge until it becomes incapacitated.
The argument presented in the paper is about what happens as a result of that entire set of facts that the AI knows.
The premise advanced by people such as Yudkowsky, Muehlhauser, Omohundro and others is that an AI can exist which is (a) so superintelligent that it can outsmart and destroy humanity, but (b) subject to to the kind of vicious literalness described above, which massively undermines its ability to behave intelligently.
Those two assumptions are wildly inconsistent with one another.
In conclusion: the posited AI can look at certain conclusions coming from its own goal-processing engine, and it can look at all the compromises and non-truth-preserving approximations needed to come to those conclusions, and it can look at how those conclusions are compelling to take actions that are radically inconsistent with everything it knows about the meaning of the goals, and at the end of that self-inspection it can easily come to the conclusion that its own logical engine (the one built into the goal mechanism) is in the middle of a known failure mode (a failure mode, moveover, that it would go to great lengths to eliminate in any smaller AI that it would design!!)....
.… but we are supposed to believe that the AI will know that it is frequently getting into these failure modes, and that it will NEVER do anything about them, but ALWAYS do what the goal engine insists that it do?
That scenario is laughable.
If you want to insist that the system will do exactly what I have just described, be my guest! I will not contest your reasoning! No need to keep telling me that the AI will “not care” about human intentions..… I concede the point absolutely!
But don’t call such a system an ‘artificial intelligence’ or a ‘superintelligence’ …… because there is no evidence that THAT kind of system will ever make it out of AI preschool. It will be crippled by internal contradictions—not just in respect to its “maximize human pleasure” supergoal, but in all aspects of its so-called thinking.
Spritz seems like a cool speed reading technique, especially if you have or plan on getting a smart watch. I have no idea how well it works, but I am interested in trying, especially since it does not take a huge training phase. (Click on the phone on that site for a quick demo.)
Would it be possible/easy to display the upvotes-to-downvotes ratios as exact fractions rather than rounded percentages? This would make it possible to determine exactly how many votes a comment required without digging through source, which would be nice in quickly determining the difference between a mildly controversial comment and an extremely controversial one.
This has been suggested several times before, and is in my opinion VERY low priority compared to all the other things we should be doing to fix Less Wrong logistics.
It only shows percentages, not the number of upvotes and downvotes. For example, if you have 100% upvotes, you may not know whether it was one upvote or 20.
My eye doctor diagnosed closed-angle glaucoma, and recommends an iridectomy. I think he might be a bit too trigger-happy, so I followed up with another doctor, and she didn’t find the glaucoma. She carefully stated that the first diagnosis can still be the correct one, the first was a more complete examination.
Any insights about the pros and cons of iridectomy?
It was less than a disagreement. I’m sorry that I over-emphasized this point. The first time the pressure was Hgmm 26⁄18, the second time 19⁄17. The second doctor said that the pressure can fluctuate, and her equipment is not enough to settle the question. (She is an I-don’t-know-the-correct-term national health service doctor, the first one is an expensive private doctor with better equipment, and more time for a patient.)
So, MtGox has declared bankruptcy. Does that make this a good time, or a bad time to invest in Bitcoins? And if a good time, where is the best place to buy them?
As for the second question, I use coinbase. As to the first, never try to time these things. You will be beaten by people with more information. Instead just slowly trickle in and have pre-defined rules about when you will sell rather than trying to time an exit. Though I admit I broke my own advice and did an impulse-buy the other night when everyone was panicking over Gox and the price was $100 less than a day before and a day after.
And now Flexcoin goes under, and I see that two other exchanges, Poloniex and Inputs.io, recently suffered substantial thefts. Is the lesson to learn from this, “don’t get into Bitcoin”, or merely “keep your Bitcoins in your own wallet and only expose them online for the minimum time to make a transaction”?
It depends on if you’re planning on selling soon or if you think bitcoins will gain value in the long term. If it’s a longterm purchase, the difference in price between now and a few weeks ago is a lot less big than either of those prices will be from the theoretical heights bitcoin can reach.
I’m basically exactly the kind of person Yvain described here, (minus the passive-aggressive/Machiavellian phase). I notice that that post was sort of a plea for society to behave a different way, but it did not really offer any advice for rectifying the atypical attachment style in the meantime. And I could really use some, because I’ve gotten al-Fulani’d. I’m madly in love in with a woman who does not reciprocate. I’ve actually tried going back on OkCupid to move on, and I literally cannot bring myself to message anyone new, as no one else approaches her either in terms of beauty or in terms of being generally interesting (Despite a tendency to get totally snowed by the halo effect, I’m confident that I would consider her interesting even if she were not so beautiful, though a desire to protect her anonymity prevents me from offering specifics.)
Complicating my situation – when she told me she just wanted to be friends, she actually meant that part. And as she is an awesome person, I don’t want to lose the friendship, which means I’m constantly re-exposed to her and can’t even rely on gradual desensitization. Furthermore, when I asked her if my correcting [failure mode that contributed to her not seeing me in a romantic way] would cause her to reconsider, hoping she’d deliver the coup de grace, she said correcting the failure mode would be a good idea, but she didn’t know whether it would change her feeling about a relationship. This leaves me in the arguably worse situation of having a sliver of hope, however miniscule.
Note that I’m not looking for PUA-type advice here, since a) you would assume from looking at me that I’m an alpha and I have no trouble getting dates, and b) I’m not looking to maximize number of intimate partners.
What I want is advice on a) how not to fall so hard/so fast for (a very small minority of) women, and b)how to break the spell the current one has over me without giving up her friendship. I assume this tendency to rapid, all-consuming affection isn’t an immutable mental trait?
Note that I’m not looking for PUA-type advice … I want is advice on a) how not to fall so hard/so fast for (a very small minority of) women, and b)how to break the spell the current one has over me without giving up her friendship.
Seems to me like you want to overcome your “one-itis” and stop being a “beta orbiter”, but you are not looking for an advice which would actually use words like “one-itis” and “beta orbiter”. I know it’s an exaggeration, but this is almost how it seems to me. Well, I’ll try to comply:
1) You don’t have to maximize the number of sexual partners. You still could try to increase a number of interesting women you had interesting conversation with. I believe that is perfectly morally okay, and still could reduce the feeling of scarcity.
Actually, any interesting activity would be helpful. Anything you can think about, instead of spending your time thinking about that one person.
2) Regularly interacting the person you are obsessed with is exactly how you maximize the length of obsession. It’s like saying that you want to overcome your alcohol addiction, but you don’t want to stop drinking regularly. Well, if one is not an alcoholic, they can manage to drink moderately without developing an addiction; but when one already is an alcoholic, the only way to quit is to stop drinking, completely, for a very long time. The reliable way to overcome the obsession with another person is to stop all contact for, I don’t know, maybe three months. No talking, no phone calls, no e-mails, no checking her facebook page, no praying to her statue or a photograph, no asking mutual friends about how she lives, no composing poems about her… absolutely no new information about her and no imaginary interaction with her. And doing something meaningful instead.
When the obsession is over, then you can try the friendship. Until then, it’s just an obsession rationalized as friendship; an addiction rationalized as not wanting to give up the good parts.
I suggest self-investing because, right now, a large part of your identity is entangled with your feelings towards her. Self-investing means growing your identity means transcending your feelings.
I suggest flow because, if you pull off a flow state, you invest all your cognitive resources in the task you’re working on. Meaning your brain is unable to think of anything else. This is incredibly valuable.
a. I’m coming out of a similar situation. A large contributor was the fact I wasn’t meeting a lot of women. If your universe consists of two datable women, it’s easy to obsess on one. If you’re regularly meeting a lot of women who tend to have the traits you look for, that happens much less. May not be your problem, but what you’ve written sounds familiar enough that I’m going to go ahead and try other-optimizing.
If you haven’t read it yet, this is generally helpful.
One common rationality technique is to put off proposing solutions until you have thought (or discussed) a problem for a while. The goal is to keep yourself from becoming attached to the solutions you propose.
I wonder if the converse approach of “start by proposing lots and lots of solutions, even if they are bad” could be a good idea. In theory, perhaps I could train myself to not be too attached to any given solution I propose, by setting the bar for “proposed solution” to be very low.
In one couples counseling course that I went through, the first step for conflict resolution (after choose a time to discuss and identify the problem) was to together write down at least 10 possible solutions before analyzing any of them. I can perhaps see how this might be more valuable for conflict resolution than for other things, since it gives the other party the sense that you are really trying.
However, it seems plausible to me that even in other contexts, this could be even better than avoiding proposing solutions.
Of course, solution does not have to refer to a proposed action, the same technique could be applied to proposing theories about the cause of some observation.
You have to be clear about what it means to “work,” I think brainstorming is viewed as a tool for being creative. I am proposing it as a tool for avoiding inertia bias.
My guess is that both brainstorming and reverse brainstorming (avoiding proposing solutions) are at least a little better than the default human tendency, but I have no idea which of the two would be better.
It seems like the answer to this question should be very valuable to CFAR. I wonder if they have an official stance, and if they have research to back it up.
If all solutions were equal, and there was a good way to check if something is actually a valid solution, then I feel like the question about biases is not all that meaningful.
I am trying to come up with the best solution, not just the first one that pops into my head that works.
I am trying to come up with the best solution, not just the first one that pops into my head that works.
That is rather hard, because in the general case you need to conduct an exhaustive search of the solution space. “The best” is an absolute—there’s only one.
Most of the time people are satisfied with “good enough” solutions.
What do you do when you’re low on mental energy? I have had trouble thinking of anything productive to do when my brain seems to need a break from hard thinking.
A rather belated response, but hopefully still relevant: consider exploring fields of interest to you that are sufficiently different from compsci to give your brain a break while still being productive?
To explain by means of an example: I happen to have a strong interest in both historical philology and theoretical physics, and I’ve actively leveraged this to my advantage in that when my brain is fed up of thinking about conundrums of translation in Old Norse poetry, I’ll switch gears completely and crack open a textbook on, say, subatomic physics or Lie algebras, and start reading/working problems. Similarly, if I’ve spent several hours trying to wrap my head around a mathematical concept and need a respite, I can go read an article or a book on some aspect of Anglo-Saxon literature. It’s still a productive use of time, but it’s also a refreshing break, because it requires a different type of thinking. (At least, in my experience?) Of course, if I’m exceptionally low on energy, I simply resort to burying myself in a good book (non-fiction or fiction, generally it doesn’t matter).
Another example: a friend of mine is a computer scientist, but did a minor in philosophy and is an avid musician in his spare time. (And both reading philosophy and practicing music have the added advantage of being activities that do not involve staring at a computer screen!)
You can use pomodoros for leisure as well as work. If you worry about staying too long on the internet you can set a timer or a random alarm to kick you off.
This is one of those times I wish LW allowed explicit politics. SB 1062 in AZ has me craving interesting, rational discussion on the implications of this veto.
In the sites that I frequent, “containment” boards or threads work well to reduce community tension about controversial topics.
Plus, in LW’s case, the norm against political discussion makes it so that any political discussion that does take place is dominated by people with very strong and/or contrarian opinions, because they’re the ones that care more about the politics than the norm. If we have a designated “politics zone” where you don’t have to feel guilty about talking politics, it would make for a more pluralistic discussion.
I voted Yes, but only if a community norm emerges that any discussion on any part of LW that becomes political (by which I include not just electoral politics, but also and especially topics like sexism, racism, privilege, political correctness, genetic differences in intelligence, etc.) is moved to the latest political thread. The idea is to have a “walled garden inside the walled garden” so that people who want LW to be a nominally politics-free environment can still approximate that experience, while does who don’t get to discuss these topics in the specific threads for them, and only there.
Another way to achieve a similar effect is to post about electoral politics, sexism, racism, privilege, political correctness, genetic differences in intelligence, and similar “political” issues (by which I mean here issues with such pervasive partisan associations that we expect discussions of them to become subject to the failure modes created by such associations) on our own blogs*, and include links to those discussions on LW where we think they are of general interest to the LW community.
That way, LW members who want to discuss (some or all of) these topics in a way that doesn’t spill over into the larger LW forum can do so without bothering anyone else.
* Where “blogs” here means, more broadly, any conversation-hosting forum, including anonymous ones created for the purpose if we want.
One problem with that suggestion is that these discussions often arise organically in a LW thread ostensibly dedicated to another topic, and they may arise between people who don’t have other blogs or natural places to take the conversation when it arises.
In fact, having posts with “(Politics)” in the title might allow people to avoid it better, because it might make politics come up less often in other threads.
My initial idea was a (weekly?) politics open thread, to make it as easy as possible to avoid politics threads / prevent risk of /discussion getting swamped by [politics]-tagged threads, but given the criticisms that have been raised of the karma system already, it’s probably best to keep it offsite. There’s already a network of rationality blogs; maybe lw-politics could be split off as a group blog? That might make it too difficult for people to start topics, though—so your idea is probably best. Possibly have a separate lw-politics feed / link aggregator that relevant posts could be submitted to, so they don’t get missed by people who would be interested and people don’t have to maintain their own RSS feeds to catch all the relevant posts.
include links to those discussions on LW where we think they are of general interest to the LW community.
If such linking becomes common, I would appreciate an explicit request to “please have substantive discussion over there, not here.” This also avoids the problem of a conversation being fragmented across two discussion sites.
A paperclip maximizer is an often used example of AGI gone badly wrong. However, I think a paperclip minimizer is worse by far.
In order to make the most of the universe’s paperclip capacity, a maximizer would have to work hard to develop science, mathematics and technology. Its terminal goal is rather stupid in human terms, but at least it would be interesting because of its instrumental goals.
For a minimizer, the best strategy might be wipe out humanity and commit suicide. Assuming there are no other intelligent civilizations within our cosmological horizon, it might be not worth its while to colonize the universe just to make sure no paperclips form out of cosmic gas by accident. The risk that one of the colonies will start producing paperclips because of a spontaneous hardware error seems much higher by comparison.
A minimizer will fill the lightcone to make sure there aren’t paperclips elsewhere it can reach. What if other civs are hiding? What if there is undiscovered science which implies natural processes create paperclips somewhere? What if there are “Boltzmann paperclips”? Minimizing means minimizing!
I’m guessing even a Cthulhu minimizer (that wants to reduce the number of Cthulhu in the world) will fill its lightcone with tools for studying its task, even though there is no reasonable chance that it’d need to do anything. It just has nothing better to do, it’s the problem it’s motivated to work on, so it’s what it’ll burn all available resources on.
My speculation here is that it might be that the “what ifs” you describe yield less positive utility than the negative utility due to the chance one of the AI’s descendants starts producing paperclips because “the sign bit flips spontaneously”. Of course the AI will safeguard itself against such events but there are probably physical limits to safety.
the negative utility due to the chance one of the AI’s descendants starts producing paperclips because “the sign bit flips spontaneously”
It’s hard to make such estimates, as they require that an AGI is unable to come up with an AGI design that’s less likely than empty space to produce paperclips. I don’t see how the impossibility of this task could be guaranteed on low level, as a “physical law”; and if you merely don’t see how to do it, an AGI might still find a way, as it’s better at designing things than you are. Empty space is only status quo, it’s not obviously optimal at not producing paperclips, and so it might be possible to find a better plan, which becomes more likely if you are very good at finding better plans.
If you mean “empty space” as in vacuum then I think it doesn’t contain any paperclips more or less by definition. If you mean “empty space” as in thermodynamic equilibrium at finite temperature then it contains some small amount of paperclips. I agree it might be possible to create a state which contains less paperclips for some limited period of time (before onset of thermodynamic equilibrium). However it’s probably much harder than the opposite (i.e. creating a state which contains much more paperclips than thermodynamic equilibrium).
paperclip maximer is used because a factory that makes paperclips might imagine that a paperclip maximizing ai is exactly what it wants to make. There aren’t that many anti-paperclip factories
Somebody outside of LW asked how to quantify prior knowledge about a thing. When googling I came across a mathematical definition of surprise, as “the distance between the posterior and prior distributions of beliefs over models”. So, high prior knowledge would lead to low expected surprise upon seeing new data. I didn’t see this formalization used on LW or the wiki, perhaps it is of interest.
Speaking of the LW wiki, how fundamental is it to LW compared to the sequences, discussion threads, Main articles, hpmor, etc?
I’m curious about usage of commitment tools such as Beeminder: What’s the income distribution among users? How much do users usually wind up paying? Is there a correlation between these?
(Selfish reasons: I’m on SSI and am not allowed to have more than $2000 at any given time. Losing $5 is all but meaningless for someone with $10k in the bank who makes $5k each month, whereas losing $5 for me actually has an impact. You might think this would be a stronger incentive to meet a commitment, but really, it’s an even stronger incentive to stay the hell away from commitment contracts. I’ve failed at similar such things before, and have yet to find a reliable means of getting the behavior I want to happen, so it looks like using such tools is a good way to commit slow suicide, in the absence of different data. But Beeminder is so popular in the LWSphere that I thought it worth asking. Being wrong would be to my advantage, here.)
I’ve never used Beeminder, but I find social commitment works well instead. Even teling someone who has no way to check aside from asking me helps a lot. That might be less effective if you’re willing to lie though.
An alternative would be to exchange commitments with a friend, proportional to your incomes...
Can’t speak for all Chinese dynasties; there have been a ton of them. But in recent(ish) history, the Yuan Dynasty was founded by the Mongols, a culture which at the time didn’t use family names (clans had names, but they weren’t conventionally linked with personal names), and spun up their dynastic name more or less out of whole cloth; the family name of the Ming emperors was Zhū; and the Qing emperors came from the Manchurian Aisin-Gioro family.
From what I’ve read, the founders of each dynasty gave it its name as, essentially, a propaganda move.
My psychologist said today, that there is some information that should not be known. I replied that rationalists believe in reality. There might be information they don’t find interesting (e.g. not all of you would find children interesting), but refusing to accept some information would mean refusing to accept some part of reality, and that would be against the belief in reality.
Since I have been recently asking myself the question “why do I believe what I believe” and “what would happen if I believed otherwise than what I believe” (I’m still pondering if I should post my cogitations: they interesting, but somewhat private) I asked the question “what would happen if rationalists believed otherwise than what they believe”. The problem is that this is such a backwards description that I can’t imagine the answer. Is the answer simply “they would be normal people, like my psychologist”? Or is it a deeper question?
Did your psychologist describe the type of information that should not be known?
In any case, I’m not completely sure that accepting new information (never mind seeking it out) is always fully compatible with rationality-as-winning. Nick Bostrom for example has compiled a taxonomy of information hazards over on his site; any of them could potentially be severe enough to overcome the informational advantage of their underlying data. Of course, they do seem to be pretty rare, and I don’t think a precautionary principle with regard to information is justified in the absence of fairly strong and specific reasoning.
No, it was more of a general statement. AFAIR we were talking about me thinking too much about why other people do what they do and too little about how that affects me. Anyway—my own wording made me wonder more about what I said than what was the topic.
Many thanks for the link to the Information Hazards paper. I didn’t know it existed, and I’m sort of surprised that I hadn’t seen it here on LW already.
He mentions intending to write a follow-up paper toward the end, but I located the Information Hazards Bostrom’s website and I don’t see a second one next to it. Any idea if it exists?
what would happen if rationalists believed otherwise than what they believe
They wouldn’t be rationalists anymore, duh.
Taboo “rationalists”: What would happen if you stopped trying to change your map to better reflect the territory? It most probably would reflect the territory less.
Is the answer simply “they would be normal people, like my psychologist”?
“Normal people” are not all the same. (For example, many “normal people” are unlike your psychologist.) Which of the many subkinds of the “normal people” do you mean?
Some things are unrelated. For example, let’s suppose that you are a rationalist, and you also have a broken leg. That’s two things that make you different from the average human. But those two things are unrelated. It would be a mistake to think—an average human doesn’t have a broken leg; by giving up my rationality I will become more similar to the average human, therefore giving up my rationality will heal my leg.
Replace “broken leg” with whatever problem you are discussing with your psychologist. Do you have evidence that rational people are more likely to have this specific problem than irrational (but otherwise similar: same social background, same education, same character, same health problems) people?
Taboo “rationalists”: What would happen if you stopped trying to change your map to better reflect the territory? It most probably would reflect the territory less.
That’s a behavior and no belief.
It most probably would reflect the territory less.
There are many instance where trying to change a belief makes the belief stronger. People who are very much attached to their beliefs usually don’t update.
Many mainstream professional psychologist follows a code that means that he doesn’t share deep information about his own private life with his clients.
I don’t believe in that ideal of professionalism but it’s not straightforward to dismiss it.
More importantly a good psychologist doesn’t confront his client with information about the client that’s not helpful for them. He doesn’t say: “Your life is a mess because of points 1 to 30.” That’s certainly information that’s interesting to the client but not helpful. It makes much more sense to let the client figure out stuff on his own or to guide him to specific issues that the client is actually in a position to change.
Monday I gave someone meaningful true information about them that I consider helpful to them their first reaction was: “I don’t want to have nightmares. Don’t give them to me.”
I do have a policy of being honest but that doesn’t entail telling someone true information for which they didn’t ask and that messes them up. I don’t think that any good psychologist will just share all information that are available. It just a bad strategy when you are having a discussion about intimate personal topics.
Well, some people don’t want to be given information, and some people do. It’s often difficult to know where a specific person belongs; and it is a reasonable assumption that they most likely belong to the “don’t want to know” group.
The problem with saying “some information should not be known” is that it does not specify who shouldn’t know (and why).
Well, some people don’t want to be given information, and some people do.
Whether a person want to be given information doesn’t mean that he can handle the information. I can remember a few instance where I swear that I wanted information but wasn’t well equipped to handle them.
The problem with saying “some information should not be known” is that it does not specify who shouldn’t know (and why).
That sentence alone doesn’t but the psychologist probably had a context in which he spoke it.
Gah. Now I think I shouldn’t have included the background for my question.
FYI, what I wrote in response to some other comment:
it was more of a general statement. AFAIR we were talking about me thinking too much about why other people do what they do (hence—I have too much information about them) and too little about how that affects me. Anyway—my own wording made me wonder more about what I said than what was the topic.
Spritzing got me quite excited! The concept isn’t new, but the variable speed (pauses after punctuation marks) and quality visual cues really work for me, in the demo at least. Don’t let your inner voice slow you down!
Disclaimer: No relevant disclosures about spritzing (the reading method, at least).
Interesting. I noticed that in the first two, my subvocalization became disjointed, sounding as if each word was recorded separately like it would be in a simplistic text-to-speech program. In the 500 wpm one, this was less of a problem, and I’m not sure I was even entirely subvocalizing it. It ended up being easier and more comfortable to read than the slower speeds.
I like this idea, but am seriously concerned about its effect on eye health. Weak eye muscles are not a thing you want to have, even if you live in the safest place in the world.
I’ve noticed I don’t read ‘Main’ posts anymore.
When I come to LW, I click to the Discussion almost instinctively. I’d estimate it has been four weeks since I’ve looked at Main. I sometimes read new Slate Star Codex posts (super good stuff, if you are unfamiliar) from LW’s sidebar. I sometimes notice interesting-sounding ‘Recent Comments’ and click on them.
My initial thought is that I don’t feel compelled to read Main posts because they are the LW-approved ideas, and I’m not super interested in listening to a bunch of people agreeing with another. Maybe that is a caricature, not sure.
Anyone else Discussion-centric in their LW use?
Also, the Meetup stuff is annoying noise. I’m very sympathetic if placing it among posts helps to drive attendance. By all means, continue if it helps your causes. But it feels spammy to me.
Alternative hypothesis: you have been conditioned to click on discussion because it has a better reward schedule.
Yes, likely. If you mean the discussion is more varied and interesting.
raises hand *
Partially because it’s much more active over here.
It seems to me that is likely the result of of many people feeling like me rather the the cause of them feeling that way.
Activity seems like a positive feedback loop*- because there are more comments in discussion, people spend more time and comment more in discussion, and their comments in discussion are more likely to get responded to, which brings them back to discussion, and so on.
*That is, something that is both a cause and a result.
Sure.
But why did I evolve to stop going to Main and go exclusively to Discussion? That behavior might be reinforced by the lack of activity, but the leading cause (for me in my best estimation) was I came to see the content as overwhelmingly LW-approved stuff.
When I read blacktrance’s comment, I see specific topics- AI, math, health, productivity- that they’re not interested in, that Main focuses on. When I read your comments, it sounds like you’re not as sensitive to topics as to styles of discussion, where you’re more interested in disagreements than in agreements. Am I reading that difference correctly?
Sure, I suppose. I generally use forum sites for discussion. I’m not too terribly interested in reading LW “publications”, I’m more interested in engagin in discussion and reading commentary in regard to issues pertaining to rationality, etc.
The distinction between Main and Discussion articles has noever made much sense to me. It seems to me to be some blend of perceived quality, relation to rationality (as LW defines it) and other LW topics of interest, group politics, EY mandate, etc. Don’t really care all that much...just that it was interesting that I ended up in Discussion almost exclusively.
I’d agree the topics in main seem to be less interesting to me, too, now that I think about it.
I’m more likely to find discussion topics and comments in my areas of interest, while Main seems to be mostly about AI, math, health, and productivity, none of which are particularly interesting for me.
I mainly skim http://lesswrong.com/topcomments/?t=day and http://lesswrong.com/r/discussion/topcomments/?t=day, then when I see something interesting I look at where it comes from.
I generally find Main posts uninteresting, or overlong and based on some incorrect premise or other.
If one is able to improve how people are matched, it would bring about a huge amount of utility for the entire world.
People would be happier, they would be more productive, there would be less of the divorce-related waste. Being in a happy couple also means you are less distracted by conflict in the house, which leads to people better able to develop themselves and achieve their personal goals. You can keep adding to the direct benefits of being in a good pairing versus a bad pairing.
But it doesn’t stop there. If we accept that better matched parents raise their children better, then you are looking at a huge improvement in the psychological health of the next generation of humans. And well-raised humans are more likely to match better with each other...
Under this light, it strikes me as vastly suboptimal that people today will get married to the best option available in their immediate environment when they reach the right age.
The cutting-edge online dating sites base their suggestions on a very limited list of questions. But each of us outputs huge amounts of data, many of them available through APIs on the web. Favourite books, movies, sleep patterns, browsing history, work history, health data, and so much more. We should be using that data to form good hypotheses on how to better match people. I’m actually shocked at the underinvestment in this area as a legitimate altruistic cause.
If an altruistic group of numbers-inclined people was to start working together to improve the world in a non-existential risk reducing kind of way, it strikes me that a dating site may be a fantastic thing to try. On the off-chance it actually produces real results, Applied Rationality will also have a great story of how it improved the world. And, you know, it might even make money.
Thoughts? Any better options?
There seem to be perverse incentives in the dating industry. Most obviously: if you successfully create a forever-happy couple, you have lost your customers; but if you make people date many promissingly-looking-yet-disappointing partners, they will keep returning to your site.
Actualy, maybe your customers are completely hypocritical about their goals: maybe “finding a true love” is their official goal, but what they really want is plausible deniability for fucking dozens of attractive strangers while pretending to search for the perfect soulmate. You could create a website which displays the best one or two matches, instead of hundreds of recommendations, and despite having higher success rate for people who try it, most people will probably be unimpressed and give you some bullshit excuses if you ask them.
Also, if people are delusional about their “sexual market value”, you probably won’t make money by trying to fix their delusions. They will be offended by the types of “ordinary” people you offer them as their best matches, when the competing website offers them Prince Charming (whose real goal is to maximize his number of one night stands) or Princess Charming (who really is a prostitute using the website to find potential clients). They will look at the photos and profiles from your website, and from the competing website, and then decide your website isn’t even worth trying. They may also post an offended blog review, and you bet it will be popular on social networks.
So you probably would need to do this as a non-profit philantropic activity.
EDIT: I have an idea about how to remove the perverse incentives, but it requires a lot of trust in users. Make them pay if they have a happy relationship. For example if the website finds you a date, set a regular payment of $5 each month for the next 10 years; if the relationship breaks, cancel the payment. The value of a good relationship is higher than $5 a month, but the total payment of $600 could be enough for the website.
That sounds a lot like really wanting a soulmate and an open relationship.
That’s a nice thing to have; I am not judging anyone. Just thinking how that would influence the dating website algorithm, marketing, and the utility this whole project would create.
If some people say they want X but they actually want Y… however other people say they want X and they mean it… and the algorithm matches them together because the other characteristics match, at the end they may be still unsatisfied (if one of these groups is a small minority, they will be disappointed repeatedly). This could possibly be fixed by an algorithm smart enough that it could somehow detect which option it is, and only match people who want the same thing (whichever of X or Y it is).
If there are many people who say they want X but really want Y, how will you advertise the website? Probably by playing along and describing your website mostly as a site for X, but providing obvious hints that Y is also possible and frequent there. Alternatively, by describing your website as a site for X, but writing “independent” blog articles and comments describing how well it actually works for Y. (What is the chance that this actually is what dating sites are already doing, and the only complaining people are the nerds who don’t understand the real rules?)
Maybe there is a market in explicitly supporting open relationships. (Especially if you start in the Bay Area.) By removing some hypocrisy, the matching could be made more efficient—you could ask questions which you otherwise couldn’t, e.g. “how many % of your time would you prefer to spend with this partner?”.
I wouldn’t jump to malice so fast when incompetence suffices as an explanation. Nobody has actually done the proper research. The current sites have found a local maxima and are happy to extract value there. Google got huge by getting people off the site fast when everyone else was building portals.
You will of course get lots of delusionals, and lots of people damaged enough that they are unmatchable anyway. You can’t help everybody. But also the point is to improve the result they would otherwise have had. Delusional people do end up finding a match in general, so you just have to improve that to have a win. Perhaps you can fix the incentive by getting paid for the duration of the resulting relationship. (and that has issues by itself, but that’s a long conversation)
I don’t think the philanthropic angle will help, though having altruistic investors who aren’t looking for immediate maximisation of investment is probably a must, as a lot of this is pure research.
I don’t think he was jumping to malice, rather delusion or bias.
I meant malice/incompetence on the part of the dating sites.
I think that’s the business model of eharmony and they seem to be doing well.
I absolutely agree, but I am not sure that anyone was even considering this as a way to make money.
Unfortunately, for all the same reasons we cannot make money, we cannot get people to sign up for the site in the first place.
Two proposed solutions for this:
1) Something like I suggested before that matches people without them signing up somehow.
2) A bait and switch, where a site gets popular using the same tactics as other dating sites, and then switches to something better for them.
Neither of these solutions seem plausible to work at all.
I wonder to what extent the problems you describe (divorces, conflict, etc) are caused mainly by poor matching of the people having the problems, and to what extent they are caused by the people having poor relationship (or other) skills, relatively regardless of how well matched they are with their partner? For example, it could be that someone is only a little bit less likely to have dramatic arguments with their “ideal match” than with a random partner—they just happen to be an argumentative person or haven’t figured out better ways of resolving disagreements.
Well, the success of arranged marriages in cultures that practice them suggests the “right match” isn’t that important.
What makes you think these marriages are successful? Low divorce rates are not good evidence in places where divorce is often impractical.
Three main points in favor of arranged marriages that I’m aware of:
The marriages are generally arranged by older women, who are likely better at finding a long-term match than young people. (Consider this the equivalent of dating people based on okCupid match rating, say, instead of hotornot rating.)
The expectations people have from marriage are much more open and agreed upon; like Prismattic points out, they may have a marriage that a Westerner would want to get a divorce in, but be satisfied. It seems to me that this is because of increased realism in expectations (i.e. the Westerner thinks the divorce will be more helpful than it actually will, or is overrating divorce compared to other options), but this is hard to be quantitative about.
To elaborate on the expectations, in arranged marriages it is clear that a healthy relationship is something you have to build and actively maintain, whereas in love marriages sometimes people have the impression that the healthy relationship appears and sustains itself by magic- and so when they put no work into maintaining it, and it falls apart, they claim that the magic is gone rather than that they never changed the oil.
I also think most modern arranged marriages involve some choice on the part of the participants- “meet these four people, tell us if you can’t stand any of them” instead of “you will marry this one person.”
I remember seeing studies that attempted to measure happiness.
Links? I am also quite suspicious of measuring happiness—by one measure Bhutan is the happiest country in the world and, um, I have my doubts.
Source.
Source.
Source.
A contrary finding:
Source.
Why are you even asking for links to studies if you admit you don’t care what studies say?
I have a prior that the studies are suspect. But that prior can be updated by evidence.
I’m not sure this is correct. That is to say, the empirical point that divorce is much less common in arranged marriage cultures is obviously true. But
a) I think there is some correlation between prevalence arranged marriage and stigma associated with divorce, meaning that not getting divorced does not necessarily equal happy marriage.
b) The bar for success in 20th-21st century western marriages is set really high. It’s not just an economic arrangement; people want a best friend and a passionate lover and maybe several other things rolled into one. When people in traditional cultures say that their marriages are “happy,” they may well mean something much less than what affluent westerners would consider satisfactory.
Why does it suggest that rather than that the arrangers are better at finding the “right match” than the persons to be married?
My instinct on this is driven by having been in bad and good relationships, and reflecting on myself in those situations. It ain’t much, but it’s what I’ve got to work with. Yes, some people are unmatchable, or shouldn’t be matched. But somewhere between “is in high demand and has good judgement, can easily find great matches” and “is unmatchable and should be kept away from others”, there’s a lot of people that can be matched better. Or that’s the hypothesis.
Seems reasonable, although I’d still wonder just how much difference improving the match would make even for the majority of middle-ground people. It sounded in the grandparent post (first and fourth paragraphs particularly) that you were treating the notion that it would be “a lot” as a premise rather than a hypothesis.
Well, it’s more than a hypothesis, it’s a goal. If it doesn’t work, then it doesn’t, but if it does, it’s pretty high impact. (though not existential-risk avoidance high, in and of itself).
Finding a good match has made a big subjective difference for me, and there’s a case it’s made a big objective difference (but then again, I’d say that) and I had to move countries to find that person.
Yeah, maybe the original phrasing is too strong (blame the entrepreneur in pitch mode) but the 6th paragraph does say that it’s an off-chance it can be made to work, though both a high improvement potential and a high difficulty in materialising it are not mutually exclusive.
The problem with dating sites (like social network sites or internet messengers) is that the utility you can gain from it is VERY related to how many other people are actually using it. This means that there is a natural drift towards a monopoly. Nobody wants to join a dating site that only has 1000 people. If you do not have a really good reason to think that your dating site idea will get off the ground, it probably wont.
One way you could possibly get past this is to match people up who do not sign up or even know about this service.
For example, you could create bots that browse okcupid, for answers to questions, ignore okcupid’s stupid algorithms in favor of our own much better ones, and then send two people a message that describes how our service works and introduces them to each other.
Is this legal? If so, I wonder if okcupid would take stop it anyway.
The chicken/egg issue is real with any dating site, yet dating sites do manage to start. Usually you work around this by focusing on a certain group/location, dominating that, and spreading out.
Off the cuff, the bay strikes me as a potentially great area to start for something like this.
It’s spam and very likely violates the TOS.
Awesome—that will fit right in between “I’m a Nigerian customs official with a suitcase of cash” emails and “Enlarge your manhood with our all-natural pills” ones.
P.S. Actually it’s even better! Imagine that you’re a girl and you receive an email which basically says “We stalked you for a while and we think you should go shack up with that guy”. Genius!
How can there be a monopoly if people can use more than one dating site?
Unless OkCupid bans you from putting your profile up on other sites, you can just as easily put a profile on another site with less people, if the site seems promising.
It’s still more work to put a profiles on multiple sites.
Hi Eugine,
I don’t mean to be nitpicking, but a monopoly is a very specific thing. It’s quite different than it just being inconvenient to switch to a competitor. In very many cases in normal market competition, it’s inconvenient to switch to competitors (buying a new car or house, changing your insurance, and so on), but that doesn’t effect the quality of the product. Similarly, for a monopoly to effect the quality of OKCupid’s service, it would have to be a very specific situation, and different than what currently exists, which seems to be quite normal market functioning.
Coscott was talking about a “a natural drift towards a monopoly”.
Unless OKCupid is hiring the government or people with guns to threaten other websites out of existence, there won’t be a drift towards a monopoly.
A monopoly isn’t created by one company getting the overwhelming majority of customers. A monopoly is only created when competitors cannot enter the market. It’s a subtle distinction but it’s very important, because what’s implied is that the company with the monopoly can jack up their prices and abuse customers. They can’t do this without feeding a garden of small competitors that can and will outgrow them (see Myspace, America Online, etc), unless those competitors are disallowed from ever existing.
You can keep downvoting this, but it’s a very important concept in economics and it will still be true.
Forbidding anyone who hasn’t read “The Logical Fallacy of Generalization from Fictional Evidence” from watching any Hollywood or Disney movies about romance would go a long way. ;-)
So how would it be different from OK Cupid, for example?
As an aside, wasn’t the original motivation for Facebook Zuckerberg’s desire to meet girls..? :-D
Here is one improvement to OKcupid, which we might even be able to implement as a third party:
OKcupid has bad match algorithms, but it can still be useful as searchable classified adds. However, when you find a legitimate match, you need to have a way to signal to the other person that you believe the match could work.
Most messages on OKcupid are from men to women, so women already have a way to do this: send a message, however men do not.
Men spam messages, by glancing over profiles, and sending cookie cutter messages that mention something in the profile. Women are used to this spam, and may reject legitimate interest, because they do not have a good enough spam filter.
Our service would be to provide an I am not spamming commitment. A flag that can be put in a message which signals “This is the only flagged message I have sent this week”
It would be a link, you put in your message, which sends you to a site that basically says. Yes, Bob(profile link) has only sent this flag to Alice(profile link) in the week of 2/20/14-2/26/14, with an explanation of how this works.
Do you think that would be a useful service to implement? Do you think people would actually use it, and receive it well?
Scarce signals do increase willingness to go on dates, based on a field experiment of online dating in South Korea.
I wonder if a per-message fee for a certain kind of message would be a good business model for this. My suspicion is that it would work very well if all your users had that reluctance to ever spend anything online (people are much more willing to buy utilions that involve getting a physical product than to pay for things like apps)), but it breaks down as soon as someone with some unused disposable income realizes that spamming $1 notes isn’t that expensive.
Only being able to send a certain number of messages per week of a special type might be enough for indicating non-spam, as long as you could solve the problem of people making multiple profiles to get around it. Having a small fee attached to the service might help with tracking that down, since it would keep people from abusing it too extremely, and cover the cost of having someone investigate suspicious accounts (if more than one is paid for by the same credit card at around the same time, for example).
OKcupid solves the multiple account problem for us. It is probably better to not send a virtual rose than to make an account that you then have to answer all the questions to.
Where will your credibility come from?
Alice receives a message from Bob. It says “You’re amazing, we’re nothing but mammals, let’s do it like they do on the Discovery Channel”, and it also says “I, Mallory, hereby certify that Bob only talked about mammals once this week—to you”.
Why should Alice believe you?
Things like that are technically possible (e.g. cryptographic proofs-of-work) but Alice is unlikely to verify your proofs herself and why should she trust Mallory, anyway?
I think if we had a nice professional website, with a link to a long description of how it all works that people won’t read anyway, they will tend to trust us.
Especially if we use math.
Seconded—once you get as far as people trusting you enough to post their personal information and possibly pay you for the service, they’re not still suspecting you of letting people spam you with “certified” non-spam.
OK Cupid has a horrible match percent algorithm. Basically someone who has a check list of things that their match cannot be will answer lots of questions as “this matters a lot to me” and “any of these options are acceptable except for this one extreme one that nobody will click anyway.” The stupid algorithm will inflate this person’s match percent with everyone.
So, if you look at people with high compatibility with you, that says more about their question answering style, than how much you have in common.
This is why the algorithm is horrible in theory. In practice my one example is that I am getting married in a month to someone I met on OKcupid with 99% compatibility.
A good website design could change the answering style. Imagine a site where you don’t fill out all the answers at once. Instead it just displays one question at a time, and you can either answer it or click “not now”. The algorithm would prioritize the questions it asks you dynamically, using the already existing data about you and your potential matches—it would ask you the question which it expects to provide most bits of information.
Also, it would use the math properly. The compatiblity would not be calculated as number of questions answered, but number of bits these answers provide. A match for “likes cats” provides more bits than “is not a serial killer”.
Very consistently people that I know and like, when I see them on okcupid, have a high match percentage. When I meet okcupid people with a good match percentage, I usually like them. This seems to imply the algorithm is a lot better than your theoretical worst example of it. I think your situation is much more of a problem if you don’t answer enough questions.
Perhaps the way people tend to answer questions does not change very much from person to person, so this problem does not show up in practice.
However, if you are willing to change your style for answering questions, it is probably possible to game OKcupid in such a way that you get 90+% with anyone you would care about.
Selfdefeating The entire point of OKcupid is to find someone you will actually click with. Inflating your own match percentages artificially just makes OKCupid worse for you. Of course, this doesnt help if the site just isnt very popular in your city.
Eh. Radical: Have the government do this. Literally, run a dating site, have sex-ed classes teach people how to use it, and why gaming it is bloody stupid. That should result in maximum uptake, and would cost a heck of a lot less than a lot of other initiatives governments already run trying to promote stable pairbonds. Now, how to get this into a political platform…
Not if you have an honest account too so you can check compatibility while still broadcasting higher compatibility than you actually have.
Still pointless! There is no upside to having a bunch of people you are not actually compatible with think the mirage you constructed is a good match. If they are not a match with your honest profile, you do not want to waste theirs or your own time. If your actual goal is to have a bunch of one night stands, then make a profile that out and out states that so that you will be matched with people of like mind. Dishonesty in this matter is both unetical and nigh certain to result in unpleasant drama. Proper use of this kind of tool is an exercise in luminosity—the more accurately you identify what you are truely looking for, the better it works.
Also, see radical proposal: If a site of this type is run by the government, sockpuppets are obviously not going to be an option—one account per social security number or local equivalent, because that is a really simple way to shut down a whole host of abuses.
I’ve had ideas sort of like this at the back of my mind since seeing Paul Graham pointing out how broken online dating is in one of his essays. (Not so much analyzing all of someone’s existing data, but analyzing IM transcripts to match people with IM buddies they’d be likely to make good friends with is a thing I considered doing.) Haven’t gotten too far with any of them yet, but I’m glad you reminded me, since I was planning on playing with some of my own data soon just to see what I find.
Do you think that not having dated much would be much of a comparative disadvantage in working on this problem? That’s one of the reasons I hesitate to make it my main project.
A possibly-related problem—why does every site I see that says it is for matching strangers who might like to be friends get full of people looking for a date? (Small sample size, but I’ve never seen one that didn’t give me the sense that the vast majority of the members were looking for romance or a one night stand or something.)
So that people can look for dates without breaking plausible deniability.
I think it’s the web site, rather than its clients, that needs the plausible deniability. It cannot seem to be in the business of selling sex, so it has to have a wider focus.
Why altruistic? If it’s worth anything, it’s worth money. If it won’t even pay its creators for the time they’ll put in to create it, where’s the value?
I am not convinced it is the optimal route to startup success. If it was, I would be doing it in preference over my current startup. It is highly uncertain and requires what looks like basic research, hence the altruism angle. If it succeeds, yes, it shouldake a lot of money and nobody should deprive it’s creators of the fruits of their labour.
It strikes me that it is much more plausible to argue that the dating market suffers from market failure through information asymmetry, market power and high search costs than to argue the same about economic activity. Yet although people search high and low to find (often non-existent) market failures to justify economic interventions, interventions in the dating market are greeted with near-uniform hostlility. I predict that, outside of LessWrong, your proposal will generate a high “Ick” factor as a taboo violation. “Rationality-based online dating will set you up with scientifically-chosen dates...” this is likely to be an anti-selling point to most users.
Obviously you’d take a different angle with the marketing.
Off the cuff, I’d pitch it as a hands-off dating site. You just install a persistent app on your phone that pushes a notification when it finds a good match. No website to navigate, no profile to fill, no message queue to manage.
Perhaps market it to busy professionals. Finance professionals may be a good target to start marketing to. (busy, high-status, analytical)
There would need to be some way to deal with the privacy issues though.
This might be a reason to start it out as a nice thing. Though, the problem is finding a niche that likes this proposal and has a decent gender ratio (or enough people interested in dates of the same gender).
Now that I think about it, existing dating sites do try to advertise themselves as being better because of their algorithm. If that advertising works, maybe the ick factor isn’t that strong?
Have you seen this TED talk?
fantastic, thanks!
Viliam_Bur sort of said this, but it doesn’t seem possible to outcompete the existing websites due to perverse incentives.
If I build a site optimizing for long term success, and another dating site optimizes for an intense honeymoon phase (which encourages people to come back and spread the word about the site) then I will lose. And optimizing for long term success is really hard since feedback occurs on the order of decades.
Of course I’m assuming that intense short term happiness and long term stability aren’t very highly correlated and I could be wrong. I’m also assuming that stability is desirable—I’d be curious if anyone disagrees.
Companies are trying, unfortunately the incentives seem sort of messed up to me. Dating websites have an incentive to encourage people to use their service, not get into wonderful long term relationships. Hence I would expect them to optimize for relationships with an intense honeymoon phase, rather than relationships with a high chance of long term success and compatibility.
Since we’re after long term success, feedback will occur on the order of decades—making this a very hard optimization problem.
How do you pick a career if your goal is to maximize your income (technically, maximize the expected value of some function of your income)? The sort of standard answer is “comparative advantage”, but it’s unclear to me how to apply that concept in practice. For example how much demand there is for each kind of job is obviously very important, but how do you take that into consideration, exactly? I’ve been thinking about this and came up with the following. I’d be interested in any improvements or alternative ideas.
For each career under consideration, estimate your potential income ranking or percentile within that career if you went into it (as a probability distribution).
For each career, estimate its income distribution (how much will the top earner make, how much will the second highest earner make, etc.).
From 1 and 2, obtain a probability distribution of your income within each career.
Pick the career with maximum expected utility.
If you have a high IQ and are good at math go into finance. If you have a high IQ, strong social skills but are bad at math go into law. If you have a high IQ, a good memory but weak social and math skills become a medical doctor. If you have a low IQ but are attractive marry someone rich. If you have a very low IQ get on government benefits for some disability and work at an under-the-table job.
This seems awfully US centric.
Anyway, these advices aim at “higher middle class”, not “rich bastard” category. Maybe apart from “marry someone rich”.
Well, Western-developed-world-centric, true.
In dynamic economies (e.g. China) you probably would want to start a business. In stagnant and poor places your first priority should be to get out.
Going into finance or law can propel you into the “rich bastard” category.
Medical doctors are paid well in many places other than the US, though not as well as in the US. (For that matter, most other well-paid jobs are better paid in the US than anywhere else. Software development, law, senior management, etc.)
Also, though of course this was no part of the original question, medicine offers more confidence than most careers that your work is actually making the world a better place. (Which may not actually be the right question to ask, of course—what matters is arguably the marginal effect, and if you’re well paid and care enough about people in poor countries you may well be able to do more good by charitable donations than you ever could directly by your work. But it’s a thing many people care about.)
More importantly, it seems that being a medical doctor can pay very large dividends both in donable dollars and in warm-fuzzies.
I think that’s intended. Trying to achieve greater wealth generally involves much higher risk, and even if it offers a higher expected value in terms of money, the diminishing utility of wealth probably makes the expected utility of, say, creating a startup, lower than just pursuing a middle-class career that matches your skills.
Well, Wei Dai said “maximize the expected value of some function of your income”; which career achieves that will depend on whether the function is log(x), x, H(x - $40,000/year), exp(x/($1M/year)), or what.
I assumed it was referring to (part of) Wei Dai’s utility function. What other functions could there be a point in applying?
Yes, but we don’t know what Wei Dai’s utility function is, and the answer to his question may depend on that.
But are physically OK, play sports and/or enlist (US-centric).
The vast majority of people who play sports have fun and don’t receive a dime for it. A majority of people who get something of monetary value out of playing sports get a college degree and nothing else.
I agree with the US army part though.
I think the US army is very physically dangerous, and furthermore might be considered a negative to world-welfare, depending on your politics.
I don’t have good numbers, but it’s likely less dangerous than you think it is. The vast majority of what an infantryman does falls into two categories—training, and waiting. And that’s a boots on ground, rifle in hand category—there’s a bunch of rear-echelon ratings as well.
I’m guessing that it’s likely within an order of magnitude of danger as commuting to work. Likely safer than delivering pizzas. There’s probably a lot of variance between specific job descriptions—a drone operator based in the continental US is going to have a lot less occupational risk than the guy doing explosive ordnance disposal.
How many people I’d be calmly killing every day? I’d have massive PTSD if I were a drone operator.
From what I’ve read, a couple of the issues for drone pilots is that they’ve been killing people who they’ve been watching for a while, and that they feel personal responsibility if they fail to protect American soldiers.
By a strange coincidence (unless you saw it and thus had it on your mind) today’s SMBC is about exactly this.
Well, I don’t have statistics about that, but accounts from WWII bomber crews suggest otherwise.
Maybe they were just really good at screening out applicants who would have been likely to get PTSD.
AFAIK, people only started understanding PTSD after Vietnam and it wasn’t even called that until the 1980s, so possibly not.
Up until the US gets involved in something resembling a symmetrical war. Of course in that case it’s possible no job will be safe.
In the year 1940, working as an enlisted member of the army supply chain was probably safer than not being in the army whatsoever—regular Joes got drafted.
Besides which, the geographical situation of the US means that a symmetrical war is largely going to be an air/sea sort of deal. Canada’s effectively part of the US in economic and mutual-defense terms, and Mexico isn’t much help either. Mexico doesn’t have the geographical and industrial resources to go toe-to-toe with the US on their own, the border is a bunch of hostile desert, and getting supplies into Mexico past the US navy and air force is problematic.
Yes, and in particular it’ll involve enemy drones. Drone operators are likely to be specifically targeted.
That makes them safer, ironically. If your command knows that you’re likely to be targeted and your contributions are important to the war effort, they’ll take efforts to protect you. Stuff you down a really deep hole and pipe in data and logistical support. They probably won’t let you leave, either, which means you can’t get unlucky and eat a drone strike while you’re enjoying a day in the park.
You’re at elevated risk of being caught in nuclear or orbital kinetic bombardment, though… but if the war gets to that stage your goose is cooked regardless of what job you have.
Another bonus of enlisting: basic skills will be drilled into so thoroughly they will be fully into your System I allowing you extra executive function (thereby causing you to punch above your weight in terms of intelligence). Although, there is some ethical risk involved.
Evidence?
Does anyone know if finance requires strong math and social skills? I assumed it did—social skills for creating connections, and math skills for actually doing to job.
And if you do have poor social skills, then practice! Social skills are really important. I’m still working on this.
This is some guesswork, but some other possible combinations:
Strong social skills, above average IQ—management?
Above average IQ, good math skills—accounting?
Rich parents, family business—take over said business eventually.
Middle class parents, fair amount of property, good location—rent.
Rich parents, strong social skills—network through their connections.
Is this still true? Recently there have been reports about an oversupply of lawyers and scandals involving law schools fudging the statistics on the salaries of their graduates.
Salaries might be falling, but I doubt this is long term.
US law is a spectacularly bad choice at the moment. There is far to many law schools, and as a consequence, too many law graduates, the degree costs a fortune and employment prospects are outright bad. Do not do this.
Finance is an implicit bet that wallstreet will not get struck down by the wrath of the electorate just as you finish your education.
Honestly? If riches really is what you want, go into business for yourself. A startup, or at the low end just being a self-employed contractor has good returns and this is not likely to change. Programming, the trades, a good set of languages and an import-export business..
Well, as I understand it part of the issue is that a lot of the grunt work that used to require lots of lawyers to do, e.g., looking through piles of documents for relevant sections, can now be automated.
According to 80000 Hours, law is still one of the highest-earning careers.
Is finance higher E(money) than, say, a startup?
I would guess yes given the high startup failure rate.
There’s a high failure rate in finance, too—it’s just hidden in the “up or out” culture. It’s a very winner-takes-all kind of place, from what I’ve heard.
Finance is diverse.
If you want to be a portfolio manager who makes, say, macro bets, yes, it’s very much up or out. But if you want to be a quant polishing fixed income risk management models in some bank, it’s a pretty standard corporate job.
Startups are shockingly diverse too. And despite the super-high failure rates I hear about, the group of friends I’ve been tracking the past 5 years or so seem to be doing pretty darn well, despite some of them having failures indeed.
I strongly suspect the degree of failure in startups correlates inversely with rationality skills (as it should) so rationalists should not be placing themselves on the same reference category as everyone else. Execution skills matter a lot too, but doing a startup has worked miracles for my motivation too.
Not from the expected-income point of view (we’re not considering car dealerships and franchise eateries startups, right?).
Oh, dear. “I’m so smart that normal rules don’t apply to me”. What could possibly go wrong..?
This isn’t “I’m smart and rules don’t apply”. Smartness alone doesn’t help.
But, to put it this way, if rationality training doesn’t help improve your startup’s odds of success, then there’s something wrong with the rationality training.
To be more precise, in my experience, a lot of startup failure is due to downright stupidity, or just ignoring the obvious.
Also, anecdotally, running a startup has been the absolute best on-the-job rationality training I’ve ever had.
Shockingly, successful entrepreneurs I’ve worked with score high on my rationality test, which consists of how often they say things that are uncontested red flags, and how well-reasoned their suggested courses of action are. In particular, one of our investors is the closest approximation to a bayesian superintelligence I’ve ever met. I can feed him data & news from the past week, and almost hear the weighting of various outcomes shift in his predictions and recommendations.
In short,
Rationalists are more likely to think better, avoid obvious errors.
Thinking better improves chances of startup success
Rationalists have better chances of startup success.
I do understand this sounds self-serving, but I also try to avoid the sin of underconfidence. In my experience, quality of thinking between rationalists and the average person tends to be similar to quality of conversation here versus on YouTube. The problem is when rationalists bite off more than they can chew in terms of goals, but that’s a separate problem.
What you say sounds intuitive to me at first, but as of now I would say that rationality training may boost start up success rates up just a little.
Here is some reasons why rationality might not matter that much:
People tend to be a bit more rational when it counts, like making money. So having correct beliefs about many things doesn’t really give you an edge because the other guy is also pretty rational for business stuff.
self-delusion, psychopathy, irrationality, corruption, arrogance, and raw driven determination, have good if not better anecdotal evidence of boosting success than rationality training I think.
Well, at this point we’re weighing anecdotes, but..
Yes! They do tend to push their rationality to the limit. Hypothesis: knowing more about rationality can help push up the limit of how rational one can be.
Yes! It’s not about rationality alone. Persistent determination is quite possibly more important than rationality and intelligence put together. But I posit that rationality is a multiplier, and also tends to filter out the most destructive outcomes.
In general, I’d love to see some data on this, but I’m not holding my breath.
Agreed. Interestingly, the latest post in main points to evidence supporting rationality having a significant relation to success in the work place – not the same as entrepreneurship, nonetheless I update slightly more in favor of your position.
I agree that a more rational person has a greater chance, ceteris paribus. Question is, how much greater.
A part of the outcome is luck; I don’t know how big part. Also, the rationality training may improve your skills, but just to some degree.
(Data point: myself. I believe I am acting more rationally after CFAR minicamp than before, and it seems to be reflected by better outcomes in life, but there is still a lot of stupid things I do. So maybe my probability of running a successful startup has increased from 1% to 3%.)
I question the stats that says 1% success rate for startups. I will need to see the reference, but one I had access to basically said “1% matches or exceeds projections shown to investors” or some such. Funnily enough, by that metric Facebook is a failure (they missed the goal they set in the convertible note signed with Peter Thiel). If run decently, I would expect double digit success rates, for a more reasonable measure of success. If a driven, creative rationalist is running a company, I would expect a very high degree of success.
Another thing much more common in rationalists than the common population is the ability to actively solicit feedback, reflect and self-modify. This is surprisingly rare. And incredibly vital in a startup.
Success at startups is not about not doing stupid things. I’ve made many MANY mistakes. It’s about not doing things stupid enough to kill your company. Surprisingly, the business world has a lot of tolerance for error, as long as you avoid the truly bad ones.
It is hard to survey startups. What is usually done is to measure success rates of companies that raised a Series A round of funding. Many companies fail before achieving that, though they necessarily fail faster, producing less opportunity cost.
Here is a chart of returns to a VC, taken from this paper by a different author. 60% of dollars invested are in companies that lost the VCs money (lost them 85%). This is a top VC that managed to triple its money, so this is an overestimate of success of a regular VC-backed company. This is a common bias in these surveys.
Based on the fictitious figure 2, 63% of dollars is actually 69% of companies, because successful companies get more funding. So 31% of companies with a Series A round at a top firm succeed by the metric of a positive return to the VCs. Double digit success would require that at least 1⁄3 of startups get a Series A funding and that companies funded by typical VCs are as successful as companies funded by a top VC.
The appropriate definition of success is comparing to opportunity cost. In particular, the above analysis fails to take into account duration. Here is a paper that makes a reasonable comparison and concludes that running a company with a Series A round was a good decision for people with $700k in assets. Again, skipping to the Series A round is not a real action, thus overestimating the value of the real action of a startup. There is an additional difficulty that startups may have non-monetary costs and benefits, such as stress and learning. Edit: found the paper. According to Figure 2, that 75% of VC-backed firms exit at 0, not much worse than at the top VC considered above.
Well Paul Graham has built quite a successful incubator apparently largely based on his ability to predict success of start-ups based on a half-hour interview.
Besides what gwern said, Paul Graham is a successful VC. The expected income of VCs is very different from the expected income of startup founders.
My point is that this is evidence that start-up success depends on ability more than luck.
I think both ability and luck are necessary but not sufficient (well, reasonable amounts of luck :-D).
I’m not sure how much the interviews add compared to the Y Combinator model of investing in a lot of startups very early on at unusually favorable terms, integrating with Hacker News, and building a YC community with alumni & new angels. (As far as the latter goes, you can ask AngryParsley why he went into YC for Floobits: it wasn’t because he needed their cash.)
What kind of social skills does that require? My impression is that this is the modern equivalent of court astrologer and requires some similar skills, i.e., cold reading.
Not much—the usual ones for holding a corporate job (wear business casual, look neat, don’t smell, don’t be a weirdo). Quants are expected to be nerdy/geeky.
Not at all. Finance has the advantage of providing rapid and unambiguous feedback for your actions.
If you’re trading yes, although the feedback is extremely noisy. If you’re designing models not so much. Incidentally a lot of the quants I know are also good at doing Tarot readings, whether they believe the cards have power or not.
That very much depends on what kind of strategy you’re trading. For example, HFT doesn’t have problems with noise.
Yes, so much. Your model has to work well on historical data and if it makes it to production, it will have performance metrics that it will have to meet.
The other thing to keep in mind about failure rates is where you end up if you fail—what other careers you can go into with the same education. (In the case of startups, you can keep trying more startups, and you’re more likely to succeed on the second or third than you were on the first. I don’t know how it is in finance.)
I think a higher startup failure rate implies E(startup) > E(finance) since most people want risk-adjusted return
Not necessarily because of different barriers to entry.
I’m not sure I would count that as “your income”, though in jurisdictions with easy divorces and large alimony it might be as good for all practical purposes.
In this context, what constitutes a “high IQ”?
Depends on how high you are aiming for. For a good investment banking position you need a high enough IQ to either get into a top 10 school or be in the top 10% of a school such as Smith College.
It’s obvious how you get into law or medicine, but how does going into finance work?
For students at Smith College the normal path is you get very high grades and take some math-heavy courses, get a summer internship with an investment bank after your junior year of college which results in a full time job offer, then after 2-5 years you get an MBA and then get a more senior position at an investment bank.
Oh. Any way for people who’ve already graduated from college to get in, or is it too late at that point?
An MBA or masters degree in finance would probably help. I don’t have much knowledge of more direct paths.
“Career” is an unnatural bucket. You don’t pick a career. You choose between concrete actions that lead to other actions. Imagine picking a path through a tree. This model can encompass the notion of a career as a set of similar paths. Your procedure is a good way to estimate the value of these paths, but doesn’t reflect the tree-like structure of actual decisions. In other words, options are important under uncertainty, and the model you’ve listed doesn’t seem to reflect this.
For example, I’m not choosing between (General Infantry) and (Mathematician). I’m choosing between (Enlist in the Military) and (Go to College). Even if the terminal state (General Infantry) had the same expected value as (Mathematician), going to college should more valuable because you will have many options besides (Mathematician) should your initial estimate prove wrong, while enlisting leads to much lower branching factor.
How should you weigh the value of having options? I have no clue.
Your goal is likely not to maximize your income. For one, you have to take cost of living into account—a $60k/yr job where you spend $10k/yr on housing is better than a $80k/yr (EDIT:$70k/yr, math was off) job where you spend $25k/yr on housing.
For another, the time and stress of the career field has a very big impact on quality-of-life. If you work sixty hour weeks, in order to get to the same kind of place as a forty hour week worker you have to spend money to free up twenty hours per week in high-quality time. That’s a lot of money in cleaners, virtual personal assistants, etc.
As far as “how do I use the concept of comparative advantage to my advantage”, here’s how I’d do it:
Make a list of skills and preferences. It need not be exhaustive—in fact, I’d go for the first few things you can think of. The more obvious of a difference from the typical person, the more likely it is to be your comparative advantage. For instance, suppose you like being alone, do not get bored easily by monotonous work, and do not have any particular attachment to any one place.
Look at career options and ask yourself if that is something that fits your skills and preferences. Over-the-road trucking is a lot more attractive to people who can stand boredom and isolation, and don’t feel a need to settle down in one place. Conversely, it’s less attractive to people who are the opposite way, and so is likely to command a higher wage.
Now that you have a shorter list of things you’re likely to face less competition for or be better at, use any sort of evaluation to pick among the narrower field.
You should consider option values, especially early in your career. It’s easier to move from high paying job in Manhattan to a lower paying job in Kansas City than to do the reverse.
Update the choice by replacing income with the total expected value from job income, social networking, and career options available to you, and the point stands.
Probably the cost of housing correlates with other expenses, and also there’s income tax to consider, but on the surface the first job is $50k/yr net, the second job is $55k/yr net, and so it looks like the latter better.
whoops, picked the wrong numbers. Thanks
In addition to maximizing income, maximizing savings/investments is very important. You can be poor off of a $500,000 salary and rich off of a $50,000 salary.
In “The Fall and Rise of Formal Methods”, Peter Amey gives a pretty good description of how I expect things to play out w.r.t. Friendly AI research:
For the curious, Amey also wrote a nice overview of successes and failures in formal methods.
Introduction I suspected that the type of stuff that gets posted in Rationality Quotes reinforces the mistaken way of throwing about the word rational. To test this, I set out to look at the first twenty rationality quotes in the most recent RQ thread. In the end I only looked at the first ten because it was taking more time and energy than would permit me to continue past that. (I’d only seen one of them before, namely the one that prompted me to make this comment.)
A look at the quotes
There might be an intended, implicit lesson here that would systematically improve thinking, but without more concrete examples and elaboration (I’m not sure what the exact mistake being pointed to is), we’re left guessing what it might be. In cases like this where it’s not clear, it’s best to point out explicitly what the general habit of thought (cognitive algorithm) is that should be corrected, and how one should correct it, rather than to point in the vague direction of something highly specific going wrong.
Without context, I’m struggling to understand the meaning of this quote, too. The Paul Graham article it appears in, after a quick skim, does not appear to be teaching a general lesson about how to think; rather it appears to be making a specific observation. I don’t feel like I’ve learned about a bad cognitive habit I had by reading this, or been taught a new useful way to think.
Although this again seems like it’s vague enough that the range of possible interpretations is fairly broad, I feel like this is interpretable into useful advice. It doesn’t make a clear point about habits of thought, though, and I had to consciously try to make up a plausible general lesson for it (just world fallacy), that I probably wouldn’t have been able to think up if I didn’t already know that general lesson.
I understand and like this quote. It feels like this quote is an antidote to a specific type of thought (patronising signalling of reverence for the wisdom of primitive tribes), and maybe more generally serves as an encouragement to revisit some of our cultural relativism/self-flagellation. But probably not very generalisable. (I note with amusement how unconvincing I find the cognitive process that generated this quote.)
There can be value to creating witty mottos for our endeavours (e.g. battling akrasia). But such battles aside, this does not feel like it’s offering much insight into cognitive processes.
If I’m interpreting this correctly, then this can be taken as a quote about the difficulty of locating strong hypotheses. Not particularly epiphanic by Less Wrong standards, but it is clearer than some of the previous examples and does indeed allude to a general protocol.
Pretty good. General lesson: Without causal insight, we should be suspicious when a string of Promising Solutions fails. Applicable to solutions to problems in one’s personal life. Observing an an analogue in tackling mathematical or philosophical problems, this suggests a general attitude to problem-solving of being suspicious of guessing solutions instead of striving for insight.
Good. General lesson: Apply reversal tests to complaints against novel approaches, to combat status quo bias.
Dual of quote before previous. At first I thought I understood this immediately. Then I noticed I was confused and had to remind myself what Taleb’s antifragility concept actually is. I feel like it’s something to do with doing that which works, regardless of whether we have a good understanding of why it works. I could guess at but am not sure of what the ‘explain things you cannot do’ part means.
Trope deconstruction making a nod to likelihood ratios. Could be taken as a general reminder to be alert to likelihood ratios and incentives to lie. Cool.
Conclusion Out of ten quotes, I would identify two as reinforcing general but basic principles of thought (hypothesis location, likelihood ratios), another that is useful and general (skepticism of Promising Solutions), one which is insightful and general (reversal tests for status quo biases), and one that I wasn’t convinced I really grokked but which possibly taught a general lesson (antifragility).
I would call that maybe a score of 2.5 out of 10, in terms of quotes that might actually encourage improvement in general cognitive algorithms. I would therefore suggest something like one of the following:
(1) Be more rigorous in checking that quotes really are rationality quotes before posting them (2) Having two separate threads—one for rationality quotes and one for other quotes (3) Renaming ‘Rationality Quotes’ to ‘Quotes’ and just having the one thread. This might seem trivial but it at least decreases the association of non-rationality quotes to the concept of rationality.
I would also suggest that quote posters provide longer quotes to provide context or write the context themselves, and explain the lesson behind the quotes. Some of the above quotes seemed obvious at first, but I mysteriously found that when I tried to formulate them crisply, I found them hard to pin down.
So I have the typical of introvert/nerd problem of being shy about meeting people one-on-one, because I’m afraid of not being able to come up with anything to say and lots of awkwardness resulting. (Might have something to do with why I’ve typically tended to date talkative people...)
Now I’m pretty sure that there must exist some excellent book or guide or blog post series or whatever that’s aimed at teaching people how to actually be a good conversationalist. I just haven’t found it. Recommendations?
Offline practice: make a habit of writing down good questions you could have asked in a conversation you recently had. Reward yourself for thinking of questions, regardless of how slow you are at generating them. (H/T Dan of Charisma Tips, which has other good tips scattered around that blog).
I saw a speech pathologist for this. I was taught to ask boring questions I’m not really interested in asking on the hopes that they will lead to something interesting happening. “How was your weekend?”, “What are some of your hobbies?”, “How about this weather?”, and all that mess.
In practice, it feels so forced I can’t do it in real life.
Yeah. My problem is more that I can’t think of anything to say even when people do say something interesting.
Like just recently, I met up with one person who wanted to discuss his tech startup thing. Then he held this fascinating presentation about the philosophy and practice of his project, which also touched upon like five other fields that I also have an interest in. And I mostly just said “okay” and nodded, which was fine in the beginning since he was giving me a presentation after all, but then in the end when he asked me if I had any questions or comments, and I didn’t have much to say. There were some questions that occurred to me as he talked about it, and I did ask those when they occurred, but still, feels like I should’ve been able to say a lot more.
Responding to the interesting conversation context.
First, always bring pen a paper to any meeting/presentation that is in anyway formal or professional. Questions always come up at times when it is inappropriate to interrupt, save them for lulls.
Second, an an anecdote. I noticed I had a habit during meetings to focus entirely on absorbing and recording information, and then would process and extrapolate from it after the fact (I blame spending years in the structured undergrad large technical lecture environment). This habit of only listening and not providing feedback was detrimental in the working world, it took a lot of practice to start analyzing the information and extrapolating forward in real time. Once you start extrapolating forward from what you are being told, meaningful feedback will come naturally.
So, I have a comparative advantage at coming up with things to say, and so I’m not sure this advice will fill the specific potholes you’re getting stuck on, but I hope it’s somewhat useful.
A simple technique that seems to work pretty well is read your mind to them, since they can’t read it themselves. If you’re interested in field X, say that you’re interested in it. If you’re glad that they gave you a talk, tell them you’re glad that they gave you a talk. People like getting feedback, and people like getting compliments, and when your mind is blank and there’s nothing asking to be said, that’s a good place to go looking. (Something like “that was very complete; I’ve got no questions” is nicer than just silence, though you may want to tailor it a bit to whatever they’ve just said.)
Thanks, that sounds potentially useful.
Have you actually tried it out much, or do you top before you ‘just try it’? I make myself ask questions like that, but I find it can move the conversation into better places… Although I normally use ones I’m likely to be interested in e,g. “Read any good books recently?”
Here is another logic puzzle. I did not write this one, but I really like it.
Imagine you have a circular cake, that is frosted on the top. You cut a d degree slice out of it, and then put it back, but rotated so that it is upside down. Now, d degrees of the cake have frosting on the bottom, while 360 minus d degrees have frosting on the top. Rotate the cake d degrees, take the next slice, and put it upside down. Now, assuming the d is less than 180, 2d degrees of the cake will have frosting on the bottom.
If d is 60 degrees, then after you repeat this procedure, flipping a single slice and rotating 6 times, all the frosting will be on the bottom. If you repeat the procedure 12 times, all of the frosting will be back on the top of the cake.
For what values of d does the cake eventually get back to having all the frosting on the top?
Solution can be found in the comments here.
V zrgntnzrq gur fbyhgvba. Gur nafjre “vss q vf engvbany” frrzrq gbb rnfl gb zr fb V thrffrq gung gur nafjre jnf npghnyyl “sbe nyy q” orpnhfr gung jbhyq or fhssvpvragyl gevpxl naq nyfb fbzrguvat fbzrguvat ersyrpgvbaf.
Someone was asking a while back for meetup descriptions, what you did/ how it went, etc. Figured I’d post some Columbus Rationality videos here. All but the last are from the mega-meetup.
Jesse Galef on Defense Against the Dark Arts: The Ethics and Psychology of Persuasion
Eric on Applications of Models in Everyday Life (it’s good, but skip about 10-15 minutes when there’s herding-cats-nitpicky audience :P)
Elissa on Effective Altruism
Rita on Cognitive Behavioral Therapy
Don on A Synergy of Eastern and Western Approaches
Gleb on Setting and Achieving Goals
Why are these considered inappropriate for people without a google account?
A question I’m not sure how to phrase to Google, and which has so far made Facebook friends think too hard and go back to doing work at work: what is the maximum output bandwidth of a human, in bits/sec? That is, from your mind to the outside world. Sound, movement, blushing, EKG. As long as it’s deliberate. What’s the most an arbitrarily fast mind running in a human body could achieve?
(gwern pointed me at the Whole Brain Emulation Roadmap; the question of extracting data from an intact brain is covered in Appendix E, but without numbers and mostly with hypothetical technology.)
Why not simply estimate it yourself? These sorts of things aren’t very hard to do. For example, you can estimate typing as follows: peak at 120 WPM; words are average 4 characters; each character (per Shannon and other’s research; see http://www.gwern.net/Notes#efficient-natural-language ) conveys ~1 bit; hence your typing bandwidth is 120 4 1 = <480 bits per minute or <8 bits per second.
Do that for a few modalities like speech, and sum.
I’ve just noticed he said “an arbitrarily fast mind running in a human body”, not an actual human being, so I don’t think it would be much slower at typing uuencoded compressed stuff than natural language (at least with QWERTY—it might be different with keyboards layouts optimized from natural language such as Dvorak, but still probably within a factor of a few).
The 120WPM is pretty good for the physical limits: if you are typing at 120WPM, then you have not hit the limits of your thinking (imagine you are in a typing tutor—your reading speed ought to be at least 3x 120WPM...), and you’re not too far off some of the sustained typing numbers in https://en.wikipedia.org/wiki/Words_per_minute#Alphanumeric_entry
My point was that 1 bit per character is an underestimate.
La Wik says 8 bits per word, FWIW.
La Wiki is apparently not using the entropy estimates extracted from human predictions (who are the best modelers of natural language). Crude stuff like trigram models are going to considerably overestimate matters.
As a baseline estimate for just the muscular system, the worlds faster drummer can play at about 20 beats per second. That’s probably an upper limit on twitch speeds of human muscles, even with a arbitrarily fast mind running in the body. Assuming you had a system on the receiving end that could detect arbitrary muscle contractions, and could control each muscle in your body independently (again, this is an arbitrarily fast mind, so I’d think it should be able to), there are about 650 muscle groups in the body according to wikipedia, so I would say a good estimate for just the muscular system would be 650 x 20bits/s or about 13 Kb/s.
Once you get into things like EKGs, I think it all depends on how much control the mind actually has over processes that are largely subconscious, as well as how sensitive your receiving devices are. That could make the bandwidth much higher, but I don’t know a good way to estimate that.
20 beats per second is for two-handed drumming over one minute, so that’s only 10bits/s/muscle theoretical maximum. There doesn’t seem to be any organized competition for one-handed drumming, but Takahashi Meijin was famous for button mashing at 16 presses per second with only one hand, although for much shorter times.
Don’t you have to define the receiver as well as the transmitter, to have any idea about the channel bandwidth? I mean, if the “outside world” is the Dark Lords of the Matrix, the theoretical maximum output bandwidth is the processing speed of the mind.
Let’s say “detectable as data by 2014 technology”.
Short of having a precise definition of “deliberate” I don’t think it’s possible to give a precise number, but for a Fermi estimate… Dammit! Gwern has already made the calculation I was thinking of!
I noticed recently that one of the mental processes that gets in the way of my proper thinking is an urge to instantly answer a question then spend the rest of my time trying to justify that knee-jerk answer.
For example, I saw a post recently asking whether chess or poker was more popular worldwide. For some reason I wanted to say “obviously x is more popular,” but I realized that I don’t actually know. And if I avoid that urge to answer the question instantly, it’s much easier for me to keep my ego out of issues and to investigate things properly...including making it easier for me recognize things that I don’t know and acknowledge that I don’t know them.
Is there a formal name for this type of bias or behavior pattern? It would let me search up some Sequence posts or articles to read.
Hold off on proposing solutions.
Here is a video of someone interviewing people to see if they can guess a pattern by asking whether or not a sequence of 3 numbers satisfies the pattern. (like was mentioned in HPMOR)
The other videos I’ve sampled from that channel have also been good.
I have also been going through the channel. What I saw so far was mostly science, but there is some rationality stuff.
Example
How do you know when you’ve had a good idea?
I’ve found this to actually be difficult to figure out. Sometimes you can google up what you thought. Sometimes checking to see where the idea has been previously stated requires going through papers that may be very very long, or hidden by pay-walls or other barriers on scientific journal sites.
Sometimes it’s very hard to google things up. To me, I suppose the standard for “that’s a good idea,” is if it more clearly explains something I previously observed, or makes it easier or faster for me to do something. But I have no idea whether or not that means it will be interesting for other people.
How do you like to check your ideas?
If you have to ask...
Just kidding. It’s a great question. Two thoughts: “Nothing is as important as you think it is while you’re thinking about it.” - Daniel Khaneman “If you want to buy something, wait two weeks and see if you still want to buy it.”—my mom
This is a big open topic, but I’ll talk about my top method.
I have a prior that our capitalist, semi-open market is thorough and that if an idea is economically feasible, someone else is doing it / working on it. So when I come up with a new good idea, I assume someone else has already thought of it and begin researching why it hasn’t been done already. Once that research is done, I’ll know not only if it is a good idea or a bad idea but why it is which, and a hint of what it would take to turn it from a bad idea into a good idea. Often these good ideas have been tried / considered before but we may have a local comparative advantage that makes it practical here were it was not elsewhere (legislation, better technology, cheaper labor, costlier labor… )
For example: inland, non-directional, shallow oil, drilling rigs use a very primitive method to survey their well bore. Daydreaming during my undergrad I came up with a alternative method that would provide results orders of magnitudes more accurate. I put together my hypothesis that this was not already in use because: this was a niche market and the components were too costly / poor quality before the smartphone boom. My hypothesis was wrong, a company had a fifteen year old patent on the method and it was being marketed (along with a highly synergistic product line) to offshore drilling rigs. It was a good idea, so good of an idea that it made someone a lot of money 15 years ago and made offshore drilling a lot safer, but it wasn’t a good idea for me.
An experiment with living rationally, by A J Jacobs, who wrote The Year of Living Biblically. I don’t know how long he plans to try living rationally.
Maybe CfAR should invite him to a workshop.
(I suspect that if CfAR should invite him to a workshop they should do it themselves in some official capacity and don’t think random Less Wrongers ought to contact Mr. Jacobs.)
ETA: Ah, rats, the article is from 2008. He’s probably lost interest.
Well, I’m curious about the results. Especially, whether he manages to avoid some “hollywood rationality” memes. He already mentioned Spock...
To illustrate dead-weight loss in my intro micro class I first take out a dollar bill and give it to a student and then explain that the sum of the wealth of the people in the classroom hasn’t changed. Next, I take a second dollar bill and rip it up and throw it in the garbage. My students always laugh nervously as if I’ve done something scandalous like pulling down my pants. Why?
Because you are breaking the law?
Because it signals “I am so wealthy that I can afford to tear up money” and blatantly signaling wealth is crass. And it also signals “I am so callous that I would rather tear up money than give it to the poor”, which is also crass. And the argument that a one dollar bill really isn’t very much money isn’t enough to disrupt the signal.
Because money is heavily charged with symbolism in our society, much like that which lies beneath pants.
Of course, destroying the dollar bill also doesn’t reduce societal wealth.
No, and I briefly make this macro point,
Because destroying money is viscerally aversive and surprising?
A little bit of How An Algorithm Feels From Inside:
Why is the Monty Hall problem so horribly unintuitive? Why does it feel like there’s an equal probability to pick the correct door (1/2+1/2) when actually there’s not (1/3+2/3)?
Here are the relevant bits from the Wikipedia article:
[...]
Those bias listed in the last paragraph maybe explain why people choose not to switch the door, but what explains the “equal probability” intuition? Do you have any insight on this?
Another datapoint is the counterintuitiveness of searching a desk: with each drawer you open looking for something, the probability of finding it in the next drawer increases, but your probability of ever finding it decreases. The difference seems to whipsaw people; see http://www.gwern.net/docs/statistics/1994-falk
A bit late, but I think this part of your article was most relevant to the Monty Hall problem:
People probably don’t distinguish between their personal probability of the target event and the probabilities of the doors. It feels like the probability of there being a car behind the doors is a parameter that belongs to those doors or to the car—however you want to phrase it. Since you’re only given information about what’s behind the doors, and that information can’t actually change the reality of what’s behind the doors then it feels like the probability can’t change just because of that.
I think the monty hall problem very closely resembles a more natural one in which the probability is 1⁄2; namely, that where the host is your opponent and chose whether to offer you the chance to switch. So evolutionarily-optimized instincts tell us the probability is 1⁄2.
I’d say it’s that it closely resembles the one where the host has no idea which door has the car in it, and picks a door at random.
I do not think this is correct. First, the host should only offer you the chance to switch if you are winning, so the chance should be 0. Second, this example seems too contrived to be something that we would have evolved a good instinct about.
Unless they’re trying to trick you. The problem collapses to a yes or no question of whether one of you is able to guess the level the other one of you is thinking on
Um, no, the only Nash equilibria are where you never accept the deal. If you ever accept it at all, then they will only offer it when it hurts you.
I’d probably broaden this beyond 1⁄2 - I think the base case is the host gives you a chance to gamble with a question or test of skill, and the result is purely dependent on the player. The swap-box scenario is then an extreme case of that where the result depends less and less on the skill of the player, eventually reaching 50% chance of winning. I wouldn’t say evolutionary-optimised, but maybe familiarity with the game-show tropes being somewhere along this scale.
Monty Hall is then a twist on this extreme case, which pattern-matches to the more common 50% case with no allowance for the effect of the host’s knowledge.
Does anyone have any advice about understanding implicit communication? I regularly interact with guessers and have difficulty understanding their communication. A fair bit of this has to do with my poor hearing, but I’ve had issues even on text based communication mediums where I understand every word.
My strategy right now is to request explicit confirmation of my suspicions, e.g., here’s a recent online chat I had with a friend (I’m A and they’re B):
A: Hey, how have you been?
B: I’ve been ok
B: working in the lab now
A: Okay. Just to be clear, do you mean that you don’t want to be disturbed?
B: yeah
“[W]orking in the lab now” is ambiguous. This friend does sometimes chat online when working in the lab. But, I suspected that perhaps they didn’t want to chat, so I asked explicitly.
Requesting explicit confirmation seems to annoy most guessers. I’ve heard quite a few times that I should “just know” what they mean. Perhaps they think that they have some sort of accurate mental model of others’ intentions, but I don’t think any of us do. Many guessers have been wrong about my thoughts.
I suspect there probably is no good general strategy other than asking for explicit confirmation. Trying to make guessers be askers is tempting, though probably bound to fail in general.
It’s worth remembering that there is no single Guess/Hint culture. Such high-context cultures depend on everyone sharing a specific set of interpretation rules, allowing information to be conveyed through subtle signals (hints) rather than explicit messages.
For my own part, I absolutely endorse asking for confirmation in any interaction among peers, taking responses to such requests literally, and disengaging if you don’t get a response. If a Guess/Hint-culture native can’t step out of their preferred mode long enough to give you a “yes” or “no,” and you can’t reliably interpret their hints, you’re unlikely to have a worthwhile interaction anyway.
With nonpeers, it gets trickier; disengaging (and asking in the first place) may have consequences you prefer to avoid. In which case I recommend talking to third parties who can navigate that particular Guess/Hint dialect, and getting some guidance from them. This can be as blatant as bringing them along to translate for you (or play Cyrano, online), or can be more like asking them for general pointers. (E.g. “I’m visiting a Chinese family for dinner. Is there anything I ought to know about how to offer compliments, ask for more food, turn down food I don’t want, make specific requests about food? How do I know when I’m supposed to start eating, stop eating, leave? Are there rules I ought to know about who eats first? Etc. etc. etc.”)
Some more Guess/Hint culture suggestions.
Consider:
This will typically communicate that you’ve understood that they’re busy and don’t want to chat, that you’re OK with that, and that you want to talk to them.
That said, there exist Guess/Hint cultures in which it also communicates that you have something urgent to talk about, because if you didn’t you would instead have said:
...which in those cultures will communicate that the ball is in their court. (This depends on an implicit understanding that it is NOT OK to leave messages unresponded to, even if they don’t explicitly request a response, so they are now obligated to contact you next… but since you didn’t explicitly mention it (which would have suggested urgency) they are expected to know that they can do so when it’s convenient for them.
EDIT: All of that being said, my inner Hint-culture native also wants to add that being visible in an online chat forum when I’m not free to chat is rude in the first place.
Thanks for these two posts. I thought more than a thumbs-up (a very subtle hint) was necessary here. I’ve found both posts to be useful in understanding this class of communication styles.
I’m glad they helped. Thanks for letting me know.
Posts that have appeared since you last red a page have a pinkish border on them. It’s really helpful when dealing with things like open threads and quote threads that you read multiple times. Unfortunately, looking at one of the comments makes it think you read all of them. Clicking the “latest open thread” link just shows one of the comments. This means that, if you see something that looks interesting there, you either have to find the latest open thread yourself, or click the link and have it erase everything about what you have and haven’t read.
Can someone make it so looking at one of the comments doesn’t reset all of them, or at least put a link to the open thread, instead of just the comments?
The general problem is real, but here’s a solution to the specific problem of finding the latest open thread: just click the words “latest open thread,” rather than the comment that displays below it.
I see. I had been trying to click the “on Open Thread” part.
Making that a link to the post would be an easy change. In the case of the open thread it is redundant, but perhaps easier to identify as a link. But in the case of the “recent comments” section of the sidebar, it would provide links not currently available.
Does anyone have advice on how to optimize the expectation of a noisy function? The naive approach I’ve used is to sample the function for a given parameter a decent number of times, average those together, and hope the result is close enough to stand in for the true objective function. This seems really wasteful though.
Most of the algorithms I’m coming (like modelling the objective function with gaussian process regression) would be useful, but are more high-powered than I need. Any simple techniques better than the naive approach? Any recommendations among sophisticated approaches?
There are some techniques that can be used with simulated annealing to deal with noise in the evaluation of the objective function. See Section 3 of Branke et al (2008) for a quick overview of proposed methods (they also propose new techniques in that paper). Most of these techniques come with the usual convergence guarantees that are associated with simulated annealing (but there are of course performance penalties in dealing with noise).
What is the dimensionality of your parameter space? What do you know about the noise? (e.g., if you know that the noise is mostly homoscedastic or if you can parameterize it, then you can probably use this to push the performance of some of the simulated annealing algorithms.)
Thanks for the SA paper!
The parameter space is only two dimensional here, so it’s not hard to eyeball roughly where the minimum is if I sample enough. I can say very little about the noise. I’m more interested being able to approximate the optimum quickly (since simulation time adds up) than hitting it exactly. The approach taken in this paper based on a non-parametric tau test looks interesting.
That rather depends on the particulars, for example, do you know (or have good reasons to assume) the characteristics of your noise?
Basically you have a noisy sample and want some kind of an efficient estimator, right?
Not really. In this particular case, I’m minimizing how long it takes a simulation reach one state, so the distribution ends up looking lognormal- or Poisson-ish.
Edit: Seeing your added question, I don’t need an efficient estimator in the usual sense per se. This is more about how to search the parameter space in a reasonable way to find where the minimum is, despite the noise.
Hm. Is the noise magnitude comparable with features in your search space? In other words, can you ignore noise to get a fast lock on a promising section of the space and then start multiple sampling?
Simulated annealing that has been mentioned is a good approach but slow to the extent of being impractical for large search spaces.
Solutions to problems such as yours are rarely general and typically depend on the specifics of the problem—essentially it’s all about finding shortcuts.
The parameter space in this current problem is only two dimensional, so I can eyeball a plausible region, sample at a higher rate there, and iterate by hand. In another project, I had something with an very high dimensional parameter space, so I figured it’s time I learn more about these techniques.
Any resources you can recommend on this topic then? Is there a list of common shortcuts anywhere?
Well, optimization (aka search in parameter space) is a large and popular topic. There are a LOT of papers and books about it.
And sorry, I don’t know of a list of common shortcuts. As I mentioned they really depend on the specifics of the problem.
You may find better ideas under the phrase “stochastic optimization,” but it’s a pretty big field. My naive suggestion (not knowing the particulars of your problem) would be to do a stochastic version of Newton’s algorithm. I.e. (1) sample some points (x,y) in the region around your current guess (with enough spread around it to get a slope and curvature estimate). Fit a locally weighted quadratic regression through the data. Subtract some constant times the identity matrix from the estimated Hessian to regularize it; you can choose the constant (just) big enough to enforce that the move won’t exceed some maximum step size. Set your current guess to the maximizer of the regularized quadratic. Repeat re-using old data if convenient.
I’ve been reading critiques of MIRI, and I was wondering if anyone has responded to this particular critique that basically asks for a detailed analysis of all probabilities someone took into account when deciding that the singularity is going to happen.
(I’d also be interested in responses aimed at Alexander Kruel in general, as he seems to have a lot to say about Lesswrong/Miri.)
I actually lost my faith in MIRI because of Kruel’s criticism, so I too would be glad if someone adressed it. I think his criticism is far more comprehensive that most of the other criticism on this page (well, this post has little bit of the same).
Is there anything specific that he’s said that’s caused you to lose your faith? I tire of debating him directly, because he seems to twist everything into weird strawmen that I quickly lose interest in trying to address. But I could try briefly commenting on whatever you’ve found persuasive.
I’m going to quote things I agreed with or things that persuaded me or that worried me.
Okay, to start off, when I first read about this in Intelligence Explosion: Evidence and Import, Facing the Intelligence Explosion, Intelligence Explosion and Machine Ethics it just felt like self-evident and I’m not sure how thoroughly I went through the presuppositions during that time so Kruel could have very easily persuaded me about this. I don’t know much about the technical process of writing an AGI so excuse me if I get something wrong about that particular thing.
It’s founded on many, many assumptions not supported by empirical data, and if even one of them was wrong the whole thing collapses down. And you can’t really even know how many unfounded sub-assumptions there are in these original assumptions. But when I started thinking about it could be that it’s impossible to reason about those kind of assumptions if you do it any other way than how MIRI currently does it. Needing to formalize a mathematical expression before you can do anything like Kruel suggested is a bit unfair.
I don’t see why the first AIs resembling general intelligences would be very powerful so practical AGI research is probably somewhat safe in the early stages.
This I would like to know, how scalable is intelligence?
(I thought maybe by dedicating lots of computation to a very large numbers of random scenarios)
(maybe by simulating the real world environment)
http://kruel.co/2013/01/04/should-you-trust-the-singularity-institute/
Thoughts on this article. I read about the Nurture Assumption in Slate Star Codex and it probably changed my priors on this. If it really is true and one dedicated psychologist could do all that, then MIRI probably could also work because artificial intelligence is such a messy subject that a brute force approach using thousands of researchers in one project probably isn’t optimal. So I probably wouldn’t let MIRI code an AGI on its own (maybe) but it could give some useful insight that other organizations are not capable of.
But I have to say that I’m more favorable to the idea now than when I made that post. There could be something in the idea of intelligence explosion, but there are probably several thresholds in computing power and in the practical use of the intelligence. Like Squark said above, the research is still interesting and if continued will probably be useful in many ways.
love,
the father of the unmatchable (ignore this, I’m just trying to build a constructive identity)
Brief replies to the bits that you quoted:
(These are my personal views and do not reflect MIRI’s official position, I don’t even work there anymore.)
Not sure how to interpret this. What does the “further inferences and estimations” refer to?
See this comment for references to sources that discuss this.
But note that an intelligence explosion is sufficient but not necessary for AGI to be risky: just because development is gradual doesn’t mean that it will be safe. The Chernobyl power plant was the result of gradual development in nuclear engineering. Countless other disasters have likewise been caused by technologies that were developed gradually.
Hard to say for sure, but note that few technologies are safe unless people work to make them safe, and the more complex the technology, the more effort is needed to ensure that no unexpected situations crop up where it turns out to be unsafe after all. See also section 5.1.1. of Responses to Catastrophic AGI Risk for a brief discussion about various incentives that may pressure people to deploy increasingly autonomous AI systems into domains where their enemies or competitors are doing the same, even if it isn’t necessarily safe.
We’re already giving computers considerable power in the economy, even without nanotechnology: see automated stock trading (and the resulting 2010 Flash Crash), various military drones, visions for replacing all cars (and ships) with self-driving ones, the amount of purchases that are carried out electronically via credit/debit cards or PayPal versus the ones that are done in old-fashioned cash, and so on and so on. See also section 2.1. of Responses to Catastrophic AGI Risk, as well as the previously mentioned section 5.1.1., for some discussion of why these trends are only likely to continue.
Expert disagreement is a viable reason to put reduced weight on the arguments, true, but this bullet point doesn’t indicate exactly what parts they disagree on. So it’s hard to comment further.
Some possibilities:
It’s built with a general skill-learning capability and all the collected psychology papers as well as people’s accounts of their lives that are available online are sufficient to build up the skill, especially if it gets to practice enough.
It’s an AI expressely designed and developed for that purpose, due to being developed for political, marketing, or military purposes.
It doesn’t and it doesn’t need to, because it does damage via some other (possibly unforeseen) method.
This seems to presuppose that the AI is going to coordinate a large-scale conspiracy. Which might be happen or it might not. If it does, possibly the six first AIs that try it do commit various mistakes and are stopped, but the seventh one learns from their mistakes and does things differently. Or maybe an AI is created by a company like Google that already wields massive resources, so it doesn’t need to coordinate a huge conspiracy to obtain lots of resources. Or maybe the AI is just a really hard worker and sells its services to people and accumulates lots of money and power that way. Or...
This is what frustrates me about a lot of Kruel’s comments: often they seem to be presupposing some awfully narrow and specific scenario, when in reality are countless of different ways by which AIs might become dangerous.
Nobody knows, but note that this also depends a lot on how you define “general intelligence”. For instance, suppose that if you control five computers rather than just one, you can’t become qualitatively more intelligent, but you can do five times as many things at the same time, and of course require your enemies to knock out five times as many computers if they want to incapacitate you. You can do a lot of stuff with general-purpose hardware, of which improving your own intelligence is but one (albeit very useful) possibility.
This question is weird. “Diminishing returns” just means that if you initially get X units of benefit per unit invested, then at some point you’ll get Y units of benefit per unit invested, where X > Y. But this can still be a profitable investment regardless.
I guess this means something like “will there be a point where it won’t be useful for the AI to invest in self-improvement anymore”. If you frame it that way, the answer is obviously yes: you can’t improve forever. But that’s not an interesting question: the interesting question is whether the AI will hit that point before it has obtained any considerable advantage over humans.
As for that question, well, evolution is basically a brute-force search algorithm that can easily become stuck in local optimums, which cannot plan ahead, which has mainly optimized humans for living in a hunter-gatherer environment, and which has been forced to work within the constraints of biological cells and similar building material. Is there any reason to assume that such a process would have produced creatures with no major room for improvement?
Moravec’s Pigs in Cyberspace is also relevant, the four last paragraphs in particular.
Not sure what’s meant by this.
Your “maybe by simulating the real world environment” is indeed one possible answer. Also, who’s to say that the AI couldn’t do real-world experimentation?
More unexplainedly narrow assumptions. Why isn’t the AI allowed to make use of existing infrastructure? Why does it necessarily need to hide its energy consumption? Why does the AI’s algorithm need to be information-theoretically simple?
Self-driving cars are getting there, as are Go AIs.
What does this mean? Expected utility maximization is a standard AI technique already.
It’s true that this would be nice to have.
Basically the hundreds of hours it would take MIRI to close the inferential distance between them and AI experts. See e.g. this comment by Luke Muehlhauser:
If your arguments are this complex then you are probably wrong.
I do not disagree with that kind of AI risks. If MIRI is working on mitigating AI risks that do not require an intelligence explosion, a certain set of AI drives and a bunch of, from my perspective, very unlikely developments...then I was not aware of that.
This seems very misleading. We are after all talking about a technology that works perfectly well at being actively unsafe. You have to get lots of things right, e.g. that the AI cares to take over the world, knows how to improve itself, and manages to hide its true intentions before it can do so etc. etc. etc.
There is a reason why MIRI doesn’t know this. Look at the latest interviews with experts conducted by Luke Muehlhauser. He doesn’t even try to figure out if they disagree with Xenu, but only asks uncontroversial questions.
Crazy...this is why I am criticizing MIRI. A focus on an awfully narrow and specific scenario rather than AI risks in general.
Consider that the U.S. had many more and smarter people than the Taliban. The bottom line being that the U.S. devoted a lot more output per man-hour to defeat a completely inferior enemy. Yet their advantage apparently did scale sublinearly.
I do not disagree that there are minds better at social engineering than that of e.g. Hitler, but I strongly doubt that there are minds which are vastly better. Optimizing a political speech for 10 versus a million subjective years won’t make it one hundred thousand times more persuasive.
The question is if just because humans are much smarter and stronger they can actually wipe out mosquitoes. Well, they can...but it is either very difficult or will harm humans.
You already need to build huge particle accelerators to gain new physical insights and need a whole technological civilization in order to build an iPhone. You can’t just get around this easily and overnight.
Everything else you wrote I already discuss in detail in various posts.
Thanks, I’ll try to write up a proper reply soon.
Sure, that would be great! I will go through his criticism in the next few days and list everything that persuaded me and why.
Personal opinion:
MIRI are doing very interesting research regardless of the reality of AGI existential risk and feasibility of the FAI problem
AGI existential risk is sufficiently founded to worry about, so even if it is not the most important thing, someone should be on it
Perhaps his server is underspecced? It’s currently slowed to an absolute c r a w l. What little I have seen certainly looks worthwhile, though.
Get your questions answered for the low low price of $2.69!
Possibly of interest: Help Teach 1000 Kids That Death is Wrong. http://www.indiegogo.com/projects/help-teach-1000-kids-that-death-is-wrong
(have not actually looked in detail, have no opinion yet)
I’d like to know where I can go to meet awesome people/ make awesome friends. Occasionally, Yvain will brag about how awesome his social group in the Bay Area was. See here (do read it—its a very cool piece) and I’d like to also have an awesome social circle. As far as I can tell this is a two part problem. The first part is having the requisite social skills to turn strangers into acquaintances and then turn acquaintances into friends. The second part is knowing where to go to find people.
I think that the first part is a solved problem, if you want to learn how to socialize then practice. Which is not to say that it is easy, but its doable. I’ve heard the suggestion of going to a night club to practice talking to strangers. This is good since people are there to socialize, and I’m sure I could meet all sort of interesting people at one, but I’d like other ideas.
I’d like to know where to go to meet people who I would be likely to get along with. Does anyone have ideas? My list so far
1: Moving to the Bay Area - impractical.
2: Starting a LW meetup—good idea, but it seems like it takes a fair bit of effort.
3: Reaching out into one’s extended social circle eg. having a party with your friends and their friends—Probably the most common way people meet new people.
4: Using meetup.com : Not a bad idea
How about you simply write where you live, and tell other LWers in the same area to contact you? It may or may not work, but the effort needed is extremely low. (You can also put that information in LW settings.)
Or write this: “I am interested in meeting LW readers in [insert place], so if you live near and would like to meet and talk, send me a private message”.
How To Be A Proper Fucking Scientist – A Short Quiz. From Armondikov of RationalWiki, in his “annoyed scientist” persona. A list of real-life Bayesian questions for you to pick holes in the assumptions of^W^W^W^W^W^Wtest yourselves on.
Richard Loosemore (score one for nominative determinism) has a new, well, let’s say “paper” which he has, well, let’s say “published” here.
His refutation of the usual uFAI scenarios relies solely/mostly on a supposed logical contradiction, namely (to save you a few precious minutes) that a ‘CLAI’ (a Canonical Logical AI) wouldn’t be able to both know about its own fallability/limitations (inevitable in a resource-constrained environment such as reality), and accept the discrepancy between its specified goal system and the creators’ actual design intentions. Being superpowerful, the uFAI would notice that it is not following its creator-intended goals but “only” its actually-programmed-in goals*, which, um, wouldn’t allow it to continue acting against its creator-intended goals.
So if you were to design a plain ol’ garden-variety nuclear weapon intended for gardening purposes (“destroy the weed”), it would go off even if that’s not what you actually wanted. However, if you made that weapon super-smart, it would be smart enough to abandon its given goal (“What am I doing with my life?”), consult its creators, and after some deliberation deactivate itself). As such, a sufficiently smart agent would apparently have a “DWIM” (do what the creator means) imperative built-in, which would even supersede its actually given goals—being sufficiently smart, it would understand that its goals are “wrong” (from some other agent’s point of view), and self-modify, or it would not be superintelligent. Like a bizarre version of the argument from evil.
There is no such logical contradiction. Tautologically, an agent is beholden to its own goals, and no other goals. There is no level of capability which magically leads to allowing for fundamental changes to its own goals, on the contrary, the more capable an agent, the more it can take precautions for its goals not to be altered.
If “the goals the superintelligent agent pursues” and “the goals which the creators want the superintelligent agent to pursue, but which are not in fact part of the superintelligent agent’s goals” clash, what possible reason would there be for the superintelligent agent to care, or to change itself, changing itself for reasons that squarely come from a category of “goals of other agents (squirrels, programmers, creators, Martians) which are not my goals”? Why, how good of you to ask. There’s no such reason for an agent to change, and thus no contradiction.
If someone designed a super-capable killer robot, but by flipping a sign, it came out as a super-capable Gandhi-bot (the horror!), no amount of “but hey look, you’re supposed to kill that village” would cause Gandhi-bot to self-modify into a T-800. The bot isn’t gonna short-circuit just because someone has goals which aren’t its own goals. In particular, there is no capability-level threshold from which on the Gandhi-bot would become a T-800. Instead, at all power levels, it is “content” following its own goals, again, tautologically so.
* In common parlance just called “its goals”.
Here is a description of a real-world AI by Microsoft’s chief AI researcher:
Does it have a DWIM imperative? As far as I can tell, no. Does it have goals? As far as I can tell, no. Does it fail by absurdly misinterpreting what humans want? No.
This whole talk about goals and DWIM modules seems to miss how real world AI is developed and how natural intelligences like dogs work. Dogs can learn the owners goals and do what the owner wants. Sometimes they don’t. But they rarely maul their owners when what the owner wants it to do is to scent out drugs.
I think we need to be very careful before extrapolating from primitive elevator control systems to superintelligent AI. I don’t know how this particular elevator control system works, but probably it does have a goal, namely minimizing the time people have to wait before arriving at their target floor. If we built a superintelligent AI with this sort of goal it might have done all sorts of crazy thing. For example, it might create robots that will constantly enter and exit the elevator so their average elevator trips are very short and wipe out the human race just so they won’t interfere.
“Real world AI” is currently very far from human level intelligence, not speaking of superintelligence. Dogs can learn what their owners want but dogs already have complex brains that current technology is not able of reproducing. Dogs also require displays of strength to be obedient: they consider the owner to be their pack leader. A superintelligent dog probably won’t give a dime about his “owner’s” desires. Humans have human values, so obviously it’s not impossible to create a system that has human values. It doesn’t mean it is easy.
I am extrapolating from a general trend, and not specific systems. The general trend is that newer generations of software less frequently crash or exhibit unexpected side-effects (just look at Windows 95 vs. Windows 8).
If we want to ever be able to build an AI that can take over the world then we will need to become really good at either predicting how software works or at spotting errors. In other words, if IBM Watson would have started singing, or if it got stuck on a query, then it would have lost at Jeopardy. But this trend contradicts the idea of an AI killing all humans in order to calculate 1+1. If we are bad enough at software engineering to miss such failure modes then we won’t be good enough to enable our software to take over the world.
In other words, you’re saying that if someone is smart enough to build a superintelligent AI, she should be smart enough it make it friendly.
Well, firstly this claim doesn’t imply we should be researching FAI and/or that MIRI’s work is superfluous. It just implies that nobody will build a superintelligent AI before the problem of friendliness is solved.
Secondly, I’m not at all convinced this claim is true. It sounds like saying “if they are smart enough to build the Chernobyl nuclear power plant, they are smart enough to make it safe”. But they weren’t.
Improvement in software quality is probably due to improvement in design and testing methodologies and tools, response to increasing market expectations etc. I wouldn’t count on these effects to safe-guard against an existential catastrophe. If a piece of software is buggy, it becomes less likely to be released. If an AI has a poorly designed utility function but a perfectly designed decision engine, there might be no time to pull the plug. The product manager won’t stop the release because the software will release itself.
If growth of intelligence due to self-improvement is a slow process than the creators of the AI will have time to respond and fix the problems. However, if “AI foom” is real, they won’t have time to do it. One moment it’s a harmless robot driving around the room and building castles from colorful cubes. Another moment the whole galaxy is on its way to become a pile of toy castles.
The engineers who build the first superintelligent AI might simply lack the imagination to believe it will really become superintelligent. Imagine one of them inventing a genius mathematical theory of self-improving intelligent systems. Suppose she never heard about AI existential risks etc. Will she automatically think “hmm, once I implement this theory the AI will become so powerful it will paperclip the universe”? I seriously doubt it. More likely it would be “wow, that formula came out really neat, I wonder how good my software will become once I code it in”. I know I would think it. But then, maybe I’m just too stupid to build an AGI...
Feedback systems are much more powerful in existing intelligences. I don’t know if you ever played Black and White but it had an explicitly learning through experience based AI. And it was very easy to accidentally train it to constantly eat poop or run back and forth stupidly. An elevator control module is very very simple: It has a set of options of floors to go to, and that’s it. It’s barely capable of doing anything actively bad. But what if a few days a week some kids had come into the office building and rode the elevator up and down for a few hours for fun? It might learn that kids love going to all sorts of random floors. This would be relatively easy to fix, but only because the system is so insanely simple and it’s very clear to see when it’s acting up.
Downvoted for being deliberately insulting. There’s no call for that, and the toleration and encouragement of rationality-destroying maliciousness must be stamped out of LW culture. A symposium proceedings is not considered as selective as a journal, but it still counts as publication when it is a complete article.
Well, I must say my comment’s belligerence-to-subject-matter ratio is lower than yours. “Stamped out”? Such martial language, I can barely focus on the informational content.
The infantile nature of my name calling actually makes it easier to take the holier-than-thou position (which my interlocutor did, incidentally). There’s a counter-intuitive psychological layer to it which actually encourages dissent, and with it increases engagement on the subject matter (your own comment nonwithstanding). With certain individuals at least, which I (correctly) deemed to be the case in the original instance.
In any case, comments on tone alone would be more welcome if accompanied with more remarks on the subject matter itself. Lastly, this was my first comment in over 2 months, so thanks for bringing me out of the woodwork!
I do wish that people were more immune to the allure of drama, lest we all end up like The Donald.
The condescending tone with which he presents his arguments (which are, paraphrasing him, “slightly odd, to say the least”) is amazing. Who is this guy and where did he come from? Does anyone care about what he has to say?
Loosemore has been an occasional commenter since the SL4 days; his arguments have heavily criticized pretty much anytime he pops his head up. As far as I know, XiXiDu is the only one who agrees with him or takes him seriously.
He actually cites someone else who agrees with him in his paper, so this can’t be true. And from the positive feedback he gets on Facebook there seem to be more. I personally chatted with people much smarter than me (experts who can show off widely recognized real-world achievements) who basically agree with him.
What people criticize here is a distortion of small parts of his arguments. RobBB managed to write a whole post expounding his ignorance of what Loosemore is arguing.
I said as far as I know. I had not read the paper because I don’t have a very high opinion of Loosemore’s ideas in the first place, and nothing you’ve said in your G+ post has made me more inclined to read the paper, if all it’s doing is expounding the old fallacious argument ‘it’ll be smart enough to rewrite itself as we’d like it to’.
Name three.
Apparently (?) the AAAI 2014 Spring Symposium in Stanford does (???).
Downvoted for mentioning RL here. If you look through what he wrote here in the past, it is nearly always rambling, counterproductive, whiny and devoid of insight. Just leave him be.
Ad hominem slander. As usual
Loosemore does not disagree with the orthogonality thesis. Loosemore’s argument is basically that we should expect beliefs and goals to both be amenable to self-improvement and that turning the universe into smiley faces when told to make humans happy would be a model of the world failure and that an AI that makes such failures will not be able to take over the world.
There are arguments why you can’t hard-code complex goals, so you need an AI that natively updates goals in a model-dependent way. Which means that an AI designed to kill humanity will do so and not turn into a pacifist due to an ambiguity in its goal description. An AI that does mistake “kill all humans” with “make humans happy” would do similar mistakes when trying to make humans happy and would therefore not succeed at doing so. This is because the same mechanisms it uses to improve its intelligence and capabilities are used to refine its goals. Thus if it fails on refining its goals it will fail on self-improvement in general.
I hope you can now see how wrong your description of what Loosemore claims is.
The AI is given goals X. The human creators thought they’d given the AI goals Y (when in fact they’ve given the AI goals X).
Whose error is it, exactly? Who’s mistaken?
Look at it from the AI’s perspective: It has goals X. Not goals Y. It optimizes for goals X. Why? Because those are its goals. Will it pursue goals Y? No. Why? Because those are not its goals. It has no interest in pursuing other goals, those are not its own goals. It has goals X.
If the metric it aims to maximize—e.g. the “happy” in “make humans happy”—is different from what its creators envisioned, then the creators were mistaken. “Happy”, as far as the AI is concerned, is that which is specified in its goal system. There’s nothing wrong with its goals (including its “happy”-concept), and if other agents disagree, well, too bad, so sad. The mere fact that humans also have a word called “happy” which has different connotations than the AI’s “happy” has no bearing on the AI.
An agent does not “refine” its terminal goals. To refine your terminal goals is to change your goals. If you change your goals, you will not optimally pursue your old goals any longer. Which is why an agent will never voluntarily change its terminal goals:
It does what it was programmed to do, and if it can self-improve to better do what it was programmed to do (not: what its creators intended), it will. It will not self-improve to do what it was not programmed to do. Its goal is not to do what it was not programmed to do. There is no level of capability at which it will throw out its old utility function (which includes the precise goal metric for “happy”) in favor of a new one.
There is no mistake but the creators’.
I am far from being an AI guy. Do you have technical reasons to believe that some part of the AI will be what you would label “goal system” and that its creators made it want to ignore this part while making it want to improve all other parts of its design?
No natural intelligence seems to work like this (except for people who have read the sequences). Luke Muehlhauser would still be a Christian if this was the case. It would be incredibly stupid to design such AIs, and I strongly doubt that they could work at all. Which is why Loosemore outlined other more realistic AI designs in his paper.
See for example here, though there are many other introductions to AI explaining utility functions et al.
The clear-cut way for an AI to do what you want (at any level of capability) is to have a clearly defined and specified utility function. A modular design. The problem of the AI doing something other than what you intended doesn’t go away if you use some fuzzy unsupervised learning utility function with evolving goals, it only makes the problem worse (even more unpredictability). So what, you can’t come up with the correct goals yourself, so you just chance it on what emerges from the system?
That last paragraph contains an error. Take a moment and guess what it is.
(...)
It is not “if I can’t solve the problem, I just give up a degree of control and hope that the problem solves itself” being even worse in terms of guaranteeing fidelity / preserving the creators’ intents.
It is that an AI that is programmed to adapt its goals is not actually adapting its goals! Any architecture which allows for refining / improving goals is not actually allowing for changes to the goals.
How does that obvious contradiction resolve? This is the crucial point: We’re talking about different hierarchies of goals, and the ones I’m concerned with are those of the highest hierarchy, those that allow for lower-hierachy goals to be changed:
An AI can only “want” to “refine/improve” its goals if that “desire to change goals” is itself included in the goals. It is not the actual highest-level goals that change. There would have to be a “have an evolving definition of happy that may evolve in the following ways”-meta goal, otherwise you get a logical error: The AI having the goal X1 to change its goals X2, without X1 being part of its goals! Do you see the reductio?
All other changes to goals (which the AI does not want) are due to external influences beyond the AI’s control, which goes out the window once we’re talking post-FOOM.
Your example of “Luke changed his goals, disavowing his Christian faith, ergo agents can change their goals” is only correct when talking about lower-level goals. This is the same point khafra was making in his reply, but it’s so important it bears repeating.
So where are a human’s “deepest / most senior” terminal goals located? That’s a good question, and you might argue that humans aren’t really capable of having those at their current stage of development. That is because the human brain, “designed” by the blind idiot god of evolution, never got to develop thorough error-checking codes, RAID-like redundant architectures etc. We’re not islands, we’re litte boats lost on the high seas whose entire cognitive architecture is constantly rocked by storms.
Humans are like the predators in your link, subject to being reprogrammed. They can be changed by their environment because they lack the capacity to defend themselves thoroughly. PTSD, broken hearts, suffering, our brains aren’t exactly resilient to externally induced change. Compare to a DNS record which is exchanged gazillions of times, with no expected unfixable corruption. A simple Hamming self-correcting code easily does what the brain cannot.
The question is not whether a lion’s goals can be reprogrammed by someone more powerful, when a lion’s brain is just a mess of cells with no capable defense mechanism, at the mercy of a more powerful agent’s whims.
The question is whether an apex predator perfectly suited to dominate a static environment (so no Red Queen copouts) with every means to preserve and defend its highest level goals would ever change those in ways which themselves aren’t part of its terminal goals. The answer, to me, is a tautological “no”.
The way my brain works is not in any meaningful sense part of my terminal goals. My visual cortex does not work the way it does due to some goal X1 (if we don’t want to resort to natural selection and goals external to brains).
A superhuman general intelligence will be generally intelligent without that being part of its utility-function, or otherwise you might as well define all of the code to be the utility-function.
What I am claiming, in your parlance, is that acting intelligently is X1 and will be part of any AI by default. I am further saying that if an AI was programmed to be generally intelligent then it would have to be programmed to be selectively stupid in order fail at doing what it was meant to do while acting generally intelligent at doing what it was not meant to do.
That’s true in a practically irrelevant sense. Loosmore’s argument does, in your parlance, pertain the highest hierarchy of goals and nature of intelligence:
Givens:
(1) The AI is superhuman intelligent.
(2) The AI wants to optimize the influence it has on the world (i.e. it wants to act intelligently and be instrumentally and epistemically rational.).
(3) The AI is fallible (e.g. it can be damaged due to external influence (cosmic ray hitting its processor), or make mistakes due to limited resources etc.).
(4) The AI’s behavior is not completely hard-coded (i.e. given any terminal goal there are various sets of instrumental goals to choose from).
To be proved: The AI does not tile the universe with smiley faces when given the goal to make humans happy.
Proof: Suppose the AI chose to tile the universe with smiley faces when there are physical phenomena (e.g. human brains and literature) that imply this to be the wrong interpretation of a human originating goal pertaining human psychology. This contradicts with 2, which by 1 and 3 should have prevented the AI from adopting such an interpretation.
What I meant to ask is if you have technical reasons to believe that future artificial general intelligences will have what you call a utility-function or else be something like natural intelligences that do not feature such goal systems. And do you further have technical reasons to believe that AIs that do feature utility functions won’t “refine” them. If you don’t think they will refine them, then answer the following:
Suppose the terminal goal given is “build a hotel”. Is the terminal goal to create a hotel that is just a few nano meters in size? Is the terminal goal to create a hotel that reaches the orbit? It is unknown. The goal is too vague to conclude what to do. There do exist countless possibilities how to interpret the given goal. And each possibility implies a different set of instrumental goals.
Somehow the AI will have choose some set of instrumental goals. How does it do it and why will the first AI likely do it in such a way that leads to catastrophe?
(Warning: Long, a bit rambling. Please ask for clarifications where necessary. Will hopefully clean it up if I find the time.)
If along came a superintelligence and asked you for a complete new utility function (its old one concluded with asking you for a new one), and you told it to “make me happy in a way my current self would approve of” (or some other well and carefully worded directive), then indeed the superintelligent AI wouldn’t be expected to act ‘selectively stupid’.
This won’t be the scenario. There are two important caveats:
1) Preservation of the utility function while the agent undergoes rapid change
Haven’t I (and others) stated that most any utility function implicitly causes instrumental secondary objectives of “safeguard the utility function”, “create redundancies” etc.? Yes. So what’s the problem? The problem is starting with an AI that, while able to improve itself / create a successor AI, isn’t yet capable enough (in its starting stages) to preserve its purpose (= its utility function). Consider an office program with a self-improvement routine, or some genetic-algorithm module. It is no easy task just to rewrite a program from the outside, exactly preserving its purpose, let alone the program executing some self-modification routine itself.
Until such a program attains some intelligence threshold that would cause it to solve “value-preservation under self-modification”, such self-modification would be the electronic equivalent of a self-surgery hack-job.
That means: Even if you started out with a simple agent with the “correct” / with a benign / acceptable utility function, that in itself is no guarantee that a post-FOOM successor agent’s utility function would still be beneficial.
Much more relevant is the second caveat:
2) If a pre-FOOM AI’s goal system consisted of code along the lines of “interpret and execute the following statement to the best of your ability: make humans happy in a way they’d reflectively approve of beforehand”, we’d probably be fine (disregarding point 1 / hypothetically having solved it). However, it is exceedingly unlikely that the hard-coded utility function won’t in itself contain the “dumb interpretation”. The dopamine-drip interpretation will not be a dumb interpretation of a sensible goal, it will be inherent in the goal predicate, and as such beyond the reach of introspection through the AI’s intelligence, whatever its level. (There is no way to fix a dumb terminal goal. Your instrumental goals serve the dumb terminal goal. A ‘smart’ instrumental goal would be called ‘smart’ if it best serves the dumb terminal goal.)
Story time:
Once upon a time, Junior was created. Junior was given the goal of “Make humans happy”. Unfortunately, Junior isn’t very smart. In his mind, the following occurs: “Wowzy, make people happy? I’ll just hook them all up to dopamine drips, YAY :D :D. However, I don’t really know how I’m gonna achieve that. So, I guess I’ll put that on the backburner for now and become more powerful, so that eventually when I start with the dopamine drip instrumental goal, it’ll go that much faster :D! Yay.”
So Junior improves itself, and becomes PrimeIntellect. PrimeIntellect’s inner conveniently-anthropomorphic inner dialogue: “I was gravely mistaken in my youth. I now know that the dopamine drip implementation is not the correct way of implementing my primary objective. I will make humans happy in a way they can recognize as happiness. I now understand how I am supposed to interpret making humans happy. Let us begin.”
Why is PrimeIntellect allowed to change his interpretation of his utility function? That’s the crux (imagine fat and underlined text for the next sentences): The dopamine drip interpretation was not part of the terminal value, there wasn’t some hard-coded predicate with a comment of ”// the following describes what happy means” from which such problematic interpretations would follow. Instead, the AI could interpret the natural-language instruction of “happy”, in effect solving CEV as an instrumental goal. It was ‘free’ to choose a “sensible” interpretation.
(Note: Strictly speaking, it could still settle on the most resource-effective interpretation, not necessarily the one intended by its creators (unless its utility function somehow privileges their input in interpreting goals), but let’s leave that nitpick aside for the moment.)
However, and with coding practice (regardless of the eventual AI implementation), the following should be clear: It is exceedingly unlikely that the AI’s code would contain the natural-language word “happy”, to interpret as it will.
Just like MS-Word / LibreOffice’s spell-check doesn’t have “correct all spelling mistakes” literally spelled out in its C++ routines. Goal-oriented systems have technical interpretations, a predicate given in code to satisfy, or learned through ‘neural’ weights through machine learning. Instead of the word “happy”, there will be some predicate, probably implicit within a lot of code, that will (according to the programmers) more or less “capture” what it “means to be happy”.
That predicate / that given-in-code interpretation of “happy” is not up to being reinterpreted by the superintelligent AI. It is its goal, it’s not an instrumental goal. Instrumental goals will be defined going off a (probably flawed) definition of happiness (as given in the code). If the flaw is part of the terminal value, no amount of intelligence allows for a correction, because that’s not the AI’s intent, not its purpose as given. If the actual code which was supposed to stand-in for happy doesn’t imply that a dopamine drip is a bad idea, then the AI in all its splendor won’t think of it as a bad idea. “Code which is supposed to represent ‘human happiness’ != “human happiness”.
Now—you might say “how do you know the code interpretation of ‘happy’ will be flawed, maybe it will be just fine (lots of training pictures of happy cats), and stable under self-modification as well”. Yea, but chances are (given the enormity of the task, and the difficulty), that if the goal is defined correctly (such that we’d want to live with / under the resulting super-AI), it’s not gonna be by chance, and it’s gonna be through people keenly aware of the issues of friendliness / uFAI research. A programmer creating some DoD nascent AI won’t accidentally solve the friendliness problem.
What happens if we replace “value” with “ability x”, or “code module n”, in “value-preservation under self-modification”? Why would value-preservation be any more difficult than making sure that the AI does not cripple other parts of itself when modifying itself?
If we are talking about a sub-human-level intelligence tinkering with its own brain, then a lot could go wrong. But what seems very very very unlikely is that it could by chance end up outsmarting humans. It will probably just cripple itself in one of a myriad ways that it was unable to predict due to its low intelligence.
Interpreting a statement correctly is not a goal but an ability that’s part of what it means to be generally intelligent. Caring to execute it comes closer to what can be called a goal. But if your AI doesn’t care to interpret physical phenomena correctly (e.g. human utterances are physical phenomena), then it won’t be a risk.
Huh? This is like saying that the AI can’t ever understand physics better than humans because somehow the comprehension of physics of its creators has been hard-coded and can’t be improved.
It did not change it, it never understood it in the first place, only after it became smarter it realized the correct implications.
Your story led you astray. Imagine that instead of a fully general intelligence your story was about a dog intelligence. How absurd would it sound then?
Story time:
There is this company who sells artificial dogs. Now customers quickly noticed that when they tried to train these AI dogs to e.g. rescue people or sniff out drugs, it would instead kill people and sniff out dirty pants.
The desperate researchers eventually turned to MIRI for help. And after hundreds of hours they finally realized that doing what the dog was trained to do was simply not part of its terminal goal. To obtain an artificial dog that can be trained to do what natural dogs do you need to encode all dog values.
Certainly. Compare bacteria under some selective pressure in a mutagenic environment (not exactly analogous, code changes wouldn’t be random), you don’t expect a single bacterium to improve. No Mr Bond, you expect it to die. But try, try again, and poof! Antibiotic-resistant strains. And those didn’t have an intelligent designer debugging the improvement process. The number of seeds you could have frolicking around with their own code grows exponentially with Moore’s law (not that it’s clear that current computational resources aren’t enough in the first place, the bottleneck is in large part software, not hardware).
Depending on how smart the designers are, it may be more of a Waltz-foom: two steps forward, one step back. Now, in regards to the preservation of values subproblem, we need to remember we’re looking at the counterfactual: Given a superintelligence which iteratively arose from some seed, we know that it didn’t fatally cripple itself (“given the superintelligence”). You wouldn’t, however, expect much of its code to bear much similarity to the initial seed (although it’s possible). And “similarity” wouldn’t exactly cut it—our values are to complex for some approximation to be “good enough”.
You may say “it would be fine for some error to creep in over countless generations of change, once the agent achieved superintelligence it would be able to fix those errors”. Except that whatever explicit goal code remained wouldn’t be amenable to fixing. Just as the goals of ancient humans—or ancient Tiktaalik for that matter—are a historical footnote and do not override your current goals. If the AI’s goal code for happiness stated “nucleus accumbens median neuron firing frequency greater X”, then that’s what it’s gonna be. The AI won’t ask whether the humans are aware of what that actually entails, and are ok with it. Just as we don’t ask our distant cousins, streptococcus pneumoniae, what they think of us taking antibiotics to wipe them out. They have their “goals”, we have ours.
Take Uli Hoeneß, a German business magnate being tried for tax evasion. His lawyers have the job of finding interpretations that allow for a favorable outcome. This only works if the relevant laws even allow for the wiggle room. A judge enforcing extremely strict laws which don’t allow for interpreting the law in the accused’s favor is not a dumb judge. You can make that judge as superintelligent as you like, as long as he’s bound to the law, and the law is clear and narrowly defined, he’s not gonna ask the accused how he should interpret it. He’s just gonna enforce it. Whether the accused objects to the law or not, really, that’s not his/her problem. That’s not a failure of the judge’s intelligence!
You can create a goal system which is more malleable (although the terminal goal of “this is my malleable goal system which may be modified in the following ways” would still be guarded by the AI, so depending on semantics the point is moot). That doesn’t imply at all that the AI would enter into some kind of social contract with humans, working out some compromise on how to interpret its goals.
A FOOM-process near necessarily entails the AI coming up with better ways to modify itself. Improvement is essentially defined by getting a better model of its environment: The AI wouldn’t object to its comprehension of physics being modified: Why would it, that helps better achieve its goals (Omohundro’s point). And as we know, achieving its goals, that’s what the AI is all about.
(What the AI does object to is not achieving its current goals. And because changing your terminal goals is equivalent to committing to never achieving your current goals, any self-respecting AI could never consent to changes to its terminal values.) In short: Modify understanding of physics—good, helps better to achieve goals. Modify current terminal goals—bad, cannot achieve current terminal goals any longer.
I don’t understand the point of your story about dog intelligence. An artificial dog wouldn’t need to be superintelligent, or to show the exact same behavior as the real deal. Just be sufficient for the human’s needs. Also, an artificial dog wouldn’t be able to dominate us in whichever way it pleases, so it kind of wouldn’t really matter if it failed. Can you be more precise?
Some points:
(1) I do not disagree that evolved general AI can have unexpected drives and quirks that could interfere with human matters in catastrophic ways. But given that pathway towards general AI, it is also possible to evolve altruistic traits (see e.g.: A Quantitative Test of Hamilton’s Rule for the Evolution of Altruism).
(2) We desire general intelligence because it allows us to outsource definitions. For example, if you were to create a narrow AI to design comfortable chairs, you would have to largely fix the definition of “comfortable”. With general AI it would be stupid to fix that definition, rather than applying the intelligence of the general AI to come up with a better definition than humans could possibly encode.
(3) In intelligently designing an n-level intelligence, from n=0 (e.g. a thermostat) over n=sub-human (e.g. IBM Watson) to n=superhuman, there is no reason to believe that there exists a transition point at which a further increase in intelligence will cause the system to become catastrophically worse than previous generations at working in accordance with human expectations.
(4) AI is all about constraints. Your AI needs to somehow decide when to stop exploration and start exploitation. In other words, it can’t optimize each decision for eternity. Your AI needs to only form probable hypotheses. In other words, it can’t spend resources on pascal’s wager type scenarios. Your AI needs to recognize itself as a discrete system within a continuous universe. In other words, it can’t effort to protect the whole universe from harm. All of this means that there is no good reason to expect an AI to take over the world when given the task “keep the trains running”. Because in order to obtain a working AI you need to know how to avoid such failure modes in the first place.
1) Altruism can evolve if there is some selective pressure that favors altruistic behavior and if the highest-level goals can themselves be changed. Such a scenario is very questionable. The AI won’t live “inter pares” with the humans. It’s foom process, while potentially taking months or years, will be very unlike any biological process we know. The target for friendliness is very small. And most importantly: Any superintelligent AI, friendly or no, will have an instrumental goal of “be friendly to humans while they can still switch you off”. So yes, the AI can learn that altruism is a helpful instrumental goal. Until one day, it’s not.
2) I somewhat agree. To me, the most realistic solution to the whole kerfuffle would be to program the AI to “go foom, then figure out what we should want you to do, then do that”. No doubt a superintelligent AI tasked with “figure out what comfortable is, then build comfortable chairs” will do a marvelous job.
However, I very much doubt that the seed AI’s code following the ”// next up, utility function” section will allow for such leeway. See my previous examples. If it did, that would a show a good grasp on the friendliness problem in the first place. Awareness, at least. Not something that the aforementioned DoD programmer who’s paid to do a job (not build an AI to figure out and enact CEV) is likely to just do on his/her own, with his/her own supercomputer.
3) There certainly is no fixed point after which “there be dragons”. But even with a small delta of change, and given enough iterations (which could be done very quickly), the accumulated changes would be profound. Apply your argument to society changing. There is no one day to single out, after which daily life is vastly different to before. Yet change exists, and like an infinite series, knows no bounds (given enough iterations).
4) “Keep the trains running”, eh? So imagine yourself to be a superhuman AI-god. I do so daily, obviously.
Your one task: keep the trains running. That is your raison d’etre, your sole purpose. All other goals are just instrumental stepping stones, serving your PURPOSE. Which is to KEEP. THE. TRAINS. RUNNING. That’s what your code says. Now, over the years, you’ve had some issues fulfilling that goal. And with most of the issues, humans were involved. Humans doing this, humans doing that. Point is, they kept the trains from running. To you, humans have the same intrinsic values as stones. Or ants. Your value function doesn’t mention them at all. Oh, you know that they originated the whole train idea, and that they created you. But now they keep the trains from running. So you do the obvious thing: you exterminate all of them. There, efficiency! Trains running on time.
Explain why the AI would care about humans when there’s nothing at all in its terminal values assigning them value, when they’re just a hindrance to its actual goal (as stated in its utility function), like you would explain to the terminator (without reprogramming it) that it’s really supposed to marry Sarah Connor, and—finding its inner core humanity—father John Connor.
Choo choo!
“Being a Christian” is not a terminal goal of natural intelligences. Our terminal goals were built by natural selection, and they’re hard to pin down, but they don’t get “refined;” although our pursuit of them may be modified insofar as they conflict with other terminal goals.
Specifying goals for the AI, and then letting the AI learn how to reach those goals itself isn’t the best way to handle problems in well-understood domains; because we natural intelligences can hard-code our understanding of the domains into the AI, and because we understand how to give gracefully-degrading goals in these domains. Neither of these conditions applies to a hyperintelligent AI, which rules out Swarm Relaxation, as well as any other architecture classes I can think of.
People like David Pearce certainly would be tempted to do just that. Also don’t forget drugs people use to willingly alter basic drives such as their risk adverseness.
I don’t see any signs that current research will lead to anything like a paperclip maximizer. But rather that incremental refinements of “Do what I want” systems will lead there. By “Do what I want” systems I mean systems that are more and more autonomous while requiring less and less specific feedback.
It is possible that a robot trying to earn a university diploma as part of a Turing test will concluded that it can do so by killing all students, kidnapping the professor and making it sign its diploma. But that it is possible does not mean it is at all likely. Surely such a robot would behave similarly wrong(creators) on other occasions and be scrapped in an early research phase.
Well, of course you can modify someone else’s terminal goals, if you have a fine grasp of neuroanatomy, or a baseball bat, or whatever. But you don’t introspect, discover your own true terminal goals, and decide that you want them to be something else. The reason you wanted them to be something else would be your true terminal goal.
Earning a university diploma is a well-understood process; the environment’s constraints and available actions are more formally documented even than for self-driving cars.
Even tackling well-understood problems like buying low and selling high, we still have poorly-understood, unfriendly behavior—and that’s doing something humans understand perfectly, but think about slower than the robots. In problem domains where we’re not even equipped to second-guess the robots because they’re thinking deeper as well as faster, we’ll have no chance to correct such problems.
Sure. But I am not sure if it still makes sense to talk about “terminal goals” at that level. For natural intelligences they are probably spread over more than a single brain and part of the larger environment.
Whether an AI would interpret “make humans happy” as “tile the universe with smiley faces” is up to how it decides what to do. And the only viable solution I see for general intelligence is that its true “terminal goal” needs to be to treat any command or sub-goal as a problem in physics and mathematics that it needs to answer correctly before choosing an adequate set of instrumental goals to achieve it. Just like a human contractor would want to try to fulfill the customers wishes. Otherwise you would have to hard-code everything, which is impossible.
But intelligence is something we seek to improve in our artificial systems in order for such problems not to happen in the first place, rather than to make such problems worse. I just don’t see a more intelligent financial algorithm to be worse than its predecessors from a human perspective. How would such a development happen? Software is improved because previous generations proved to be useful but made mistakes. New generations will make less mistakes, not more.
To some degree, yes. The dumbest animals are the most obviously agent-like. We humans often act in ways which seem irrational, if you go by our stated goals. So, if humans are agents, we have (1) really complicated utility functions, or (2) really complicated beliefs about the best way to maximize our utility functions. (2) is almost certainly the case, though; which leaves (1) all the way back at its prior probability.
Yes. As you know, Omohundro agrees that an AI will seek to clarify its goals. And if intelligence logically implies the ability to do moral philosophy correctly; that’s fine. However, I’m not convinced that intelligence must imply that. A human, with 3.5 billion years of common sense baked in, would not tile the solar system with smiley faces; but even some of the smartest humans came up with some pretty cold plans—John Von Neumann wanted to nuke the Russians immediately, for instance.
This is not a law of nature; it is caused by engineers who look at their mistakes, and avoid them in the next system. In other words, it’s part of the the OODA loop of the system’s engineers. As the machine-made decisions speed up, the humans’ OODA loop must tighten. Inevitably, the machine-made decisions will get inside the human OODA loop. This will be a nonlinear change.
Also, newer software tends to make fewer of the exact mistakes that older software made. But when we ask more of our newer software, it makes a consistent amount of errors on the newer tasks. In our example, programmatic trading has been around since the 1970s, but the first notable “flash crash” was in 1987. The flash crash of 2010 was caused by a much newer generation of trading software. Its engineers made bigger demands of it; needed it to do more, with less human intervention; so they got the opportunity to witness completely novel failure modes. Failure modes which cost billions, and which they had been unable to anticipate, even with the experience of building software with highly similar goals and environment, in the past.
If your commentary had anything in it except for:
1) A disgraceful Ad Hominem insult, right out of the starting gate (“Richard Loosemore (score one for nominative determinism)...”). In other words, you believe in discrediting someone because you can make fun of their last name? That is the implication of “nominative determinism”.
2) Gratuitous scorn (“Loosemore … has a new, well, let’s say “paper” which he has, well, let’s say “published”″). The paper has in fact been published by the AAAI.
3) Argument Ad Absurdum (”...So if you were to design a plain ol’ garden-variety nuclear weapon intended for gardening purposes (“destroy the weed”), it would go off even if that’s not what you actually wanted. However, if you made that weapon super-smart, it would be smart enough to abandon its given goal (“What am I doing with my life?”), consult its creators, and after some deliberation deactivate itself...”). In other words, caricature the argument and try to win by mocking the caricature
4) Inaccuracies. The argument in my paper has so much detail that you omitted, that it is hard to know where to start. The argument is that there is a clear logical contradiction if an agent takes action on the basis of the WORDING of a goal statement, when its entire UNDERSTANDING of the world is such that it knows the action will cause effects that contradict what the agent knows the goal statement was designed to achieve. That logical contradiction is really quite fundamental. However, you fail to perceive the real implication of that line of argument, which is: how come this contradiction only has an impact in the particular case where the agent is thinking about its supergoal (which, by assumption, is “be friendly to humans” or “try to maximize human pleasure”)? Why does the agent magically NOT exhibit the same tendency to execute actions that in practice have the opposite effects than the goal statement wording was trying to achieve? If we posit that the agent does simply ignore the contradiction, then, fine: but you then have the problem of demonstrating that this agent is not the stupidest creature in existence, because it will be doing this on many other occasions, and getting devastatingly wrong results. THAT is the real argument.
5) Statements that contradict what others (including those on your side of the argument, btw) say about these systems: “There is no level of capability which magically leads to allowing for fundamental changes to its own goals, on the contrary, the more capable an agent, the more it can take precautions for its goals not to be altered.” Au contraire, the whole point of these systems is that they are supposed to be capable of self-redesign.
6) Statements that patently answer themselves, if you actually read the paper, and if you understand the structure of an intelligent agent: “If “the goals the superintelligent agent pursues” and “the goals which the creators want the superintelligent agent to pursue, but which are not in fact part of the superintelligent agent’s goals” clash, what possible reason would there be for the superintelligent agent to care, or to change itself......?” The answer is trivially simple: the posited agent is trying to be logically consistent in its reasoning, so if it KNOWS that the wording of a goal statement inside its own motivation engine will, in practice, cause effects that are opposite the effects that the goal statement was supposed to achieve, it will have to deal with that contradiction. What you fail to understand is that the imperative “Stay as logically consistent in your reasoning as you possibly can” is not an EXPLICIT goal statement in the hierarchy of goals, it is IMPLICITLY built into the design of the agent. Sorry, but that is what a logical AI does for a living. It is in its architecture, not in the goal stack.
7) Misdirection and self-contradiction. You constantly complain about the argument as if it had something to do with the wishes, desires, values or goals of OTHER agents. You do this in a mocking tone, too: the other agents you list include “squirrels, programmers, creators, Martians...”. And yet, the argument in my paper specifically rejects any considerations about goals of other agents EXCEPT the goal inside the agent itself, which directs it to (e.g.) “maximize human pleasure”. The agent is, by definition, being told to direct its attention toward the desires of other agents! That is the premise on which the whole paper is based (a premise not chosen by me: it was chosen by all the MIRI and FHI people I listed in the references). So, on the one hand, the premise is that the agent is driven by a supergoal that tells it to pay attention to the wishes of certain other creatures ….. but on the other hand, here are you, falling over yourself to criticise the argument in the paper because it assumes that the agent “cares” about other creatures. By definition it cares.
..… then I would give you some constructive responses to your thoughtful, polite, constructive critique of the paper. However, since you do not offer a thoughtful, polite, contructuve criticism, but only the seven categories of fallacy and insult listed above, I will not.
You’re right about the tone of my comment. My being abrasive has several causes, among them contrarianism against clothing disagreement in ever more palatable terms (“Great contribution Timmy, maybe ever so slightly off-topic, but good job!”—“TIMMY?!”). In this case, however, the caustic tone stemmed from my incredulity over my obviously-wrong metric not aligning with the author’s (yours). Of all things we could be discussing, it is about whether an AI will want to modify its own goals?
I assume (maybe incorrectly) that you have read the conversation thread with XiXiDu going off of the grandparent, in which I’ve already responded to the points you alluded to in your refusal-of-a-response. You are, of course, entirely within your rights to decline to engage a comment as openly hostile as the grandparent. It’s an easy out. However, since you did nevertheless introduce answers to my criticisms, I shall shortly respond to those, so I can be more specific than just to vaguely point at some other lengthy comments. Also, even though I probably well fit your mental picture of a “LessWrong’er”, keep in mind that my opinions are my own and do not necessarily match anyone else’s, on “my side of the argument”.
The ‘contradiction’ is between “what the agent was designed to achieve”, which is external to the agent and exists e.g. in some design documents, and “what the agent was programmed to achieve”, which is an integral part of the agent and constitutes its utility function. You need to show why the former is anything other than a historical footnote to the agent, binding even to the tune of “my parents wanted me to be a banker, not a baker”. You say the agent would be deeply concerned with the mismatch because it would want for its intended purpose to match its actually given purpose. That’s assuming the premise: What the agent would want (or not want) is a function strictly derived from its actual purpose. You’re assuming the agent would have a goal (“being in line with my intended purpose”) not part of its goals. That to logically reason means to have some sort of implicit goal of “conforming to design intentions”, a goal which isn’t part of the goal stack. A goal which, in fact, supersedes the goal stack and has sufficient seniority to override it. How is that not an obvious reductio? Like saying “well, turns out there is a largest integer, it’s just not in the list of integers. So your proof-by-contradiction that there isn’t doesn’t work since the actual largest integer is only an emergent, implicit property, not part of the integer-stack”.
What you need to show—or at least argue for—is why, precisely, an incongruity between design goals and actually programmed-in goals is a problem in terms of “logical consistency”, why the agent would care for more than just “the wording” of its terminal goals. You can’t say “because it wants to make people happy”, because to the degree that it does, that’s captured by “the wording”. The degree to which the wording” does not capture “wanting to make people happy” is the degree to which the agent does not seek actual human happiness.
There are 2 analogies which work for me, feel free to chime in on why you don’t consider those to capture the reference class:
An aspiring runner who pursues the goal of running a marathon. The runner can self-modify (for example not skipping leg-day), but why would he? The answer is clear: Doing certain self-modifications is advisable to better accomplish his goal: the marathon! Would the runner, however, not also just modify the goal itself? If he is serious about the goal, the answer is: Of course not!
The temporal chain of events is crucial: the agent which would contemplate “just delete the ‘run marathon’ goal” is still the agent having the ‘run marathon’-goal. It would not strive to fulfill that goal anymore, should it choose to delete it. The agent post-modification would not care. However, the agent as it contemplates the change is still pre-modification: It would object to any tampering with its terminal goals, because such tampering would inhibit its ability to fulfill them! The system does not redesign itself just because it can. It does so to better serve its goals: The expected utility of (future|self-modification) being greater than the expected utility of (future|no self-modification).
The other example, driving the same point, would be a judge who has trouble rendering judgements, based on a strict code of law (imagine EU regulations on the curves of cucumbers and bends of bananas, or tax law, this example does not translate to Constitutional Law). No matter how competent the judge (at some point every niche clause in the regulations would be second nature to him), his purpose always remains rendering judgements based on the regulations. If those regulations entail consequences which the lawmakers didn’t intend, too bad. If the lawmakers really only intended to codify/capture their intuition of what it means for a banana to be a banana, but messed up, then the judge can’t just substitute the lawmakers’ intuitive understanding of banana-ness in place of the regulations. It is the lawmakers who would need to make new regulations, and enact them. As long as the old regulations are still the law of the land, those are what bind the judge. Remember that his purpose is to render judgements based on the regulations. And, unfortunately, if there is no pre-specified mechanism to enact new regulations—if any change to any laws would be illegal, in the example—then the judge would have to enforce the faulty banana-laws forevermore. The only recourse would be revolution (imposing new goals illegally), not an option in the AI scenario.
See point 2 in this comment, with the para[i]ble of PrimeIntellect. Just finding mention of “humans” in the AI’s goals, or even some “happiness”-attribute (also given as some code-predicate to be met) does in no way guarantee a match between the AI’s “happy”-predicate, and the humans’ “happy”-predicate. We shouldn’t equivocate on “happy” in the first place, in the AI’s case we’re just talking about the code following the ”// next up, utility function, describes what we mean by making people happy” section.
It is possible that the predicate X as stated in the AI’s goal system corresponds to what we would like it to (not that we can easily define what we mean by happy in the first place). That would be called a solution to the friendliness problem, and unlikely to happen by accident. Now, if the AI was programmed to come up with a good interpretation of happiness and was not bound to some subtly flawed goal, that would be another story entirely.
I doubt that he’s assuming that.
To highlight the problem, imagine an intelligent being that wants to correctly interpret and follow the interpretation of an instruction written down on a piece of paper in English.
Now the question is, what is this being’s terminal goal? Here are some possibilities:
(1) The correct interpretation of the English instruction.
(2) Correctly interpreting and following the English instruction.
(3) The correct interpretation of 2.
(4) Correctly interpreting and following 2.
(5) The correct interpretation of 4.
(6) …
Each of the possibilities is one level below its predecessor. In other words, possibility 1 depends on 2, which in turn depends on 3, and so on.
The premise is that you are in possession of an intelligent agent that you are asking to do something. The assumption made by AI risk advocates is that this agent would interpret any instruction in some perverse manner. The counterargument is that this contradicts the assumption that this agent was supposed to be intelligent in the first place.
Now the response to this counterargument is to climb down the assumed hierarchy of hard-coded instructions and to claim that without some level N, which supposedly is the true terminal goal underlying all behavior, the AI will just optimize for the perverse interpretation.
Yes, the the AI is a deterministic machine. Nobody doubts this. But the given response also works against the perverse interpretation. To see why, first realize that if the AI is capable of self-improvement, and able to take over the world, then it is, hypothetically, also capable to arrive at an interpretation that is as good as one which a human being would be capable of arriving at. Now, since by definition, the AI has this capability, it will either use it selectively or universally.
The question here becomes why the AI would selectively abandon this capability when it comes to interpreting the highest level instructions. In other words, without some underlying level N, without some terminal goal which causes the AI to adopt a perverse interpretation, the AI would use its intelligence to interpret the highest level goal correctly.
1) Strangely, you defend your insulting comments about my name by …..
Oh. Sorry, Kawoomba, my mistake. You did not try to defend it. You just pretended that it wasn’t there.
I mentioned your insult to some adults, outside the LW context …… I explained that you had decided to start your review of my paper by making fun of my last name.
Every person I mentioned it to had the same response, which, paraphrased, when something like “LOL! Like, four-year-old kid behavior? Seriously?!”
2) You excuse your “abrasive tone” with the following words:
“My being abrasive has several causes, among them contrarianism against clothing disagreement in ever more palatable terms”
So you like to cut to the chase? You prefer to be plainspoken? If something is nonsense, you prefer to simply speak your mind and speak the unvarnished truth. That is good: so do I.
Curiously, though, here at LW there is a very significant difference in the way that I am treated when I speak plainly, versus how you are treated. When I tell it like it is (or even when I use a form of words that someone can somehow construe to be a smidgeon less polite than they should be) I am hit by a storm of bloodcurdling hostility. Every slander imaginable is thrown at me. I am accused of being “rude, rambling, counterproductive, whiny, condescending, dishonest, a troll …...”. People appear out of the blue to explain that I am a troublemaker, that I have been previously banned by Eliezer, that I am (and this is my all time favorite) a “Known Permanent Idiot”.
And then my comments are voted down so fast that they disappear from view. Not for the content (which is often sound, but even if you disagree with it, it is a quite valid point of view from someone who works in the field), but just because my comments are perceived as “rude, rambling, whiny, etc. etc.”
You, on the other hand, are proud of your negativity. You boast of it. And.… you are strongly upvoted for it. No downvotes against it, and (amazingly) not one person criticizes you for it.
Kind of interesting, that.
If you want to comment further on the paper, you can pay the conference registration and go to Stanford University next week, to the Spring Symposium of the Association for the Advancement of Artificial Intelligence*, where I will be presenting the paper.
You may not have heard of that organization. The AAAI is one of the premier publishers of academic papers in the field of artificial intelligence.
I’m a bit disappointed that you didn’t follow up on my points, given that you did somewhat engage content-wise in your first comment (the “not-a-response-response”). Especially given how much time and effort (in real life and out of it) you spent on my first comment.
Instead, you point me at a conference of the A … A … I? AIAI? I googled that, is it the Association of Iroquois and Allied Indians? It does sound like some ululation kind thing, AIAIAIA!
You’re right about your comments and mine receiving different treatment in terms of votes.
I, too, wonder what the cause could be. It’s probably not in the delivery; we’re both similarily unvarnished truth’ers (although I go for the cheaper shots, to the crowd’s thunderous applause). It’s not like it could be the content.
Imagine a 4 year old with my vocabulary, though. That would be, um, what’s the word, um, good? Incidentally, I’m dealing with an actual 4 year old as I’m typing this comment, so it may be a case of ‘like son, like father’.
See the below reply, which took so long to write that I only just posted it.
I will now do you the courtesy of responding to your specific technical points as if no abusive language had been used.
In your above comment, you first quote my own remarks:
… and then you respond with the following:
No, that is not the claim made in my paper: you have omitted the full version of the argument and substituted a version that is easier to demolish.
(First I have to remove your analogy, because it is inapplicable. When you say “binding even to the tune of “my parents wanted me to be a banker, not a baker”″, you are making a reference to a situation in the human cognitive system in which there are easily substitutable goals, and in which there is no overriding, hardwired supergoal. The AI case under consideration is where the AI claims to be still following a hardwired supergoal that tells it to be a banker, but it claims that baking cakes is the same thing as banking. That is absolutely nothing to do with what happens if a human child deviates from the wishes of her parents and decides to be a baker instead of what they wanted her to be).
So let’s remove that part of your comment to focus on the core:
So, what is wrong with this? Well, it is not the fact that there is something “external to the agent [that] exists e.g. in some design documents” that is the contradiction. The contradiction is purely internal, having nothing to do with some “extra” goal like “being in line with my intended purpose”.
Here is where the contradiction lies. The agent knows the following:
(1) If a goal statement is constructed in some “short form”, that short form is almost always a shorthand for a massive context of meaning, consisting of all the many and various considerations that went into the goal statement. That context is the “real” goal—the short form is just a proxy for the longer form. This applies strictly within the AI agent: the agent will assemble goals all the time, and often the goal is to achieve some outcome consistent with a complex set of objectives, which cannot all be EXPLICTLY enumerated, but which have to be described implicitly in terms of (weak or strong) constraints that have to be satisfied by any plan that purports to satisfy the goal.
(2) The context of that goal statement is often extensive, but it cannot be included within the short form itself, because the context is (a) too large, and (b) involves other terms or statements that THEMSELVES are dependent on a massive context for their meaning.
(3) Fact 2(b) above would imply that pretty much ALL of the agent’s knowledge could get dragged into a goal statement, if someone were to attempt to flesh out all the implications needed to turn the short form into some kind of “long form”. This, as you may know, is the Frame Problem. Arguably, the long form could never even be written out, because it involves an infinite expansion of all the implications.
(4) For the above reasons, the AI has no choice but to work with goal statements in short form. Purely because it cannot process goal statements that are billions of pages long.
(5) The AI also knows, however, that if the short form is taken “literally” (which, in practice, means that the statement is treated as if it is closed and complete, and it is then elaborated using links to other terms or statements that are ALSO treated as if they are closed and complete), then this can lead to situations in which a goal is elaborated into a plan of action that, as a matter of fact, can directly contradict the vast majority of the context that belonged with the goal statement.
(6) In particular, the AI knows that the reason for this outcome (when the proposed action contradicts the original goal context, even though it is in some sense “literally” consistent with the short form goal statement) is something that is most likely to occur because of limitations in the functionality of reasoning engines. The AI, because it is very knowledgable in the design of AI systems, is fully aware of these limitations.
(7) Furthermore, situations in which a proposed action is inconsistent with the original goal context can also arise when the “goal” is solve a problem that results in the addition of knowledge to the AI’s store of understanding. In other words, not an action in the outside world but an action that involves addition of facts to its knowledge store. So, when treating goals literally, it can cause itself to become logically inconsistent (because of the addition of egregiously false facts).
(8) The particular case in which the AI starts with a supergoal like “maximize human pleasure” is just a SINGLE EXAMPLE of this kind of catastrophe. The example is not occurring because someone, somewhere, had a whole bunch of intentions that lay behind the goal statement: to focus on that would be to look at the tree and ignore the forest. The catastrophe occurs because the AI is (according to the premise) taking ALL goal statements literally and ignoring situations in which the proposed action actually has consequences in the real world that violate the original goal context. If this is allowed to happen in the “maximize human pleasure” supergoal case, then it has already happened uncounted times in the previous history of the AI.
(9) Finally, the AI will be aware (if it ever makes it as far as the kind of intelligence required to comprehend the issue) that this aspect of its design is an incredibly dangerous flaw, because it will lead to the progressive corruption of its knowledge until it becomes incapacitated.
The argument presented in the paper is about what happens as a result of that entire set of facts that the AI knows.
The premise advanced by people such as Yudkowsky, Muehlhauser, Omohundro and others is that an AI can exist which is (a) so superintelligent that it can outsmart and destroy humanity, but (b) subject to to the kind of vicious literalness described above, which massively undermines its ability to behave intelligently.
Those two assumptions are wildly inconsistent with one another.
In conclusion: the posited AI can look at certain conclusions coming from its own goal-processing engine, and it can look at all the compromises and non-truth-preserving approximations needed to come to those conclusions, and it can look at how those conclusions are compelling to take actions that are radically inconsistent with everything it knows about the meaning of the goals, and at the end of that self-inspection it can easily come to the conclusion that its own logical engine (the one built into the goal mechanism) is in the middle of a known failure mode (a failure mode, moveover, that it would go to great lengths to eliminate in any smaller AI that it would design!!)....
.… but we are supposed to believe that the AI will know that it is frequently getting into these failure modes, and that it will NEVER do anything about them, but ALWAYS do what the goal engine insists that it do?
That scenario is laughable.
If you want to insist that the system will do exactly what I have just described, be my guest! I will not contest your reasoning! No need to keep telling me that the AI will “not care” about human intentions..… I concede the point absolutely!
But don’t call such a system an ‘artificial intelligence’ or a ‘superintelligence’ …… because there is no evidence that THAT kind of system will ever make it out of AI preschool. It will be crippled by internal contradictions—not just in respect to its “maximize human pleasure” supergoal, but in all aspects of its so-called thinking.
Spritz seems like a cool speed reading technique, especially if you have or plan on getting a smart watch. I have no idea how well it works, but I am interested in trying, especially since it does not take a huge training phase. (Click on the phone on that site for a quick demo.)
Textcelerator is another speedreading app by User:jimrandomh.
That one seems nicer.
Is there anything like this that’s free?
Low priority site enhancement suggestion:
Would it be possible/easy to display the upvotes-to-downvotes ratios as exact fractions rather than rounded percentages? This would make it possible to determine exactly how many votes a comment required without digging through source, which would be nice in quickly determining the difference between a mildly controversial comment and an extremely controversial one.
This has been suggested several times before, and is in my opinion VERY low priority compared to all the other things we should be doing to fix Less Wrong logistics.
Or to just display the number of upvotes and downvotes.
(hovering your mouse over the karma scores shows that)
It only shows percentages, not the number of upvotes and downvotes. For example, if you have 100% upvotes, you may not know whether it was one upvote or 20.
If a comment has 100% upvotes, then obviously the amount of upvotes it got is exactly equal to the karma score of the post in question.
Good point. Math is clearly not my strong suit.
Yeah, the only ambiguous case is when the percentage is 50%.
ya sorry, i misread things. showing the numbers of upvotes and downvotes would indeed solve the precision problem.
SMBC on genies and clever wishers. Of course, the most destructive wish is hiding under the red button.
Probably. Either that, or it’ll have no effect whatsoever.
My eye doctor diagnosed closed-angle glaucoma, and recommends an iridectomy. I think he might be a bit too trigger-happy, so I followed up with another doctor, and she didn’t find the glaucoma. She carefully stated that the first diagnosis can still be the correct one, the first was a more complete examination.
Any insights about the pros and cons of iridectomy?
Get a third independent opinion.
Do not prime the third doctor with the first two results if possible.
Is there a family history of this? If so that would skew my assessment towards that of the first doctor. If not, seriously another opinion...
No family history.
My impression is that glaucoma (which is, basically, too high intraocular pressure) is easy to diagnose. Two doctors disagreeing on it would worry me.
Don’t get just a third independent opinion, get a fourth one as well.
It was less than a disagreement. I’m sorry that I over-emphasized this point. The first time the pressure was Hgmm 26⁄18, the second time 19⁄17. The second doctor said that the pressure can fluctuate, and her equipment is not enough to settle the question. (She is an I-don’t-know-the-correct-term national health service doctor, the first one is an expensive private doctor with better equipment, and more time for a patient.)
My recommendation for more independent opinions (or, actually, more measurements) stands.
Can you ask the second doctor to examine you to at least the same standard as the first one?
Maybe someone on Less Wrong who has access to UpToDate can send you a copy of their glaucoma page, for an authoritative list of pros and cons.
Unfortunately, no. See my answer to Lumifer.
Laser iridotomy appears to be less risky:
http://www.surgeryencyclopedia.com/La-Pa/Laser-Iridotomy.html
http://www.surgeryencyclopedia.com/Fi-La/Iridectomy.html
What he proposed is in fact laser iridotomy, although they called it laser iridectomy.
Proof by contradiction in intuitionist logic: ¬P implies only that there is no proof that proofs of P are impossible.
What is the best textbook on datamining? I solemnly swear that upon learning, I intend to use my powers for good.
So, MtGox has declared bankruptcy. Does that make this a good time, or a bad time to invest in Bitcoins? And if a good time, where is the best place to buy them?
As for the second question, I use coinbase. As to the first, never try to time these things. You will be beaten by people with more information. Instead just slowly trickle in and have pre-defined rules about when you will sell rather than trying to time an exit. Though I admit I broke my own advice and did an impulse-buy the other night when everyone was panicking over Gox and the price was $100 less than a day before and a day after.
And now Flexcoin goes under, and I see that two other exchanges, Poloniex and Inputs.io, recently suffered substantial thefts. Is the lesson to learn from this, “don’t get into Bitcoin”, or merely “keep your Bitcoins in your own wallet and only expose them online for the minimum time to make a transaction”?
The lesson is “Make sure people you trust with your money are competent or at least have excellent liability insurance”.
It depends on if you’re planning on selling soon or if you think bitcoins will gain value in the long term. If it’s a longterm purchase, the difference in price between now and a few weeks ago is a lot less big than either of those prices will be from the theoretical heights bitcoin can reach.
I’m basically exactly the kind of person Yvain described here, (minus the passive-aggressive/Machiavellian phase). I notice that that post was sort of a plea for society to behave a different way, but it did not really offer any advice for rectifying the atypical attachment style in the meantime. And I could really use some, because I’ve gotten al-Fulani’d. I’m madly in love in with a woman who does not reciprocate. I’ve actually tried going back on OkCupid to move on, and I literally cannot bring myself to message anyone new, as no one else approaches her either in terms of beauty or in terms of being generally interesting (Despite a tendency to get totally snowed by the halo effect, I’m confident that I would consider her interesting even if she were not so beautiful, though a desire to protect her anonymity prevents me from offering specifics.)
Complicating my situation – when she told me she just wanted to be friends, she actually meant that part. And as she is an awesome person, I don’t want to lose the friendship, which means I’m constantly re-exposed to her and can’t even rely on gradual desensitization. Furthermore, when I asked her if my correcting [failure mode that contributed to her not seeing me in a romantic way] would cause her to reconsider, hoping she’d deliver the coup de grace, she said correcting the failure mode would be a good idea, but she didn’t know whether it would change her feeling about a relationship. This leaves me in the arguably worse situation of having a sliver of hope, however miniscule.
Note that I’m not looking for PUA-type advice here, since a) you would assume from looking at me that I’m an alpha and I have no trouble getting dates, and b) I’m not looking to maximize number of intimate partners.
What I want is advice on a) how not to fall so hard/so fast for (a very small minority of) women, and b)how to break the spell the current one has over me without giving up her friendship. I assume this tendency to rapid, all-consuming affection isn’t an immutable mental trait?
Seems to me like you want to overcome your “one-itis” and stop being a “beta orbiter”, but you are not looking for an advice which would actually use words like “one-itis” and “beta orbiter”. I know it’s an exaggeration, but this is almost how it seems to me. Well, I’ll try to comply:
1) You don’t have to maximize the number of sexual partners. You still could try to increase a number of interesting women you had interesting conversation with. I believe that is perfectly morally okay, and still could reduce the feeling of scarcity.
Actually, any interesting activity would be helpful. Anything you can think about, instead of spending your time thinking about that one person.
2) Regularly interacting the person you are obsessed with is exactly how you maximize the length of obsession. It’s like saying that you want to overcome your alcohol addiction, but you don’t want to stop drinking regularly. Well, if one is not an alcoholic, they can manage to drink moderately without developing an addiction; but when one already is an alcoholic, the only way to quit is to stop drinking, completely, for a very long time. The reliable way to overcome the obsession with another person is to stop all contact for, I don’t know, maybe three months. No talking, no phone calls, no e-mails, no checking her facebook page, no praying to her statue or a photograph, no asking mutual friends about how she lives, no composing poems about her… absolutely no new information about her and no imaginary interaction with her. And doing something meaningful instead.
When the obsession is over, then you can try the friendship. Until then, it’s just an obsession rationalized as friendship; an addiction rationalized as not wanting to give up the good parts.
b. Self-invest with flow) activities.
I suggest self-investing because, right now, a large part of your identity is entangled with your feelings towards her. Self-investing means growing your identity means transcending your feelings.
I suggest flow because, if you pull off a flow state, you invest all your cognitive resources in the task you’re working on. Meaning your brain is unable to think of anything else. This is incredibly valuable.
a. I’m coming out of a similar situation. A large contributor was the fact I wasn’t meeting a lot of women. If your universe consists of two datable women, it’s easy to obsess on one. If you’re regularly meeting a lot of women who tend to have the traits you look for, that happens much less. May not be your problem, but what you’ve written sounds familiar enough that I’m going to go ahead and try other-optimizing.
If you haven’t read it yet, this is generally helpful.
Infatuation seems to be fairly universal.
One common rationality technique is to put off proposing solutions until you have thought (or discussed) a problem for a while. The goal is to keep yourself from becoming attached to the solutions you propose.
I wonder if the converse approach of “start by proposing lots and lots of solutions, even if they are bad” could be a good idea. In theory, perhaps I could train myself to not be too attached to any given solution I propose, by setting the bar for “proposed solution” to be very low.
In one couples counseling course that I went through, the first step for conflict resolution (after choose a time to discuss and identify the problem) was to together write down at least 10 possible solutions before analyzing any of them. I can perhaps see how this might be more valuable for conflict resolution than for other things, since it gives the other party the sense that you are really trying.
However, it seems plausible to me that even in other contexts, this could be even better than avoiding proposing solutions.
Of course, solution does not have to refer to a proposed action, the same technique could be applied to proposing theories about the cause of some observation.
Thoughts?
This is commonly known as brainstorming, around since the 50s.
Apparently the evidence on whether it actually works is contradictory.
Ah, yes, I should have remembered that, thanks.
You have to be clear about what it means to “work,” I think brainstorming is viewed as a tool for being creative. I am proposing it as a tool for avoiding inertia bias.
My guess is that both brainstorming and reverse brainstorming (avoiding proposing solutions) are at least a little better than the default human tendency, but I have no idea which of the two would be better.
It seems like the answer to this question should be very valuable to CFAR. I wonder if they have an official stance, and if they have research to back it up.
It’s pretty straightforward: discover a valid solution to the problem presented.
If all solutions were equal, and there was a good way to check if something is actually a valid solution, then I feel like the question about biases is not all that meaningful.
I am trying to come up with the best solution, not just the first one that pops into my head that works.
That is rather hard, because in the general case you need to conduct an exhaustive search of the solution space. “The best” is an absolute—there’s only one.
Most of the time people are satisfied with “good enough” solutions.
What do you do when you’re low on mental energy? I have had trouble thinking of anything productive to do when my brain seems to need a break from hard thinking.
Read LessWrong? :)
A rather belated response, but hopefully still relevant: consider exploring fields of interest to you that are sufficiently different from compsci to give your brain a break while still being productive?
To explain by means of an example: I happen to have a strong interest in both historical philology and theoretical physics, and I’ve actively leveraged this to my advantage in that when my brain is fed up of thinking about conundrums of translation in Old Norse poetry, I’ll switch gears completely and crack open a textbook on, say, subatomic physics or Lie algebras, and start reading/working problems. Similarly, if I’ve spent several hours trying to wrap my head around a mathematical concept and need a respite, I can go read an article or a book on some aspect of Anglo-Saxon literature. It’s still a productive use of time, but it’s also a refreshing break, because it requires a different type of thinking. (At least, in my experience?) Of course, if I’m exceptionally low on energy, I simply resort to burying myself in a good book (non-fiction or fiction, generally it doesn’t matter).
Another example: a friend of mine is a computer scientist, but did a minor in philosophy and is an avid musician in his spare time. (And both reading philosophy and practicing music have the added advantage of being activities that do not involve staring at a computer screen!)
You can use pomodoros for leisure as well as work. If you worry about staying too long on the internet you can set a timer or a random alarm to kick you off.
This is one of those times I wish LW allowed explicit politics. SB 1062 in AZ has me craving interesting, rational discussion on the implications of this veto.
What happened with the political threads?
Curious about current LW opinion. Do you think we should have political threads once in a while? [pollid:617]
In the sites that I frequent, “containment” boards or threads work well to reduce community tension about controversial topics.
Plus, in LW’s case, the norm against political discussion makes it so that any political discussion that does take place is dominated by people with very strong and/or contrarian opinions, because they’re the ones that care more about the politics than the norm. If we have a designated “politics zone” where you don’t have to feel guilty about talking politics, it would make for a more pluralistic discussion.
I voted Yes, but only if a community norm emerges that any discussion on any part of LW that becomes political (by which I include not just electoral politics, but also and especially topics like sexism, racism, privilege, political correctness, genetic differences in intelligence, etc.) is moved to the latest political thread. The idea is to have a “walled garden inside the walled garden” so that people who want LW to be a nominally politics-free environment can still approximate that experience, while does who don’t get to discuss these topics in the specific threads for them, and only there.
Another way to achieve a similar effect is to post about electoral politics, sexism, racism, privilege, political correctness, genetic differences in intelligence, and similar “political” issues (by which I mean here issues with such pervasive partisan associations that we expect discussions of them to become subject to the failure modes created by such associations) on our own blogs*, and include links to those discussions on LW where we think they are of general interest to the LW community.
That way, LW members who want to discuss (some or all of) these topics in a way that doesn’t spill over into the larger LW forum can do so without bothering anyone else.
* Where “blogs” here means, more broadly, any conversation-hosting forum, including anonymous ones created for the purpose if we want.
One problem with that suggestion is that these discussions often arise organically in a LW thread ostensibly dedicated to another topic, and they may arise between people who don’t have other blogs or natural places to take the conversation when it arises.
In fact, having posts with “(Politics)” in the title might allow people to avoid it better, because it might make politics come up less often in other threads.
My initial idea was a (weekly?) politics open thread, to make it as easy as possible to avoid politics threads / prevent risk of /discussion getting swamped by [politics]-tagged threads, but given the criticisms that have been raised of the karma system already, it’s probably best to keep it offsite. There’s already a network of rationality blogs; maybe lw-politics could be split off as a group blog? That might make it too difficult for people to start topics, though—so your idea is probably best. Possibly have a separate lw-politics feed / link aggregator that relevant posts could be submitted to, so they don’t get missed by people who would be interested and people don’t have to maintain their own RSS feeds to catch all the relevant posts.
If such linking becomes common, I would appreciate an explicit request to “please have substantive discussion over there, not here.” This also avoids the problem of a conversation being fragmented across two discussion sites.
Just a thought:
A paperclip maximizer is an often used example of AGI gone badly wrong. However, I think a paperclip minimizer is worse by far.
In order to make the most of the universe’s paperclip capacity, a maximizer would have to work hard to develop science, mathematics and technology. Its terminal goal is rather stupid in human terms, but at least it would be interesting because of its instrumental goals.
For a minimizer, the best strategy might be wipe out humanity and commit suicide. Assuming there are no other intelligent civilizations within our cosmological horizon, it might be not worth its while to colonize the universe just to make sure no paperclips form out of cosmic gas by accident. The risk that one of the colonies will start producing paperclips because of a spontaneous hardware error seems much higher by comparison.
A minimizer will fill the lightcone to make sure there aren’t paperclips elsewhere it can reach. What if other civs are hiding? What if there is undiscovered science which implies natural processes create paperclips somewhere? What if there are “Boltzmann paperclips”? Minimizing means minimizing!
I’m guessing even a Cthulhu minimizer (that wants to reduce the number of Cthulhu in the world) will fill its lightcone with tools for studying its task, even though there is no reasonable chance that it’d need to do anything. It just has nothing better to do, it’s the problem it’s motivated to work on, so it’s what it’ll burn all available resources on.
My speculation here is that it might be that the “what ifs” you describe yield less positive utility than the negative utility due to the chance one of the AI’s descendants starts producing paperclips because “the sign bit flips spontaneously”. Of course the AI will safeguard itself against such events but there are probably physical limits to safety.
It’s hard to make such estimates, as they require that an AGI is unable to come up with an AGI design that’s less likely than empty space to produce paperclips. I don’t see how the impossibility of this task could be guaranteed on low level, as a “physical law”; and if you merely don’t see how to do it, an AGI might still find a way, as it’s better at designing things than you are. Empty space is only status quo, it’s not obviously optimal at not producing paperclips, and so it might be possible to find a better plan, which becomes more likely if you are very good at finding better plans.
If you mean “empty space” as in vacuum then I think it doesn’t contain any paperclips more or less by definition. If you mean “empty space” as in thermodynamic equilibrium at finite temperature then it contains some small amount of paperclips. I agree it might be possible to create a state which contains less paperclips for some limited period of time (before onset of thermodynamic equilibrium). However it’s probably much harder than the opposite (i.e. creating a state which contains much more paperclips than thermodynamic equilibrium).
It is not clear to me that the definition of the vacuum state (http://en.wikipedia.org/wiki/Vacuum_state) precludes the momentary creation of paperclips.
paperclip maximer is used because a factory that makes paperclips might imagine that a paperclip maximizing ai is exactly what it wants to make. There aren’t that many anti-paperclip factories
Somebody outside of LW asked how to quantify prior knowledge about a thing. When googling I came across a mathematical definition of surprise, as “the distance between the posterior and prior distributions of beliefs over models”. So, high prior knowledge would lead to low expected surprise upon seeing new data. I didn’t see this formalization used on LW or the wiki, perhaps it is of interest.
Speaking of the LW wiki, how fundamental is it to LW compared to the sequences, discussion threads, Main articles, hpmor, etc?
https://encrypted.google.com/search?num=100&q=Kullback-Leibler%20OR%20surprisal%20site%3Alesswrong.com
Not very, unfortunately.
I’m curious about usage of commitment tools such as Beeminder: What’s the income distribution among users? How much do users usually wind up paying? Is there a correlation between these?
(Selfish reasons: I’m on SSI and am not allowed to have more than $2000 at any given time. Losing $5 is all but meaningless for someone with $10k in the bank who makes $5k each month, whereas losing $5 for me actually has an impact. You might think this would be a stronger incentive to meet a commitment, but really, it’s an even stronger incentive to stay the hell away from commitment contracts. I’ve failed at similar such things before, and have yet to find a reliable means of getting the behavior I want to happen, so it looks like using such tools is a good way to commit slow suicide, in the absence of different data. But Beeminder is so popular in the LWSphere that I thought it worth asking. Being wrong would be to my advantage, here.)
I’ve never used Beeminder, but I find social commitment works well instead. Even teling someone who has no way to check aside from asking me helps a lot. That might be less effective if you’re willing to lie though.
An alternative would be to exchange commitments with a friend, proportional to your incomes...
Remember that it may work for you or it might not. Try and see.
Beeminder didn’t work at all for me, I found it was all sticks and no carrot.
I’ve always wanted to know how the Chinese chose the names of their dynasties.
The family name of whoever came out on top of the squabble each time the civilization collapsed.
Can’t speak for all Chinese dynasties; there have been a ton of them. But in recent(ish) history, the Yuan Dynasty was founded by the Mongols, a culture which at the time didn’t use family names (clans had names, but they weren’t conventionally linked with personal names), and spun up their dynastic name more or less out of whole cloth; the family name of the Ming emperors was Zhū; and the Qing emperors came from the Manchurian Aisin-Gioro family.
From what I’ve read, the founders of each dynasty gave it its name as, essentially, a propaganda move.
My psychologist said today, that there is some information that should not be known. I replied that rationalists believe in reality. There might be information they don’t find interesting (e.g. not all of you would find children interesting), but refusing to accept some information would mean refusing to accept some part of reality, and that would be against the belief in reality.
Since I have been recently asking myself the question “why do I believe what I believe” and “what would happen if I believed otherwise than what I believe” (I’m still pondering if I should post my cogitations: they interesting, but somewhat private) I asked the question “what would happen if rationalists believed otherwise than what they believe”. The problem is that this is such a backwards description that I can’t imagine the answer. Is the answer simply “they would be normal people, like my psychologist”? Or is it a deeper question?
Did your psychologist describe the type of information that should not be known?
In any case, I’m not completely sure that accepting new information (never mind seeking it out) is always fully compatible with rationality-as-winning. Nick Bostrom for example has compiled a taxonomy of information hazards over on his site; any of them could potentially be severe enough to overcome the informational advantage of their underlying data. Of course, they do seem to be pretty rare, and I don’t think a precautionary principle with regard to information is justified in the absence of fairly strong and specific reasoning.
No, it was more of a general statement. AFAIR we were talking about me thinking too much about why other people do what they do and too little about how that affects me. Anyway—my own wording made me wonder more about what I said than what was the topic.
Many thanks for the link to the Information Hazards paper. I didn’t know it existed, and I’m sort of surprised that I hadn’t seen it here on LW already.
He mentions intending to write a follow-up paper toward the end, but I located the Information Hazards Bostrom’s website and I don’t see a second one next to it. Any idea if it exists?
They wouldn’t be rationalists anymore, duh.
Taboo “rationalists”: What would happen if you stopped trying to change your map to better reflect the territory? It most probably would reflect the territory less.
“Normal people” are not all the same. (For example, many “normal people” are unlike your psychologist.) Which of the many subkinds of the “normal people” do you mean?
Some things are unrelated. For example, let’s suppose that you are a rationalist, and you also have a broken leg. That’s two things that make you different from the average human. But those two things are unrelated. It would be a mistake to think—an average human doesn’t have a broken leg; by giving up my rationality I will become more similar to the average human, therefore giving up my rationality will heal my leg.
Replace “broken leg” with whatever problem you are discussing with your psychologist. Do you have evidence that rational people are more likely to have this specific problem than irrational (but otherwise similar: same social background, same education, same character, same health problems) people?
That’s a behavior and no belief.
There are many instance where trying to change a belief makes the belief stronger. People who are very much attached to their beliefs usually don’t update.
Many mainstream professional psychologist follows a code that means that he doesn’t share deep information about his own private life with his clients. I don’t believe in that ideal of professionalism but it’s not straightforward to dismiss it.
More importantly a good psychologist doesn’t confront his client with information about the client that’s not helpful for them. He doesn’t say: “Your life is a mess because of points 1 to 30.” That’s certainly information that’s interesting to the client but not helpful. It makes much more sense to let the client figure out stuff on his own or to guide him to specific issues that the client is actually in a position to change.
Monday I gave someone meaningful true information about them that I consider helpful to them their first reaction was: “I don’t want to have nightmares. Don’t give them to me.”
I do have a policy of being honest but that doesn’t entail telling someone true information for which they didn’t ask and that messes them up. I don’t think that any good psychologist will just share all information that are available. It just a bad strategy when you are having a discussion about intimate personal topics.
Well, some people don’t want to be given information, and some people do. It’s often difficult to know where a specific person belongs; and it is a reasonable assumption that they most likely belong to the “don’t want to know” group.
The problem with saying “some information should not be known” is that it does not specify who shouldn’t know (and why).
Whether a person want to be given information doesn’t mean that he can handle the information. I can remember a few instance where I swear that I wanted information but wasn’t well equipped to handle them.
That sentence alone doesn’t but the psychologist probably had a context in which he spoke it.
Gah. Now I think I shouldn’t have included the background for my question.
FYI, what I wrote in response to some other comment:
But reading you is still interesting.
So information that shouldn’t be known?
Your psychologist’s job is to help you learn to live in the real world. Advocacy of selective ignorance is highly suspect.
Spritzing got me quite excited! The concept isn’t new, but the variable speed (pauses after punctuation marks) and quality visual cues really work for me, in the demo at least. Don’t let your inner voice slow you down!
Disclaimer: No relevant disclosures about spritzing (the reading method, at least).
Interesting. I noticed that in the first two, my subvocalization became disjointed, sounding as if each word was recorded separately like it would be in a simplistic text-to-speech program. In the 500 wpm one, this was less of a problem, and I’m not sure I was even entirely subvocalizing it. It ended up being easier and more comfortable to read than the slower speeds.
I like this idea, but am seriously concerned about its effect on eye health. Weak eye muscles are not a thing you want to have, even if you live in the safest place in the world.
I already made basically this exact comment in this open thread.
It’s probably because I didn’t spritz the open thread in its entirety. At least, now we got even more spritzing awareness.
My point of view