simon

Karma: 982

No, really, it predicts next tokens.

simon18 Apr 2023 3:47 UTC

58 points

37 comments3 min readLW link

simon 27 Jan 2024 0:03 UTC
28 points
0
on: Without fundamental advances, misalignment and catastrophe are the default outcomes of training powerful AI
I’m not convinced by the argument that AI science systems are necessarily dangerous.
It’s generically* the case that any AI that is trying to achieve some real-world future effect is dangerous. In that linked post Nate Soares used chess as an example, which I objected to in a comment. An AI that is optimizing within a chess game isn’t thereby dangerous, as long as the optimization stays within the chess game. E.g., an AI might reliably choose strong chess moves, but still not show real-world Omohundro drives (e.g. not avoiding being turned off).
I think scientific research is more analogous to chess than trying to achieve a real-world effect in this regard (even if the scientific research has real-world side effects), in that you can, in principle, optimize for reliably outputting scientific insights without actually leading the AI to output anything based on its real-world effects. (the outputs are selected based on properties aligned with “scientific value”, but that doesn’t necessarily require the assessment to take into account how it will be used, or any other effect on the future of the world. You might need to be careful not to evaluate in such a way that it will wind up optimizing for real-world effects, though).
Note: an AI that can “build a fusion rocket” is generically dangerous. But an AI that can design a fusion rocket, if that design is based on general principles and not tightly tuned on what will produce some exact real-world effect, is likely not dangerous.
*generically dangerous: I use this to mean, an AI with this properties is going to be dangerous unless some unlikely-by-default (and possibly very difficult) safety precautions are taken.

simon 1 Jul 2022 6:53 UTC
24 points
2
on: The Track Record of Futurists Seems … Fine
There’s a lot of room for debate on the correctness of the resolutions of these predictions:
e.g. Heinlein in 1949:
Space travel we will have, not fifty years from now, but much sooner. It’s breathing down our necks.
This is marked as incorrect, due to the marker assuming that this meant mass space travel, but I wouldn’t interpret this as mass space travel unless there’s some relevant context I’m missing here—keep in mind that this was from 1949, 8 years before Sputnik.^[1]
On the other hand:
All aircraft will be controlled by a giant radar net run on a continent-wide basis by a multiple electronic “brain.”
This is marked as correct, apparently due to autopilot and the “USAF Airborne Command Post”? But I would interpret it as active control of the planes by a centralized computer and mark it as incorrect.^[2]
Edited to add: there were a bunch i could have mentioned but want to remark on this one where my interpretation was especially different from the marker’s:
Interplanetary travel is waiting at your front door — C.O.D. It’s yours when you pay for it.

This is also from 1949. The marker interprets this as a prediction of “Commercial interplanetary travel”. I see it rather as a conditional prediction of interplanetary travel (not necessarily commercial), given the willingness to fund it, i.e. a prediction that the necessary technology would be available but not necessarily that it would be funded. If this is the right interpretation, it seems correct to me. Again, I could be completely wrong depending on the context. ^[3]
1. ^
  Edited to add: I realized I actually have a copy of Heinlein’s “Expanded Universe” which includes “Where To?” and followup 1965 and 1980 comments. In context, this statement comes right in the middle of a discussion of hospitals for old people on the moon, which considerably shifts the interpretation towards it being intended to refer to mass space travel, though if Heinlein were still here he could argue it literally meant any space travel.
2. ^
  In context, it’s not 100% clear that he meant a single computer, though I still think so. But he definitely meant full automation outside of emergency or unusual situations; from his 1980 followup: “But that totally automated traffic control system ought to be built. … all routine (99.9%+ )takeoffs and landings should be made by computer.”
3. ^
  And now seeing the context, I stand by this interpretation: It’s a standalone comment from the original, but Heinlein’s 1965 followup includes “and now we are paying for it and the cost is high”, confirming that government space travel counted in his view...but, given that he did assert we were paying for it, and interplanetary space travel has not occurred (I interpret the prediction as meaning human space travel), this actually might cut against counting this as a correct prediction.

simon 10 Apr 2009 22:20 UTC
23 points
on: Akrasia and Shangri-La

The metabolically privileged don’t believe in metabolic privilege, since they are able to lose weight by trying!

Some of us do believe in it since we are able to stay very thin without trying. I have never dieted and never needed to.

But, we probably don’t post very much on diet blogs.

I come from a family of thin people who eat fairly unhealthily but are quite active. When I first stopped living with my parents, I basically stopped exercising and ate even more unhealthily. I became very unfit in the sense of e.g., not being able to run a block without getting out of breath, but gained very little weight. So I figure the causation is probably not mainly exercise → thinness, but more on the lines of genes → (thinness & athleticism) or genes → thinness → athleticism.

simon 6 Sep 2021 18:10 UTC
16 points
on: D&D.Sci Pathfinder: Return of the Gray Swan
Some (late relative to others) initial remarks:
As others have noted, all but one datapoint is consistent with Galleon/Carrack having 30hp, Barquentine/Dhow having 20hp, The one exception being a single case of Carrack taking 5% dmg from Reef. Also as abstractapplic has noted, Barquentines tend to be effected by things similarly to Galleons and Dhow similarly to Carracks. abstractapplic claims that they are the same other than hp, but it would take a massive confounder to account for e.g. the different encounter probabilities (as found by measure, also remarked on below).
Note, per-hex encounter probabilities below don’t account for selection effects except that I tended to round up if close call to round up or down. I do count only out-of-port ships that didn’t get destroyed in the denominator. Damage numbers don’t account for selection effects either.
Encounters:
Reef, Kraken, Iceberg Mefolk and Wyrd Majick Fyre have location dependence as noted by others.
Reef:
Reef always does at least 1 damage, exponentialish decline with long tail, 3.5-4 average
As noted by abstractapplic, Reefs occur on hexes adjacent to land but not adjacent to ports. I haven’t seen anyone mention that for the purpose of this rule, L16 is a land hex. I guess it’s a seamount.
The probability of receiving a reef encounter if going through a reef hex is about ~20% for non-Dhow’s, and ~4% for Dhows. Combined with the potentially high damage this makes these a high priority to avoid if not using a Dhow.
Kraken:
Kraken: spiky damage histogram. Spikes decline for higher values (but selection effects?), worse for Carrack/Dhow, and Carrack/Dhow also seem to lack a low damage component present in the Barquentine/Galleon distribution . ~3.5 average for Galleon/Barquentine, ~6.5 for Dhow, ~8 for Carrack.
As others have noted Kraken have “territories”. These “territories” actually are just a simple rule as with Reefs:
Kraken territories = spaces at least a 2 hex gap away from land (where land has the same definition as for Reefs, i.e. L16 is a land hex).
Around 25-30% encounter probability per relevant hex. Combined with the high damage, high priority to avoid for all ships but especially Carrack/Dhow.
You can always avoid Kraken+Reefs by keeping a 1 hex gap between you and land (or L16) when not adjacent to a port. There are minimum length paths that follow this rule between most port combinations except between South Point and either Norwatch and Eastmarch, where a detour is required (a quite significant one for Eastmarch/South Point).
As noted by others, the target points are in Kraken territory (IMO this is likely a coincidence since that just means they are far at sea). We can avoid going into any additional Kraken territory, but this will require an additional detour (relative to just avoiding reefs) for the western target particularly if avoiding E7 for which we have no data.
Iceberg:
Icebergs: dmg roughly consistent with 1d6 as reported by abstractapplic, so about 3.5 average damage. However, Galleons and Carracks seem to take 1 damage more often and Barquentines and Dhows take 1 damage less often. Maybe coincidence?
Icebergs are found from rows 0-2 in summer (Jun-Aug), in rows 0-6 in spring and fall(Sept-May), and in rows 0-10 in winter (Dec-Feb). (Others have remarked on Iceberg seasonality/northernness more generally).
Icebergs are not particularly high probability (<10%) per hex, but would add up if far enough north. Since this voyage will occur in summer, we don’t have to worry about icebergs unless taking a significant detour to the north.
Merfolk:
Merfolk: Do 0 dmg a lot of time, though unlike abstracapplic I am not convinced it is exactly half. Exponentialish? decline if they do do damage, which can go to high values. About ~2.6 average damage for Galleons, ~1.7 for Barquentines, ~6 for Carracks, ~4 for Dhows.
As noted by abstractapplic Merfolk have two zones.
Most Merfolk reports form a giant triangularish donut centered around the northeast corner of J8. The donut looks like it should include F7 and L11, but there are no reports from there, and looks like it should not include O5, but O5 does have one Merfolk report. In the case of F7 this is probably just chance, since it’s not visited a lot. All other reports are in another Merfolk zone southwest of Westengard. The giant donut occupies most of the center of the map and is hard to avoid, so should be analyzed further.
Merfolk have a ~9% probability per hex for Galleons, ~2.7% for Barquentines, and ~1.8% for Carrack/Dhow.
I have also noticed that relative to the low popularity of these hexes, Merfolk are significantly more likely to be encountered in the southwest region. I have not checked if this is connected with the ship type stats, but I will leave this for now since we don’t need to go to that region.
Wyrd Macjick Fyre:
Wyrd Majick Fyre: high damage, mostly 7 or less, but with tail (exponentialish?) 4-5 average for all except Dhow, which gets ~1.2.
As others have reported, Wyrd Majick Fyre mostly occurs around J8 (almost but not quite aligned with the Merfolk donut hole), with a few random-looking other instances. It is a >10% encounter in these hexes making them important to avoid for non-Dhows even if they did not also have Reefs (which they do).
Pirates:
Pirates: as abstractapplic noted does not do 1 dmg (but does do zero, very often, or 2d3? but with a long tail. ~3.1 average for Galleons/barquentine, ~4.4 for Dhow and ~5.29 for Carrack.
As measure noted, Galleons receive more pirate attacks. Per-hex encounter probability of ~12% for Galleons and ~4% for everything else. Todo: check to see if this depends e.g. on mission type.
Storm:
Storm: usually 0-7dmg. some tail. 2.5-2.6 average damage. Around ~7% chance per hex regardless of ship type.
Sharks:
Sharks: as abbstractapplic noted, dmg is consistent with min(2d4)-1. As with Pirates, Per-hex encounter probability of ~12% for Galleons and ~4% for everything else. While not as damaging as Pirates, adds a reason to avoid Galleons.
Harpies:
Harpies: Galleons and Barquentines seem to take 0 dmg ²⁄₃ of the time, and 1-2 damage ¹⁄₃ of the time. Carracks and Dhows seem to take 0 dmg ¹⁄₃ of the time, and 1-4 dmg ²⁄₃ of the time. So, theoretically 0.5 average for Galleons/Barquentines and 1.7 average for Carracks/Dhows
Per-hex encounter probability is ~9% for Galleons and ~4% for others, but per-hex damage from Harpies is still less for Galleons than for Carracks and Barquentines.
Dragon:
Dragon: long tail in damage; peaks later for Carracks/Dhows; ~3.4 average for Barquentine, ~3.7 Galleon, ~5 for Dhow, ~7.5 for Carracks.
Per-hex encounter probability is ~2% for Galleons, ~0.06% for barquentines, ~0.3% for Carracks and Dhows. In terms of average damage per hex the extra encounter probability for Galleons more than makes up for them taking less damage per event than Carracks and Dhows.
Route and ship selection (analysis):
Looking at the encounter types that are initally seem location-independent, we have an average per hex movement cost in damage points, by ship type, of:
Ship/encounter| Pirates | Storm | Sharks | Harpies | Dragon | Total
Galleon | 0.36 | 0.17 | 0.11 | 0.039 | 0.073 | 0.75
Carrack | 0.19 | 0.18 | 0.34 | 0.069 | 0.021 | 0.50
Barquentine | 0.11 | 0.18 | 0.034 | 0.021 | 0.019 | 0.37
Dhow | 0.16 | 0.16 | 0.036 | 0.058 | 0.017 | 0.44
Taking into account ship hp, the Carrack looks the best here, with ~60 hexes of movement.
We are also likely going to go into Merfolk territory though, which adds an additional cost:
Galleon: 0.24
Carrack: 0.11
Barquentine: 0.044
Dhow: 0.072
It’s looking a lot more even here between Carrack/Barquentine, but still slightly favouring Carrack. Since not all of the trip will be in Merfolk territory, might as well go for the Carrack?
One other thing—this cost assumes uniformity of Merfolk, though I actually think the southwest merfolk are more aggressive. Should adjust to account for that later.
We also want to minimize chance of sinking, not damage to be repaired in port. If confident average damage will be tolerable, we might want to reduce long tails rather than average damage. This could favour the Barquentine.
Dhow has less chance of hitting Reefs. Going to the east target, we can take a shortcut through Reefs and might want to consider a Dhow for that.
Additional dmg per Reef hex (v. non-Reef):
Galleon: 0.79
Carrack: 0.76
Barquentine: 0.68
Dhow: 0.17
Going to the West target, we might want to take a shortcut through Kraken territory, for which a Barquentine might be more suitable than a Carrack.
Additional damage per Kraken hex (v. non-Kraken):
Galleon: 0.99
Carrack: 2.04
Barquentine: 0.95
Dhow 1.59
We also might want to avoid E7, for which we have no data. There be dragons. I mean … in-universe hypothetical squared dragons.
Also, early on I noted down some hexes where >1/5 of ships passing through were destroyed. They include some hexes which should not be especially dangerous from the above info, but this could just be that the routes also pass through dangerous hexes. Anyway, something to look at with further analysis, and maybe avoid if not costly to do so.
With all the above in mind, candidate routes and other info messily drawn on the map:
EditL map deleted and moved to imgur since it wasn’t being spoiler properly
imgur link
When counting hexes, I don’t count the port since these seem safe from the data.
For the west target:
Route A is the obvious choice taking all the above at face value. With a return trip, it will involve 27 hexes, of which 22 are Merfolk hexes and 1 Kraken.
Route B avoids the unknowns of E7. It’s the same overall length including Merfolk length as Route A, but has 5 Kraken hexes on the round trip.
Route C also expensively avoids E7. It’s 37 hexes, of which 20 are Merfolk hexes and 1 Kraken. No way that’s going to be worth it.
I also added route G later which minimizes distance (and avoids E7) at the cost of additional Kraken hexes. 25 hexes, of which 20 Merfolk and 9 Kraken.
All of these routes involve >1/5 destroyed hexes, but I’m not prioritizing avoiding these super hard atm on the theory that these hexes will turn out to just be on paths that go through other dangerous stuff or are long.
For the east target:
Route D avoids reefs, but is long and goes through Merfolk territory. It also goes through some >1/5 destroyed hexes. 21 hexes round trip, of which 18 Merfolk and 1 Kraken.
Route E takes 2 reefs to shortcut. 19 hexes round trip, or which 4 reefs and 1 Kraken.
Route F takes 3 reefs to shorten the path a bit more. 17 hexes round trip, of which 6 reefs and 1 Kraken. This is the shortest possible path given the constraint that you can’t go across land.
So, expected damage for each route :
route/ship type | Galleon | Carrack | Barquentine | Dhow
Route A (west) | 26.5 | 17.9 | 11.8 | 15.0
Route B (west) | 30.5 | 26.0 | 15.6 | 21.3
Route C (west) | 33.5 | 22.6 | 15.4 | 19.2
Route G (west) | 32.4 | 33.0 | 18.6 | 26.6
Route D (east) | 21.1 | 14.4 | 9.5 | 12.0
Route E (east) | 18.4 | 14.6 | 10.7 | 10.6
Route F (east) | 18.4 | 15.1 | 11.3 | 10.0
Route A looks so much better than the other western choices that I am willing to have the sailors brave the squared dragons. Barquentine looking like the best choice even with only 20 hp.
For the eastern target, Route D looks good with either a Barquentine or a Carrack, or Route F with a Dhow. Some considerations: Route D does go through >1/5 destroyed hexes, so I should try to find out if that really is a problem. On the other hand, the Dhow has low chance to hit a Reef but not low damage if it does get hit—high variance is risky. On balance, I pick the Barquentine on D for now.
Current route and ship choice:
So, for now I pick:
“The Bloody Diamond, a Barquentine captained by Angus MacDougal” on Route A (Q6-P6-O6-N6-M5-L5-K5-J5-I5-H5-G5-F5-F6-E7-E8) and back by the same route.
“The Saucy Heart, a Barquentine captained by Erin Aubrey” on Route D (Q6-P6-O6-N6-M7-M8-L8-K9-K10-K11-L12-L13) and back by the same route.
Comparing to others’ selections:
My selected routes A and D are the same ones chosen by abstractapplic, but I use two Barquentines whereas abstractapplic uses a Galleon and a Barquentine.
Yonge selected Route F to go to the east target and for the west first selected something that looks like it should be equivalent to my Route B, in terms of length and types of hexes it goes in, but at the bottom of the comment changes it (why?) to add some additional dilly-dallying in Kraken territory. Yonge chose a Galleon and a Carrack.
measure picked two Dhows (unconventional!), and sent one of them on a route equivalent-seeming to Route E, which looks sensible to me, but the other one is going to the west target starting out at (up to the last hex) the same route (so, super long route), and is a Dhow cutting through Kraken territory, which looks not so sensible.
todo:
Look at Merfolk donut only, check to see if that affects merfolk stats
Look to see if expected damage can reasonably account for observed losses, check where excess losses are occuring (is Jemist right that there are unexpected losses?)
check to see if Captains affect anything
check to see if time docked affects anything
check to see if voyage purpose affects anything
additional remarks:
As Yonge notes, there are 19 encounters not on the planned route I did not see a pattern and attribute this to noise in the data. Note that it is possible that, even if something was displaced by noise, it would still end up on the planned route. I am inclined to attribute the Merfolk event on O5 to such noise, the event probably having really occurred on N5, which was also on the ship’s route.

simon 10 Jan 2024 15:49 UTC
15 points
5
on: Saving the world sucks
I don’t want to tile the universe with hedonium.
Good! I don’t want to be replaced with tiled hedonium either.
Perhaps some of the issue might be with perceptions of what is supposed to be “good” not matching your own values.

simon 15 Jun 2023 2:53 UTC
14 points
3
in reply to: RatsWrongAboutUAP’s comment on: UFO Betting: Put Up or Shut Up
This is to publicly confirm that I have received approximately $2000 USD equivalent.
Unless you dispute what timing is appropriate for the knowledge cutoff, I will consider the knowledge cutoff for the paradigm-shattering UAP-related revelations for me to send you $100k USD to be 11:59pm, June 14, 2028 UTC time.

simon 6 Apr 2021 9:32 UTC
14 points
in reply to: mike_hawke’s comment on: I’m from a parallel Earth with much higher coordination: AMA
I think the idea is that the different Earths have the same landmasses, and in particular the landmass corresponding to our Japan has in Dath Ilan an exile-location function that is vaguely analogous to Australia in the past in our timeline.

simon 26 May 2023 7:30 UTC
13 points
−11
on: Book Review: How Minds Change
This post has a lot of great points.
But one thing that mars it for me to some extent is the discussion of OpenAI.
To me, the criticism of OpenAI feels like it’s intended as a tribal signifier, like “hey look, I am of the tribe that is against OpenAI”.
Now maybe that’s unfair and you had no intention of anything like that and my vibe detection is off, but if I get that impression, I think it’s reasonably likely that OpenAI decisionmakers would get the same impression, and I presume that’s exactly what you don’t want based on the rest of the post.
And even leaving aside practical considerations, I don’t think OpenAI warrants being treated as the leading example of rationality failure.
First, I am not convinced that the alternative to OpenAI existing is the absence of a capabilities race. I think, in contrast, that a capabilities race was inevitable and that the fact that the leading AI lab has as decent a plan as it does is potentially a major win by the rationality community.
Also, while OpenAI’s plans so far look inadequate, to me they look considerably more sane than MIRI’s proposal to attempt a pivotal act with non-human-values-aligned AI. There’s also potential for OpenAI’s plans to be improved as more knowledge on mitigating AI risk is obtained, which is helped by their relatively serious attitude as compared to, for example, ~~Google after their recent reorganization.~~ Meta.
And while OpenAI is creating a race dynamic by getting ahead, IMO MIRI’s pivotal act plan would be creating a far worse race dynamic if they were showing signs of being able to pull it off anytime soon.
I know many others don’t disagree, but I think that there is enough of a case for OpenAI being less bad than potential alternatives to feel using it as if it were an uncontroversial bad thing detracts from the post.

simon 13 Jan 2022 20:54 UTC
13 points
on: Omicron Post #14
Not sure what reasons the Wall Street Journal gave (haven’t seen the article), but they’re probably right. Faster spread doesn’t change the total number of cases all that much v. slow spread but it does reduce the average number of sequential infections each lineage experiences before the disease burns out, thus the evolutionary distance possible for the disease to achieve.

simon 14 Apr 2021 2:34 UTC
13 points
on: D&D.Sci April 2021: Voyages of the Gray Swan
I haven’t looked much into dependence on time and direction yet, apart from
noticing that the pirates decline in relative frequency
but, I would like some clarification about whether the 100gp budget is for a single set of interventions we’ll use on all trips, or is spent each time on a (potentially varying) arrangement. (edit: I see abstractapplic already responded to such a question from Measure: it’s a single set obtained up front and not changed.)
My current thoughts:
From looking at the probability distributions, I mostly agree with gjm; my current recommendations are the same as GuySrinivasan’s and Measure’s.
Demon Whale distribution scares the crap out of me, and I probably panic buy all 20 oars allowed. I mostly agree that the distribution looks like it could be peaking near 100% damage, but I am not at all confident of this and think it could be consistent with something growing a lot bigger beyond the cutoff. I expect 250-1000 of the destroyed vessels to have been destroyed by demon whales. While it looks like it will be in control before needing 20 oars, I am uncertain enough to value oars pretty high. Budget spent = 20gp.
Merpeople have a very wierd looking distribution. It doesn’t seem to tailing off at the end, and so (like gjm) I am very uncertain about what happens after the 100% cutoff. I think (like GuySrinivasan) there’s a possibility that it has a multimodal distribution with another peak beyond 100% (not saying bimodal since there even looks like there might be a small peak around 25% distorting the main peak of about 50%, though this could very easily be random). I figure merpeople are responsible for around 200 (assuming no extra peak) or potentially vastly more (assuming an extra peak) of the losses. Merpeople are potentially another solid choice for mitigation imo, unless removing them from the encounter table puts in demon whales in as the substitute. Budget spent = 45+20=65gp
I do not agree with gjm that only about 1% of crabmonster encounters are terminal. The distribution seems to be tailing off very slowly, visually more or less consistent with a triangle-shaped distribution. A simple linear extrapolation would suggest a few percent which I would take as a lower bound. But it might not be linear, but slowing in how it tapers off, so it might be much much more than this. For all I know (apart from the finite number of sinkings) it might not even sum to a finite value. On the other hand, we only really care about crabmonster attacks that do less than 200% damage, since the only relevant intervention reduces damage by 50%. I estimate that between about 50 and about 250 of the destroyed vessels to have been destroyed by the relevant part of the crabmonster damage distribution, with potentially unlimited numbers destroyed by crabmonsters outside that range. Despite the expected max of about 250 mitigatable losses I consider arming carpenters a pretty solid choice for 20gp. Budget spent = 20+65=85gp.
Nessie looks like a pretty straightforward distribution where I assume about 10-20 losses are from the bit of the distribution we can’t see. A single cannon (which is all that we can afford after the other purchases) should suffice. Budget spent = 10+85=95gp.
Conclusion: 20 oars (20gp) + pay off merpeople (45gp) + arm carpenters (20gp) + one cannon (10gp), total 95 gp spent. This is the same plan previously recommended by GuySrinivasan and Measure.
Nothing else looks like it can kill us, unless e.g. some bimodal distribution has one of its humps located entirely within the >=100% zone.
However, stepping out of the pure data analysis and into reasoning about the fantasy world, it seems strange that pirates would bother to attack us if they only ever do 64% damage. They are intelligent, after all. Maybe they run away if in a hard fight, not sticking around to do more damage than that, and take over the ship entirely if they win? I might consider a second cannon (also provides extra insurance in case nessie’s distribution extends further than expected). Dropping 3 oars would provide the funds, and will probably still be enough for the demon whale.
(and...also anticipated by Measure on the pirate theory).

simon 13 Oct 2020 0:47 UTC
13 points
on: Have the lockdowns been worth it?

By ‘lockdown’ we refer to the thing that the US, UK and China have been doing, and what Sweden and Italy didn’t

It seemed to me that Italy (after the initially hit areas were pretty much saturated) did a much harder lockdown than much of the US has been doing.

https://en.wikipedia.org/wiki/COVID-19_pandemic_lockdown_in_Italy

simon 4 Jan 2015 12:15 UTC
12 points
in reply to: Baisius’s comment on: 2014 Survey Results
There isn’t necessarily any problem with a small positive correlation between masculinity and femininity. The abstract of what I think is the original paper (I couldn’t find an ungated version) says that “The dimensions of masculinity and femininity are empirically and logically independent.”

simon 3 Dec 2023 2:56 UTC
11 points
9
on: Taking Into Account Sentient Non-Humans in AI Ambitious Value Learning: Sentientist Coherent Extrapolated Volition
A thought experiment: the mildly xenophobic large alien civilization.
Imagine at some future time we encounter an expanding grabby aliens civilization. The civilization is much older and larger than ours, but cooperates poorly. Their individual members tend to have a mild distaste for the existence of aliens (such as us). It isn’t that severe, but there are very many of them, so their total suffering at our existence and wish for us to die outweighs our own suffering if our AI killed us, and our own will to live.
They aren’t going to kill us directly, because they co-operate poorly, individually don’t care all that much, and defense has the advantage over offense.
But, in this case, the AI programmed as you proposed will kill us once if finds out about these mildly xenophobic aliens. How do you feel about that? And do you feel that, if I don’t want to be killed in this scenario, that my opposition is unjustified?

simon 1 Mar 2023 21:58 UTC
11 points
3
on: To MIRI-style folk, you can’t simulate the universe from the beginning
Can you link specifically an example of what you are criticizing here? Which formalisms require simulating the universe from the beginning to identify a specific being? (as opposed to, e.g. some probabilistic approximation?)

(trying to think of examples, maybe Aumann’s theorem qualifies? But I’m not sure, and not sure what else does?)

simon 8 May 2022 5:43 UTC
11 points
on: D&D.Sci Divination: Nine Black Doves
Thanks for setting up the problem aphyer.
Some observations on plague:
The 9 provinces in the dataset have the following adjacency graph:
8 of the 9 provinces are arranged in a loop:
Britannia <-> Hispania <-> Italia <-> Grecia <-> Anatolia <-> Parthia <-> Scythia <-> Scandinavia <-> Britannia
The last province, Germania, is adjacent to Italia and Scandinavia.
Plague occurs in epidemics of one or more instances of plague. The epidemic usually originates in a single province, but sometimes in multiple. Each year, any provinces adjacent to a province with plague in the previous year have a fairly high chance of getting plague. A province is immune to plague in the 5 years following an instance of plague.
In some cases, a province that is not adjacent (according to my graph described above) to a province that has plague gets plague during an epidemic. This is sufficiently rare that I am assuming that this is due to independent originations rather than due to additional connections on the graph. I can’t tell from the dataset if there is any separate rule (beyond the 5 year immunity rule) preventing a province from receiving plague twice from the same origination. There is an apparent case of this in the dataset, but that is sufficiently rare that it could be due to an independent origination that happened to occur adjacent to a plague province.
Plague seems to have some association with Earthquakes and Black Doves (todo: investigate this).
In the previous year, a Plague started in Anatolia. This makes plague protection in the adjacent province of Grecia look prudent this year.
There were also Earthquakes in Britannia, Italia and Parthia, as well as the black doves (everywhere except Germania) as mentioned in the problem description. Todo: investigate if this makes independent plague originations look likely enough to get plague protection for other provinces than Grecia. On average given base rates, plague prevention is not particularly cost effective.
Miscellaneous other observations:
Pillaging follows the same adjacency graph as plague (pillaging a target empire pillages the province of the target that the pillaging empire is adjacent to, which is always unique). All empire pairs except Britain/Persia can pillage each other.
Pillaging an empire (afflicting target province, from “jump-off” province in pillaging empire adjacent to target province) seems to be associated with:
- an increased chance of a two-headed baby the same year in the jump-off province
- an increased chance of fire the next year in the jump-off province.
A two headed baby is also associated with fire in the same province the next year (todo—untangle which aspects of this triangle are primary)
Wolves howling at the sun is also associated with fire in the next year in the same province.
Note, Anatolia was pillaged by Romans last year. So, there may be an increased chance of fire in Grecia this year.
Famine is associated with Earthquake or Fire in the previous year. There was no disaster however before the Famine in Italia which occurred last year. There was an earthquake in Italia last year, making famine relief there potentially more attractive.
Geese Flying Backwards is associated with increased chance of Famine in the same province in the next year. (Note: this was observed last year in Hispania).
So, initially at least, Famine relief in Italia and Hispania and Fire protection in Grecia look like possible choices, along with plague prevention in Grecia.

simon 11 Jul 2020 15:48 UTC
11 points
in reply to: jimv’s comment on: Maximal Ventilation
I think it’s just that 1 mile per hour = 88 feet per minute which is close to 90.

simon 3 Dec 2023 17:51 UTC
10 points
9
in reply to: Adrià Moret’s comment on: Taking Into Account Sentient Non-Humans in AI Ambitious Value Learning: Sentientist Coherent Extrapolated Volition
If your view is that you only have reasons to include those, whom you have instrumental reasons to include, on your view: the members of an AGI lab that developed ASI ought to include only themselves if they believe (in expectation) that they can successfully do so. This view is implausible, it is implausible that this is what they would have most moral reasons to do.
I note that not everyone considers that implausible, for example Tamsin Leake’s QACI takes this view.
I disagree with both Tamsin Leake and with you: I think that humans-only, but only humans, makes the most sense. But for concrete reasons, not for free-floating moral reasons.
I was writing the following as a response to NicholasKees’ comment, but I think it belongs better as a response here:
...imagine you are in a mob in such a “tyranny of the mob” kind of situation, with mob-CEV. For the time being, imagine a small mob.
You tell the other mob members: “we should expand the franchise/function to other people not in our mob”.
OK, should the other mob members agree?
- maybe they agree with you that it is right that the function should be expanded to other humans. In which case mob-CEV would do it automatically.
- Or they don’t agree. And still don’t agree after full consideration/extrapolation.
If they don’t agree, what do you do? Ask Total-Utility-God to strike them down for disobeying the One True Morality?
At this point you are stuck, if the mob-CEV AI has made the mob untouchable to entities outside it.
But there is something you could have done earlier. Earlier, you could have allied with other humans outside of the mob, to pressure the would-be-mob members to pre-commit to not excluding other humans.
And in doing so, you might have insisted on including all humans, not specifically the humans you were explicitly allying with, even if you didn’t directly care about everyone, because:
- the ally group might shift over time, or people outside the ally group might make their own demands
- if the franchise is not set to a solid Schelling point (like all humans) then people currently inside might still worry about the lines being shifted to exclude them.
Thus, you include the Sentinelese, not because you’re worried about them coming over to demand to be included, but because if you draw the line to exclude them then it becomes more ambiguous where the line should be drawn, and relatively low (but non-zero) influence members of the coalition might be worried about also being excluded. And, as fellow humans, it is probably relatively low cost to include them—they’re unlikely to have wildly divergent values or be utility monsters etc.
You might ask, is it not also a solid Schelling point to include all entities whatsoever?
First, not really, we don’t have good definitions of “all sentient beings”, not nearly as good as “all humans”. It might be different if, e.g., we had time travel, such that we would also have to worry about intermediate evolutionary steps between humans and non-human-animals, but we don’t.
In the future, we will have more ambiguous cases, but CEV can handle it. If someone wants to modify themselves into a utility monster, maybe we would want to let them do so, but discount their weighting in CEV to a more normal level when they do it.
And second, it is not costless to expand the franchise. If you allow non-humans preemptively you are opening yourself up to, as an example, the xenophobic aliens scenario, but also potentially who-knows-what other dangerous situations since entities could have arbitrary values.
And that’s why expanding the franchise to all humans makes sense, even if individuals don’t care about other humans that much, but expanding to all sentients does not, even if people do care about other sentients.
In response to the rest of your comment:
If you want to argue that s-risks would be prevented for certain, please address the object-level arguments I present.
If humans would want to prevent s-risks, then they would be prevented. If humans would not want to prevent s-risks, they would not be prevented.
If you want to argue that the occurrence of s-risks would not be bad, you want to argue for a particular view in normative and practical ethics.
You’re the one arguing that people should override their actual values, and instead of programming an AI to follow their actual values, do something else! Without even an instrumental reason to do so (other than alleged moral considerations that aren’t in their actual values, but coming from some other magical direction)!
Asking someone to do something that isn’t in their values, without giving them instrumental reasons to do so, makes no sense.
It is you who needs a strong meta-ethical case for that. It shouldn’t be the objector who has to justify not overriding their values!

simon 3 Dec 2023 2:23 UTC
10 points
9
in reply to: Adrià Moret’s comment on: Taking Into Account Sentient Non-Humans in AI Ambitious Value Learning: Sentientist Coherent Extrapolated Volition
Thanks for the reply.
We don’t work together with animals—we act towards them, generously or not.
That’s key because, unlike for other humans, we don’t have an instrumental reason to include them in the programmed value calculation, and to precommit to doing so, etc. For animals, it’s more of a terminal goal. But if that terminal goal is a human value, it’s represented in CEV. So where does this terminal goal over and above human values come from?
Regarding 2:
There is (at least) a non-negligible probability that an adequate implementation of the standard CEV proposal results in the ASI causing or allowing the occurrence of risks of astronomical suffering (s-risks).
You don’t justify why this is a bad thing over and above human values as represented in CEV.
Regarding 2.1:
The normal CEV proposal, like CEO-CEV and men-CEV, excludes a subset of moral patients from the extrapolation base.
You just assume it, that the concept of “moral patients” exists and includes non-humans. Note, to validly claim that CEV is insufficient, it’s not enough to say that human values include caring for animals—it has to be something independent of or at least beyond human values. But what?
Regarding 4.2:
However, as seen above, it is not the case that there are no reasons to include sentient non-humans since they too can be positively or negatively affected in morally relevant ways by being included in the extrapolation base or not.
Again, existence and application of the “moral relevance” concept over and above human values just assumed, not justified.
regarding 3.2:
At any given point in time t, the ASI should take those actions that would in expectation most fulfil the coherent extrapolated volition of all sentient beings that exist in t.
Good, by focusing at the particular time at least you aren’t guaranteeing that the AI will replace us with utility monsters. But if utility monsters do come to exist or be found (e.g. utility monster aliens) for whatever reason, the AI will still side with them, because:
Contrary to what seems to be the case in the standard CEV proposal, the interests of future not-yet-existing sentient beings, once they exist, would not be taken into account merely to the extent to which the extrapolated volitions of currently existing individuals desire to do so.
Also, I have to remark on:
Finally, it should also be noted that this proposal of SCEV (as CEV) is not intended as a realist theory of morality, it is not a description of the metaphysical nature of what constitutes the ‘good’. I am not proposing a metaethical theory but merely what would be the most morally desirable ambitious value learning proposal for an ASI.
You assert your approach is “the most morally desirable” while disclaiming moral realism. So where does that “most morally desirable” come from?
And in response to your comment:
Yes, but (as I argue in 2.1 and 2.2) there are strong reasons to include all sentient beings. And (to my knowledge) there are no good reasons to support any religion.
The “reasons” are simply unjustified assumptions, like “moral relevance” existing (independent of our values, game theoretic considerations including pre-commitments, etc.) (and yes, you don’t explicitly say it exists independent of those things in so many words, but your argument doesn’t hold unless they do exist independently).

simon 10 Jul 2022 7:12 UTC
10 points
0
on: Ars D&D.sci: Mysteries of Mana
Thanks for making this!
Initially I’ve been looking at the mana data (which I’ve already figured out according to the story, but is only presented in the data in oracular fashion):
Typically, all mana strengths stay the same for some number of duels, then multiple mana strengths change. The 232501 total duels can be separated into 14475 groups of duels where the mana strengths stay the same in the group. The most common number of duels in such a group is 1 (871 examples) but the number of duels can go as high as 144. Since 1 duel before a change is common, probably 0 is as well meaning we are likely missing a bunch of mana data points which do not have a corresponding duel.
All mana strengths are anticorrelated, but fire and water are especially anticorrelated and so are earth and air.
Graphing fire+water and earth+air (by group number) shows what visually looks like slowly varying sine waves about 120 degrees out of phase. These sums are in the range from 34 to 66. The individual mana types vary much more frequently and look random. So, I guess that the sine waves are deterministic, then an allocation into individual mana types is made in a relatively random manner.
But what about light? It seems symmetrical with the other mana types apart from not being in a group… so a natural hypothesis is that it is part of a group with another non-shown mana type to complete the symmetry, let’s call the hypothesized mana type “dark”.
Since all mana types average to about 25, I set “dark” to 150 minus the sum of all known mana types. This indeed produced the expected effect of a third slowly varying sine wave like shape for light+dark 120 degrees out of phase with the other two. The statistics for “dark” mana do look similar to those for known types, except that while known types vary from 7 to 53, “dark” as calculated varies from 6 to 54. I am inclined to put this down to rounding, though haven’t checked the actual instances of 6 and 54 to verify if this is likely.
Conclusion on mana strengths: Mana strength including a hypothetical “dark” mana type has a total budget of 150 (though the sum will not necessarily be this due to rounding) allocated between fire+water, air+earth and light+dark sine waves 120 degrees out of phase (each sine wave varying from 34 to 66, average 50), and some kind of random-looking allocation occurs between the mana types of each wave, so the individual mana types then vary from 7 to 53 (average 25). Then, different numbers of duels occur at each mana strength data point, which could be 0 resulting in some mana strength data points not showing up.

simon

No, re­ally, it pre­dicts next to­kens.

No, really, it predicts next tokens.