harfe

• harfe 19 Apr 2024 20:19 UTC

  1 point

  0

  on: What is the best way to talk about prob­a­bil­ities you ex­pect to change with ev­i­dence/​ex­per­i­ments?

  A lot of the probabilities we talk about are probabilities we expect to change with evidence. If we flip a coin, our p(heads) changes after we observe the result of the flipped coin. My p(rain today) changes after I look into the sky and see clouds. In my view, there is nothing special in that regard for your p(doom). Uncertainty is in the mind, not in reality.

  However, how you expect your p(doom) to change depending on facts or observation is useful information and it can be useful to convey that information. Some options that come to mind:

  1. describe a model: If your p(doom) estimate is the result of a model consisting of other variables, just describing this model is useful information about your state of knowledge, even if that model is only approximate. This seems to come closest to your actual situation.

  2. describe your probability distribution of your p(doom) in 1 year (or another time frame): You could say that you think there is a 25% chance that your p(doom) in 1 year is between 10% and 30%. Or give other information about that distribution. Note: your current p(doom) should be the mean of your p(doom) in 1 year.

  3. describe your probability distribution of your p(doom) after a hypothetical month of working on a better p(doom) estimate: You could say that if you were to work hard for a month on investigating p(doom), you think there is a 25% chance that your p(doom) after that month is between 10% and 30%. This is similar to 2., but imo a bit more informative. Again, your p(doom) should be the mean of your p(doom) after a hypothetical month of investigation, even if you don’t actually do that investigation.

• harfe 10 Feb 2024 16:47 UTC

  1 point

  in reply to: Yitz’s comment on: Yitz’s Shortform

  This sounds like https://​​www.super-linear.org/​​trumanprize. It seems like it is run by Nonlinear and not FTX.

• harfe 10 Jan 2024 19:25 UTC

  1 point

  on: Ba­sic In­framea­sure Theory

  I think Proposition 1 is false as stated because the resulting functional $f^{+}:M^{\pm }\left(X\right)\oplus R\to R$ is not always continuous (wrt the KR-metric). The function $f:[0,1]\to [0,1]$, $x\mapsto x^{1/3}$ with $X=[0,1]$ should be a counterexample. However, the non-continuous functional $f^{+}$ should still be continuous on the set of sa-measures.

  Another thing: the space of measures $M^{\pm }\left(X\right)$ is claimed to be a Banach space with the KR-norm (in the notation section). Afaik this is not true, while the space is a Banach space with the TV-norm, with the KR-metric/​norm it should not be complete and is merely a normed vector space. Also the claim (in “Basic concepts”) that $M^{\pm }\left(X\right)$ is the dual space of $C\left(X\right)$ is only true if equipped with TV-norm, not with KR-metric.

  Another nitpick: in Theorem 5, the type of $h$ in the assumption is probably meant to be $C\left(X\right)\to R\cup \{-\infty \}$, instead of $C\left(X\right)\to R$.

• harfe 9 Jan 2024 16:21 UTC

  LW: 3 AF: 3

  0

  AF

  on: The Learn­ing-The­o­retic Agenda: Sta­tus 2023

  Regarding direction 17: There might be some potential drawbacks to ADAM. I think its possible that some very agentic programs have relatively low $g$ score. This is due to explicit optimization algorithms being low complexity.

  (Disclaimer: the following argument is not a proof, and appeals to some heuristics/​etc. We fix $M=M_0$ for these considerations too.) Consider an utility function $\hat{U}$. Further, consider a computable approximation of the optimal policy (AIXI that explicitly optimizes for $\hat{U}$) and has an approximation parameter n (this could be AIXI-tl, plus some approximation of $\hat{U}$; higher $n$ is better approximation). We will call this approximation of the optimal policy ${\pi }_n^{\hat{U}}$. This approximation algorithm has complexity $K\left({\pi }_n^{\hat{U}}\right)=C+K\left(\hat{U}\right)+K\left(n\right)$, where $C>0$ is a constant needed to describe the general algorithm (this should not be too large).

  We can get better approximation by using a quickly growing function, such as the Ackermann function with $n=A(k,k)$. Then we have $K\left({\pi }_{A(k,k)}^{\hat{U}}\right)=C+K\left(\hat{U}\right)+K\left(A\right(k,k\left)\right)\le C+K\left(\hat{U}\right)+\log \left(k\right)$.

  What is the $g$ score of this policy? We have $g\left({\pi }_{A(k,k)}^{\hat{U}}\right)={max}_U\left({min}_{{\pi }^{\prime }:\dots }K\right({\pi }^{\prime })-K(U\left)\right)$. Let $\overline{U}$ be maximal in this expression. If $K\left(\overline{U}\right)\ge K\left(\hat{U}\right)-C$, then $$g\left({\pi }_{A(k,k)}^{\hat{U}}\right)=\underset{{\pi }^{\prime }:E_{{\zeta }_{M_0}{\pi }^{\prime }}\left(\overline{U}\right)\ge E_{{\zeta }_{M_0}{\pi }_{A(k,k)}^{\hat{U}}}\left(\overline{U}\right)}{min}K\left({\pi }^{\prime }\right)-K\left(\overline{U}\right)\le K\left({\pi }_{A(k,k)}^{\hat{U}}\right)-K\left(\hat{U}\right)+C\le 2C\log \left(k\right)$$.

  For the other case, let us assume that if $K\left(\overline{U}\right)<K\left(\hat{U}\right)-C$, the policy ${\pi }_{A(k,k)}^{\overline{U}}$ is at least as good at maximizing $\overline{U}$ than ${\pi }_{A(k,k)}^{\hat{U})}$. Then, we have $$g\left({\pi }_{A(k,k)}^{\hat{U}}\right)=\underset{{\pi }^{\prime }:E_{{\zeta }_{M_0}{\pi }^{\prime }}\left(\overline{U}\right)\ge E_{{\zeta }_{M_0}{\pi }_{A(k,k)}^{\hat{U}}}\left(\overline{U}\right)}{min}K\left({\pi }^{\prime }\right)-K\left(\overline{U}\right)\le K\left({\pi }_{A(k,k)}^{\overline{U}}\right)-K\left(\overline{U}\right))\le C+\log (k)$$.

  I don’t think that the assumption ($({\pi }_{A(k,k)}^{\overline{U}}$ maximizes $barU$ better than $({\pi }_{A(k,k)}^{\hat{U}}$) is true for all $\hat{U}$ and $k$, but plausibly we can select $\hat{U}$ such that this is the case (exceptions, if they exist, would be a bit weird, and if ADAM working well due to these weird exceptions feels a bit disappointing to me). A thing that is not captured by approximations such as AIXI-tl are programs that halt but have insane runtime (longer than $A(k,k)$). Again, it would feel weird to me if ADAM sort of works because of low-complexity extremely-long-running halting programs.

  To summarize, maybe there exist policies ${\pi }_{A(k,k)}^{\hat{U}}$ which strongly optimize a non-trivial utility function $\hat{U}$ with approximation parameter $n=A(k,k)$, but where $g\left({\pi }_{A(k,k)}^{\hat{U}}\right)\le 2C+\log \left(k\right)$ is relatively small.

• harfe 21 Dec 2023 11:59 UTC

  3 points

  0

  on: How Would an Utopia-Max­i­mizer Look Like?

  I think the “deontological preferences are isomorphic to utility functions” is wrong as presented.

  Firts, the formula has issues with dividing by zero and not summing probabilities to one (and re-using variable $x$ as a local variable in the sum). So you probably meant something like $$P\left(x\right)=\frac{e^{u\left(x\right)}}{{\sum }_{y\in X}{e}^{u\left(y\right)}}.$$ Even then, I dont think this describes any isomorphism of deontological preferences to utility functions.

  • Utility functions are invariant when multiplied with a positive constant. This is not reflected in the formula.

  • utility maximizers usually take the action with the best utility with probability $1$, rather than using different probabilities for different utilities.

  • modelling deontological constraints as probability distributions doesnt seem right to me. Let’s say I decide between drinking green tea and black tea, and neither of those violate any deontological constraints, then assigning some values (which ones?) to P(“I drink green tea”) or P(“I drink black tea”) doesnt describe these deontological constraints well.

  • any behavior can be encoded as utility functions, so finding any isomorphisms to utility functions is usually possible, but not always meaningful.

• harfe 13 Nov 2023 13:12 UTC

  2 points

  0

  in reply to: [deactivated]’s comment on: TESCREAL and so­cial jus­tice: a synthesis

  Some of the downvotes were probably because of the unironic use of the term TESCREAL. This term mixes a bunch of different things together, which makes your writing less clear.

• harfe 16 Oct 2023 15:46 UTC

  1 point

  in reply to: Buck’s comment on: Buck’s Shortform

  Sure, I’d be happy to read a draft

• harfe 15 Oct 2023 22:31 UTC

  LW: 3 AF: 2

  AF

  in reply to: Buck’s comment on: Buck’s Shortform

  I am going to assume that in the code, when calculating p_alice_win_given_not_caught, we do not divide the ${\lambda }_2$ term by two (since this is not that consistent with the description. I am also assuming that $0.25-t_2$ is a typo and $0.5-t_2$ is meant, which would also be more consistent with other stuff). So I am going to assume assume a symmetrical version.

  Here, P(Alice wins) is $e^{-{\lambda }_1{t}_1-{\lambda }_2{t}_2}-e^{-{\lambda }_1/2-{\lambda }_2/2}=:f({\lambda }_1,{\lambda }_2,t_1,t_2)$. Wlog we can assume $0<t<1$ (otherwise Bob will run everything or nothing in shielded mode).

  We claim that $\left(\right({\lambda }_1^{\ast },{\lambda }_2^{\ast }),(t_1^{\ast },t_2^{\ast }\left)\right):=\left(\right({\mu }^{\ast },{\mu }^{\ast }),(t/2,t/2)$ is a (pure) Nash equilibrium, where ${\mu }^{\ast }=\log t/(t-1)$.

  To verify, lets first show that Alice cannot make a better choice if Bob plays $(t_1^{\ast },t_2^{\ast })=(t/2,t/2)$. We have $f({\lambda }_1,{\lambda }_2,t_1^{\ast },t_2^{\ast })=e^{-({\lambda }_1+{\lambda }_2)t/2}-e^{-({\lambda }_1+{\lambda }_2)/2}$. Since this only depends on the sum, we can make the substitution $\mu =({\lambda }_1+{\lambda }_2)/2$. Thus, we want to maximize $\tilde{f}(\mu ,t)=e^{-\mu t}-e^{-\mu }$. We have ${\tilde{f}}^{\prime }(\mu ,t)=-t{e}^{-\mu t}+e^{-\mu }=0$. Rearranging, we get $t{e}^{-\mu t}=e^{-\mu }$. Taking logs, we get $\log t-\mu t=-\mu$. Rearranging, we get $\mu =\log t/(t-1)$. Thus, $\mu ={\mu }^{\ast }$ is the optimal choice. This means, that if Bob sticks to his strategy, Alice cannot do better than $f({\mu }^{\ast },{\mu }^{\ast },t/2,t/2)$.

  Now, lets show that Bob cannot do better. We have $f({\lambda }_1^{\ast },{\lambda }_2^{\ast },t_1,t_2)=e^{-{\mu }^{\ast }{t}_1-{\mu }^{\ast }{t}_2}-e^{-{\mu }^{\ast }/2-{\mu }^{\ast }/2}=e^{-{\mu }^{\ast }t}-e^{-{\mu }^{\ast }}$. This does not depend on $t_1$ and $t_2$ anymore, so any choice of $t_1$ and $t_2$ is optimal if Alice plays $({\mu }^{\ast },{\mu }^{\ast })$.

  (If I picked the wrong version of the question, and you actually want some symmetry: I suspect that the solution will have similarities, or that in some cases the solution can be obtained by rescaling the problem back into a more symmetric form.)

• harfe 13 Oct 2023 17:57 UTC

  1 point

  0

  on: To open-source or to not open-source, that is (an over­sim­plifi­ca­tion of) the ques­tion.

  This article talks a lot about risks from AI. I wish the author would be more specific what kinds of risks they are thinking about. For example, it is unclear which parts are motivated by extinction risks or not. The same goes for the benefits of open-sourcing these models. (note: I haven’t read the reports this article is based on, these might have been more specific)

• harfe 5 Oct 2023 23:06 UTC

  5 points

  4

  on: Prov­ably Safe AI

  Thank you for writing this review.

  The strategy assumes we’ll develop a good set of safety properties that we’re demanding proof of.

  I think this is very important. From skimming the paper it seems that unfortunately the authors do not discuss it much. I imagine that actually formally specifying safety properties is actually a rather difficult step.

  To go with the example of not helping terrorists spread harmful virus: How would you even go about formulating this mathematically? This seems highly non-trivial to me. Do you need to mathematically formulate what exactly are harmful viruses?

  The same holds for Asimov’s three laws of robotics, turning these into actual math or code seems to be quite challenging.

  There’s likely some room for automated systems to figure out what safety humans want, and turn it into rigorous specifications.

  Probably obvious to many, but I’d like to point out that these automated systems themselves need to be sufficiently aligned to humans, while also accomplishing tasks that are difficult for humans to do and probably involve a lot of moral considerations.

• harfe 24 Sep 2023 3:10 UTC

  3 points

  0

  on: Five ne­glected work ar­eas that could re­duce AI risk

  A common response is that “evaluation may be easier than generation”. However, this doesn’t mean evaluation will be easy in absolute terms, or relative to one’s resources for doing it, or that it will depend on the same resources as generation.

  I wonder to what degree this is true for the human-generated alignment ideas that are being submitted LessWrong/​Alignment Forum?

  For mathematical proofs, evaluation is (imo) usually easier than generation: Often, a well-written proof can be evaluated by reading it once, but often the person who wrote up the proof had to consider different approaches and discard a lot of them.

  To what degree does this also hold for alignment research?

• harfe 24 Sep 2023 1:22 UTC

  9 points

  4

  on: The Dick Kick’em Paradox

  The setup violates a fairness condition that has been talked about previously.

  From https://​​arxiv.org/​​pdf/​​1710.05060.pdf, section 9:

  We grant that it is possible to punish agents for using a specific decision proce- dure, or to design one decision problem that punishes an agent for rational behavior in a different decision problem. In those cases, no decision theory is safe. CDT per- forms worse that FDT in the decision problem where agents are punished for using CDT, but that hardly tells us which theory is better for making decisions. [...]

  Yet FDT does appear to be superior to CDT and EDT in all dilemmas where the agent’s beliefs are accurate and the outcome depends only on the agent’s behavior in the dilemma at hand. Informally, we call these sorts of problems “fair problems.” By this standard, Newcomb’s problem is fair; Newcomb’s predictor punishes and rewards agents only based on their actions. [...]

  There is no perfect decision theory for all possible scenarios, but there may be a general-purpose decision theory that matches or outperforms all rivals in fair dilem- mas, if a satisfactory notion of “fairness” can be formalized

• harfe 17 Aug 2023 12:25 UTC

  1 point

  2

  on: If we had known the at­mo­sphere would ignite

  Is the organization who offers the prize supposed to define “alignment” and “AGI” or the person who claims the prize? this is unclear to me from reading your post.

  Defining alignment (sufficiently rigorous so that a formal proof of (im)possibility of alignment is conceivable) is a hard thing! Such formal definitions would be very valuable by themselves (without any proofs). Especially if people widely agree that the definitions capture the important aspects of the problem.

• harfe 10 Jul 2023 15:08 UTC

  9 points

  0

  in reply to: Vanessa Kosoy’s comment on: The Learn­ing-The­o­retic Agenda: Sta­tus 2023

  I think the conjecture is also false in the case that utility functions map from $O^{\omega }$ to $[0,1]$.

  Let us consider the case of $A=\{a_1,a_2\}$ and $O=\{o_1,o_2\}$. We use $U_1\left(o\right)=1-2^{-k}$, where $k$ is the largest integer such that $o$ starts with $o_1^k$ (and $U_1\left(o_1^{\omega }\right)=1$). As for $U_2$, we use $U_2\left(o\right)=1-3^{-k}$, where $k$ is the largest integer such that $o$ starts with $o_1^k$ (and $U_2\left(o_1^{\omega }\right)=1$). Both $U_1$ and $U_2$ are computable, but they are not locally equivalent. Under reasonable assumptions on the Solomonoff prior, the policy $\pi$ that always picks action $a_1$ is the optimal policy for both $U_1$ and $U_2$ (see proof below).

  Note that since the policy is computable and very simple, $g_0\left(\pi \right)=\infty$ is not true, and we have $g_0\left(\pi \right)=O\left(1\right)$ instead. I suspect that the issues are still present even with an additional $g_0\left(\pi \right)=\infty$ condition, but finding a concrete example with an uncomputable policy is challenging.

  proof: Suppose that $U_1$ and $U_2$ are locally equivalent. Let $V$ be an open neighborhood of the point $x=o_1^{\omega }$ and $\alpha >0$, $\beta \in R$ be such that $U_1\left(y\right)=\alpha U_2\left(y\right)+\beta$ for all $y\in V$.

  Since $x\in V$, we have $1=U_1\left(x\right)=\alpha U_2\left(x\right)+\beta =\alpha +\beta$. Because $V$ is an open neighborhood of $o_1^{\omega }$, there is an integer $N$ such that $o_1^n{o}_2^{\omega }\in V$ for all $n\ge N$. For such $n\ge N$, we have $$1-2^{-n}=U_1\left(o_1^n{o}_2^{\omega }\right)=\alpha U_2\left(o_1^n{o}_2^{\omega }\right)+\beta =\alpha (1-3^{-n})+\beta =\alpha +\beta -\alpha 3^{-n}=1-\alpha 3^{-n}.$$ This implies $\alpha =(2/3{)}^{-n}$. However, this is not possible for all $n\ge N$. Thus, our assumption that $U_1$ and $U_2$ are locally equivalent was wrong.

  Assumptions about the solomonoff prior: For all $n$, the sequence of actions that produces the sequence of $o_1^n$ with the highest probability is $a_1^{n-1}$ (recall that we start with observations in this setting). With this assumption, it can be seen that the policy that always picks action $a_1$ is among the best policies for both $U_1$ and $U_2$.

  I think this is actually a natural behaviour for a reasonable Solomonoff prior: It is natural to expect that $o_1{a}_1{o}_1$ is more likely than $o_1{a}_2{o}_1$. It is natural to expect that the sequence of actions that leads to $o_1$ over $o_2$ has low complexity. Always picking $a_1$ is low complexity.

  It is possible to construct an artificial UTM that ensures that “always take $a_1$” is the best policy for $U_1$, $U_2$: An UTM can be constructed such that the corresponding Solomonoff prior assigns ³⁄₄ probability to the program/​environment “start with o_1. after action a_i, output o_i”. The rest of the probability mass gets distributed according to some other more natural UTM.

  Then, for $U_1$, in each situation with history $o_1^n$ the optimal policy has to pick $a_1$ (the actions outside of this history have no impact on the utility): With ³⁄₄ probability it will get utility of at least $1-2^{-(n+1)}$. And with $1/4$ probability at least $1-2^{-n}$. Whereas, for the choice of $a_2$, with probability $3/4$ it will have utility of $1-2^{-n}$, and with probability $1/4$ it can get at most $1$. We calculate $(1-2^{-(n+1)})3/4+(1-2^{-n})1/4=1-52^{-(n+3)}1-3\cdot 2^{-(n+2)}=(1-2^{-n})3/4+1/4$, ie. taking action $a_1$ is the better choice.

  Similarly, for $U_2$, the optimal policy has to pick $a_1$ too in each situation with history $o_1^n$. Here, the calculation looks like $(1-3^{-(n+1)})3/4+(1-3^{-n})1/4=1-3^{-n}/21-\cdot 3^{-n+1}/4=(1-3^{-n})3/4+1/4$.

• harfe 5 Jul 2023 20:07 UTC

  1 point

  0

  in reply to: Quinn’s comment on: In­fra-Bayesian Logic

  “inclusion map” refers to the map $i_X$, not the coproduct $X\coprod Y$. The map $i_X$ is a coprojection (these are sometimes called “inclusions”, see https://​​ncatlab.org/​​nlab/​​show/​​coproduct).

  A simple example in sets: We have two sets $X$, $Y$, and their disjoint union $X\bigsqcup Y$. Then the inclusion map $i_X$ is the map that maps $x$ (as an element of $X$) to $x$ (as an element of $X\bigsqcup Y$).

In­fra-Bayesian Logic

• harfe 19 Jun 2023 1:06 UTC

  1 point

  0

  in reply to: Daniel Kokotajlo’s comment on: Daniel Koko­ta­jlo’s Shortform

  What is an environmental subagent? An agent on a remote datacenter that the builders of the orginal agent don’t know about?

  Another thing that is not so clear to me in this description: Does the first agent consider the alignment problem of the environmental subagent? It sounds like the environmental subagents cares about paperclip-shaped molecules, but is this a thing the first agent would be ok with?

• harfe 12 Jun 2023 13:09 UTC

  10 points

  8

  on: UK PM: $125M for AI safety

  This does not sound very encouraging from the perspective of AI Notkilleveryoneism. When the announcement of the foundation model task force talks about safety, I cannot find hints that they mean existential safety. Rather, it seems about safety for commercial purposes.

  A lot of the money might go into building a foundation model. At least they should also announce that they will not share weights and details on how to build it, if they are serious about existential safety.

  This might create an AI safety race to the top as a solution to the tragedy of the commons

  This seems to be the opposite of that. The announcement talks a lot about establishing UK as a world leader, e.g. “establish the UK as a world leader in foundation models”.

• harfe 9 Jun 2023 22:32 UTC

  5 points

  4

  in reply to: jimrandomh’s comment on: Trans­for­ma­tive AGI by 2043 is <1% likely

  There is an additional problem where one of the two key principles for their estimates is

  Avoid extreme confidence

  If this principle leads you to picking probability estimates that have some distance to 1 (eg by picking at most 0.95).

  If you build a fully conjunctive model, and you are not that great at extreme probabilities, then you will have a strong bias towards low overall estimates. And you can make your probability estimates even lower by introducing more (conjunctive) factors.

• harfe 7 Jun 2023 23:57 UTC

  2 points

  1

  on: Ar­ti­cle Sum­mary: Cur­rent and Near-Term AI as a Po­ten­tial Ex­is­ten­tial Risk Factor

  Nitpick: The title the authors picked (“Current and Near-Term AI as a Potential Existential Risk Factor”) seems to better represent the content of the article than the title you picked for this LW post (“The Existential Risks of Current and Near-Term AI”).

  Reading the title I was expecting an argument that extinction could come extremely soon (eg by chaining GPT-4 instances together in some novel and clever way). The authors of the article talk about something very different imo.