riceissa

Karma: 463

riceissa 28 Oct 2014 7:52 UTC
31 points

on: 2014 Less Wrong Census/Survey
I took the survey.

riceissa 14 May 2019 20:04 UTC
15 points

on: Flashcards for AI Safety?
I’ve made around 250 Anki cards about AI safety. I haven’t prioritized sharing my cards because I think finding a specific card useful requires someone to have read the source material generating the card (e.g. if I made the card based on a blog post, one would need to read that exact blog post to get value out of reviewing the card; see learn before you memorize). Since there are many AI safety blog posts and I don’t have the sense that lots of Anki users read any particular blog post, it seems to me that the value generated from sharing a set of cards about a blog post isn’t high enough to overcome the annoyance cost of polishing, packaging, and uploading the cards.

More generally, from a consumer perspective, I think people tend to be pretty bad at making good Anki cards (I’m often embarrassed at the cards I’ve created several months ago!), which makes it unexciting for me to spend a lot of effort trying to collaborate with others on making cards (because I expect to receive poorly-made cards in return for the cards I provide). I think collaborative card-making can be done though, e.g. Michael Nielsen and Andy Matuschak’s quantum computing guide comes with pre-made cards that I think are pretty good.

Different people also have different goals/interests so even given a single source material, the specifics one wants to Ankify can be different. For example, someone who wants to understand the technical details of logical induction will want to Ankify the common objects used (market, pricing, trader, valuation feature, etc.), the theorems and proof techniques, and so forth, whereas someone who just wants a high-level overview and the “so what” of logical induction can get away with Ankifying much less detail.

Something I’ve noticed is that many AI safety posts aren’t very good at explaining things (not enough concrete examples, not enough emphasis on common misconceptions and edge cases, not enough effort to answer what I think of as “obvious” questions); this fact is often revealed by the comments people make in response to a post. This makes it hard to make Anki cards because one doesn’t really understand the content of the post, at least not well enough to confidently generate Anki cards (one of the benefits of being an Anki user is having a greater sensitivity to when one does not understand something; see “illusion of explanatory depth” and related terms). There are other problems like conflicting usage of terminology (e.g. multiple definitions of “benign”, “aligned”, “corrigible”) and the fact that some of the debates are ongoing/some of the knowledge is still being worked out.

For “What would be a good strategy for generating useful flashcards?”: I try to read a post or a series of posts and once I feel that I understand the basic idea, I will usually reread it to add cards about the basic terms and ask myself simple questions. Some example cards for iterated amplification:
- what kind of training does the Distill step use?
- in the pseudocode, what step gets repeated/iterated?
- how do we get A[0]?
- write A[1] in terms of H and A[0]
- when Paul says IDA is going to be competitive with traditional RL agents in terms of time and resource costs, what exactly does he mean?
- advantages of A[0] over H
- symbolic expression for the overseer
- why should the amplified system (of human + multiple copies of the AI) be expected to perform better than the human alone?

riceissa 25 Apr 2019 14:33 UTC
15 points

on: When is rationality useful?

We can model success as a combination of doing useful things and avoiding making mistakes. As a particular example, we can model intellectual success as a combination of coming up with good ideas and avoiding bad ideas. I claim that rationality helps us avoid mistakes and bad ideas, but doesn’t help much in generating good ideas and useful work.

Eliezer Yudkowsky has made similar points in e.g. “Unteachable Excellence” (“much of the most important information we can learn from history is about how to not lose, rather than how to win”, “It’s easier to avoid duplicating spectacular failures than to duplicate spectacular successes. And it’s often easier to generalize failure between domains.”) and “Teaching the Unteachable”.

riceissa 19 Nov 2018 2:33 UTC
LW: 14 AF: 4
AF
on: Topological Fixed Point Exercises
My solution for #3:

Define $g : [0, 1] \to R$ by $g (x) = f (x) - x$ . We know that $g$ is continuous because $f$ and the identity map both are, and by the limit laws. Applying the intermediate value theorem (problem #2) we see that there exists $x \in [0, 1]$ such that $g (x) = 0$ . But this means $f (x) = x$ , so we are done.

Counterexample for the open interval: consider $f : (0, 1) \to (0, 1)$ defined by $f (x) = x / 2$ . First, we can verify that if $0 < x < 1$ then $0 < x / 2 < 1 / 2 < 1$ , so $f$ indeed maps to $(0, 1)$ . To see that there is no fixed point, note that the only solution to $x / 2 = x$ in $R$ is $0$ , which is not in $(0, 1)$ . (We can also view this graphically by plotting both $y = x$ and $y = x / 2$ and checking that they do not intersect in $(0, 1)$ .)

riceissa 17 Sep 2019 23:04 UTC
12 points

on: Why Subagents?
When I initially read this post, I got the impression that “subagents = path-dependent/incomplete DAG”. After working through more examples, it seems like all the work is being done by “committee requiring unanimous agreement” rather than by the “subagents” part.

Here are the examples I thought about:
1. Same as the mushroom/pepperoni situation, with the same two agents, but now each side can retaliate/hijack the rest of the mind if it doesn’t get what it wants. For example, if it starts at pepperoni, the mushroom-preferring agent will hijack the rest of the mind to remove the pepperoni, ending up at cheese. But if the agent starts at the “both” node, it will stay there (because both agents are satisfied). The preference relation can be represented as $pepperoni \to cheese \leftarrow mushroom$ with an extra arrow from $cheese \to both$ . This is still a DAG, and it’s still incomplete (in the sense that we can’t compare pepperoni vs mushroom) but it’s no longer path-dependent, because no matter where we start, we end up at cheese or “both” (I am assuming that toppings-removal can always be done, whereas acquiring new toppings can’t).
2. Same as the previous example, except now only the mushroom-preferring agent can retaliate/hijack (because the pepperoni-preferring agent is weak or nice). Now the preferences are $pepperoni \to cheese \to mushroom \to both$ . This is still a DAG, but now the preferences are total, so we can also view it as a (somewhat weird) single agent. A realistic example of this is given by Andrew Critch, where pepperoni=work, cheese=burnout (i.e. neither work nor friendship), mushroom=friendship, and both=friendship-and-work.
3. A modified version of the Zyzzx Prime planet by Scott Alexander. Now whenever we start out at pepperoni, the pepperoni-preferring agent becomes stupid/weak, and loses dominance, so now there are edges from pepperoni to mushroom and “both”. (And similarly, mushroom points to both pepperoni and “both”.) Now we no longer have a DAG because of the cycle between pepperoni and mushroom.
It seems like when people talk about the human mind being composed of subagents, the deliberation process is not necessarily “committee requiring unanimous agreement”, so the resulting preference relations cannot necessarily be represented using path-dependent DAGs.

It also seems like the general framework of viewing systems as subagents (i.e. not restricting to “committee requiring unanimous agreement”) is broad enough that it can basically represent any kind of directed graph. On one hand, this is suspicious (if everything can be viewed as a bunch of subagents, then maybe the subagents framework isn’t adding anything after all). On the other hand, this suggests that claims of subagents are not really about the resulting behavior/preference ordering of the system, but rather about the internal dynamics of the system.

riceissa 28 Jul 2019 21:06 UTC
12 points

on: AALWA: Ask any LessWronger anything
I was surprised to see, both on your website and the white paper, that you are part of Mercatoria/ICTP (although your level of involvement isn’t clear based on public information). My surprise is mainly because you have a couple of comments on LessWrong that discuss why you have declined to join MIRI as a research associate. You have also (to my knowledge) never joined any other rationality-community or effective altruism-related organization in any capacity.

My questions are:
1. What are the reasons you decided to join or sign on as a co-author for Mercatoria/ICTP?
2. More generally, how do you decide which organizations to associate with? Have you considered joining other organizations, starting your own organization, or recruiting contract workers/volunteers to work on things you consider important?

riceissa 3 Jan 2019 7:13 UTC
12 points

on: What exercises go best with 3 blue 1 brown’s Linear Algebra videos?
Some other sources of exercises you might want to check out (that have solutions and that I have used at least partly):
- Multiple choice quizzes (the ones related to linear algebra are determinants, elementary matrices, inner product spaces, linear algebra, linear systems, linear transformations, matrices, and vector spaces)
- Vipul Naik’s quizzes (disclosure: I am friends with Vipul and also do contract work for him)
Regarding Axler’s book (since it has been mentioned in this thread): there are several “levels” of linear algebra, and Axler’s book is at a higher level (emphasis on abstract vector spaces and coordinate-free ways of doing things) than the 3Blue1Brown videos (more concrete, working in $R^{n}$ ). Axler’s book also assumes that the reader has had exposure to the lower level material (e.g. he does not talk about row reduction and elementary matrices). So I’m not sure I would recommend it to someone starting out trying to learn the basics of linear algebra.

Gratuitous remarks:
- I think different resources covering material in a different order and using different terminology is in some sense a feature, not a bug, because it allows one to look at the subject from different perspectives. For instance, the “done right” in Axler’s book comes from one such change in perspective.
- I find that learning mathematics well takes an unintuitively long time; it might be unrealistic to expect to learn the material well unless one puts in a lot of effort.
- I think there is a case to be made for the importance of struggling in learning (disclosure: I am the author of the page).

riceissa 19 Nov 2018 1:54 UTC
LW: 12 AF: 2
AF
on: Topological Fixed Point Exercises

Here is my attempt, based on Hoagy’s proof.

Let $n \geq 1$ be an integer. We are given that $f (0 / n) = f (0) \leq 0$ and $f (n / n) = f (1) \geq 0$ . Now consider the points $0 / n, 1 / n, \dots, (n - 1) / n, n / n$ in the interval $[0, 1]$ . By 1-D Sperner’s lemma, there are an odd number of $j \in {0, \dots, n}$ such that $f (j / n) \geq 0$ and $f ((j - 1) / n) \leq 0$ (i.e. an odd number of “segments” that begin below zero and end up above zero). In particular, $0$ is an even number, so there must be at least one such number $j$ . Choose the smallest and call this number $j_{n}$ .

Now consider the sequence $(j_{n} / n)_{n = 1}^{\infty}$ . Since this sequence takes values in $[0, 1]$ , it is bounded, and by the Bolzano–Weierstrass theorem there must be some subsequence $(k_{n} / n)_{n = 1}^{\infty}$ that converges to some number $x \in [0, 1]$ .

Consider the sequences $(f ((k_{n} - 1) / n))_{n = 1}^{\infty}$ and $(f (k_{n} / n))_{n = 1}^{\infty}$ . We have $f ((k_{n} - 1) / n) \leq 0 \leq f (k_{n} / n)$ for each $n \geq 1$ . By the limit laws, $(k_{n} - 1) / n \to x$ as $n \to \infty$ . Since $f$ is continuous, we have $f ((k_{n} - 1) / n) \to f (x)$ and $f (k_{n} / n) \to f (x)$ as $n \to \infty$ . Thus $f (x) \leq 0$ and $f (x) \geq 0$ , showing that $f (x) = 0$ , as desired.

riceissa 14 Apr 2019 19:57 UTC
11 points

on: Diagonalization Fixed Point Exercises
Thoughts on #10:

I am confused about this exercise. The standard/modern proof of Gödel’s second incompleteness theorem uses the Hilbert–Bernays–Löb derivability conditions, which are stated as (a), (b), (c) in exercise #11. If the exercises are meant to be solved in sequence, this seems to imply that #10 is solvable without using the derivability conditions. I tried doing this for a while without getting anywhere.

Maybe another way to state my confusion is that I’m pretty sure that up to exercise #10, nothing that distinguishes Peano arithmetic from Robinson arithmetic has been introduced (it is only with the introduction of the derivability conditions in #11 that this difference becomes apparent). It looks like there is a version of the second incompleteness theorem for Robinson arithmetic, but the paper says “The proof is by the construction of a nonstandard model in which this formula [i.e. formula expressing consistency] is false”, so I’m guessing this proof won’t work for Peano arithmetic.

riceissa 14 Apr 2019 19:41 UTC
11 points

on: Diagonalization Fixed Point Exercises
My solution for #12:

Suppose for the sake of contradiction that such a formula $ϕ$ exists. By the diagonal lemma applied to $\neg ϕ$ , there is some sentence $ψ$ such that, provably, $\neg ϕ (┌ ψ ┐) \leftrightarrow ψ$ . By the soundness of our theory, in fact $\neg ϕ (┌ ψ ┐) \leftrightarrow ψ$ . But by the property for $ϕ$ we also have $ϕ (┌ ψ ┐) \leftrightarrow ψ$ , which means $\neg ϕ (┌ ψ ┐) \leftrightarrow ϕ (┌ ψ ┐)$ , a contradiction.

This seems to be the “semantic” version of the theorem, where the property for $ϕ$ is stated outside the system. There is also a “syntactic” version where the property for $ϕ$ is stated within the system.

riceissa 19 Nov 2018 1:29 UTC
LW: 11 AF: 2
AF
on: Topological Fixed Point Exercises

I’m having trouble understanding why we can’t just fix $n = 2$ in your proof. Then at each iteration we bisect the interval, so we wouldn’t be using the “full power” of the 1-D Sperner’s lemma (we would just be using something close to the base case).

Also if we are only given that $f$ is continuous, does it make sense to talk about the gradient?

riceissa 7 Jun 2019 3:20 UTC
10 points

on: Arbital scrape
The scrape seems to be missing the Solomonoff induction dialogue, which was available at https://arbital.com/p/solomonoff_induction/?l=1hh (at the moment I just get an error).

ETA (2019-06-28): The new version of the scrape has this page, and can be viewed on the GreaterWrong version.

riceissa 14 Mar 2019 23:13 UTC
LW: 10 AF: 1
AF
on: Open Problems Regarding Counterfactuals: An Introduction For Beginners
The link no longer works (I get “This project has not yet been moved into the new version of Overleaf. You will need to log in and move it in order to continue working on it.”) Would you be willing to re-post it or move it so that it is visible?

riceissa 1 Apr 2019 18:56 UTC
9 points

on: Diagonalization Fixed Point Exercises
Attempted solution and some thoughts on #9:

Define a formula $ξ (v)$ taking one free variable to be $f (┌ ϕ ┐, ┌ f (v, v) ┐)$ .

Now define $ψ$ to be $f (┌ ξ ┐, ┌ ξ ┐)$ . By the definition of $f$ we have $ψ \leftrightarrow ξ (┌ ξ ┐)$ .

We have $ϕ (┌ ψ ┐) \leftrightarrow f (┌ ϕ ┐, ┌ ψ ┐) \leftrightarrow f (┌ ϕ ┐, ┌ f (┌ ξ ┐, ┌ ξ ┐) ┐) \leftrightarrow ξ (┌ ξ ┐) \leftrightarrow ψ$

The first step follows by the definition of $f$ , the second by the definition of $ψ$ , the third by the definition of $ξ$ , and the fourth by the property of $ψ$ mentioned above. Since $ψ \in S_{0}$ by the type signature of $f$ , this completes the proof.

Things I’m not sure about:

It’s a little unclear to me what the $S y n (A, B)$ notation means. In particular, I’ve assumed that $f$ takes as inputs Gödel numbers of formulas rather than the formulas themselves. If $f$ takes as inputs the formulas themselves, then I don’t think we can assume that the formula $ξ$ exists without doing more arithmetization work (i.e. the equivalent of $ξ$ would need to know how to convert from the Gödel number of a formula to the formula itself).

If the biconditional “ $\leftrightarrow$ ” is a connective in the logic itself, then I think the same proof works but we would need to assume more about $f$ than is given in the problem statement, namely that the theory we have can prove the substitution property of $f$ .

The assumption about the quantifier complexity of $S_{0}$ and $S_{1}$ was barely used. It was just given to us in the type signature for $f$ , and the same proof would have worked without this assumption, so I am confused about why the problem includes this assumption.

riceissa 29 Nov 2018 7:44 UTC
9 points

on: Turning Up the Heat: Insights from Tao’s ‘Analysis II’
Can you say more about why exercise 17.6.3 is wrong?

If we define $f : [0, 1] \to R$ by $f (x) := x / (1 + x)$ then for distinct $x, y \in [0, 1]$ we have $| f (x) - f (y) | = ∣ ∣ ∣ \frac{x}{1 + x} - \frac{y}{1 + y} ∣ ∣ ∣ = ∣ ∣ ∣ \frac{x - y}{(1 + x) (1 + y)} ∣ ∣ ∣ < | x - y |$

We also have $f^{'} (0) = 1$ since $lim x \to 0; x \in (0, 1] \frac{f (x) - f (0)}{x - 0} = lim x \to 0; x \in (0, 1] \frac{x}{(x + 1) x} = 1$

In general, the derivative is $f^{'} (x) = 1 / (1 + x)^{2}$ , which is continuous on $[0, 1]$ .

riceissa 25 Jul 2018 22:09 UTC
8 points

on: Probability is Real, and Value is Complex
I was confused about this too, but now I think I have some idea of what’s going on.
Normally probability is defined for events, but expected value is defined for random variables, not events. What is happening in this post is that we are taking the expected value of events, by way of the conditional expected value of the random variable (conditioning on the event). In symbols, if $A \subset Ω$ is some event in our sample space, we are saying $E (A) = E (X ∣ A)$ , where $X : Ω \to R$ is some random variable (this random variable is supposed to be clear from the context, so it doesn’t appear on the left hand side of the equation).
Going back to cousin_it’s lottery example, we can formalize this as follows. The sample space can be $Ω = {win, lose}$ and the probability measure is defined as $P ({win}) = 1 / 10^{6}$ and $P ({lose}) = 1 - 1 / 10^{6}$ . The random variable $L : Ω \to R$ represents the lottery, and it is defined by $L (win) = 10^{6}$ and $L (lose) = 0$ .
Now we can calculate. The expected value of the lottery is:
$E (L) = L (win) P ({win}) + L (lose) P ({lose}) = 10^{6} / 10^{6} + 0 \cdot (1 - 10^{6}) = 1$
The expected value of winning is:
$\begin{matrix} E ({win}) & = E (L ∣ {win}) = L (win) P (win ∣ {win}) + L (lose) P (lose ∣ {win}) = 10^{6} \cdot 1 + 0 \cdot 0 = 10^{6} \end{matrix}$
The “probutility” of winning is:
$Q ({win}) = P ({win}) E ({win}) = \frac{1}{10^{6}} \cdot 10^{6} = 1$
So in this case, the “probutility” of winning is the same as the expected value of the lottery. However, this is only the case because the situation is so simple. In particular, if $L (lose)$ was not equal to zero (while winning and losing remained exclusive events), then the two would have been different (the expected value of the lottery would have changed while the “probutility” would have remained the same).

riceissa 12 Sep 2019 3:19 UTC
6 points

on: Conversation with Paul Christiano

It’s lots of saving throws, you know? And you multiply the saving throws together and things look better. And they interact better than that because– well, in one way worse because it’s correlated: If you’re incompetent, you’re more likely to fail to solve the problem and more likely to fail to coordinate not to destroy the world. In some other sense, it’s better than interacting multiplicatively because weakness in one area compensates for strength in the other. I think there are a bunch of saving throws that could independently make things good, but then in reality you have to have a little bit here and a little bit here and a little bit here, if that makes sense.

I don’t understand this part. Translating to math, I think it’s saying something like, if $p_{i}$ is the probability that saving throw $i$ works, then the probability that at least one of them works is $1 - \prod_{i} (1 - p_{i})$ (assuming the saving throws are independent), which is higher the more saving throws there are; but due to correlation, the saving throws are not independent, so we effectively have fewer saving throws. I don’t understand what “weakness in one area compensates for strength in the other” or “a little bit here and a little bit here and a little bit here” mean.

riceissa 12 Sep 2019 2:26 UTC
6 points

on: Conversation with Paul Christiano

Christiano cares more about making aligned AIs that are competitive with unaligned AIs, whereas MIRI is more willing to settle for an AI with very narrow capabilities.

Looking at the transcript, it seems like “AI with very narrow capabilities” is referring to the “copy-paste a strawberry” example. It seems to me that the point of the strawberry example (see Eliezer’s posts 1, 2, and Dario Amodei’s comment here) is that by creating an AGI that can copy and paste a strawberry, we necessarily solve most of the alignment problem. So it isn’t the case that MIRI is aiming for an AI with very narrow capabilities (even task AGI is supposed to perform pivotal acts).

riceissa 19 Aug 2019 22:22 UTC
5 points

on: GreaterWrong Arbital Viewer
The page https://arbital.greaterwrong.com/p/AI_safety_mindset/ is blank in the GreaterWrong version, but has content in the obormot.net version.

riceissa 19 May 2019 15:44 UTC
5 points

on: Announcement: AI alignment prize round 4 winners
Was this post ever published? (Looking at Zvi’s posts since January, I don’t see anything that looks like it.)