rotatingpaguro

Karma: 456

rotatingpaguro 7 Jan 2024 17:40 UTC
17 points
19
in reply to: Rafael Harth’s comment on: Bayesians Commit the Gambler’s Fallacy
I have the impression the OP is using “gambler’s fallacy” as a conventional term for a strategy, while you are taking “fallacy” to mean “something’s wrong”. The OP does write about this contrast, e.g., in the conclusion:
Maybe the gambler’s fallacy doesn’t reveal statistical incompetence at all. After all, it’s exactly what we’d expect from a rational sensitivity to both causal uncertainty and subtle statistical cues.
So I think the adversative backbone of your comment is misdirected.

rotatingpaguro 28 Feb 2024 5:10 UTC
16 points
−1
on: Counting arguments provide no evidence for AI doom
There is also a hazy counting argument for overfitting:
1. It seems like there are “lots of ways” that a model could end up massively overfitting and still get high training performance.
2. So absent some additional story about why training won’t select an overfitter, it feels like the possibility should be getting substantive weight.
While many machine learning researchers have felt the intuitive pull of this hazy overfitting argument over the years, we now have a mountain of empirical evidence that its conclusion is false. Deep learning is strongly biased toward networks that generalize the way humans want— otherwise, it wouldn’t be economically useful.
I don’t know well NN history, but I have the impression good NN training is not trivial. I expect that the first $n$ attempts at NN training went bad in some way, including overfitting. So, without already knowing how to train an NN without overfitting, you’d get some overfitting in your experiments. The fact that now, after someone already poured their brain juice over finding techniques that avoid the problem, you don’t get overfitting, is not evidence that you shouldn’t have expected overfitting before.
The analogy with AI scheming is: you don’t already know the techniques to avoid scheming. You can’t use as counterargument a case in which a problem has already deliberately been solved. If you take that same case, and put yourself in the shoes of someone who doesn’t already have the solution, you see you’ll get the problem in your face a few times before solving it.
Then, it is a matter of whether it works like Yudkowsky says, that you may only get one chance to solve it.
The title says “no evidence for AI doom in counting arguments”, but the article mostly talks about neural networks (not AI in general), and the conclusion is
In this essay, we surveyed the main arguments that have been put forward for thinking that future AIs will scheme against humans by default. We find all of them seriously lacking. We therefore conclude that we should assign very low credence to the spontaneous emergence of scheming in future AI systems— perhaps 0.1% or less.
“main arguments”: I don’t think counting arguments completely fill up this category. Example: the concept of scheming originates from observing it in humans.
Overall, I have the impression of some overstatement. It can also be that I’m missing some previous discussion context/assumptions, so other background theory from you may say “humans don’t matter as examples”, and also “AI will be NNs and not other things”.

rotatingpaguro 16 Oct 2023 3:16 UTC
16 points
6
on: On Frequentism and Bayesian Dogma
What do you think about Wald’s complete class theorems and other similar decision-theoretic results that say that, under a fixed frequentist setting, the set of admissible algorithms coincides (barring messes with infinities) with the set of Bayesian procedures as all possible priors are considered? In other words, if you think it makes sense to strive for the “best” procedure in a context, for any fixed even if unknown definition of what’s best, and you have a frequentist procedure you think is statistically good, then there must be a corresponding Bayesian prior.
(This is an argument I’d always like to see addressed as basic disclaimer in frequentist vs bayesian discussions, I think it helps a lot to put down the framework people are reasoning under, e.g., if it’s more practical vs. theoretical.)
My own opinion on the topic (I’m pro-Bayes):
1. Many standard frequentist things can be derived as easily or more easily in a Bayesian way; that they are conventionally considered frequentist is an irrelevant accident of history.
2. In tree methods, the frequentist version comes first, but the Bayesian version, when it arrives, is better, and usable in practice.
3. Practically all real Bayesian methods are not purely Bayesian, there are many ad hockeries. The point is using Bayes as a guide. Even with an algorithm pulled out of the hat, it’s useful to know if it has a Bayesian interpretation, because it makes it clearer.
4. ML is frequentist only in the sense of trying algorithms without set rules, I don’t think that should be counted as frequentist success! It’s too generic. I have the impression the mindset of the people working in ML that know their shit is closer to Bayesian, but I am not confident in this since it’s an indirect impression. Example: information theoretic stuff is more natural with Bayes.

rotatingpaguro 13 Jul 2023 8:53 UTC
14 points
0
on: Are there any good, easy-to-understand examples of cases where statistical causal network discovery worked well in practice?
I’m doing causal inference in academia. I do not work on causal discovery, neither does anyone of my colleagues, but I can tell my impression about it from occasional seminars on the topic.
The few causal discovery seminars I have seen belong to these categories:
1. Purely theoretical work that does not show actual applications,
2. Applied work that does not work well,
3. Applied work that (maybe) worked but required years and a team.
Consider this a skewed but actual slice of the field.
My own thoughts on the subject matter:
In practice you don’t have the information to completely reconstruct the causal relationships, neither to do it with low enough uncertainty that you can pretend you knew the graph, in cases where you have enough constraints to converge in principle to a single graph with infinite i.i.d. data. So an ideal method would provide you a list of graphs with a posterior probability for each, and then you would carry out the inference conditional on each graph. This is what Bayes tells you to do.
However, a graph with less arcs leads naturally to a lower-dimensional parameter space than one with more arcs, when you try to specify a model. This would suggest that the graph with missing arcs has probability zero. You can try to repair this with delta distributions (i.e., probability mass given to a single point in a continuous space), but does it make sense? As Andrew Gelman sometimes says, everything has a causal effect on everything, it’s just that it can be very small. So maybe a model with shrinkage (i.e., keeping all connections in the graph, but defining a notion of “small” connection in the model and using a prior distribution that prefers simpler graphs) would make more sense.
I’ve not had these doubts answered in seminars nor by asking.
Finally, @IlyaShpitser may know something more.

rotatingpaguro 24 Oct 2023 21:28 UTC
13 points
2
in reply to: 307th’s comment on: Lying is Cowardice, not Strategy
First, this is a minor point where you’re wrong, but it’s also a sufficiently obvious point that it should hopefully make clear how wrong your world model is: AI safety community in general, and DeepMind + Anthropic + OpenAI in particular, have all made your job FAR easier. This should be extremely obvious upon reflection, so I’d like you to ask yourself how on earth you ever thought otherwise. CEOs of leading AI companies publicly acknowledging AI risk has been absolutely massive for public awareness of AI risk and its credibility. You regularly bring up how CEOs of leading AI companies acknowledge AI risk as a talking point, so I’d hope that on some level you’re aware that your success in public advocacy would be massively reduced in the counterfactual case where the leading AI orgs are Google Brain, Meta, and NVIDIA, and their leaders were saying “AI risk? Sounds like sci-fi nonsense!”

The fact that people disagree with your preferred method of reducing AI risk does not mean that they are EVIL LIARS who are MAKING YOUR JOB HARDER and DOOMING US ALL.
I disagree this is obviously wrong. I think you are not considering the correct counterfactual. From Connor L.’s point of view, the guys at the AI labs are genuinely worried about existential risk, but run 4D chess algorithms to determine they have to send mixed signals about it. Since Connor thinks these decisions run counter to the goal, counterfactually they are making his life harder by not just stating their worry and its consequence clearly. The counterfactual is not with “if AI labs did not exist”. That said, I’m not so confident I understand what he’s thinking, but you are excluding a reasonable possibility and so it’s not <obvious> as you say.
Overall, I think your comment is one of those cases where you indulge in the same sin you want to point out. See e.g. your overconfident epilogue.

rotatingpaguro 25 Jul 2023 21:04 UTC
13 points
0
on: Russian parliamentarian: let’s ban personal computers and the Internet

</div>

rotatingpaguro 24 Oct 2023 6:29 UTC
10 points
6
on: AI Pause Will Likely Backfire (Guest Post)
I disagree with so many things here that I’m overwhelmed, but thanks for putting all the arguments in one place coherently.

rotatingpaguro 14 Nov 2023 0:21 UTC

9 points

on: Game Theory without Argmax [Part 1]

Context-independence
We’ll say that a consequentialist optimiser $ψ = λ^{u : Γ \to R} . {γ \in Γ | u (γ) \in q (u)}$ is context-independent if $Image (u) = Image (u^{'}) ⟹ ψ (u) = ψ (u^{'})$ .
Context-independence is a stronger condition than consequentialism — this condition says that, once we know which payoffs are achievable in the agent’s task, then the only thing relevant to the optimality of a particular option is its payoff. For example, ${argmax}_{X}$ is context-independent, but ${better-than-average}_{π}$ is not.

Context-independence	We’ll say that a consequentialist optimiser $ψ = λ^{u : Γ \to R} . {γ \in Γ \| u (γ) \in q (u)}$ is context-independent if $Image (u) = Image (u^{'}) ⟹ ψ (u) = ψ (u^{'})$ . Context-independence is a stronger condition than consequentialism — this condition says that, once we know which payoffs are achievable in the agent’s task, then the only thing relevant to the optimality of a particular option is its payoff. For example, ${argmax}_{X}$ is context-independent, but ${better-than-average}_{π}$ is not.

${a r g m a x}_{X}$ is not context-independent according to the given definition.

Consider tasks $u, u^{'} : {0, 1} \to {0, 1}$ , with $u (x) = x$ and $u^{'} (x) = 1 - x$ . Then $a r g m a x u = {1} \neq a r g m a x u^{'} = {0}$ , despite $Image u = Image u^{'}$ .

I guess the correct definition says $q (u) = q (u^{'})$ instead of $ψ (u) = ψ (u^{'})$ .

rotatingpaguro 11 Mar 2024 1:12 UTC
8 points
3
in reply to: trevor’s comment on: lc’s Shortform
It’s AI-based, so my guess is that it uses a lot of somewhat superficial correlates that could be gamed. I expect that if it went mainstream it would be Goodharted.
I expect Goodhart would hit particularly bad if you were doing the kind of usage I guess you are implying, which is searching for a few very well selected people. A selective search is a strong optimization, and so Goodharts more.
More concrete example I have in mind, that maybe applies right now to the technology: there are people who are good at lying to themselves.

rotatingpaguro 29 Sep 2023 10:49 UTC
8 points
6
in reply to: NicholasKees’s comment on: EA Vegan Advocacy is not truthseeking, and it’s everyone’s problem
If a small group of “weirdos” is ideological and non truthseeking, I won’t listen to them. Popularity and social custom is the single most important heuristic because almost no one has the cognitive capacity to go on alone. To overcome this burden, I think being truth-seeking helps. That said, your argument could work if most people are not like me, and respond to more emotional motivations. In that case, I’d like that to not be what EA is for but something else. I’m not EA but I quite enjoy my EA friends while not other altruism advocates.

rotatingpaguro 27 May 2023 21:54 UTC
8 points
12
in reply to: shminux’s comment on: Who are the Experts on Cryonics?
If someone gives you a concrete point estimate probability of revival, their estimate is automatically untrustworthy.
This goes strongly against probabilistic forecasting. It seems a wrong principle to me.

rotatingpaguro 23 May 2023 17:54 UTC
8 points
1
on: Will Artificial Superintelligence Kill Us?
If developing interpretable ASIs is beyond us, we might need to strive towards making them extremely difficult to interpret, even for themselves.
Intuitively, I think that if developing interpretable ASI is beyond us, then developing provably-obscure-to-ASI ASI is beyond us too.

rotatingpaguro 30 Apr 2023 20:27 UTC
LW: 8 AF: 4
1
AF
on: Connectomics seems great from an AI x-risk perspective
I have a vague impression that I am not crazy to hope for whole primate-brain connectomes in the 2020s and whole human-brain connectomes in the 2030s, if all goes well.
After reading the post “Whole Brain Emulation: No Progress on C. elegans After 10 Years” I was left with the general impression that this stuff is very difficult; but I don’t know the details, and that post talks about simulation given a connectome, not getting a connectome, which maybe then is easier even for a huge primate brain, I guess? And I don’t know what probability you mean with “not crazy”.
A market here is thus apposite:

</div>

rotatingpaguro 10 Dec 2023 21:02 UTC
6 points
0
on: Some Rules for an Algebra of Bayes Nets
- Re-Rooting Rule for Markov Chains
- Joint Independence Rule
- Frankenstein Rule
- Factorization Transfer Rule
- Stitching Rule for A Shared Markov Blanket
- Swap Rule <=====
- Bookkeeping Rules
Can’t find the “Swap rule” in the post, what was that?
The more complex approximation rule requires some additional machinery. For any diagram, we can decompose its $D_{K L}$ into one term for each variable via the chain rule:
$D_{K L} (P [X_{1}, \dots, X_{n}] | | \prod_{i} P [X_{i} | X_{p a (i)}]) = D_{K L} (\prod_{i} P [X_{i} | X_{< i}] | | \prod_{i} P [X_{i} | X_{p a (i)}])$
$= \sum_{i} E [D_{K L} (P [X_{i} | X_{< i}] | | P [X_{i} | X_{p a (i)}])]$ <=====
If we know how our upper bound $ϵ$ on the diagram’s $D_{K L}$ decomposes across variables, then we can use that for more fine-grained bounds. Suppose that, for each diagram $j$ and variable $i$ , we have an upper bound
$ϵ_{j}^{i} \geq E [D_{K L} (P [X_{i} | X_{< i}] | | P [X_{i} | X_{p a_{j} (i)}])]$ <=====
I guess here there are missing expectations (indicated in red), otherwise $X_{< i}$ would remain a free RV. The second expectation is not necessary, although omitting it makes the bound unnecessarily strict.
I have never encountered this topic, and I couldn’t find it skimming the Handbook of Graphical Models (2019), did you invent it? If not, could you list references?
Do you perchance have something for conditioning or marginalization? I know that graphical marginalization requires using ADMGs (graphs with double-headed arrows) instead of DAGs, I don’t know about conditioning.
Exercise
Extend the above proofs to re-rooting of arbitrary trees (i.e. the diagram is a tree). We recommend thinking about your notation first; better choices for notation make working with trees much easier.
In a tree each node has a single parent. Re-rooting means flipping the arrow from the root to one of its children.
In the factorization I do P(old_root)P(new_root|old_root) = P(old_root|new_root)P(new_root), so the proof is the same.
The fact that notation is not involved makes my suspect I may be missing something. Maybe they mean it’s needed to write out in full the factorization expression? I guess the nice notation is using ch(X_i) instead of pa(X_i).
Joint Independence Rule: Exercise
The Joint Independence Rule can be proven using the Frankenstein Rule. This is left as an exercise. (And we mean that unironically, it is actually a good simple exercise which will highlight one or two subtle points, not a long slog of tedium as the phrase “left as an exercise” often indicates.)
Bonus exercise: also prove the conditional version of the Joint Independence Rule using the Frankenstein Rule.
Joint independence starts from the graphs
X_i {X_not i}
for all i. The Frankenstein rule can not be applied right away because the graphs are not on the same variables, since all but one appear grouped as a single tuple node.
To change that, I replace the tuple nodes with fully connected subgraphs. This is valid because cliques do not constrain the distribution, as any distribution can be factorized in any order if all variables are kept in the conditionals.
I choose the connection order of the cliques such that they respect a global ordering.
Then I apply Frankenstein, picking, for each node, the graph where that node is isolated.
Is there a way to obtain a narrower bound? Yes: I can pick only $D_{K L} (P (X_{i} | X_{< i}) ∥ P (X_{i}))$ from the i-th graph.
Conditional version: basically the same proof, with the common parent as first node in the global order. Needs more care in justifying replacing a tuple node with a clique: Conditional on the parent, the argument above still goes. The only additional independence property of the graph with the tuple is that the parent d-separates the tuple from the singlet, and that is preserved as well.
Frankenstein Rule
We’ll prove the approximate version, then the exact version follows trivially.
Without loss of generality, assume the order of variables which satisfies all original diagrams is $X_{1}, \dots, X_{n}$ . Let $P [X] = \prod_{i} P [X_{i} | X_{p a^{j} (i)}]$ be the factorization expressed by diagram $j$ , and let $σ (i)$ be the diagram from which the parents of $X_{i}$ are taken to form the Frankenstein diagram. (The factorization expressed by the Frankenstein diagram is then $P [X] = \prod_{i} P [X_{i} | X_{p a^{σ (i)} (i)}]$ .)
The proof starts by applying the chain rule to the $D_{K L}$ of the Frankenstein diagram:
$D_{K L} (P [X] | | \prod_{i} P [X_{i} | X_{p a^{σ (i)} (i)}]) = D_{K L} (\prod_{i} P [X_{i} | X_{< i} | | \prod_{i} P [X_{i} | X_{p a^{σ (i)} (i)}])$
$= \sum_{i} E [D_{K L} (P [X_{i} | X_{< i}] | | P [X_{i} | X_{p a^{σ (i)} (i)}])]$ <=====
Then, we add a few more expected KL-divergences (i.e. add some non-negative numbers) to get:
$\leq \sum_{i} \sum_{j} E [D_{K L} (P [X_{i} | X_{< i}] | | P [X_{i} | X_{p a^{j} (i)}])] = \sum_{j} D_{K L} (P [X] | | \prod_{i} P [X_{i} | X_{p a^{j} (i)}])$
$\leq \sum_{j} ϵ_{j}$
Thus, we have
$D_{K L} (P [X] | | \prod_{i} P [X_{i} | X_{p a^{σ (i)} (i)}]) \leq \sum_{j} D_{K L} (P [X] | | \prod_{i} P [X_{i} | X_{p a^{j} (i)}]) \leq \sum_{j} ϵ_{j}$ <=====
I guess you meant the green thing (first margin arrow) to appear in place of the red thing (second margin arrow)? The math is right, but I hypothesize the final line was intended to express at once the narrow and the loose bounds.

rotatingpaguro 12 May 2023 8:02 UTC
6 points
0
in reply to: Steven Byrnes’s comment on: I bet $500 on AI winning the IMO gold medal by 2026
I know a guy from the Physics Olympiads that was a mobile library of past olympiads problems. I think you’re underestimating the level of weirdness you can find around. Maybe it’s still a fraction of the existing problems, but I’d estimate enough to cover non-redundant ones.

I would expect them to find that at least a couple of the problems were sufficiently similar to something they’ve seen that they could get the answer without too much creativity.

I’ve not been to the IMO but I’d bet this already happens from comments I overheard by people who have been.

rotatingpaguro 8 May 2023 16:55 UTC
6 points
0
on: Inference Speed is Not Unbounded
I don’t completely understand your point because I don’t have a calibration for your “slow” in “training an AI must be slow”. How slow is “slow”? Compared to what? (Leaving aside Solomonoff inductors and other incomputable things.)

Do you consider the usual case that “a toddler requires fewer examples” as a reference for “not slow”? If so: human DNA is < 1GB, so humans get at most 1GB of free knowledge as inductive bias. Does your argument for “AI slow” then rely on us not getting to that <1GB of stuff to preconfigure in a ML system? If not so (humans slow too): do you think humans are a ceiling, or close to one, on data efficiency?

rotatingpaguro 8 May 2024 23:17 UTC
5 points
0
in reply to: Sable’s comment on: Observations on Teaching for Four Weeks
broken link

rotatingpaguro 13 Jan 2024 4:21 UTC
5 points
0
on: Eliminating Cookie Banners is Hard
I think this is a good demonstration of why companies generally do choose to stick with cookie banners even though they’re annoying.
Why are they annoying?
Some websites—rare, delicate lotus flowers—bother me with a small, horizontal banner on the bottom of the page. When I click “accept”, it actually goes away forever on that browser for that website.
While many others, istead, slap a bulky framed message in the middle of the page. Possibly 2-4 seconds after most of the loading, just to interrupt my initial interactions with the page in the most annoying way possible.
Is there a reason for that? Is it out of control overconservative legal worry?

rotatingpaguro 18 Dec 2023 22:54 UTC
5 points
2
on: OpenAI: Preparedness framework
OpenAI’s basic framework:
1. Do dangerous capability evals at least every 2x increase in effective training compute. This involves fine-tuning for dangerous capabilities, then doing evals on pre-mitigation and post-mitigation versions of the fine-tuned model. Score the models as Low, Medium, High, or Critical in each of several categories.
  Initial categories: cybersecurity, CBRN (chemical, biological, radiological, nuclear threats), persuasion, and model autonomy.
2. If the post-mitigation model scores High in any category, don’t deploy it until implementing mitigations such that it drops to Medium.
3. If the post-mitigation model scores Critical in any category, stop developing it until implementing mitigations such that it drops to High.
4. If the pre-mitigation model scores High in any category, harden security to prevent exfiltration of model weights. (Details basically unspecified for now.)
This outlines a very iterative procedure. If models started hitting the thresholds, and this logic was applied repeatedly, the effect could be pushing the problems under the rug. At levels of ability sufficient to trigger those thresholds, I would be worried about staying on the bleeding edge of danger, tweaking the model until problems don’t show up in evals.
I guess this strategy is intended to be complemented with the superalignment effort, and not to be pushed on indefinitely and the main alignment strategy.

rotatingpaguro 14 Dec 2023 0:19 UTC
5 points
0
in reply to: jacob_cannell’s comment on: Significantly Enhancing Adult Intelligence With Gene Editing May Be Possible
Couldn’t there be genetic effects on things that can improve the brain even once its NN structure is mostly fixed? Maybe it’s possible to have neurons work faster, or for the brain to wear less with abstract thinking, or to need less sleep.
This kind of thing is not a full intelligence improvement because it does not allow you to notice more patterns or to think with new schemes.
So maybe yes, it won’t make a difference for AI timelines, though it would still be a very big deal.

rotatingpaguro

Exercise

Joint Independence Rule: Exercise

Frankenstein Rule