Although I accept this in the spirit of a rough and dirty calculation, your “liters” example has a slight unphysical artifact, because your t-distribution assigns non-zero probability to negative volume. Granted, this disappears pretty fast for large sample sizes (when t-tails ~ Gaussian thin tails). But since you’re advocating its use in small sample sizes, it’s a relevant caveat.
In the spirit of this post, when your data is strictly non-negative (like volume) you can always handwave your way into doing your t-test on log(data).
JustinSheek
Karma: 68
I think people who believe this—and I don’t know if this includes you—usually don’t really get the bounded rationality argument. Roughly
- any cognition&agency in this physics costs negentropy
- this “selects” against length, against depth of world models, against details, against thinking too long, against being unnecessarily smartYou have to carry this argument a bit further, no? Intelligence costs negentropy, but intelligence pays dividends in negentropy too. That’s the benefit of “depth of world models, details, thinking” in the first place. That’s why “unnecessarily” does all the heavy lifting in that argument. Empirically, the (locally) “thinkiest” species has got all the (local) negentropy, so isn’t the burden of proof pointing in the other direction?
Oof, the amount of misinformation on QM even here on LW is staggering.
Physics no longer describes what can happen.
This is straightforwardly false. Maybe you meant to say “Physics no longer describes what definitely happens”? Still misleading, as that was already the case with statistical mechanics within the ontology of Boltzmann and Gibbs 50 years earlier.
Oh, and the tree is emergent by the way, it’s not even a base component of the ontology.
Coherent phenomena are definitely part of the base ontology of QM. The density matrix encodes the ensemble. (If by “the tree” you didn’t mean the ensemble, then your statement would make even less sense to me).
...QM will tell you that what it means to be a function has changed and you have to do it differently now.
No. QM has no bearing on “what it means to be a function”. Maybe you mean “QM encodes permutations in a surprising way”?
Except that QM can tell the difference between a 360 degree rotation and doing nothing. (Not 720 degree rotations though. Those are still just like doing nothing, so at least there’s that.)
Strictly speaking this is only sometimes true. It seems like you are alluding to the spin-statistics theorem or maybe the Aharonov-Bohm effect or Berry phase. Your quoted statement is specifically applicable only to fermionic states. It’s inapplicable to bosons or more exotic states like anyons (FQHE) or braid statistics.
Because of the way QM works, it’s actually possible to exploit it to perform some kinds of computations faster than seems to be possible classically. This is also kind of weird.
Indeed.
Whoever wills the end also wills (in so far as reason has decisive influence on his actions) the indispensably necessary means to it that is in his control
-Kant
It’s a fruitless endeavor to try to disentangle instrumental drives from some kind of immutable sacred telos.
Thanks for this. Yet it’s odd to see a post about mechanism design with nary a mention of mechanism design.
My apologies then, I don’t know how to (compactly) improve your understanding other than to point at my priors more vigorously.
[Bowing out.]
Non sequitur? I did not expect that statement to prove “there is a clean theory of bounded agency.” It’s a terse illustration of why correspondence principles aren’t physics-magic. I am trying to explain why I believe we are disagreeing on priors here. It appeared to me that you believe elegant grand unified theories only appear in physics a priori.
https://www.fourmilab.ch/etexts/einstein/specrel/www/
Namely§ §
I think that we are in the situation where we need to understand something (agency; metaethics) better to achieve the desired effect (safe superintelligence). I think the amount of scientific work to be done on the former far eclipses the engineering work of the latter, to the point that it’s disingenuous to call the scientific work a subproblem. So it is more precise to say that I agree with you on the factual matter, but am dismayed by your rhetorical framing.
Disagreed. In my experience, flaws in [X] paradigm usually suggests there is a better paradigm [Y] waiting to be found. Correspondence principles wherein the domain of validity of [X] is subsumed by [Y] are not limited to physics. Specifically, a comprehensive theory of bounded agency (say, with bound b) should recover a theory of non-bounded agency in the limit
On the object level, some of Einstein’s gedankenexperiments on clock synchronization are pretty hard to distinguish from “a philosopher specializing in metaphysics had invented special relativity by reasoning from first principles about time”.
More importantly, AI safety is fundamentally an engineering problem, not a scientific problem.
Disagreed. If it’s an engineering problem, then I would expect that we must already deeply understand all the relevant principles—so that it’s just a matter of exchanging cost vs. safety, i.e. how much do we want to overengineer this? That does not characterize my epistemic state on AI safety in the least. In the moon landing analogy, I think AI safety research is arguably about at the level of the Tsiolkovsky rocket equation.
Ok point taken. I’m glad to see your research—it’s clearly presented and seems reproducible. I still think the above-quoted implication is broadly misleading. Based on your comments, I’m satisfied that we’re in disagreement about what part of the attack surface is important/worrisome.
[I absolve you of the tacit duty to respond to this.]
If “price=probability”, then changing the pricing curve is equivalent to changing how the AMM updates its probability estimates (on evidence of buy/sell orders).
I will try to dig up some references for you. Sorry it really was a small side project and has been several years.
Ah so I can’t imagine a probability function for that market that isn’t . is a fine pricing function that doesn’t appear to adhere to the rules of probability theory. If I try to compose two markets, one conditional on the other, then can I multiply their prices to find the joint probability? Does this violate “price=probability”?
”price=probability is a general rule for prediction markets” is a very interesting claim. Seems obvious, but then you have to ask yourself whose probability?I’m familiar with some of those operations (I’ve skimmed the Boyd paper). Unfortunately, there are a lot of different ways of expressing the same constraints, so I can’t immediately tell whether Manifold’s implementation is equivalent to what I had imagined.
Thanks for your answers, I’ll look into some of the other ideas you referenced.
“These results cast doubt on the worry that optimization pressure on the CoT would lead to encoded reasoning.”
This interpretation of your results doesn’t hold up for me. That “less sophisticated strategies, like jailbreaking the monitor” are found first, seems more like a property of the monitor than the optimization pressure generally. If you use a weak monitor, then you will tend to see a weak strategy (e.g. jailbreak the monitor) emerge. If you use a human monitor, you should expect to see a strategy that defeats human monitoring (e.g. steganography) emerge. If you use a superhuman monitor, you should expect to see an even more sophisticated strategy emerge.
Why should I be assuaged that encoded reasoning didn’t emerge when the monitor itself applied little optimization pressure in that direction?
I’ve seen similar derivations before, but it’s been a few years since I looked at AMMs in detail. I’ve spent some time recently looking at mechanism design for prediction markets, so this is a timely reminder!
Three questions—
Would you agree that this captures your main conclusion for a binary prediction market:
CPMM “price = probability”?
I seem to recall that CPMMs easily generalize to multiple assets. Instead of you have and so on. Do you happen to know if that matches the generalization of your prediction market with binary outcome to one with categorical outcome?
You mention in your 2nd footnote that there are “different probability functions matching these desiderata”. Care to say more on this?
Your notation is clear to me! It can be shown that:
Even though and are both convex, their composition is not necessarily convex. Sorry, I don’t have any clear counterexamples at the ready. (The Hessian determinants all vanish in the graphene/bit-string model.) It’s likely that I’ve configured my numerical solvers badly for this example.
As far as binding to reality:
Power laws are ubiquitous in nature. Exponential family doesn’t admit power laws; f-div does. (It admits many other important non-exponential distributions as well—see the B-E example above.)
Symmetries in your data that are weaker than—or just different from—independence are ubiquitous in nature. Exponential family is only congruent with independence; not so, f-div.
KL / Gibbs entropy is inadequate for many observable systems. This typically appears under the heading nonextensive entropy (Gell-Mann & Tsallis w/ contributions by Touchette and others) or Tsallis entropy. This paper is far enough outside my field that I can only loosely validate it, but it’s an attempt to explain dark energy by way of applying observational data to determine Bayes factors of different nonextensive entropy models of our cosmological horizon.
I consider Jensen’s, DPI, and coarse-graining monotonicity to be pretty important desiderata (see Nielsen). f-div is the biggest family that supports these, uniquely determined.
[I could write several more posts worth of material]
Zero-math-just-words summary:
So the natural world seems to be full of this symmetry thing, which is great because I suspect otherwise we wouldn’t have any hope of making a good map of it. Here’s some math that makes constructive epistemic claims about a particularly simple example—permutation symmetry—that people have been generally confused about for 90 years. Then here’s how that idea would testably interact with a thought experiment that you could almost pull off in a lab (BYO magic box).
Remember T is a program that’s supposed to emit all the computable parts of D that are relevant for doing inference on A. That’s P(A|D) = P(A|T(D)). (Naturally, the identity function on D does a perfectly fine job of preserving the relevant parts of D, so to be non-trivial you really want T to compress D in some way.) Here is some permutation of D (this doesn’t compress D). T(D) = T((D)) just says that T doesn’t care about the ordering of D. Then you combine these facts to resolve that inference on A also doesn’t care about the ordering of D. That’s P(A|D) = P(A|(D)).
Negative example:
= “this sensory input contains a cat”
D = “the pixels of the sensory input”
T = “AlexNet without the final softmax layer”
Observe destroys inference for . P(|D) P(|(D)). Classification of natural images isn’t invariant to arbitrary reordering of pixels. AlexNet isn’t invariant to reordering of pixels because gradient descent would rapidly exit that region of weight space. T(D) T((D)).
Perhaps there are some other (maybe approximate) symmetries () like reflections, translations, blurring, recoloring, rescaling. P(|D) P(|(D)). AlexNet might be in or near those regions of weight space. T(D) T((D)). This should also apply to your cortex or aliens or whatever has a notion of a cat!
I would be very surprised if this didn’t connect to natural latents. It connects to universality. I don’t think it connects directly to infrabayes but they might be compatible. It also connects to SLT, but for reasons that are beyond the scope of this post.
In the Ising model, the inability—through local fluctuations, well below -- for the coarse-grained state to transition to/from is an example of superselection. Unfortunately, I can’t find a good primer on the subject, but here’s something: https://physics.stackexchange.com/a/56570
To be able to take your desired step, reasoning from local->global, you also have to make additional measurements to rule out non-uniform temperature and non-uniform magnetic fields.
I think the failure modes you’re pointing to are valuable to point to. I’m left with the impression that you confidently believe everything other than your agenda—which btw I can donate to—fails trivially. Yet your arguments for this are kinda weak?!