There was a recent post estimating that GTP-3 is equivalent to about 175 bees. There is also a comment there asserting that a human is about 140k bees.
I would be very interested if someone could explain where this huge discrepancy comes from. (One estimate is equating synapses with parameters, while this one is based on FLOPS. But there shouldn’t be such a huge difference.)
Indeed (as other commenters also pointed out) the ability to sexually reproduce seems to be much more prevalent than I originally thought when writing the above comment. (I thought that eukaryotes only capable of asexual reproduction were relatively common, but it seems that there may only be a very few special cases like that.)
I still disagree with you dismissing the importance of mitochondria though. (I don’t think the OP is saying that mitochondria alone are sufficient for larger genomes, but the argument for why they are at least necessary is convincing to me.)
I disagree with English (in principle at least) being inadequate for software specification.
For any commercial software, the specification basically is just “make profit for this company”. The rest is implementation detail.
(Obviously this is an absurd example, but it illustrates how you can express abstractions in English that you can’t in C++.)
I don’t think the comparison of giving a LLM instructions and expecting correct code to be output is fair. You are vastly overestimating the competence of human programmers: when was the last time you wrote perfectly correct code on the very first try?
Giving the LLM the ability to run its code and modify it until it thinks its right would be a much fairer comparison. And if, as you say, writing unit tests is easy for a LLM, wouldn’t that just make this trial-and-error loop trivial? You can just bang the LLM against the problem until the unit tests pass.
(And this process obviously won’t produce bug-free code, but humans don’t do that in the first place either.)
Not all eukaryotes employ sexual reproduction. Also prokaryotes do have some mechanisms for DNA exchange as well, so copying errors are not their only chance for evolution either.
But I do agree that it’s probably no coincidence that the most complex life forms are sexually reproducing eukaryotes.
I barely registered the difference between small talk and big talk
I am still confused about what “small talk” is after reading this post.
Sure, talking about the weather is definitely small talk. But if I want to get to know somebody, weather talk can’t possibly last for more than 30 seconds. After that, both parties have demonstrated the necessary conversational skills to move on to more interesting topics. And the “getting to know each other” phase is really just a spectrum between surface level stuff and your deepest personal secrets, so I don’t really see where you would draw the line between small and deep talk.
One situation I struggle with on the other hand is when I would rather avoid talking to a person at all, and so I want to maintain the shallowest possible level of small talk. (Ideally I could tell them that “sorry, I would rather just not talk to you right now”, but that’s not really socially accepted.)
It was actually this post about nootropics that got me curious about this. Apparently (based on self reported data) weightlifting is just straight up better than most other nootropics?
Anyway, thank you for referencing some opposing evidence on the topic as well, I might try to look into it more at some point.
(Unfortunately, the thing that I actually care about—whether it has cognitive benefits for me—seems hard to test, since you can’t blind yourself to whether you exercised.)
I think this is (and your other post about exercise) are good practical examples of situations where rational thinking makes you worse off (at least for a while).
If you had shown this post to me as a kid, my youth would probably have been better. Unfortunately no one around me was able to make a sufficiently compelling argument for caring about physical appearance. It wasn’t until much later that I was able to deduce the arguments for myself. If I just blindly “tried to fit in with the cool kids, and do what is trendy”, I would have been better off.
I wonder what similar blind spots I could have right now where the argument in favor of doing something is quite complicated, but most people in society just do it because they blindly copy others, and I am worse off as a result.
This alone trumps any other argument mentioned in the post. None of the other arguments seem universal and can be argued with on an individual basis.
I actually like doing things with my body. I like hiking and kayaking and mountain climbing and dancing.
As some other commenters noted, what if you just don’t?
I think it would be valuable if someone made a post just focused on collecting all the evidence for the positive cognitive effects of exercise. If the evidence is indeed strong, no other argument in favor of exercise should really matter.
FWIW I don’t think that matters, in my experience interactions like this arise naturally as well, and humans usually perform similarly to how Friend did here.
In particular it seems that here ChatGPT completely fails at tracking the competence of its interlocutor in the domain at hand. If you asked a human with no context at first they might give you the complete recipe just like ChatGPT tried, but any follow up question immediately would indicate to them that more hand-holding is necessary. (And ChatGPT was asked to “walk me through one step at a time”, which should be blatantly obvious and no human would just repeat the instructions again in answer to this.)
Cool! (Nitpick: You should probably mention that you are deviating from the naming in the HoTT book. AFAIK usually and types are called Pi and Sigma types respectively, while the words “product” and “sum” (or “coproduct” in the HoTT book) are reserved for and .)
I am especially looking forward to discussion on how MLTT relates to alignment research and how it can be used for informal reasoning as Alignment Research Field Guide mentions.
I always get confused when the term “type signature” is used in text unrelated to type theory. Like what do people mean when they say things like “What’s the type signature of an agent?” or “the type of agency is ”?
This argument seems a bit circular, nondeterminism is indeed a necessary condition for exfiltrating outside information, so obviously if you prevent all nondeterminism you prevent exfiltration.
You are also completely right that removing access to obviously nondeterministic APIs would massively reduce the attack surface. (AFAIK most known CPU side-channel require timing information.)
But I am not confident that this kind of attack would be “robustly impossible”. All you need is finding some kind of nondeterminism that can be used as a janky timer and suddenly all Spectre-class vulnerabilities are accessible again.
For instance I am pretty sure that rowhammer depends on the frequency of the writes. If you insert some instruction between the writes to RAM, you can suddenly measure the execution time of said instruction by looking at how many cycles it took to flip a bit with rowhammer. (I am not saying that this particular attack would work, I am just saying that I am not confident you couldn’t construct something similar that would.)
I am confident that this direction of exfiltration would be robustly impossible.
If you have some deeper reason for believing this it would probably be worth its own post. I am not saying that its impossible to construct some clever sandbox environment that ensures determinism even on a buggy CPU with unknown classes of bugs, I am just saying that I don’t know of existing solutions.
(Also in my opinion it would be much easier to just make a non-buggy CPU instead of trying to prove correctness of something executing on a buggy one. (Though proving your RAM correct seems quite hard, e.g. deriving the lack of rowhammer-like attacks from Maxwell’s laws or something.))
Yes, CPUs leak information: that is the output kind of side-channel, where an attacker can transfer information about the computation into the outside world. That is not the kind I am saying one can rule out with merely diligent pursuit of determinism.
I think you are misunderstanding this part, input side channels absolutely exist as well, Spectre for instance:
On most processors, the speculative execution resulting from a branch misprediction may leave observable side effects that may reveal private data to attackers.
Note that the attacker in this case is the computation that is being sandboxed.
This implies that we could use relatively elementary sandboxing (no clock access, no networking APIs, no randomness, none of these sources of nondeterminism, and that’s about it) to prevent a task-specific AI from learning any particular facts
It’s probably very hard to create such a sandbox though, your list is definitely not exhaustive. Modern CPUs leak information like a sieve. (The known ones are mostly patched of course but with this track record plenty more unknown vulnerabilities should exist.)
Maybe if you build the purest lambda calculus interpreter with absolutely no JIT and a deterministic memory allocator you could prove some security properties even when running on a buggy CPU? This seems like a bit of a stretch though. (And maybe while running it like this on a single thread you can prevent the computation from being able to measure time, any current practical AI needs massive parallelism to execute. With that probably all hopes of determinism and preventing timing information from leaking in go out the window.)
Also I just found that you already argued this in an earlier post, so I guess my point is a bit redundant.
Anyway, I like that this article comes with an actual example, we could probably use more examples/case studies for both sides of the argument.
Upon reading the title I actually thought the article would argue the exact opposite, that formalization affects intuition in a negative way. I like non-eucledian geometry as a particular example where formalization actually helped discovery.
But this is definitely now always true. For instance if you wanted to intuitively understand why addition of naturals is commutative, maybe to build intuition for recognizing similar properties elsewhere, would this formal proof really help?
plus_comm =
fun n m : nat =>
nat_ind (fun n0 : nat => n0 + m = m + n0)
(plus_n_O m)
(fun (y : nat) (H : y + m = m + y) =>
eq_ind (S (m + y))
(fun n0 : nat => S (y + m) = n0)
(f_equal S H)
(m + S y)
(plus_n_Sm m y)) n
: forall n m : nat, n + m = m + n
This is as formal as it gets, a 100% machine checkable proof without room for human error.
I think formalization is just a tool that may or may not be helpful depending on your goal, and the real question is how you can tell ahead of time what level of formalization will be helpful?
Isn’t this similar to a Godzilla Strategy? (One AI overseeing the other.)
That variants of this approach are of use to superintelligent AI safety: 40%.
Do you have some more detailed reasoning behind such massive confidence? If yes, it would probably be worth its own post.
This seems like a cute idea that might make current LLM prompt filtering a little less circumventable, but I don’t see any arguments for why this would scale to superintelligent AI. Am I missing something?
Collaborating with an expert/getting tutoring from an expert might be really good?
Probably. How does one go about finding such experts, who are willing to answer questions/tutor/collaborate?
(I think the usual answer to this is university, but to me this does not seem to be worth the effort. Like I maybe met 1-2 people at uni who would qualify for this? How do you find these people more effectively? And even when you find them, how do you get them to help you? Usually this seems to require luck & significant social capital expenditure.)
I unfortunately don’t have any answers, just some more related questions:
Does anyone have practical advice on this topic? In the short term we are obviously powerless to change the system as a whole. But I couldn’t in good conscience send my children to suffer through the same system I was forced to spend a large part of my youth in. Are there any better practically available alternatives?
What about socialization? School is quite poor at this, yet unilaterally removing one kid would probably make them even worse off. (Since presumably all other kids their age are still at school.)
As an adult, what actually useful methods of learning exist? I learned the vast majority of my useful knowledge through autodidactism, everything else (school, university) is pretty much noise. I would be open to alternatives, but I haven’t seen any kind of “teaching” so far that came anywhere close.
Why is this being downvoted?
From what I am seeing people here are focusing way too much on having a precisely calibrated P(doom) value.
It seems that even if P(doom) is 1% the doom scenario should be taken very seriously and alignment research pursued to the furthest extent possible.
The probability that after much careful calibration and research you would come up with a P(doom) value less than 1% seems very unlikely to me. So why invest time into refining your estimate?