Re: Aristotle. Large part of what Aristotle wrote is science and math. If you felt you didn’t learn anything from Aristotle, that’s because only non-science and non-math part of Aristotle is usually taught, because science and math are usually taught in ahistorical manner.
Barbara, Celarent, Darii, Ferio is good math. It is just that first-order logic and Venn diagrams are better, so we don’t teach Barbara etc. One thing lost by this ahistorical teaching is how much better first-order logic is, and how difficult advance it was when it was first done by Frege.
I am not aware of any single policy that can solve climate change by itself. What policy do these experts support? Let’s say it is to eliminate all coal power stations in the world by magic. That is at best 20% of global emission, so isn’t that policy synergistic with geoengineering? To think your favorite policy is competing with geoengineering, that policy should be capable of solving 100% of the problem, but I am not aware of existence of any such policy whatsoever.
Samsung is a memory play, not a TSMC competitor. Logic semiconductor is what Samsung hopes to do in the future, it is not how they make money now. In my opinion, Samsung’s technical lead in memory is no less than TSMC’s technical lead in logic. And transformative AI will require memory as much as CPU, GPU, and TPU.
I would say Corrigibility paper shares the same “feel” with certain cryptography papers. I think it is true that this feel is distinct, and not true that it means they are “not real”.
For example, what does it mean for cryptosystem to be secure? This is an important topic with impressive achievements, but it does feel different from bolts and nuts of cryptography like how to perform differential cryptanalysis. Indistinguishability under chosen plain text attack, the standard definition of semantic security in cryptography, does sound like “make up rules and then pretend they describe reality and prove results”.
In a sense, I think all math papers with focus on definitions (as opposed to proofs) feel like this. Proofs are correct but trivial, so definitions are the real contribution, but applicability of definitions to the real world seems questionable. Proof-focused papers feel different because they are about accepted definitions whose applicability to the real world is not in question.
What do you think of Paul Christiano’s argument that short timeline and fast takeoff is anti-correlated?
No. I think it is indeed the usual wisdom that slower takeoff causes shorter timeline. I found Paul Christiano’s argument linked in the article pretty convincing.
Yes. From page 34 of Supplementary Materials:
In practice, we observe that expert players tend to be very active in communicating, whereas those less experienced miss many opportunities to send messages to coordinate: the top 5% rated players sent almost 2.5 times as many messages per turn as the average in the WebDiplomacy dataset.
I am doubtful about this. I am unsure whether Cicero will score higher if it is more vindictive, so I am hesitant to call its game theory poor. A good analogy is that I am hesitant to call AlphaGo’s endgame moves poor even if they look 100% poor, because I am not sure whether AlphaGo will win more games if it plays more human like endgame.
Re 3: Cicero team concedes they haven’t overcome the challenge of maintaining coherency in chatting agents. They think they got away with it because 5 minutes are too short, and consider the game with longer negotiation periods will be more challenging.
Cicero is designed to be honest in the sense that all its messages are generated from its intents, where its intents are what moves Cicero in fact intends to play at the moment Cicero said them (Cicero can change its mind after saying things), and at the end of the turn played moves are equal to its last intents.
Not only Cicero uses its true intents to generate messages, it also tries to generate messages that correspond to intents. That is, its dialogue model is trained to imitate humans in WebDiplomacy, but when humans intend to attack Belgium, they will sometimes say things like “I won’t attack Belgium”. That is, AI can lie by forming intent to attack Belgium, and devising lying intent “won’t attack Belgium”, and generating lying messages from lying intent. Cicero doesn’t do this, its intent input to dialogue model is always truthful. AI can also lie by forming intent to attack Belgium, and generating lying messages like “I won’t attack Belgium” by imitating lying humans from truthful intent. Cicero also doesn’t do this! Dialogue model is trained to imitate only truthful humans, training data is filtered by the lie detector and 5% of turns are filtered.
That does not mean Cicero does not dissemble or mislead! There are three aspects to this. First, there is messaging model, entirely separate from dialogue model. Messaging model decides whether to send messages at all, trained to imitate humans. When humans intend to attack Belgium, held by France, they may not message France at all. Cicero copies this behavior.
Second, there is topic model, also entirely separate. Topic model decides which intent to talk about, trained to imitate humans. When humans intend to attack Belgium, held by France, and also Norway, held by Russia, they may talk to France about Norway and Russia about Belgium. Cicero also copies this behavior.
Third, there is filtering model, also entirely separate. When Cicero intends to attack Belgium, held by France, maybe messaging model decides to talk to France and topic model decides to talk to France about Belgium and dialogue model decides to say “I will attack Belgium”. That does not mean Cicero says “I will attack Belgium”, filtering model can veto it. In particular, value-based filtering model estimates how saying something will impact its own utility. Eight messages are sampled from dialogue model, their value impacts are calculated, importance is calculated from value impacts, and in 15% of most important situations, bottom three messages are dropped, and one message is picked at random.
I think it’s irrelevant to ELK. Cicero’s honesty is hardcoded in fixed architecture, it’s not latent at all.
If you’re vocally obstinate about not going along with its plan, can the dialogue side feed that info back into the planning side?
Yes. Figure 5 of the paper demonstrates this. Cicero (as France) just said (to England) “Do you want to call this fight off? I can let you focus on Russia and I can focus on Italy”. When human agrees (“Yes! I will move out of ENG if you head back to NAO”), Cicero predicts England will move out of ENG 85% of the time, moves the fleet back to NAO as agreed, and moves armies to Italy. When human disagrees (“You’ve been fighting me all game. Sorry, I can’t trust you won’t stab me”), Cicero predicts England will attack 90% of the time, moves the fleet to attack EDI, and does not move armies.
And if you’re dishonest does it learn not to trust you?
Yes. It’s also demonstrated in Figure 5. When human tries to deceive (“Yes! I’ll leave ENG if you move KIE → MUN and HOL → BEL”), Cicero judges it unreasonable. Cicero moves the fleet back to de-escalate, but does not move armies.
East Asia used counting rods for calculation for thousands of years. Counting rods use true positional numeral system. It’s just that East Asia didn’t use it for writing. In other words, there were separate systems, one to calculate numbers which was efficient, and one to write numbers which was traditional. If that sounds weird, consider that we calculate in binary but write in decimal.
It’s a plausible business model, but I assumed it’s not FTX’s business model (even before the event), because if it were so, FTX would have tried to acquire BitLicense from New York, like Coinbase did. The same applies to Binance and other exchanges uninterested in BitLicense.
I see many people saying things like “centralized cryptocurrency exchange is obviously dangerous, I told you so”, but I disagree. Yes, cryptocurrency allows you to self-custody, and option to self-custody is its main innovation, but it is an option: you don’t need to take it, and it does have significant convenience cost. I think the right lesson is something like “unregulated centralized cryptocurrency exchange has been historically dangerous”. It has been a while since BitLicense was introduced in 2015, and I think it proved to be a good regulatory framework so far.
Then scrapers looking for parallel data to train on would find these, and it took a lot of effort to screen them out.
How did Google screen them out? Is there a paper published on this? It seems potentially important.
I am not sure whether there is a technical argument, but there is a question of priority. To me, it is self-evident that algorithmic feed should be the highest priority for ActivityPub. But ActivityPub world seems to be full of people who say things like “I don’t want algorithmic feed”, or “newest first works fine”.
This seems to be the idea behind AT Protocol, which gives “Algorithmic choice: Control how you see the world through an open market of algorithms” as a goal.
From what I read they are probably not using Reverse Code Engineering (RCE) tools for their analysis; instead, they use the source of an app.
Where did you read this? Mayhem is totally using reverse code engineering. Mayhem would be pretty uninteresting and it wouldn’t have won DARPA Cyber Grand Challenge if it required source code. From their FAQ:
Q. Does Mayhem for Code require source code?
A. No, Mayhem for Code does not require source code to test an application. Mayhem for Code works with both source code and binaries. Mayhem for Code finds vulnerabilities before and after software release.
In 2016, DARPA funded Cyber Grand Challenge. (What would Earth do without DARPA? Remember DARPA funded Moderna.) Systems had to demonstrate:
Automatic vulnerability finding on previously-unknown binaries
Automatic patching of binaries without sacrificing performance
Automatic exploit generation
And they were demonstrated. Mayhem by ForAllSecure won the challenge. ForAllSecure is still in business and Mayhem is available for sales.
Looking at history, having no path to victory haven’t exactly deterred insurgencies in the past. But yes, existence of Russia is a big deal. They have some place to go, unlike, say, Koreans under Japanese rule, who had no place to go.