My top interest is AI safety, followed by reinforcement learning. My professional background is in software engineering, computer science, machine learning. I have degrees in electrical engineering, liberal arts, and public policy. I currently live in the Washington, DC metro area; before that, I lived in Berkeley for about five years.
David James
Karma: 181
I’ll share a stylized story inspired from my racing days of many years ago. Imagine you are a competitive amateur racing road cyclist. After years of consistent training racing in the rain, wind, heat, and snow, you are ready for your biggest race of the year, a 80-something mile hilly road race. Having completed your warm-up, caffeination, and urination rituals, you line up with your team and look around. Having agreed you are the leader for the day (you are best suited to win given the course and relative fitness), you chat about the course, wind, rivals, feed zones, and so on.
A hundred-ish brightly-colored participants surround you, all optimized to convert calories into rotational energy. You feel the camaraderie among this traveling band of weekend warriors. Lots of determination and shaved legs. You notice but don’t worry about the calves carved out of wood since they are relatively weak predictors of victory. Speaking of...
You hazard one of twenty some-odd contenders is likely to win. It will involve lots of fitness and some blend of grit, awareness, timing, team support, adaptability, and luck. You estimate you are in the top twenty based on previous events. So from the outside viewpoint, you estimate your chances of winning are roughly 5%.
What’s that? … You hear three out-of-state semi-pro mountain bikers are in the house. Teammates too. They have decided to “use this race for training”. Lovely. You update your P(win) to ~3%. From an outside view, sure, that sounds about right. But when the race starts, does a 3% chance demotivate you? Hell no. A low chance of winning does not lower your level of effort. You assume a mental state of “I’m going to win this” as a convinction, a way of enhancing your performance.
How are you doing to win? You don’t know exactly. You can say this: you will harness your energy and abilities. You review your plan. Remain calm, pay attention, conserve energy until the key moments, trust your team to help, play to your strengths, and when the time is right, take a calculated risk. You have some possible scenarios in mind; get in a small breakaway, cooperate, get ready for cat-and-mouse at the end, maybe open your sprint from 800 meters or farther. (You know from past experiences your chances of winning a pack sprint are very low.)
Are we starting soon? Some people are twitchy. Lots of cycling computer beeping and heart-rate monitor fiddling. Ready to burst some capillaries? Ready to drop the hammer? Turn the pedals in anger? Yes! … and no. You wait some more. This is taking a while. (Is that person really peeing down their leg? Caffeine intake is not an exact science apparently. Better now than when the pistons are motion.)
After a seemingly endless sequence of moments, a whistle blows. The start! The clicking of shoes engaging with pedals. Leading to … not much … the race starts with a neutral roll out. Slower than anyone wants. But the suspense builds .. until maybe a few minutes later … a cacophony of shifting of gears … and the first surge begins. This hurts. Getting dropped at the beginning is game-over. Even trading position to save energy is unwise right now—you have to be able to see the front of the pack until things calm down a bit. You give whatever level of energy is needed, right now.
One gets the feeling that the authors are just raising their hands and saying something like: “Look, we are doomed, and there’s no realistic way we’re getting out of this short of doing stuff we are not going to do. These proposals are the necessary consequence of accepting what is stated in the preceding chapters, so that’s that”.
First, personally, I can understand how some people could have this response, and I empathize—this topic is heavy. However, with some time and emotional distance, one can see the above interpretation doesn’t correspond with key passages from the book, such as this one from Chapter 13:
Anytime someone tells you that the Earth could not possibly manage to do anything as difficult as restricting AI research, they are really claiming to know that countries will never care. They are asserting that countries and their leaders could not possibly come to care even 1 percent as much as they cared to fight World War II.
Also, the last chapter (14, “Where There’s Life, There’s Hope”) centers around people facing the truth of their circumstances and rising to the occasion. Here is a key quote:
Humanity averted nuclear war because people who understood that the world was on track for destruction worked hard to change tracks.
Chapter 14 concludes by offering tailored suggestions to civil servants, elected officials, political leaders, journalists, or “the rest of us”. The authors are decidedly not sending up the white flag; instead, they are stating what needs to be done and advocating for it.
I composed this poem by myself (except for the final 4 lines):
## Homeward Bound wandering around they go eating grass to and fro etiquette gives them a pass sometimes ducking head under ass sometimes waddling palmatedly strutting with sass and pooping belatedly with an impact so vast if only they could be steered like a ship by mast into a loving home though weird by tempting grain cast show me, caring shepherd, how to usher them from the splattered urban sprawl so they follow me undemurred to a place quiet streams and safety callSonnet 4.5 tells me:
The practical answer: you almost certainly can’t and shouldn’t try to relocate them yourself. Legal reality: In most jurisdictions, waterfowl (especially Canada geese) are protected under migratory bird laws. Moving them without permits is illegal. Even if they’re a nuisance, relocation requires wildlife management authorities.
Biological reality: Urban waterfowl are often there because:
They’ve habituated to human environments (easy food, few predators)
They may be resident populations, not migratory
Their home range is the urban area
If you actually want to help:
Contact local wildlife authorities—They have legal authority and expertise. Some areas have humane relocation programs for problem populations.
Habitat modification—Making the current area less attractive is more effective than chasing them elsewhere:
Stop feeding (yours and others’)
Install barriers to nesting sites
Use visual deterrents during establishment season
Accept the reality—Urban waterfowl populations are generally stable because cities provide resources. The “splattered urban sprawl” is their ecological niche now.
The herding question: Yes, you can move them short distances with patience—walk slowly behind them, use your body/arms to guide direction, they’ll generally move away from pressure. But they’ll return. Geese have strong site fidelity.
Probability assessment: ~95% chance any DIY relocation effort results in them returning within days, possibly with legal consequences for you. The caring shepherd move is usually to let wildlife professionals handle it, or to modify the environment rather than the birds.
I wanted to highlight the Trustworthy Systems Group at School of Computer Science and Engineering of UNSW Sydney and two of their projects, seL4 and LionsOS.
We research techniques for the design, implementation and verification of secure and performant real-world computer systems. / Our techniques provide the highest possible degree of assurance—the certainty of mathematical proof—while being cost-competitive with traditional low- to medium-assurance systems.
seL4 microkernel
seL4 is both the world’s most highly assured and the world’s fastest operating system kernel. Its uniqueness lies in the formal mathematical proof that it behaves exactly as specified, enforcing strong security boundaries for applications running on top of it while maintaining the high performance that deployed systems need.
seL4 is grounded in research breakthroughs across multiple science disciplines. These breakthroughs have been recognised by international acclaimed awards, from the MIT Technology Review Award, to the ACM Hall of Fame Award, the ACM Software Systems Award, the DARPA Game changer award, and more.
LionsOS
They are building a modular operating system called LionsOS:
We are designing a set of system services, each made up of simple building blocks that make best use of the underlying seL4 kernel and its features, while achieving superior performance. The building blocks are simple enough for automated verification tools (SMT solvers) to prove their implementation correctness. We are furthermore employing model checkers to verify key properties of the interaction protocols between components.
Core to this approach are simple, clean and lean designs that can be well optimised, use seL4 to the best effect, and provide templates for proper use and extension of functionality. Achieving this without sacrificing performance, while keeping the verification task simple, poses significant systems research challenges.
It seems like a catastrophic civilizational failure that we don’t have confident common knowledge of how colds spread.
Given the context above (posted by bhauth), the problem seems intrinsically hard. What would make this a civilizational failure? To my eye, that label would be warranted if either:
-
in alternative timelines with the same physics and biological complexity, other civilizations sometimes figured out transmission. If the success rate is under some threshold (maybe 1%), it suggests variation in civilization isn’t enough to handle the intrinsic complexity. (This option could be summarized as “grading on a multiverse curve”.)
-
deaths from the common cold (cc) met the criteria of “catastrophic”. The cc costs lives, happiness, and productivity, yes, but relative to other diseases, the “catastrophic” label seems off-target. (This option is analogous to comparing against other risks.)
-
Are you positing an associative or causal connection from increased intelligence to decreased listening ability or motivation to those who are deemed less intelligent? This is a complex area; I agree with statisticians who promote putting one’s causal model front and center.
A browser doesn’t count as a significant business integration?
It seems to me the author’s meaning of business integration is pretty clear: “profitable workplace integrations, such as a serious integration with Microsoft office suite or the google workplace suite.”—integrations tailored to a particular company’s offerings. A browser is not business-specific nor workplace-specific.
I don’t usually venture into LessWrong culture or “feels” issues, but the above comment seems out of place to me. As written, the comment seems more like something I would see on Hacker News, where someone fires off a quick quip or a zinger. I prefer to see more explanation here on LW. I’m in some ways a HN refugee, having seen too many low-effort ping-pong sniping contests.
Asking even a good friend to take the time to read The Sequences (aka Rationality A-Z) is a big ask. But how else does one absorb the background and culture necessary if one wants to engage deeply in rationalist writing? I think we need alternative ways to communicate the key concepts that vary across style and assumed background. If you know of useful resources, would you please post them as a comment? Thanks.
Some different lenses that could be helpful:
-
“I already studied critical thinking in college, why isn’t this enough?”
-
“I’m already a practicing data scientist, what else do I need to know and why?
-
“I’m already interested in prediction markets… how can I get better?”
-
“I’m not a fan of parables, can you teach me aspiring rationality in a different way?”
-
“I would never admit this except to close friends, but weird or fringe movements make me nervous, but as do think of myself as a critical thinker, help me ramp up without getting into uncomfortable areas.”
-
“I would never admit this to others — and may not even recognize it in myself — but my primary motivation is to use evidence and rationality as a cudgel against my opponents. Can you teach me how to be better at this?” (And this reminds me of Harry teaching Malfoy in HPMOR with an ulterior motive!)
-
It seems retroactive if-then commitments could be one mechanism for mitigating the Boiling Frog or Camel’s Nose metaphors.
I’m interpreting your comment as fitting into a kind of “game theory for rationalists who want to effect change in the real world.” This is complex and situational, and I’m skeptical I can say much that is interesting, actionable, and generalizable. Still, the following statements seem true to me, but I haven’t vetted them:
A strategy of aiming for high epidemic standards has many advantages:
It is a durable way to build and maintain trust.
It is resilient to changing circumstances and knowledge.
intellectual honesty is an endowment that should be developed, not degraded, because it is necessary for wise decisions, and we will have to make many smart decisions to survive and thrive.
However, this might:
limit one’s audience and reach.
not be sufficient.
not be effective against other strategies in certain environments or communication channels.
There’s something peculiar about the motte and bailey fallacy compared to many other fallacies. The former requires change (often via the passage of time) whereby one sneakily changes the contents of their central claim box*, hoping others won’t notice.
I have a theory: the motte and bailey fallacy would be harder to perpetrate if one’s argument had to be written out visually. This would freeze it in time and help people to detect central claim box-switching shenanigans.
* originally instead of “box” I was going to say variable in the programming language sense, but I didn’t want to get into the semantics of mutable versus immutable variables, which might lead to Rich Hickey’s famous talk on identity and state.
To varying degrees, people immersed in the study of economics may adopt more than just an analytical toolkit; they may hold some economic concepts as moral foundations.
To give some context, I have bolded two parts from Teaching Economics as if Ethics Mattered (2004) by Charles K. Wilber:
I have spent the past thirty-five years as a professor of economics. Over the course of my tenure at The American University, in Washington, DC (1965-75) and the University of Notre Dame (1975 present), however, I became ever more disenchanted with the capacity of traditional economic theory to enable people to lead a good life. I found myself unable to accept the values embedded in economic theory, particularly the elevation of self-interest, the neglect of income distribution, and the attempts to export these values into studies of the family, the role of the state and so on. As a result I started researching and writing on the nature of economics and the role of ethics in economic theory.
This work has led me to three important conclusions. First is the conviction that economic theory is not value-free as is so often claimed. Rather, it presupposes a set of value judgments upon which economic analysis is erected. Second is the realization that the economy itself requires that the self interest of economic actors be morally constrained. Third is the recognition that economic institutions and policies impact people’s lives requiring that both efficiency and equity be assessed. Teachers of economics need to make use of these insights.
Though Wilber (above) focuses on neoclassical economics as entangled with libertarianism, there are other couplings across different flavors of economic history: Marxism, Keynesianism, Development Economics, and more.
I have a working theory, which I have reflected on imperfectly: mentioning the difference between an analytical toolkit (aka modeling approach) and a value system goes a long way towards orienting myself in a discussion with someone I don’t know well. In some ways, it can be a test balloon to assess someone’s awareness of the complexity of economics.
Here are the axes I see in the original comment:
axis 1: works on: capabilities versus safety
axis 2: international cooperation: seems hopeless versus seems possible
We could certainly debate/discuss more about it: (a) the sharpness of the quadrant boundaries; (b) how well the bullet points map cleanly to the idealized quadrants (e.g. perhaps the second bullet point makes additional assumptions beyond what the quadrants purely require); (c) if other conclusions should be included (the “so we need to” part).
First, thanks for sharing—this is an insightful taxonomy. Second, to get into one detail, it seems indentured servitude has more variation and complexity than the description above captures:
Indentured servitude is a form of labor in which a person is contracted to work without salary for a specific number of years. The contract, called an “indenture”, may be entered voluntarily for a prepaid lump sum, as payment for some good or service (e.g. travel), purported eventual compensation, or debt repayment. An indenture may also be imposed involuntarily as a judicial punishment. -Wikipedia
Here are two compound nouns that I’ve found useful for high-bandwidth communication: economics-as-toolkit versus economics-as-moral-foundation.
Is power shifting away from software creators towards attention brokers?
That said, I don’t understand your question. Do you think that in future, AI will write the code, and the job of the developer will be to persuade the employer to use technology X instead of technology Y? Why not ask AI instead?
Upon further reflection, my question is more about function (or role) than who has the power. It seems like the function of persuasion is more important than ever, relative to creation. It might be helpful to think of the persuasion role being done by some combination of people, AI decision support, and AI agents.
The writing above could be clearer as to what I mean. Here are some different ways of asking the question. (I abbreviate software creator as creator.)
Is the median creator less powerful today…
overall?
relative to the median influencer?
Is the set of all software creators less powerful today…
overall?
relative to the set of all influencers?
Since there is variation across industries or products (call them value chains):
For a particular value chain, is a median creator less powerful today relative to a media influencer?
Or maybe we want to ask the question from the POV of an individual:
For a power-seeking individual, would they be better off becoming a creator or an influencer?
Is power shifting away from software creators towards attention brokers? I think so...
Background: Innovation and Compositionality
How does innovation work? Economists, sociologists, and entrepreneurs sometimes say things like:
new scientific ideas are born when the stars align
the same idea pops up in many places because an idea is in the air
some ideas become products by way of tech transfer
pain points spur solutions
software companies sometimes dogfood their own technologies to vet them
hardware companies who understand and control the technology of their production pipeline often can better innovate at a process level
Software engineers would probably add to the list by saying practical innovation is driven by the awareness and availability of useful building blocks such as libraries, APIs, data structures, and algorithms. Software developers know this. These blocks are the raw material for their work.
But, I don’t know if the rest of the world gets it. Off the top of my head, I don’t think I’ve yet seen a compelling account of this—how compositionally in software feels
differentcompletely bonkers compared with other industries. Just keeping up is exciting (if you are lucky) but often disorienting. If you look backwards, previous practices seem foreign, clunky, quaint, or even asinine.Imagine if this rate of change applied to dentistry. Imagine a dentist sitting down with a patient. “Hello, how are you today?” The patient answers nervously, “Not too bad...” and mentally appends ”...yet”. The dentist says “Well, let’s have a look-see...” and glances over at her tray of tools. Eeek. Nothing looks familiar. She nervously calls for an assistant. “Where is my Frobnicator???” The assistant answers: “That technology was superseded yesterday. Here. Use the Brofnimator instead.”
Software development feels like this.
To the extent software is modular, components can be swapped or improved with relatively little cost. Elegant software abstractions reduce the cost of changing implementation details. It is striking how much intellectual energy goes into various software components. Given the size and scope of software industry, maybe we shouldn’t be surprised.
Example: seemingly over the course of only a few years, there seems to a widespread acceptance (in circles I read, like Hacker News) that embedded databases can play key roles. Rather than reaching for, say, a server-based database management system (DBMS) like PostgreSQL, developers increasingly choose embedded (in-process) data storage libraries like SQLite or one of the many embedded K/V stores (RocksDB, LMDB, etc). Many of the newer K/V stores have been developed and adopted quite rapidly, such as redb.
Tension Between Building and Adoption
Now, to abstract a bit, here is my recent insight about software. When I think of the balance between:
-
the cost (time, money, labor) of designing & building, versus:
-
the steps needed to socialize, persuade, trial, adopt, integrate
… it is clear the cost (1) is dropping fast. And large parts of (2) are dropping too. The following are getting easier: (a) surveying options; (b) assessing fitness for purpose; (c) prototyping; (d) integrating.
This means that attention, socialization, persuasion are increasingly important. This kind of trend is perhaps nothing new in the domains of politics, advertising, fashion, and so on. But it seems notable for software technology. In a big way it shifts the primary locus of power away from the creators to the attention brokers and persuaders.
Logically speaking, “soon” is not required for your argument. With regards to persuasiveness, saying “soon” is double-edged: those who think soon applies will feel more urgency, but the rest are given an excuse to tune out. A thought experiment by Stuart Russell goes like this: if we know with high confidence a meteor will smack Earth in 50 years, what should we do? This is an easy call. Prepare, starting now.
In Chapter 6 of his 2019 book, Human Compatible: Artificial Intelligence and the Problem of Control, Russell lists various objections to what one would hope would be well underway as of 2019:
The 50-year-away meteor is discussed by Human Compatible on page 151 as well as in
Reason 2 of 10 Reasons to Ignore AI Safety by Robert Miles (adapted from H.C. Chapter 6)
Provably Beneficial Artificial Intelligence by Russell (PDF, free)