Tldr;
Arbital should put design and engineering effort into better ways to structure discussions, such as defining terms, adding evidence for or against a claim, declaring cruxes, proposing tests, and posting challenges (e.g. “how might we…”). These structures will improve the quality of discussion and can plug together for collaborative truth-seeking.
Problem
I see three structures of community-created content out on the Interwebs today:
Blogs and articles (long-ish form usually written by one author, permanent but sinks out of awareness quickly.)
Discussions that flow quickly (threaded discussions like reddit or Q&A sites like StackExchange), and
Wikis (permanent but appearing static and the evolution is hidden).
What’s missing is multi-user collaboration about a permanent topic, seeking to arrive at a high-quality, trusted, and agreed upon statement of truth.
Reasoned disagreements driven by data, insights, and rationale help the group to test and refine the accuracy of arguments and decisions. Hashing out a complex problem like this demands diverse viewpoints and sources of knowledge.
But coming to that agreement is hard because you have to keep track of a complex and nested web of points and counterpoints, some that really matter and some that are nit-picks.
Which leaves people within a group feeling either that their point was missed and ignored and a bad decision was made, or that the group takes too long to come to agreement.
Proposal
A truth-seeking platform can help groups:
focus on specific claims or questions,
track how they want to decide (decomposing cases and flagging crux arguments)
track evidence for and against different claims,
support nested evidence about whether each piece of evidence is true or not.
Some claims will not be decided absolutely but will have evidence weighted more on one side or another.
Software can integrate the quality of the evidence (perhaps measured by reputation-based votes) across the collected evidence to provide an “estimated epistemic strength.”
How does this relate to other discussion features?
In trying to create algorithmic ways to surface the best content, contemporary discussion platforms have focused on measures of positive reader engagement (often up or down votes) and calculating writer reputation based on that engagement.
For a collaborative truth-seeking platform, the voting and reputation system are necessary. But it’s demonstrably not enough to shift a forum from being a casual watering hole to being a community that develops lasting agreement. In the face of tribal allegiances and confirmation bias, simple voting has little leverage to improve the quality of a complex and intricate debate. See Scalable ways to associate evidence (pro or con) with claims will be more valuable in elevating accuracy than complex voting and reputation systems
One question I posed was “Without argument structuring features, how is Arbital different than Reddit or Stack Exchange?”
In reply, Eric Bruylent wrote:
Reddit’s reputation system gives new arrivals equal weight to long-standing highly trusted members of the community, and does not include priors about content quality based on poster’s history. It’s the simplest thing which could barely work, and does not allow for high quality discussion to scale without relying heavily on moderators or other social things not present in all communities and not able to resist certain forms of attack. It also lacks adequate indexing and browsing by topic, making discussions temporary rather than able to produce lasting artifacts and be continued easily.
SE’s reputation system is a little better (you need to prove to the system you can productively engage with the topic before your votes have any weight), but it’s very focused on QA, which is not a great format for extended truth-seeking discussion.
Cool argument structuring seems like an optional bonus (still great to have, but not necessary for the thing to work), but features that give users reason to expect their high-quality content gets more eyeballs (particularly the eyeballs which most need that specific content) seem core and essential.
Perhaps what I’m proposing is to combine Stack Exchange’s robust reputation system with a format that’s better than QA. How can we make extended truth-seeking discussion easier?
P.S.
How not to do this: TruthSift
UI is WAY too heavy and slow
Writing and reading have to be the core enjoyable task. The system can’t get in the way of laying out a clear and interesting case. (OK, maybe posts are central)
Visual webs of information have long been dreamt of and yet never succeeded.
I’ve been looking for something like this for a long time now. I hope Arbital can be the platform that does it well.
I think the Semantic Web solves a lot of this, and could solve all of it.
Some argument structuring tools using AIF—I haven’t evaluated their UI’s yet.
The Provenance ontology lets us assert how information was produced and how it has been evaluated.
The goal should be the ability to filter conclusions based on arbitrary epistemic standards.
In Jimrandomh’s scenario, one could create their own argument structure, linking to appropriate concepts, and linking to the existing argument only where appropriate. Possibly only the conclusion. I think AIF would also allow citing any post or comment as an argument, so freeform stuff can still be connected even if it isn’t automatically filterable. Other people could do the structuring and validation.
The novel David’s Sling by Marc Stiegler features a hyper-rationality organization (kind of like a CFAR CEV) and this organization has a truth/argument-mapping process. Notably, it features both a software component as well as a skill/form that people are trained in. This could be considered to be the thing that Double Crux wants to evolve into.
You’re thinking about this problem and you like to read novels, I would recommend it. It doesn’t go into enough detail that I would say it’s a necessary source though.
I’ve come across a number of argument-structuring tools in the past. I think doing this right is much harder than people give it credit for.
The core problem is that most of the action in a quality discussion doesn’t actually consist of claims and counterclaims. For example, I went to Truthsift and the first claim I found on their randomized front page was “vaccines are safe” (http://truthsift.com/search_view?topic=Are-Vaccines-Safe-?&id=406&nid=4083). The response I wanted to make is that vaccine is too broad a category; a well-managed discussion would first clarify that the discussion was limited to the sort of FDA-approved vaccines one would normally be prescribed, not to random research chemicals, then break into sections about specific vaccines and vaccine additives, the trustworthiness of the FDA approval process, and so on. But Truthsift wouldn’t let me say that, first because I’d have to somehow mash that into a pro-and-con structure, and second because I’d have to merge that into an incompatible structure that someone else set up. This is entirely representative of my experiences with argument-structuring sites: I show up, find an elaborate structure that partitions the question in a way I think is wrong, and bounce.
Arbital is currently better because it doesn’t try to structure everything; it leaves space for all the hard-to-tag irregular stuff to happen in comment threads. I think there’s a large valley in between comment threads and a structure that can incorporate all the irregular things that happen in discussions, and that TruthSift’s mistake was that it tried to squeeze out the irregular stuff.
After reading the Doc(tm), I think there is still design space to explore. For most readers, and many authors, keeping track of the points in an argument is actually pretty hard mental work. Just a little help would make them be better at it.
The trick would be to create “just enough” structure that is easy to fold into the process of authoring a long post or writing a quick comment. I don’t think we need to reproduce all the elements of formal analysis here (and if we tried, it would be unusable).
In the end, I might argue myself back into the model where there are just “claims”, not “claims” and “evidence”.
It’s very likely that because we haven’t seen any solution to this problem, the ones that do exist are not low-effort. This is part of Eliezer’s argument (as you’ll probably get from the doc).
Great analysis of problems with TruthSift. Perhaps we should start a list of irregular and natural behavior that Arbital needs to handle gracefully. Hmm, I wonder what kind of page that should go on?
Can you point to other argument-structuring tools?
I see the point about this being a hard problem—it increases the likely investment needed to get the return, so my “high ROI for improving discussion quality” is a claim at risk. But if we can keep our eyes out for a low-effort way to solve the problem, the return still feels high.
I basically agree with everything here.