Something that I think would make rationalists more effective at convincing people is if we had explicit models of the things we care about.
Currently we are at the stage of physicists arguing that the atom bomb might ignite the atmosphere without concrete math and models of how that might happen.
If we do this for lots of issues and have a norm of making models composable this would have further benefits.
People would use the models to make real world decisions with more accuracy
We would create frameworks for modelling that would be easily composable, that other people would use
Both would raise the status and knowledge of the rationalist community.
Found “The Future of Man” by Pierre Teilhard de Chardin in a bookshop. Tempted to wite a book review. It discusses some interesting topics, like the planetisation of Mankind. However it treats them as inevitable, rather as something contingent on us getting our act together. Anyone interested in a longer review?
Edit: I think his faith in the super natural plays a part in the assumption of inevitability.
One of my factors in thinking about the development of AI is self-managing systems, as humans and animals self manage.
It is possible that they will be needed to manage the complexity of AI, once we move beyond LLMs. For example they might be needed to figure out when to train on new data in an efficient way and how much resources to devote to different AI sub processes in real time depending upon the problems being faced.
They will change the AI landscape making it easier for people to run their own AIs, for this reason it is unlikely that corporations will develop them or release them to the outside world (much like corporations cloud computing infra is not open source) as it will erode their moats.
Modern computer systems have and rely on the concept of a super user. It will take lots of engineering effort to remove that and replace it with something new.
With innovation being considered the purview of corporations are we going to get stuck in a local minima of cloud compute based AI, that is easy for corporations to monetise?
By corporation I am mainly thinking about current cloud/SaaS providers. There might be a profitable hardware play here, if you can get enough investment to do the R&D.
Code nowadays can do lots of things, from buying items to controlling machines. This presents code as a possible coordination mechanism, if you can get multiple people to agree on what code should be run in particular scenarios and situations, that can take actions on behalf of those people that might need to be coordinated.
This would require moving away from the “one person committing code and another person reviewing” code model.
This could start with many people reviewing the code, people could write their own test sets against the code or AI agents could be deputised to review the code (when that becomes feasible). Only when an agreed upon number of people thinking the code should it be merged into the main system.
Code would be automatically deployed, using gitops and the people administering the servers would be audited to make sure they didn’t interfere with running of the system without people noticing.
Code could replace regulation in fast moving scenarios, like AI. There might have to be legal contracts that you can’t deploy the agreed upon code or use the code by itself outside of the coordination mechanism.
Can you give a concrete example of a situation where you’d expect this sort of agreed-upon-by-multiple-parties code to be run, and what that code would be responsible for doing? I’m imagining something along the lines of “given a geographic boundary, determine which jurisdictions that boundary intersects for the purposes of various types of tax (sales, property, etc)”. But I don’t know if that’s wildly off from what you’re imagining.
I’ve been thinking about non AI catastrophic risks.
One that I’ve not seen talked about is the idea of cancerous ideas. That is ideas that spread throughout a population and crowd out other ideas for attention and resources.
This could lead to civilisational collapse due to basic functions not being performed.
Safeguards for this are partitioning the idea space and some form of immune system that targets ideas that spread uncontrollably.
Relatedly I am thinking about improving the wikipedia page on recursive self-improvement. Does anyone have any good papers I should include? Ideally with models.
Proposal for new social norm—explicit modelling
Something that I think would make rationalists more effective at convincing people is if we had explicit models of the things we care about.
Currently we are at the stage of physicists arguing that the atom bomb might ignite the atmosphere without concrete math and models of how that might happen.
If we do this for lots of issues and have a norm of making models composable this would have further benefits.
People would use the models to make real world decisions with more accuracy
We would create frameworks for modelling that would be easily composable, that other people would use
Both would raise the status and knowledge of the rationalist community.
Found “The Future of Man” by Pierre Teilhard de Chardin in a bookshop. Tempted to wite a book review. It discusses some interesting topics, like the planetisation of Mankind. However it treats them as inevitable, rather as something contingent on us getting our act together. Anyone interested in a longer review?
Edit: I think his faith in the super natural plays a part in the assumption of inevitability.
Trying something new a hermetic discussion group on computers.
https://www.reddit.com/r/computeralchemy/s/Fin62DIVLs
Self-managing computer systems and AI
One of my factors in thinking about the development of AI is self-managing systems, as humans and animals self manage.
It is possible that they will be needed to manage the complexity of AI, once we move beyond LLMs. For example they might be needed to figure out when to train on new data in an efficient way and how much resources to devote to different AI sub processes in real time depending upon the problems being faced.
They will change the AI landscape making it easier for people to run their own AIs, for this reason it is unlikely that corporations will develop them or release them to the outside world (much like corporations cloud computing infra is not open source) as it will erode their moats.
Modern computer systems have and rely on the concept of a super user. It will take lots of engineering effort to remove that and replace it with something new.
With innovation being considered the purview of corporations are we going to get stuck in a local minima of cloud compute based AI, that is easy for corporations to monetise?
By corporation I am mainly thinking about current cloud/SaaS providers. There might be a profitable hardware play here, if you can get enough investment to do the R&D.
Agreed code as coordination mechanism
Code nowadays can do lots of things, from buying items to controlling machines. This presents code as a possible coordination mechanism, if you can get multiple people to agree on what code should be run in particular scenarios and situations, that can take actions on behalf of those people that might need to be coordinated.
This would require moving away from the “one person committing code and another person reviewing” code model.
This could start with many people reviewing the code, people could write their own test sets against the code or AI agents could be deputised to review the code (when that becomes feasible). Only when an agreed upon number of people thinking the code should it be merged into the main system.
Code would be automatically deployed, using gitops and the people administering the servers would be audited to make sure they didn’t interfere with running of the system without people noticing.
Code could replace regulation in fast moving scenarios, like AI. There might have to be legal contracts that you can’t deploy the agreed upon code or use the code by itself outside of the coordination mechanism.
Can you give a concrete example of a situation where you’d expect this sort of agreed-upon-by-multiple-parties code to be run, and what that code would be responsible for doing? I’m imagining something along the lines of “given a geographic boundary, determine which jurisdictions that boundary intersects for the purposes of various types of tax (sales, property, etc)”. But I don’t know if that’s wildly off from what you’re imagining.
Looks like someone has worked on this kind of thing for different reasons https://www.worlddriven.org/
I was thinking of having evals that controlled deployment of LLMs could be something that needs multiple stakeholders to agree upon.
Butt really it is a general use pattern.
I’ve been thinking about non AI catastrophic risks.
One that I’ve not seen talked about is the idea of cancerous ideas. That is ideas that spread throughout a population and crowd out other ideas for attention and resources.
This could lead to civilisational collapse due to basic functions not being performed.
Safeguards for this are partitioning the idea space and some form of immune system that targets ideas that spread uncontrollably.
I’m starting a new blog here. It is on modelling self-modifying systems, starting with AI. Criticisms welcome
Relatedly I am thinking about improving the wikipedia page on recursive self-improvement. Does anyone have any good papers I should include? Ideally with models.