“Lesswrong community underestimates the risk of nuclear ww3 and overestimates the chance of humanity extinction due to AI”.
Kongo Landwalker
Karma: 130
Who is a more reliant guide? The one who knows the optimal path through the labyrinth, or the one who knows all the corridors in the labyrinth?
If you are a student, You should not try to use vibecoding (or agentcoding) to try to catch up to the current level of productivity you see among researcher and programmers. That is similar to buying stocks when they are overvalued and about to crash.
Be slow. Be behind. Play with things, break things, make your own manual silly versions without libraries, try to understand something by experiments without explanations. When the current mess of uncatchable bugs created by agents explodes, somebody will have to fix things. Thinking properly for hours over four lines of code, anticipating how things can be broken instead of convincing oneself that reading through the generated code is enough.
This might sound silly, but listen: AI is only taught to do the right thing, and THAT is a big problem. When researchers publish a paper (at least how they used to 10 years ago), they play with the topic for hundreds/thousands of hours, trying dozens of alternative formulas, reactions, algorithms, and creating their own variants. They gained huge experience of how things would act in the edge cases and HOW by emergent mechanisms things can become invalid/broken. None of that wisdom is transfered into the final paper, which only shows the optimal way they found. AI can read all the articles in the world and not have enough examples of how to anticipate emergent problems in simple situations.
I find and hear about a lot of stupid bugs in google and microsoft products, and it seems the situation is similar most-where. Ai coding is the second reason behind enshittification. When the internet will crash, you would not like to be dependent on any help. You might prefer to have wisdom instead of a big portfolio.
I know that people, when start overusing some word, stop recognising its original meaning. Eventually the word’s accepted meaning can change in natural language.
Minmaxing trap is not happening. I am only allowed to do one edit per finished session, and that edit can be just an increment by 1 or a decrement by 1 of some parameter in the generator, which takes ~15 seconds. If my priorities change, the generator will eventually converge (easy-in over couple days) through the increments to the new state. That prevents “being hyped” and placing “all in” into some new exciting project. The new project will gain weight only if it keeps looking worthy.
I may adjust at the end of a session if i feel that something should happen sooner/more often, or something was promted too often over the past week, or i felt that the session time length was inappropriate to make an unregressable progress, etc.
I put anything I want to do eventually. That includes “work on the publication”, “work on fanfic”, “make the geolocation script”, “update my transformer”, “play a match of king of the hill chess”, “calisthenics”, “solve project euler problem”, “go to the cinema and watch avatar 3″. Both fun and serious stuff.
I am only generating the activity when there is a moment no scheduled/obligation activities, this way it never interferes with life even if fun activities start randomly appearing more often.
The generator is implemented in Google Sheets, using its in-cell functions, thus accessable on both my pc and phone. At some point I have added a column of calculated expectations “how many % of time is expected to be spent on the activity if the generator did not change parameters over a long run”, but it was distracting and not exactly meaningful, since i change weights every day to reflect energy/mood/inspirations.
Let’s say AI is the greatest risk.
When Nuclear weapons gained that status, it became illegal in many coutries to possess radioactive materials, and facilities got obligations to follow some rules, and special entities to observe were created.
Why is there none of that for AI? For an outsider (if he somehow manages to even follow the ai conversations) it looks absurd: researchers are complaining how risky it all is, but instead of going and protesting on the central streets to make the government introduce overseeing entities, they only make more publications about some improvements in AI.
I understand that those who complain and those who update models can be different groups, but i still do not see anyone protesting on the streets. Yet it is possible to find occasional global warming protest.
Bubble does not only mean the information is not going in, it also means information is not going out.
One of additional problems why info is not going out, is extremely complicated language of lesswrong. Some simple thought on the main page can be made into 30minute read (i believe that is subconcious attempt of people to “match the serious aesthetics of the site”), while some hard topics lack the explanation of notions they use. I just cannot send this to my friends or collegues. I thought lesswrong is supposed to be educational website.
Looking back at this take I think It became even more true. Now creation is made simplier, AI slop is quicker than persobal effort, and the internet is flooded even more.
I have an instrument that helps me mitigate decision fatigue, energy lack and conformism. I created an updatable random generator with a weighted list of all the ideas and activities that cross my mind. I exercise agency at the state of “designing” the free time passage, setting probability weights and side goals. Then I can circumvent fatigue of deciding what to do next, because i can click generate and it see the option. And since clicking generator button is a short action, a habit to actually go and execute the option can be formed.
Lesswrong is a bubble, like many others. I try to be in several bubbles at once. So, I hear from the guys right here that AI is the main risk, from osint enthusiasts that ww3 is gonna kill more people and sooner than ai, and from environment scientists that it doesnt even matter as in 200 year humanity gets significantly reduced by global warming anyway, and from computer security specialists that maybe humanity is too stupid to deserve to live.
Triple dooom and cherry
Partition of India was a good intent, but caused 200k+ dead.
According to ChatGPT 5.2 Jackson Kernion “is likely the same person” as Foreign Man in a Foreign Land.
I have used the same instance of a chat for many different topics, from music theory to scrabble ttg, from spiders to Lean. Apparently, the chat wanted to see some connection in that mess of random questions and linked long-long ago mentioned Nebula subscription to the new question.
I think we live in a world where alignment is impossible. All attention based models in my opinion are complex enough systems to be computationally irreducable (There is no shorter way to know the outcome than to run the system itself, like with rule 110). If it is impossible to predict the outcome with certainty, the impossibility to force some desired outcome follows logically.
Humanity has not solved even the allignment of humans (children).
Do you think that people here would notice whether some accounts are controlled by OpenClaw agents? It has the technical opportunity to join the discussion.
When we learn, we do many mistakes in our approach and adapt our strategy. And when we succeed, we tend to post only the final good version, not mentioning how many fails we had.
And thus AI learns only on successes. It gets the wisdom of how to do things right, but not about what could go wrong.
I mostly use ChatGPT and Claude, and I have noticed that sometimes to solve my niche question they propose code that just does not work. It looks like a reasonable piece of code (according to intuition formed by mainstream languages), but appears to be unreadable by compiler due to some undocumented language inconsistency. Several times I managed to find relevant questions on stackoverflow, but those questions had nonsense replies (where people instead of acknowledging there is a problem just say “this question has been asked, look here” and link to an irrelevant question, which uses similar concepts).
As soon as I hit such a problem in development while talking to a chat, the chat becomes useless. AI pushes back with “are you sure? it must work”. It feels like at this moment AI inherits overconfidence of those gaslighting stack overflow incorrect answerers, and continues to push against any feedback, not being able to comprehend it is not right.
I am interested in cooperation/teamwork scores/metrics. Is there any research paper trying to map cooperation into any metric that is designed for single humans?
For example, what chess Elo rating would two 1000 rated humans achieve, if they are allowed to duscuss strategy during the game? How much higher IQ score of a pair of cooperating people is expected to be compared to the max of their individual scores? How much faster a team of people would solve computer puzzle game Witness?
Upd. How does a human+LLM score compare to single human score?
What if instead of cooperation and tools we use handicap. How much worse blindfolded chess players perform?
The minimal change that in my opinion tilts the situation in the healthy direction (but is absolutely not economically profitable and thus unrealistic), is to limit the posting frequency. Can’t make more than 1 publication per day/week. I bet most people would think just a bit more between they post, and they would not be able to spam content into twitter/reels etc.
Limitation of internet exposure time is the complement idea that is already quite spread. Timers, minimalist phones. But the person is controlling exposure, something about themselves.
Controlling/limiting the production is something about others. People, who would most benefit from thinking more between talking, would not go to such a website. And that us why I said it is unrealistic. Although it is possible to create a social meme to “make it cool” to use a website with limitations, and promote that “feed becomes less saturated with slop”.
Continuation of my previous quick take about teachers.
Long ago I read a thing claiming that some types of anecdotes can be hard to remember because of their absurdity. Things that do not make sense are funny specifically because they do not match correctly into your knowledge. I think that is also why it is difficult to recollect confusions in education: at least one side of the conversation was having a wrong model, which cannot be remembered by just a natural association. It takes more space in memory to remember other person’s incorrect image of the world.
I am trying to recollect as many confusions as i can. I had an mechanical engineering class (materials, geometry, kinematics, loads, surfaces). Over the semester we had to design a gearbox. In the middle of the semester I noticed 2d technical drawing, front and up views. The girl was apperently ready to be defending her design. But I stopped her. She had a fatal flaw, which is only apparent when you have 3d understanding what those drawings mean. She did not have that, even though drawing looked good in other aspects (clean lines, correct sizes, top and front allign). The teachers were so strict about those clean lines that the student optimised to make it look good, not to match common sense. The shaft, which goes through the whole thing, was intersecting a gear of another shaft. Matter was passing through matter.
The crazy thing is, I am 80% sure teachers in my first uni would/did not notice that (her handmade kinematics draft must have been approved, since she was on the second stage), because of how they used to rigor for small imperfections (“You chose the wrong smootheness for this surface, go remake”) and to not watch the drawing for longer.
Both students and teachers can have faults in everything, even in common sense.
My teacher in Strength of materials also did not posess 3d imagination. When drawing a crystal structures, she only succeeded in flat images, failing immediately to draw tetrahedron and more complex projections. The students did not understand that she is drawing nonsense, and were just memorising, not questioning, really handicapping their understanding.
One of my teachers once made a silly mistake, so i thought. She forgot to place a minus in the exponent. So instead of 10^-11 there was 10^11. Knowing, that my classmates will blindly memorise it like this, I came to point that out. When I pointed that out, I was met with “I teach this formula like that for eleven years! Do you think I have been making a mistake for eleven years, or maybe you made a mistake?”. Her argument was basically authority and tradition! She was not willing to discuss with proper arguments, not willing to see how her formula implies molecules of galactic scale. Even if I was the one that had the confusion, the teacher still has to let the student describe his thought process, to see where is the mistake.
I once found a mistake in the lab book of a PhD professor. I came to tell that he has all indices swapped on couple pages. Like, the speed of electron 1 was multiplied by the mass of electron 2. Couple pages later another mistake, which magically cancelled it out and the resulting formula was correct. If this is indeed a book’s mistake, I knew most students will not understand, will just assume that they are too stupid to grasp and just go on, and thus the whole chapter is dangerous.
The teacher shouted at me. “You should have better went to the art school, like your mother...” (he overheard my private conversation with other student over break before) ”… look at the next page and everything will be clear, because the final formula is correct. I can’t handle that, every year some stupid students come with this question, learn to read first”. The “every year” basically showed me that there is indeed a mistake in his book and not in my understanding.
(The teachers were indeed outrageously stupid in my first uni, so I went to get an education abroad. Why am I thinking about the past and about teaching? I am in Masters program, and my faculty proposed that I teach Math2 in the next semester, and that is quite a responsibility, since I value proper teaching so high)
Are you familiar with game Nomic? It has original ruleset with immutable/mutable ruled and new rules are decided by democracy. https://www.nomic.net/deadgames/Nomic-Game/Rules/first.html
If i have to prove my rule was not too hardcore, that means I demostrate the exact strategy that opponent has to use to overcome the puzzle. I do not get the full benefit of the rule, since it is “solved”.
This test does not seem easy to automate. Who decides the win? Who resolves the rules’ collision? Whatever algorithm does those two things will be the blind optimisation goal. You do not have to be smarter or more creative than opponent, if you can optimise for “judge resolving conflicts in your favor”.
Couple of my own confusions. During aerodynamics the lecturer made a drawing of a cube, and named it elementary particle. I, to fit it better into my knowledge base aquired from physics course, asked whether it is the same as elementary volume. Perfect particle is a point, but the image had three dimensions, was squeezed and rotated later into lectures. For some reason teacher failed to understand what i meant, said couple things about particles, but not related to volumes and concepts from physics at all, and asked “did i answer your question?”. I had to say no, and since then the teacher hated me.
In my 5th grade during geometry we studied the concept of orthogonality. The teacher said (translating into english word for word) “line is orthogonal to plane if it is orthogonal to any line in that plane”. In my language the word corresponding to “any” means “take one arbitrary thing and test”, like in “bring me any apple”. My teacher, i assume was used to english papers, where “any” and “every” can both mean “forall”, failed to see why I opposed.
I want to speculate too.
When a toy is not the favourite, it is still alive. If it is broken, it is still alive. Same if it is given to other kid or when the person grows out of them. But if the kid gets hooked and stops having “unstructured play”, he loses fantasy and then it is the scenario when the toy dies.
I would make the story the first tragedy in the series. The abandonned-for-ipad toys lose magic and color, become truly dead even when nobody watches.