Hi there! I didn’t sign up before because this community tends to comment what I want to say most of the time anyway, and because signup hurdles are a thing and lack of OpenID support makes me frustrated.
I’ve been reading LW intermittently for about one and a half years now; whilst integrating these concepts in my life is something I tend to find hard, I have picked some of these up. Specifically anchoring effects and improving my ability to spot “the better action”. It’s still hard to actually take such actions; I’ll find myself coming up with a better plan of action and then executing the inferior plan of action anyway.
I’ve been horrified at a few of my past mistakes; one of them was accidental p-hacking. (Long story!)
One of the things I had to do for my college degree was performing research. I picked a topic (learning things) and got asked to focus on a key area (I picked best instructional method for learning how to play a game). We had to use two data collection methods; I wanted to do an experiment because that was cool, and I added a survey because if I’m going to have to ask lots of people to do something for me, I might as well ask those same people to do something else. Basically I’m lazy.
My experiment consisted of a few levels (15) in which you have to move a white box to various shapes by dragging it about. I had noticed that teaching research focused on “reading” “doing” “listening” and “seeing” types, (I forgot the specific words, something about Kinestetic, Audititive, Visual… - learning). So I translated to “written text”, “imagery”, “sounds and spoken text”, and “interactivity” to model the reading, seeing, listening and doing respectively.
Then I made each level test a combination of learning methods. First “learning by doing” only. Here’s a box. Here’s a green circle. Here’s a red star. Go.
Most people passed in 5 seconds or in 1 minute. This after I added a background which was dotted so that you’d see a clear white box and not a black rectangle, and a text “this is level 1, experiment!”. Some people would think it was still loading without this text. I didn’t include the playtesters in the research result data.
After that it showed you 4 colored shapes and a arrow underneath, and a button “next” below it. Hitting next moves you to level 2, where a white box is in the center of the screen, and various colored shapes are surrounding the white box. Dragging the white box over the wrong shape sends you back to the screen with the 4 colored shapes and the arrow.
This was supposed to be “imagery”.
Then the next screen after that was an audio icon and a “next button”. I had recorded myself saying various colored shapes, and people were told at this screen something like “black circle, red triangle, blue star, green square”. The idea being you’d have to remember various instructions and act upon them. Hitting the next button brings you to the surrounded white box again. Each level had a different distribution of shapes to prevent memorizing the locations.
Then the 4th text level was just text instructions (“drag the white box over the green circle, then the red star …”)
Then after that came combinations—voiced text, text where I had put the shapes in images on the screen as well, shapes + voice saying what they were… for interactivity, I skipped the instruction screen and just went with text appearing in the center of the screen, and then the text changes when you perform the correct action (else level resets). This to simulate tutorials like “press C to crouch” whenever you hit the first crouch obstacle.
I had recorded the time spent on the instruction screen, the total time for each level, and per attempt, the time between each progress step and failure. So 1.03 seconds to touch the first shape, 0.7 to touch the second, 0.3 to touch a third wrong one, then 0.5 to touch the first, 0.4 to touch the second, 0.8 to touch the third and 1.0 to touch the fourth and level complete.
The idea was that I could use this to see how “efficient” people were at understanding the instructions, both in speed and correctness.
(FYI, N=75 or so, out of a gaming forum with 700 users)
Then I committed my grave sin and took the data, took excel’s “correlate” function, and basically compared various columns until I got something with a nice R. This after trying a few things I had thought I would find and seeing non-interesting results.
I “found” that apparently showing text and images in interactive form “learn as you go” was best—audio didn’t help much, it was too slow. Interactivity works as a force multiplier and does poorly on its own.
But these findings are likely to be total bogus because, well, I basically compared statistics until I found something with a low chance to randomly occur.
… What scares me not is not that I did this. What scares me is that I turned this in, got told off for “not including everything I checked”, thought this was a stupid complaint because look I found a correlation, voiced said opinion, and still got a passing grade (7/10) anyway. And then thought “Look, I am a fancy researcher.”
I could dig it up if people were interested—the experiment is in English, the research paper is in Dutch, and the data is in an SQL database somewhere.
This is probably a really long post now, so I’ll write more if needed instead of turning this into a task to be pushed down todo lists forever.
I believe the whole idea of learning styles is a pseudoscience, so you not finding more correlations could actually be the correct answer… which almost no one cares about, because negative findings are boring.
Publication bias is probably even greater sin than p-hacking, because in theory any study that found some result using p-hacking could be follow by a few failed attempts at replication. Except that those failed attempts at replication usually don’t get published.
The idea of learning styles as “fits better to a specific person” wasn’t interesting to me—instead I took it as inspiration for natural division of “ways people could learn this thing in general”.
As for publication bias, I don’t think anyone published their research. … but if there had been a really interesting result, I bet someone would have tried to get their research published somehow.
Hi there! I didn’t sign up before because this community tends to comment what I want to say most of the time anyway, and because signup hurdles are a thing and lack of OpenID support makes me frustrated.
I’ve been reading LW intermittently for about one and a half years now; whilst integrating these concepts in my life is something I tend to find hard, I have picked some of these up. Specifically anchoring effects and improving my ability to spot “the better action”. It’s still hard to actually take such actions; I’ll find myself coming up with a better plan of action and then executing the inferior plan of action anyway.
I’ve been horrified at a few of my past mistakes; one of them was accidental p-hacking. (Long story!)
One of the things I had to do for my college degree was performing research. I picked a topic (learning things) and got asked to focus on a key area (I picked best instructional method for learning how to play a game). We had to use two data collection methods; I wanted to do an experiment because that was cool, and I added a survey because if I’m going to have to ask lots of people to do something for me, I might as well ask those same people to do something else. Basically I’m lazy.
My experiment consisted of a few levels (15) in which you have to move a white box to various shapes by dragging it about. I had noticed that teaching research focused on “reading” “doing” “listening” and “seeing” types, (I forgot the specific words, something about Kinestetic, Audititive, Visual… - learning). So I translated to “written text”, “imagery”, “sounds and spoken text”, and “interactivity” to model the reading, seeing, listening and doing respectively.
Then I made each level test a combination of learning methods. First “learning by doing” only. Here’s a box. Here’s a green circle. Here’s a red star. Go.
Most people passed in 5 seconds or in 1 minute. This after I added a background which was dotted so that you’d see a clear white box and not a black rectangle, and a text “this is level 1, experiment!”. Some people would think it was still loading without this text. I didn’t include the playtesters in the research result data.
After that it showed you 4 colored shapes and a arrow underneath, and a button “next” below it. Hitting next moves you to level 2, where a white box is in the center of the screen, and various colored shapes are surrounding the white box. Dragging the white box over the wrong shape sends you back to the screen with the 4 colored shapes and the arrow. This was supposed to be “imagery”.
Then the next screen after that was an audio icon and a “next button”. I had recorded myself saying various colored shapes, and people were told at this screen something like “black circle, red triangle, blue star, green square”. The idea being you’d have to remember various instructions and act upon them. Hitting the next button brings you to the surrounded white box again. Each level had a different distribution of shapes to prevent memorizing the locations.
Then the 4th text level was just text instructions (“drag the white box over the green circle, then the red star …”)
Then after that came combinations—voiced text, text where I had put the shapes in images on the screen as well, shapes + voice saying what they were… for interactivity, I skipped the instruction screen and just went with text appearing in the center of the screen, and then the text changes when you perform the correct action (else level resets). This to simulate tutorials like “press C to crouch” whenever you hit the first crouch obstacle.
I had recorded the time spent on the instruction screen, the total time for each level, and per attempt, the time between each progress step and failure. So 1.03 seconds to touch the first shape, 0.7 to touch the second, 0.3 to touch a third wrong one, then 0.5 to touch the first, 0.4 to touch the second, 0.8 to touch the third and 1.0 to touch the fourth and level complete.
The idea was that I could use this to see how “efficient” people were at understanding the instructions, both in speed and correctness.
(FYI, N=75 or so, out of a gaming forum with 700 users)
Then I committed my grave sin and took the data, took excel’s “correlate” function, and basically compared various columns until I got something with a nice R. This after trying a few things I had thought I would find and seeing non-interesting results.
I “found” that apparently showing text and images in interactive form “learn as you go” was best—audio didn’t help much, it was too slow. Interactivity works as a force multiplier and does poorly on its own.
But these findings are likely to be total bogus because, well, I basically compared statistics until I found something with a low chance to randomly occur.
… What scares me not is not that I did this. What scares me is that I turned this in, got told off for “not including everything I checked”, thought this was a stupid complaint because look I found a correlation, voiced said opinion, and still got a passing grade (7/10) anyway. And then thought “Look, I am a fancy researcher.”
I could dig it up if people were interested—the experiment is in English, the research paper is in Dutch, and the data is in an SQL database somewhere.
This is probably a really long post now, so I’ll write more if needed instead of turning this into a task to be pushed down todo lists forever.
I believe the whole idea of learning styles is a pseudoscience, so you not finding more correlations could actually be the correct answer… which almost no one cares about, because negative findings are boring.
Publication bias is probably even greater sin than p-hacking, because in theory any study that found some result using p-hacking could be follow by a few failed attempts at replication. Except that those failed attempts at replication usually don’t get published.
The idea of learning styles as “fits better to a specific person” wasn’t interesting to me—instead I took it as inspiration for natural division of “ways people could learn this thing in general”.
As for publication bias, I don’t think anyone published their research. … but if there had been a really interesting result, I bet someone would have tried to get their research published somehow.