With some training, it was pretty easy for me to double my performance in some of these cognitive tests. I observed no generalized improvement in my cognitive abilities whatsoever.
Teaching to the test is bad, but teaching without any feedback is even worse, isn’t it?
Sorry if it sounds like a false dilemma; the idea is that we could see any kind of feedback as a “test” and some people will find a way to optimize for this “test”. Even if we would define the goal of education as “children are happy and have fun”, some teachers would optimize for their students reporting being happy and having fun, even if it is harmful for their education. If the teacher is determined to optimize for what is measured at the cost of the nameless goal, replacing one test with another is just replacing one set of perverse incentives with another set.
I think the long-term solution is inventing more reliable tests. Most likely, that would be a lot of work. For example, if G is fixed, but there is a training that increases the performance in the test, then we could conclude that the test does not measure G well and we need a better test. Making such test is probably horribly expensive; but if IQ is a good predictor of success, it could be money well invested. Perhaps with some smart approach the costs could be reduced. Or maybe one day we will measure the activity of the brain directly, without having to find out the right words to put on the paper.
For education, the fix seems much simpler. If the problem of “teaching to the test” is that students forget their lessons soon after the test, add more tests in longer intervals (Anki-style). If the problem is memorizing without understanding, add more complex questions which are impossible to answer without understanding. If the problem is teachers telling students the specific questions in advance, make a large pool of questions and choose a random subset of them at the last moment. If the problem is teachers helping students during the test, require a presence of an independent person in the room. Etc.
There are multiple issues with teaching to the test.
Take an issue such as writing a political essay.
A student that writes a well argued text that provides other reasons than the ones that are covered in the curiculum will fail. Standardised testing doesn’t lend itself for judging critical reflection of material.
If you want to standardised the text you can say, you teach a student knowledge X, Y and Z about the American revolution. Every student that can integrate X,Y and Z get’s maximum marks.
A student who integrates two passes. A student who only gets one or zero fails.
If you want to judge a text by a checklist where everyone who has the checklist gets a similar result it’s hard to test whether someone can meaningfully think about politics and go beyond just reciting knowledge.
As far as feedback for writing goes I think peer feedback should be encouraged. If a student manages to convince his peers with a political essay the essay is good.
While we are at the topic of peer feedback, the ability to take feedback and revise your text based on the feedback is a very valuable skill that you can’t effectively test in standardised testing.
Peer Feedback should also be good for teachers. I would suggest that every teacher spends one hour per week sitting in the classroom of another teacher. At best every week another teacher.
Human’s learn best through feedback that doesn’t come after a large time gap. I’m not sure that an individual teacher can learn much about improving his own teaching from the standardised test scores.
Instead of teaching to the test, a teacher could try to teach what his students find interesting.
The teacher who teaches to the test doesn’t work with the natural curiosity of the children but pushes the children in the direction of the preset goals of a test.
At Sudbury Valley one child might be interested in learning reading at 6 years and another at 10 years. That doesn’t mean that one child is smarter than the other. Both can graduate from college.
The idea that every child has to learn the same thing at the same age is problematic. It wastes a lot of energy on trying to teach things to children for which they just aren’t ready in the moment the curriculum thinks they should learn something.
Instead of teaching to the test, a teacher could try to teach what his students find interesting.
For some kinds of students this would work beautifully, for other kinds of students this would fail horribly.
Also the “students find interesting” part may be impossible to evaluate if students of both kinds are present in the same classroom. Could possibly be solved by teachers announcing lectures on different subjects, and students choosing which lecture they will attend.
The idea that every child has to learn the same thing at the same age is problematic.
I completely agree. It’s just easier to organize education this way. Imagine a school with dozen teachers, dozen subjects, a few hundred students, where each student wants to learn different parts of different subjects at different pace.
A part of this mess could be solved by heavy use of computers, but it wouldn’t be easy. For example every subject could have defined its “tech tree” (the structure of lecture dependencies), and for each student the computer would remember which lectures they already passed. The teachers would put their schedules in the computer, and each student would see a list of lectures they are ready to attend. But there would be probably a lot of “holes” in the student’s schedules, when for all lectures given at this moment the student already attended them, or didn’t attend the prerequisities. There could be some activities for these “holes”, such as doing a homework, or an unstructured discussion with other students.
(The “tech trees” are necessary, because otherwise there would be a lot of frustration caused by many students choosing a lecture X and then not understanding it because they lack the necessary knowledge Y; especially in the cases where students didn’t know that X requires Y, or were overconfident about their knowledge of Y-related things.)
A part of this mess could be solved by heavy use of computers, but it wouldn’t be easy. For example every subject could have defined its “tech tree” (the structure of lecture dependencies), and for each student the computer would remember which lectures they already passed.
SRS can tell you very exactly what a student knows and what he doesn’t know. It’s much better than looking at which lectures a student attended to know where they are at.
I completely agree. It’s just easier to organize education this way. Imagine a school with dozen teachers, dozen subjects, a few hundred students, where each student wants to learn different parts of different subjects at different pace.
Sudbury Valley School is effectively such a place, that’s why I used it as example. If you force every school to perform well on yearly standardised tests a school like Sudbury Valley School is practically impossible.
You take a bunch of assumptions about how schools should organise themselves for granted. Standardized testing and curricula force schools to play according to those assumptions. That prevents experimentation with different core assumptions about what school should be about.
Take an issue like emotional management. Even CFAR thinks it’s a highly valuable skill to learn.
It’s not something that a modern standard test where a student tries to score high measures meaningfully.
I could imagine a school where nearly all subjects that get taught are different from the subjects that get taught in our Western curriculum and that still provide kids with meaningful skills for their live.
Reading and basic math could be taught on the side in one our per day in a optimized computer program.
But there would be probably a lot of “holes” in the student’s schedules, when for all lectures given at this moment the student already attended them, or didn’t attend the prerequisities. There could be some activities for these “holes”, such as doing a homework, or an unstructured discussion with other students.
I think we probably want that ever student spends a hour or day on a SRS system anyway. I think looking at Anki 2.X is very interesting. There are so many raw edges and unexplored issues that you could optimize to make it more effective.
We shouldn’t live in a world where the mnemosyne database sits around for years without any researcher analysing it and applying some modern math to build a good model around learning.
I really don’t understand it. I mean from an academic perspective, the data is there. The mnemosyne guys are open to sharing it. Math talent is there.
If you throw good math at learning, it seems like you can get a decent paper out of it. Possibly even a paper that highly cited, when you add some mathematical technique to it that nobody before used on learning.
Calculating SRS repetition dates could be a problem with the same priority as designing computers that play Go or maybe even on the level of protein folding.
The fact that few people care about the problem illustrates well how little interest there is in approaching new ways of learning. I think that lack of diversity between most school is a cause. Getting rid of standarized curricula and standardized testing would open the possibility of schools trying out approaches that differ very much from each other. Different school persuing different concepts then opens up resources to figure out how different ways of learning work.
I think lectures could be replaced by videos and interactive material, and teachers could take more the role of an instructor instead of a lecturer. I think it’s easier to be a high quality instructor than a high quality lecturer so this kind of centralization of lecturing would be good. Because students would have different strengths and weaknesses, it could be arranged that they also instruct each other.
This is pretty much the premise behind Khan Academy. There’s tech trees with motivating level ups, videos, interaction, I think also peer instruction.
I think lectures could be replaced by videos and interactive material, and teachers could take more the role of an instructor instead of a lecturer.
Exactly. Videos are easy to replicate. Humans are interactive. So why not have the best of the both worlds and see the best possible video lecture first, and then discuss it with a teacher?
With the additional advantage that if you miss the school for whatever reason, you could watch the video anyway. Or you could choose to use the same videos for homeschooling.
Gwern has done some literature review on this subject and pretty much concluded that G isn’t trainable.
See also http://www.cambridgebrainsciences.com/
With some training, it was pretty easy for me to double my performance in some of these cognitive tests. I observed no generalized improvement in my cognitive abilities whatsoever.
This reminds me of a common complaint of the US school system, “Teaching to the test.”
Teaching to the test is bad, but teaching without any feedback is even worse, isn’t it?
Sorry if it sounds like a false dilemma; the idea is that we could see any kind of feedback as a “test” and some people will find a way to optimize for this “test”. Even if we would define the goal of education as “children are happy and have fun”, some teachers would optimize for their students reporting being happy and having fun, even if it is harmful for their education. If the teacher is determined to optimize for what is measured at the cost of the nameless goal, replacing one test with another is just replacing one set of perverse incentives with another set.
I think the long-term solution is inventing more reliable tests. Most likely, that would be a lot of work. For example, if G is fixed, but there is a training that increases the performance in the test, then we could conclude that the test does not measure G well and we need a better test. Making such test is probably horribly expensive; but if IQ is a good predictor of success, it could be money well invested. Perhaps with some smart approach the costs could be reduced. Or maybe one day we will measure the activity of the brain directly, without having to find out the right words to put on the paper.
For education, the fix seems much simpler. If the problem of “teaching to the test” is that students forget their lessons soon after the test, add more tests in longer intervals (Anki-style). If the problem is memorizing without understanding, add more complex questions which are impossible to answer without understanding. If the problem is teachers telling students the specific questions in advance, make a large pool of questions and choose a random subset of them at the last moment. If the problem is teachers helping students during the test, require a presence of an independent person in the room. Etc.
There are multiple issues with teaching to the test. Take an issue such as writing a political essay.
A student that writes a well argued text that provides other reasons than the ones that are covered in the curiculum will fail. Standardised testing doesn’t lend itself for judging critical reflection of material.
If you want to standardised the text you can say, you teach a student knowledge X, Y and Z about the American revolution. Every student that can integrate X,Y and Z get’s maximum marks. A student who integrates two passes. A student who only gets one or zero fails.
If you want to judge a text by a checklist where everyone who has the checklist gets a similar result it’s hard to test whether someone can meaningfully think about politics and go beyond just reciting knowledge.
As far as feedback for writing goes I think peer feedback should be encouraged. If a student manages to convince his peers with a political essay the essay is good.
While we are at the topic of peer feedback, the ability to take feedback and revise your text based on the feedback is a very valuable skill that you can’t effectively test in standardised testing.
Peer Feedback should also be good for teachers. I would suggest that every teacher spends one hour per week sitting in the classroom of another teacher. At best every week another teacher.
Human’s learn best through feedback that doesn’t come after a large time gap. I’m not sure that an individual teacher can learn much about improving his own teaching from the standardised test scores.
Instead of teaching to the test, a teacher could try to teach what his students find interesting.
The teacher who teaches to the test doesn’t work with the natural curiosity of the children but pushes the children in the direction of the preset goals of a test.
At Sudbury Valley one child might be interested in learning reading at 6 years and another at 10 years. That doesn’t mean that one child is smarter than the other. Both can graduate from college. The idea that every child has to learn the same thing at the same age is problematic. It wastes a lot of energy on trying to teach things to children for which they just aren’t ready in the moment the curriculum thinks they should learn something.
For some kinds of students this would work beautifully, for other kinds of students this would fail horribly.
Also the “students find interesting” part may be impossible to evaluate if students of both kinds are present in the same classroom. Could possibly be solved by teachers announcing lectures on different subjects, and students choosing which lecture they will attend.
I completely agree. It’s just easier to organize education this way. Imagine a school with dozen teachers, dozen subjects, a few hundred students, where each student wants to learn different parts of different subjects at different pace.
A part of this mess could be solved by heavy use of computers, but it wouldn’t be easy. For example every subject could have defined its “tech tree” (the structure of lecture dependencies), and for each student the computer would remember which lectures they already passed. The teachers would put their schedules in the computer, and each student would see a list of lectures they are ready to attend. But there would be probably a lot of “holes” in the student’s schedules, when for all lectures given at this moment the student already attended them, or didn’t attend the prerequisities. There could be some activities for these “holes”, such as doing a homework, or an unstructured discussion with other students.
(The “tech trees” are necessary, because otherwise there would be a lot of frustration caused by many students choosing a lecture X and then not understanding it because they lack the necessary knowledge Y; especially in the cases where students didn’t know that X requires Y, or were overconfident about their knowledge of Y-related things.)
SRS can tell you very exactly what a student knows and what he doesn’t know. It’s much better than looking at which lectures a student attended to know where they are at.
Sudbury Valley School is effectively such a place, that’s why I used it as example. If you force every school to perform well on yearly standardised tests a school like Sudbury Valley School is practically impossible.
You take a bunch of assumptions about how schools should organise themselves for granted. Standardized testing and curricula force schools to play according to those assumptions. That prevents experimentation with different core assumptions about what school should be about.
Take an issue like emotional management. Even CFAR thinks it’s a highly valuable skill to learn. It’s not something that a modern standard test where a student tries to score high measures meaningfully.
I could imagine a school where nearly all subjects that get taught are different from the subjects that get taught in our Western curriculum and that still provide kids with meaningful skills for their live.
Reading and basic math could be taught on the side in one our per day in a optimized computer program.
I think we probably want that ever student spends a hour or day on a SRS system anyway. I think looking at Anki 2.X is very interesting. There are so many raw edges and unexplored issues that you could optimize to make it more effective.
We shouldn’t live in a world where the mnemosyne database sits around for years without any researcher analysing it and applying some modern math to build a good model around learning.
I really don’t understand it. I mean from an academic perspective, the data is there. The mnemosyne guys are open to sharing it. Math talent is there. If you throw good math at learning, it seems like you can get a decent paper out of it. Possibly even a paper that highly cited, when you add some mathematical technique to it that nobody before used on learning.
Calculating SRS repetition dates could be a problem with the same priority as designing computers that play Go or maybe even on the level of protein folding.
The fact that few people care about the problem illustrates well how little interest there is in approaching new ways of learning. I think that lack of diversity between most school is a cause. Getting rid of standarized curricula and standardized testing would open the possibility of schools trying out approaches that differ very much from each other. Different school persuing different concepts then opens up resources to figure out how different ways of learning work.
I think lectures could be replaced by videos and interactive material, and teachers could take more the role of an instructor instead of a lecturer. I think it’s easier to be a high quality instructor than a high quality lecturer so this kind of centralization of lecturing would be good. Because students would have different strengths and weaknesses, it could be arranged that they also instruct each other.
This is pretty much the premise behind Khan Academy. There’s tech trees with motivating level ups, videos, interaction, I think also peer instruction.
Exactly. Videos are easy to replicate. Humans are interactive. So why not have the best of the both worlds and see the best possible video lecture first, and then discuss it with a teacher?
With the additional advantage that if you miss the school for whatever reason, you could watch the video anyway. Or you could choose to use the same videos for homeschooling.