Another treatment of Direct Instruction getting more into the technical details of the theory
[NOTE: This was a discussion post asking if anyone would mind giving feedback on a very rough draft in progress.
If you are downvoting it because you do not want to see discussion posts asking for feedback like this, then by all means, that’s a valid use of a downvote.
But if you are downvoting it in order to express your opinion of the quality of the draft, I urge you to reconsider]
This is another work in progress coming at the DI issue from a somewhat different direction. It’s contained in the comments of the original, and I’m posting this to ask for more wonderful beta-reader critics to tell me if it’s a step in the right direction. (It’s still very informal writing, but it’s the ideas I’m dealing with now.)
And about what I’m looking for in the LW audience, someone asked me in a private message:
… who is the audience, here? Are you hoping that LW readers are school administrators, who will introduce DI into their schools? Are you hoping that they are teachers, who will introduce DI into their classes? Are you hoping that they are students, who will be able to seek out instructors using DI? …
I’m personally interested because I have an interest in alternate education methods; I think the method sounds promising and what I know matches up with what I know about solid epistemology. The people who are interested in epistemic rationality (another group you could tailor posts towards) would probably be interested in learning about epistemic methods that are quantitatively superior to others.
And I said:
Oh, I dunno about DI involving “epistemic methods that are quantitatively superior to others”. The founder recently wrote a book about what John Stuart Mill could have done for education, so that’s the epistemology that DI is applying.
So actually, another reason I keep using Newton’s laws analogies is that I suspect there’s an analogious ‘general relativity’ to be found.
So what I really want is for the people from the LW audience to learn DI theory themselves, because I think they could improve the theory.
Well, that’s the major part of what I want that’s important here. I also had to add:
you remember how I mentioned ‘creative strategic twists’ [for how we could help DI win, and how DI could help us win], and indicated that the inspiration for that came from comparing the Michel Thomas lessons with DI proper (the internal details of each and their separate histories)?
That’s another long-inferential distance topic...
But that’s not important here (except to disclose that is where I’m coming from). LWers would first have to understand DI to fully grasp that. And I am significantly less certain of my current beliefs about those ‘strategic twists’ (although still pretty certain), and LWers proficient in DI would be the best to evaluate the ideas.
There is exactly one thing you could do that would be more effective in improving your communication about DI than anything else, by an order of magnitude:
Explain, accurately and briefly but with enough detail that someone could give the method a go if they wanted, just what the hell Direct Instruction actually is.
Because (1) that would actually be useful to us and (2) failing to do so sets off (for me, and I’m guessing for lots of others) all sorts of mental alarms of the “someone’s trying to scam you” variety. Enthusing about how wonderful something is while not saying just what it is is an absolutely standard behaviour of snake-oil salesmen, proselytizers for cults, etc., and you are not putting yourself in good company by adopting a similar approach.
(For the avoidance of doubt: I am not calling you a snake-oil salesman or cultist. I am still assigning a fairly high probability to their being something useful behind the puffery. But if you don’t very quickly cut to the chase and tell us enough about what you’re talking about for us to understand it, that’s going to change. I don’t think I’m alone in this.)
A quick Google search turned up this promising report on Direct Instruction. Unfortunately it reads very similarly to Owen’s posts—lots of trying to convince me Direct Instruction is great great great, practically no information about what DI is, even in the section entitled What Is Direct Instruction? I’ve updated in the “DI is legitimately hard to summarize” direction.
Thank you very much for the IM convo you had with me resolving all those confusions I had about how I had miscommunicated various points. I just thought I should post this comment before I go and use what you gave me to write a much better clarification post for discussion, since the comment was almost done when you IM’d me.
I hope that report doesn’t read too similarly to my posts! I only did the quickest skim of the “what is DI” section, but it seems to be like an abridgement of “Rubric for Identifying Authentic Direct Instruction Programs” (prologue here ) that’s even more abridged than my treatment of Theory of Instruction.
I wouldn’t even try to communicate DI theory across a large inferential distance (unless the audience already had a positive impression of it from personal experiences teaching from DI programs after failing with low performing kids, to make them patient).
But with LW, I figured that after showing the experimental evidence of effectiveness (in the way teachers who had previously been failures became successful after they started using DI programs), to convince some LW people to study the theory themselves, they would find it much easier to understand because of their prior understanding of concepts like extensional/intensional definitions, ‘looking into the dark’, and thingspace.
For instance, when studying the theory behind the design of templates for teaching ‘noun’ concepts (or ‘multi-dimensional non-comparatives’), any LWer should go, “Ohhh, this is taking the basics of what Eliezer was talking about with the example of the ‘bird-cluster’ in ‘thingspace’, and applying it make sure you’re giving enough information to someone else who doesn’t already know what the label ‘bird’ refers to that they can find the cluster!”
And of course as we discussed by IM, the way that “logically faultless communication” relates to the “2-4-6 game” and is “logically faultless” in the same way Bayes is the logically faultless way of doing induction yourself, but that doesn’t guarantee that a particular user won’t misapply it. But if they do misapply it, someone who knows how to apply it correctly can figure out exactly how the mistake-maker must change to become correct.
But yeah, I’ll go get on that clarification post for discussion now.
Thanks again!
I haven’t done so not because it hasn’t occurred to me to try, but because if it’s even possible (which I doubt), it’s hard enough to do that I still haven’t figured out how.
Did you read my short analogy explaining how this difficulty feels to me from my perspective? It feels like trying to explain what physics is and why it’s so powerful as well as interesting in itself, to someone who has never seen physics, hasn’t grown up in a world filled with obvious examples of the amazing engineering it makes possible, etc.
Really really really try yourself to: “Explain, accurately and briefly but with enough detail that someone could give the method a go if they wanted, just what the hell physics actually is”, keeping in mind that your audience is someone who has never heard of it or seen anything other than pre-physics level technology.
I wish I could find the exact source for this, but I remember once reading a quote of some famous Roman general saying that military technology was at the highest limit it could reach, with the ballista or something representing the absolute apex of possible achievement.
Imagine you’re addressing this to contemporaries of that guy.
If you can’t instruct someone as to what DI is, it might not be living up to claims its a ground-breaking instructional process.
The best quick reply I can think of to that is another analogy I already made:
So please avoid meta-judging this?
The main difficulties in giving an accurate and brief statement of what physics is for someone who’s never seen anything like it are (1) that physics is a very big subject and (2) that your stipulation about the audience seems to be intended to assume they don’t know any mathematics to speak of either.
Are you claiming that DI is a hugely varied subject like physics? (If so: How is that even possible, given how recent it apparently is?)
Are you claiming that DI depends on a large and conceptually very difficult substrate, as physics does with mathematics? (If so: What is that substrate?)
If the answers are both “no”, then I’ll adjust your question to fix those two mismatches, and attempt to give a brief summary of (not all of physics, but) Newtonian particle mechanics for (not someone completely ignorant of mathematics, but) someone who has college-level mathematics but has somehow managed not to hear that it can be applied to understanding the physical world.
“This is a theory about the motion of small rigid objects such as rocks. The same theory can actually handle large objects by considering them as aggregates of smaller ones, but for the moment we’ll only consider objects that are very small. So, suppose we have a bunch of small objects. At any instant we can describe the state of the universe by saying (1) where these objects are—we specify this by giving three coordinates for each object—and (2) how fast and in what direction each object is moving—we specify this by another set of three coordinates, which collectively we call the velocity. And the only other thing there is to know about each object is its mass, a single positive number that in some sense describes how much of it there is; it’s closely related to how heavy it feels when you hold it.
“Now the basic rules are as follows. First: the derivative of position with respect to time is the velocity. Second: the derivative of velocity with respect to time, which we call the acceleration, is completely determined by a set of rules that care only about the positions, velocities, and masses of the objects. (As long as they don’t actually crash into one another. We’ll talk about that in a moment.)
“Third: the rule for finding the acceleration is as follows. The acceleration of each object is a sum of vectors, one for each other object. What’s happening here is that all the objects are pulling on one another; objects with more mass pull harder. More precisely, the formula goes like this: the acceleration of an object with mass m is the sum over all the other objects of Mr/|r|^3 where r is the vector from our object to the other object and M is the mass of the other object. So the further away the other object is, the less effect it has; in practice this means we can mostly ignore objects very far away.
“Here’s an exercise, to give you a feeling for what the theory lets you do. Define the momentum of an object to be the mass times the velocity. Then prove that the sum of all the momenta in the system never changes.
“This system takes some getting used to, although basically it’s just fairly elementary mathematics. Let me mention a few of the things that follow from it. We just saw a “conservation law”: the total momentum is constant. There are a whole lot of other conservation laws, and in many cases one can make detailed predictions using only those. If you have one object whose mass is very large and all other nearby objects have much smaller mass and don’t get too close to one another, then you can pretend that the first object stays fixed in place, and that the others’ motions are affected only by it, and in that case it turns out that their paths are always conic sections. In particular, our solar system is like this, and the planets move in near-perfect ellipses. If you do the calculations more carefully, taking into account the planets’ influence on one another, you can get very accurate results, and in fact someone once found a previously unknown planet by noticing that another already known planet wasn’t moving quite as expected and working out where another planet would have to be to produce the observed deviations.
“That’ll do for now. If you want a more difficult challenge, try to prove the statement I made about conic sections.”
...Okay, that is honestly an impressively rapid bit of writing in its own unusual context (as a summary of newtonian particle mechanics to someone who had college math but had somehow never heard about physics, like you said.)
But my original analogy was never meant to be expanded, because it was never an argument by analogy in the first place, as I told Jem
OK, so the only point of your analogy was to explain how you feel about being asked to explain what DI is? Fair enough. Then all I can say is: It doesn’t seem to me that you’ve successfully communicated how you feel about that, since apparently you find it unreasonable to be asked to explain what DI is, whereas if I try to imagine myself in any genuinely-analogous situation then I don’t find it unreasonable to be asked the corresponding question.
I’m also rather confused about what your actual problem with explaining what DI is is. You say it’s kinda like the problem someone has who’s asked to explain what physics is. But (see above) it seems to me to be highly unlike that problem in a couple of respects that (for me) seem to be central to the difficulty of the problem. So what, in fact, is it about DI that makes it so difficult for you to say what it is?
An uncharitable explanation suggests itself: Perhaps you cannot say just what DI is because you don’t know what DI is; you’ve read things saying how wonderful it is, and you’ve experienced something that purports to be DI and found that it works well, and concluded: DI is great—without actually pinning down just what DI is. I don’t think this is terribly likely (though I suspect that situations of roughly that sort are quite common, and for sure I have been in them myself), but it might be useful for you to try explaining why you don’t think that’s what’s going on. (Assuming that you don’t.)
Yeah, this analogy-laden meta-digression is getting a bit ridiculous, I agree. Forget the physics stuff, at least for now.
Yeah, I am just a student of DI theory myself, largely just reciting outlines of my own mental notes.
If you could possibly find the time to check the online catalogs of any university libraries near you to see if they have the book… because if you could easily get your hands on a copy, it wouldn’t be too hard to just try skimming the section and chapter summaries.
Quite honestly, yes, that is how it started.
But I was actually explicitly aware of it at the time, that my emotional experience with the Michel Thomas lessons was almost surely biasing me in my initial tentative vague estimate that there was a somewhere more than 50% chance that the results from Project Follow-Through were pretty much representative of something true about DI’s effectiveness in practice.
Although just because the experience with the Michel Thomas lessons was emotional doesn’t mean it should have been discarded as evidence, does it? Especially considering that I also had some evidence that many other people had had similar experiences (my vague impression that the ‘marketing anecdotes’ surrounding them as a product were slightly more numerous and slightly more gushing than usual, especially given how the lessons were in surface appearance much less polished compared to their competitors)… so maybe the bias wasn’t so bad, but I knew I had a general human bias to underestimate my biases, and might therefore overcompensate for it… which is a line of thought that just goes into insanity, so at the time the sanest thing I could do was accept my feelings of how good my experience with the audio lessons was as evidence as valid, right? As the best working level at the time?
Anyway, yeah, my estimate of the probability of there being something to DI theory, even though I found it just as mystifyingly vague as you did at first, was obviously bumped up a lot by my slightly stronger faith in the Project Follow-Through graphs as representing something true about DI’s practical effectiveness.
And as I found that bits of DI theory that had just seemed like techno-babble at first started to actually become meaningful to me, in recursive layers, I started to get really quite sure.
At this point, I would be very surprised if any evidence I found that contradicted DI actually held up under scrutiny (and yes, give it a correspondingly greater weight if it did!)
...And from that story you could probably give me some great feedback on my current level of general strength as a rationalist. How’s my epistemic driving? (Although I realize you’re in a position where you should probably expect that if you keep looking into DI theory your probability estimate of it being valid will more than likely move from ‘somewhere in the middle?’(?) to a position much closer to either 0 or 1, and that might complicate things… or not? I’d have to think about that.)
...This is me working on less than four hours of sleep a night for three days in a row, by the way. I’ma go to bed now.
But physics and DI are different in this important way: physics is a classification of knowledge, and DI is a technique for communicating knowledge. It’s reasonable to ask for a functional description of a technique, even though it wouldn’t be reasonable to ask for a functional description of a classification of knowledge. I don’t think your analogy works.
You’re claiming that DI is a way of teaching people things. You’d like to teach us what DI is (or, at least, we’re giving you the benefit of the doubt in assuming that this is your goal). However, you’ve currently been successful in teaching (as near as I can tell) exactly zero of us what DI is. If you’ve succeeded in teaching zero people the thing you’re trying to teach, I suggest this is evidence that you don’t have a good teaching method.
Here are some things I don’t yet care about, and can’t care about until I know what DI is. If your response addresses these non-concerns, my doubts about your stated goal will increase:
I don’t care if DI is effective.
I don’t care if DI is well thought-of by others.
I don’t care if DI is the result of a new paradigm or an old one.
Here are two things that I do care about:
I’d like to know what DI is.
I’d like to know if you, personally, are a member of the Church of Scientology.
Sorry I have to include that last one, but your behavior so far absolutely mimics a common CoS MO of introducing a new program which is vague on details but turns out, in practice, to be Scientology (see, for example, Narconon).
I… if I had to list a million different things I expect I might one day be asked, I don’t think “Are you from the church of scientology?” would ever occur to me. I have no idea what ’Narconon” is.
Well, DI is a way of applying a classification of knowledge to creating effective communications of other knowledge, but that’s not the point. The analogy wasn’t meant to be expanded beyond that, because it’s not an argument by analogy, just explaining what this difficulty feels like from my perspective.
And the only obvious possible way I’ve been able to see around the difficulty so far, the only thing that could make it easier to convince you that DI is awesome than it would be to convince a Roman general (who thinks ballistas are the apex of possible military technology) that physics is awesome is that you already have generalized concepts of things like “science” and “rationality” that I can activate with verbal tags like “science” and “rationality”.
So I’m trying to leverage that.
The only thing I’m trying to sell you is that you try to get a copy of Theory of Instruction yourself from a local university library and at least try to start reading the section/chapter summaries. And even if you can’t get it at a library, the book is forty dollars at the ADI store, which is a damn cheap text book.
The reason you should care if DI is effective is that showing you the results from the Project Follow-Through graphs and the quote from the meta-analysis is the only quick clear way I’ve been able to think of to convince you that doing so is worth your time.
(And yeah, I solemnly swear that I’m not getting a cut of that $40 bucks in any way, shape, or form. I bloody well hope that if I ever tried to scam people I’d be able to come up with more effective ways than this...)
I’m pretty certain (say, 98%) that (1) you aren’t a member of the Church of Scientology and (2) the CoS has nothing to do with “Direct Instruction”, but it seems worth pointing out that after Jem asked you whether you are a member of the CoS you replied with something that strongly suggests you aren’t without actually saying so. Just for the sake of explicitness, would you care to answer the question? :-)
Ha, upvoted cuz this one made me laugh! =]
NO, I am not a member of the Church of Scientology, or in any way sympathetic to their views (although I do feel sorry for those poor messed up people as I do for every other person who’s living their life drowned deep beneath the sanity water line).
The only common idea on LW that I can think of offhand that I don’t think is really part of the correct contrarian cluster is the 3^^^3 dustspecks thing, and I’m damn sure many LWers would agree with me on that one.
My reply has so far gotten 3 upvotes, which somewhat suggests that your certainty of points (1) and (2) has risen from “say 98%”, without actually saying so. Just for the sake of explicitness, what is your current estimate? :P
As it happens, none of those upvotes was from me. But, also as it happens, your explicit answer does indeed raise my probability estimate for #1 and (indirectly) for #2, and the fact that no one else has chimed in to point out a Scientology-DI connection raises my probability estimate for #2 and (indirectly) for #1; my estimate for #1-and-#2 is now more like 99% or so. Not that I really trust estimates like these very much unless they’re backed up by credible calculation, which this one isn’t.
So, if I were to make you a bet that #1-and-#2 is true that you should rationally take if you believe that 99% estimate, it must be set up so that gjmgain*0.01>owengain*0.99...
So unless I’m making some embarrassing simple math mistake here, if I put up say $2000 (Canadian) for “gjmgain” (wish I had more to play with, but unpaid intern, no work visa here, etc), you should be willing to put up anything less than $20.20...
Ah! But what if rather than money you had to put up that you would read the entire Theory of Instruction and the entire Research on Direct Instruction?
If you seriously made such a bet, then gjm would probably update on that evidence and revise his 99% probability upwards.
[But as gjm says the bet is impractical anyway because it’s too hard to resolve]
We’d get it settled by a respected public figure. Someone like Eliezer obviously springs to mind in the context of LW.
And believe me, I thought of how me just offering the bet should update him above his current level. I’m just trying to get him to read books, obviously :P
That would (I’d have thought) take quite a lot more than CDN$20 worth of time. (Unless I were just skimming through it rather than reading it thoroughly enough to be any use.) Though I’d presumably expect some nonzero benefit from reading the book, at least in the “99%” scenario, and that would need factoring in.
Also, how would the bet actually get resolved? I mean, in the “1%” case you’re a Scientologist and willing to lie brazenly about the fact; conditional on that, other Scientologists are probably very unreliable too so I can’t just ask whatever gathering of Scientologists is closer to what you say is your physical location (note: actually I’m pretty sure Scientologists are very unreliable anyway); so how could I be sufficiently—well over 99% -- convinced?
I already thought of all that. You know I’m just trying to get you to read the books. :P
Obviously, we’d both have to give our bets (well, my money and your promise) to a respected public figure, someone like Eliezer being an obvious candidate in the context of LW, of course.
The other thing besides just trying to get you to read is that I’m annoyed that you still have such a ridiculously high estimate of me being involved in scientology as 1%. I think that from what you know of me now, and what I know of you, I should have a higher estimate of you being involved with the CoS :P
I did say that the 99% figure was very rough and I wouldn’t trust it much. But yeah, I’m probably mostly “privileging the hypothesis”.
Considering that you didn’t even try to see if I was making a bluff by offering to bet me one cent against my $2000...
S=probability of scientology involvement
2000S<0.01(1-S)
2000S<0.01-0.01S
2000.01S<0.01
S<0.000004999
Again, assuming I didn’t make any embarrassingly simple math errors, that’s an over 99.9995% confidence that the ‘scientology-related’ hypothesis was wrong.
Not that this is factoring in the hassle for both of us of setting up the judging and so on, but still, right? :P
But “the hassle of setting up the judging and so on” makes something like two orders of magnitude difference to the probability estimate here. And why would I want to call your bluff in that way?
I went to the library and checked out a copy of Theory of Instruction. There’s a three day weekend ahead, and I’m hoping that before the end of it I’ll be able to explain what DI is all about to everyone else here, and judge whether or not it’s worth anything. Wish me luck!
Thanks! If you’ve already read the AthabascaU open module, that might not be too hard (not that I’ve ever tried to skim just the section and chapter summaries without having read everything else before in order, and can’t do so now, of course.)
Honestly though, I have to keep coming back to the argument: “The results from Project Follow-Through show DI walloping all competing models (including the default ‘traditional’ education), and the meta-analysis of all the other studies doing direct experimental comparisons says that’s typical, so that phenomena needs an explanation. One very likely candidate for that necessary explanation should be that DI theory actually knows what it’s talking about, even if it seems very confusing at first.”
Which seems like a damn reasonable argument to me, but...
You seem to be having trouble explaining a theory about instruction. How much evidence does this provide me about the merits of the theory as a teaching tool?
Given that he doesn’t claim to be trying to use the theory to explain it, or claim that it’s easy for him to use the theory to produce teaching materials for a new subject, and in fact claims just the opposite… not much evidence either way.
Those givens seem to provide information all by themselves.
You don’t think my analogy with the Akashi Kaikyou bridge and all is valid?
I don’t think I said that. I’m mostly going by your own conveyed judgement and the responses you received.
It’s implied that you don’t think that analogy was valid, because if you did, you wouldn’t say “Those givens seem to provide information all by themselves”. You’d say, “Okay, at this point, I think it’s best to refrain from trying to apply a meta-judgement to give any information beyond the fact that creating DI sequences is hard enough for someone who is still a relatively new student to the theory (ie me) that Owen didn’t try to write this post as an example of a complete DI program itself.”
That doesn’t sound true to me. (But the thread is not of sufficient interest to discuss further.)
About valid downvote uses, please see the note added to the beginning of this post.