The Prediction Situation

Every so often you have the good fortune to encounter a piece of information that, although not originally written to endorse a firmly held bias, does so anyway.  While becoming completely immersed in Daniel Kahneman’s Thinking Fast and Slow, I couldn’t help feeling the urge to climb a mountain and yell: “Would everyone involved in education please listen up.  Dr. Kahneman is telling us something we REALLY need to think about!”   

Now I am the first to realize that when you yell, people begin to wonder why you feel the compulsion to be so loud.  Educators, especially those who’ve been yelled at by an irate parent, will go so far as to avoid feeding into the yeller’s need.  Regardless, what the man who enlightened us with heuristics has to say about validity, intuition and expertise should hold our attention.   If nothing else, his insights should become part of the professional conversation (even with my volume emphasis) if we are to evolve our practices in any sort of enlightened and meaningful way.

Open Thinking Fast and Slow to the middle pages and you find Dr. Kahneman appearing to repeat a statement about expertise that we can no longer afford to ignore or undervalue.  Here is what he has to say:

         IT IS NOT THE EXPERTS FAULT-THE WORLD IS DIFFICULT

         The main point of this chapter is not that people who attempt to predict the future make many errors; that goes without saying.  The first lesson is that errors of prediction are inevitable because the world is unpredictable.  The second is that high subjective confidence is not to be trusted as an indicator of accuracy (low confidence could be more informative).

         Short-term trends can be forecast, and behavior and achievements can be predicted with fair accuracy from previous behaviors and achievements.  But we should not expect performance in officer training and in combat to be predictable from behavior on an obstacle field – behavior both on the test and in the real world is determined by many factors that are specific to the particular situation (emphasis added). (Kahneman, 2011)

And then again 22 pages later:

         An experienced psychotherapist knows that she is skilled in working out what is going on in her patient’s mind and that she has good intuitions about what the patient will say next.  It is tempting for her to conclude that she can also anticipate how well the patient will do next year, but this conclusion is not equally justified.  Short-term anticipation and long-term forecasting are different tasks, and the therapist has had adequate opportunity to learn one but not the other.  [T]he clinical psychologist, the stock picker, and the pundit do have intuitive skills in some of their tasks, but they have not learned to identify the situations and the tasks in which intuition will betray them.  The unrecognized limits of professional skill help explain why experts are over confident. (Kahneman, 2011)

If you have been following along as I try to unravel how school currently works, you won’t be surprised when my immediate thoughts added educators to Dr. Kahneman’s ‘experts’ list.  You would also know that I like that the “particular situation” matters when generating behavior.  Knowing that I have spent the 10,000 hours required to gain insight into anticipatory thinking and situated learning; the statements above were the greatest validation yet on why we might want to start re-thinking how today’s learners prepare for an unpredictable future.

Both of Kahneman’s statements leave me asking a lot of questions starting with: Is there any need to further interpret a statement that says that long-term prediction failure rates by experts is inevitable and that behavior and outcomes that result from testing that were limited to representations, cannot be compared to any behavioral patterns that might be manifested later in the real world?  It would appear by how this is spelled out that these are givens.  Assuming for a moment that both statements are true, then does annotating a poem or applying exponent laws in a classroom have any connection whatsoever with the learner’s real world 10 years hence?  The learner’s argument that they don’t understand why they have to perform some “random” learning task certainly has some merit if ‘experts’ have been shown to be less than accurate when it comes to long-term prediction.  Short of the appreciation argument (you won’t know the value of poetry until you’ve studied it) and familiarity argument (decimals and fractions are now realized to be different representations of the same thing), just about everything done in school from grade eight on risks being categorized as collateral as opposed to a targeted effort. (note: Yes, I prefer 'targeted' liberal arts)

The ‘other shoe’ implication of Kahneman’s statement is that as an educational ‘expert’ I may be recognized as having exceptional abilities as a developer, organizer, and communicator of knowledge.  But should I assume from this that I have managed to evade the limitations of expertise?  Can I predict that ANY assignment that I distribute today to a grade nine class will shape the behaviors of those who took up that assignment, 10 years down the road?  The simple answer would appear to be no.  At least it would be “no” if, in the mind of the learner, the randomness question had not be resolved on the day the assignment was given.

So it would appear that even though we believe learners are expanding their appreciation and familiarity margins, the predictability piece of success, the “poetry annotation moment was when I discovered how experts interpret and translate” has no hope of materializing unless there is a bright line connection made with some known outcome being currently achieved in the real world.  (note: See Space-time learning for an explanation of how matching a learner up with a practicing professional builds a young learner’s anticipatory and adaptational skills)

…

Once again I am struck by how we have likely fooled ourselves.   What we consider to be mastery of concept, which is really mastery of the representations funneled into the curriculum, can somehow move beyond foundational learning and start to predict the anticipatory and adaptability skills that we hope are present when the learner graduates from the formal education process.  I say this knowing full well that many educators have gone to great lengths to examine what they are teaching and in turn have actively distilled the course content to get at ‘what is important’ prior to engaging their students.   At least in these thoughtful educator’s cases, those in their charge have been spared the ‘education through mental attrition exercise.”   This exercise can be summed up as: moving through the system and seeing no other choice than to reduce concentrations to one specialized area of interest or ability.

This doesn’t happen because what is being studied is progressively more complex.  Of course things get more complex, but the answer to this doesn’t always mean reductionism.  That being said, these educators manage the complexity AND try to cultivate a set of transferable skills (they aren’t just academia’s “trainers”).   As grateful as I am to the thoughtful among us in not contributing to the content pumping exercise, there is still little or no emphasis on the anticipatory and the adaptable.  Based on Dr. Kahneman’s assertions, we can’t predict too far into the future.  But can we do a better job of determining if we are giving learners an opportunity to exercise their anticipation and adaptation abilities?

I’ve been thinking about this for a while, even though reading this book has me writing about this topic…again.  I’m not going to pretend that I am a great prognosticator.  In fact it is my weak predictor gene that got me so interested in figuring out why schooling hadn’t improved my powers to divine the future.   Not that I’m looking to moonlight as a fortune-teller, it is just that I’ve never been able to conclude with confidence that what we’re told should be in the curriculum, is the thing the learner actually needs.

So maybe it IS time for us to take a look at the things we teach and ask ourselves if some of our students are ready to enter the world of anticipation and adaptation and not be limited to just appreciation and familiarity.  No predictions here, but if programming is not in place to develop and evolve thinking skills that involve recursion and iteration, then are we leaving something important out?  Who might we consider when trying to determine if the time has come to expand the recursive and iterative learning dimensions?

What I’m about to share should not be seen as a barrier to learning.  Dr. Kahneman has made a strong argument that when we allow our intuitions to fool us into thinking we can become prognosticators, things will inevitably go wrong.  This leaves educators with few options when it comes to determining who might be ready to go beyond the learning challenges that are presented in school every day.  My gut instinct is to let every learner situate their learning, as I too, often argue the appreciation and familiarity pillars of learning.  Still, I know that some learners need to show more evidence that they are actually ready to adopt a ‘field study’ component to their learning repertoire. 

 The term ‘barrier’ is mentioned because I know many an educator who has been accused of ‘denying access to learning’ on the grounds that the learner isn’t ready, at least in the opinion of the educator.  It is this murky area between teacher intuition and not knowing until success or failure has been achieved, that can generate all kinds of problems and conflicts.  To try and mitigate this to a degree, we might want to turn to data we can gather as evidence that a person might be ready to see school extending into the larger community.  Any teacher who has a gut feeling that their student can benefit from a learning enhancement opportunity has every right to exercise the option to build programming that creates this reality.  For those of us who are comfortable with data, the following Indicators can be used to see if someone is building an argument for being a situated learner.  Just remember to consider the circumstances as well as the facts you can gather below.

The table lists the indicators and accompanying arguments for why I think you can make a case for inclusion.

Indicator	Indications
Number of subjects where result is in the top quintile or bottom two quintiles	At first blush one might conclude that this is only an indicator of specific aptitude(s) and concentration of effort, but this is more than an ability measure.  Top 20% achievement demonstrates learning beyond the statistical norm generated by the class.  Grade and subject specific mastery relative to peers is not only demonstrated on a relative basis.  The more times this outcome is achieved the more likely specific mastery becomes shared mastery, which implies a reduction in transferability barriers. Bottom two quintile scores may require remediation. This then turns into adaptability within subject domain rather than emphasizing adaptability across subjects.   The emphasis will be on consolidation and refinement protocols, which are a type of rehearsal strategy more than adaptability strategy (at least in the short term).
Degree of commitment to organized activity outside of school	Fitting into learning structures that are different from school means adapting to the different kinds of expectations that are generated beyond the classroom.  The more variety in activity choices, the more exposure to adaptation necessary situations.
Parental or guardian monitoring of learning accommodative behavior	A significant adult who models and expects an accommodative attitude reinforces the obligation to remain open to different possibilities.
Commitment to homework	Study extension beyond the direct supervision setting of the classroom reinforces self-control.  Self-controlling the learning effort demonstrates the ability to connect concepts without external assistance.  This builds both subject specific competence and the sense that one can adjust to novel circumstances with less outside support.
Adherence to punctuality	The ability to address time expectations of others is one of the easiest ways to measure the ability and willingness to adapt.  It should be noted that this should be measured when the person is in control of being on time.
Initiative generation and/or identified responsibility assumption	A demonstrated willingness to propose new ideas and take ownership of tasks, whether simple or complicated, influences empathy development.  Putting into practice one’s actions so as to improve the circumstances for many, means transfer of effort is in effect.

I don’t believe these indicators need to be given a weighting or that one indicator is more important than the others.  We need to be extremely careful that this is not an averaging exercise.  What I do know from using this indicator formula is that taken together (summed…I repeat, it is critical to understand you DO NOT get a score out of six), you can start to determine if an adaptability profile is developing.  As you collect the data in each category, you begin to see why some students seem to have an easier time ‘fitting in’ while others struggle with even minor deviations from their routines. Your choice of what to include as an indication and what score to provide becomes important if more than one teacher is using the same model.

Here is what your indications table might look like once the data is gathered:

Indicator	Indications
Number of subjects where result is in the top quintile or bottom two quintiles	Student has 83% in math and 91% in science last term (score = +2) Student has 45% in Spanish (score = -1)
Degree of commitment to organized activity outside of school	Student piano lessons year round, 2 hours per week (score = +8) Student plays community outdoor soccer (score = +4)
Parental or guardian monitoring of learning accommodative behavior	Parents self score 4/5 on accommodation scale (score = +4)
Commitment to homework	Student commits to 4 hours of homework per week (score = +2)
Adherence to punctuality	Student has three late slips last term (score = -3)
Initiative generation and/or identified responsibility assumption	Student started shoe rack club (score = +5) Student babysits younger siblings 4 days per week after school       (score = +4)
SUM	= +25

The sum should be seen as the degree of development achieved thus far and should not be confused as a predictor of future performance.  As scores increase however, you might want to consider when a calculated risk can be taken to allow the learner an opportunity to introduce some autonomy into their learning routine.

None of this ‘teaches’ adaptability or anticipatory thinking skills.  I assume that anyone reading this knows how to infuse the learning with adaptation and anticipation requirements.  What Kahneman’s work has me contemplating (and now reinforcing) is the need to identify indicators of thinking transfer that are present, are consistent and are durable.  From a diagnostic perspective this won’t fly in the face of Dr. Kahneman’s claims, as we still won’t be able to predict who graduates from medical school and becomes a gastroenterologist.  What we might discover though is that on an individual level if we want to assist more learners to have the potential to become medical specialists 10 years from now we should foster a learning environment that develops learning experts who recognize both their current strengths and the limitations which only time will overcome.

Define & Design and Diagnose & Implement

I have known many an educator who is adept at D&D, but is not always so quick to take up the cause of Diagnosis and Implementation.  For some this might come down to the aversion to being a prescriptive educator, for others it might be that D&I gets its due in professional learning community work like ‘student work assessments’.  When I look at diagnosis and implementation it is from the perspective of the learner.  I think when the learner internalizes all four concepts (and not just the first two) you give him/her a better chance of evolving their ability to adapt.

One current planning model that is quite popular (and I think quite effective) is Understanding by Design by Wiggins and McTighe where define and design gets attention by the teacher and sometimes even the learner.  Diagnosis and implementation, although given some attention, are seen as an addendum to the planning effort.   The emphasis is on working it backwards to create an effective learning plan.  Let me explain.

If we agree that knowing what we want to achieve makes it easier for us to put a plan together (have the goal front of mind first) then it would seem logical to design your planning around what you believe the learning destination will be.  For many years (since at least 1998) this concept was not new to me, as coaching had primed my thinking that ‘winning a championship’ should drive all instructional decisions and pedagogical outcomes.  Thankfully I’ve changed my primary goal to ‘winning the learning battle’.  I was fully engaged and committed to the idea that a plan grows out of the learner’s ability to think iteratively (make progress) and recursively (anticipate the future and practice for it now) and to this day I still believe this is true. 

What I didn’t realize was that in my effort to move the learning along (that the ‘performance’ was polished enough for public display) I was the only one in the room who was diagnosing the system, or implementing alternative solutions when the learning refinement had either leveled off or got stuck.  I hadn’t planned for this! This got me re-thinking about planning in general, and more specifically, I wondered how learner diagnosis and learner implementation meshed with the backwards define and design framework that UBD proposed.  What I found was, that even though Wiggins and McTighe weren’t intending to confuse, the linear nature of the term ‘backwards’ had me believing that order should trump context (at least when planning…I know, my bad).  I also overlooked the roll that learner self-analysis and action-alignment played in getting the learning done faster.  The sooner I could get the learners to engage in effective outcomes diagnosis, the sooner we could plan and implement situational (re. adaptive) solutions.

So the shift was on.  From now on it was to be learner diagnostics as much as teacher diagnostics, and not the ‘grand plan’ implementation, but rather the small implementations that accommodate variables both known and unknown. I started by looking at this under one of the most common situations faced by teachers:  Proof-reading written work.  Yes, there are those detail savvy souls who gain some sort of perverse satisfaction from the editing process, but most see editing as both time consuming/repetitive and revealing in a negative sense, especially when review doesn’t uncover the mistake(s) because the mistakes are not recognized.   Add to this, that some learners get emotionally attached to their original work, and that many of us have a strong dislike for being even a little bit wrong, and you can probably appreciate why some students have built quite the aversion to the exercise. 

Regardless, being stubborn and seeing the value in polished writing, I was going to prevail.  But first I had to minimize the ‘taking the medicine’ feel that accompanies most diagnostic efforts.  I had to put the learner in the position of prescribing the medicine as opposed to being the medicine taker…but how?  In the context of writing I’d had some success with ‘writing reviews’ where kids sat in a room with rubrics and critiqued writing samples.  Although most got some sense of what a “1” represented and why a “4” was a “4”, the net effect was that the novelty wore off pretty quickly and the detached nature of the conversation among learners (“this person’s use of sentence starters could be better”) took on a somewhat mechanical tone.  When I mentioned that critiquing work should be more like ‘getting under the hood of your own car in your own garage, rather than being the guy at the dealership who plugs the car into the computer”, all I got back was blank stares, at least metaphorically speaking.

My next thought was, “if I just make personal the nature of the writing assignment maybe there would be a better chance that learners would see that their writing is a clear reflection of one’s communication skills.”  Writing music lyrics worked for some, using a persuasive argument on parents had a few hooked as well.  Slowly but surely I came around to the idea that students would have to do writing that involved reviewing an event staged as a public performance, be an op-ed piece, or be a written petition.  Any of the three channels would have to be submitted to an actual third party outside-of-school and be publication worthy.

This got people’s attention, and even made a few elevate their writing efforts as now strangers were going to be the audience of their work.  As I went back and looked at the progressions I’d employed in the instructional design, I realized that perhaps I’d increased the attention people were paying to the writing, but I wasn’t so sure that I’d really tackled the question of ‘why writing diagnosis?’  And then another a-ha moment came.  I reflected on the whole giving/taking-the-medicine frame, and realized that finding the right prescription for someone is actually very ‘contributive’ in nature.  That sense that you are making a positive difference in someone’s life was what I had to get to as the core focus of the editing process.  The ‘writing for a public audience’ assignment never really captured this, as all the writers were more focused on making their point than in providing a contribution through writing.

Now it was on to implementation.  This ‘writing as contribution’ thing wasn’t going to work as an assignment.  I’d have to come up with some pretty compelling reasons for why writing could morph into contribution.  I’d already been doing some work with learner/professional mentorship programs that contained a contribution component.  It dawned on me; what if we could make editing like riding a bike (that metaphor thing again).  The editing would equate to inertia, something that could be studied, but was more about a way of making the vehicle serve the intended purpose, in this case, make a contribution.  The need to edit messages to mentors, edit information slide decks and edit proposal plans had never been something I’d had to spend a great deal of time ‘correcting’.  I was around to collaborate on how the message should be packaged, or why certain information had to be included, but there was rarely a spelling mistake to be exorcised.  Best of all, “defining” what was correct in terms of spelling and grammar wasn’t up for debate.  The contribution would be lost if reading a message had become laborious due to errors.  In this case at least, writers had become self-diagnosticians. 

Disclaimer:

You may be asking at this point, was there transfer back to classroom written work like essays and book reviews.  The answer is: mostly.  It is difficult to say that all essays showed significant gains in terms of editing outcomes.  Oddly enough though, written work like speeches and first affirmative arguments in debates were much more polished, I assume because, the work was going to be heard by a significant audience so an editing strategy was invoked.

What I found most revealing about the ‘diagnosis as contribution’ initiative was that my writers stepped back and analyzed written work from a ‘helping’ perspective.  Yes, there were still those who were particularly self-critical, but instead of obsessing through the perfectionist cycle, there was instead an effort to find a trusted peer who would act as a second set of eyes and ears.  Having the ‘have someone proof it before it goes out’ habit cross over from the mentoring program was a pleasant surprise.

A final note:

Some readers might be asking: why not just make lots of excuses to put written work on public display?  Although some work was always requested for bulletin board examples, I found that the person who was reading the posted work was usually the original author (or a relative who’d dropped into the school), and not many others.  I also recognized that writers who weren’t keen on having their effort being put on display weren’t feeling any incentive to polish their work.  They’d just as soon keep things private anyway (no need for extra editing).

Nothing in the meantime has advanced the writing diagnostic effort better than framing the diagnosis as a contribution.  As I transferred over to other subject areas I taught it wasn’t hard to ask:  what’s wrong with government, or, why does our offense always seem to be out of sync?  Again, the premise behind this was that learners could make things better; that a fresh set of eyes on an old or evolving problem could make a real difference, especially if we didn't keep the solutions to ourselves.   These solutions always ended up being put forward as gentle 'suggestions' to those who we thought would benefit the most.  

Making opportunities available to build diagnostics and implementations into your define and design planning is tricky at first.  Learners don’t tend to believe you when you say you are looking for a difference-making contribution.  Have your learners stick around long enough that their contribution makes a difference, and you will have people who appreciate the define-design continuum even more because their contribution doesn’t just fix a problem, but actually helps others out.