how we failed and then succeeded with khan academy
How We Failed (and Then Succeeded) with Khan Academy
Last fall I wrote about the need for an ed-tech Test Kitchen – a place that might address ‘what works’ in ed-tech. It’s really a two-part question: which products work and how should they be used? That ‘how’ question is tricky. Figuring out the best way to incorporate a product into a school, with all its idiosyncrasies, usually involves a process of repeated trial-and-error. Which means some failure.
Today I’m writing about a handful of ways that we failed with Khan Academy. I’m always eager to hear how other places try and tweak and sometimes jettison things, so here’s an attempt to share how we did it.
First, a bit of context: I joined Match last year to work on a new blended learning model we’re calling Match Next. At the three existing Match schools, small-group tutoring is incorporated into the day of every student. This is often used to try and remediate skills a student never mastered when it was taught in class. We were curious how well Khan would help our kids fill in their math gaps in tutorial.
If you’re reading this blog you’ve probably heard of Khan. (If you haven’t, an explanation here.) Lots of people praise Khan. And lots of people criticize it. On one hand, I’m always wary of a Pollyanna, and technophiles have a tendency to overstate the advantages of the products they use. On the other hand, sometimes it seems like the naysayers bail too early. In my experience, it usually takes a few tries to figure out how to integrate a product just right.
I’ll describe how we — 6 kids, 3 tutors, and me — actually used Khan in a daily hour set aside for 6th-grade math tutorial. I messed it up a bunch of times in the beginning, but eventually we found a way that seemed to work reasonably well. Not perfect, but an improvement over the incumbent.
Here’s how it started:
Round One: Khan Free-for-all
Once we’d showed kids how to use the Khan website, we let them pick and choose which aims they wanted to work on. Kids picked the easy ones. They racked up tons of points and badges. They loved it. When they clicked on a more challenging aim, they got frustrated quickly, and would frequently downshift to an easier aim to work on. Not a good start. Product’s fault or my fault? My fault. Didn’t set kids up for success.
Round Two: Assigned Khan – Easy
Tutors selected aims for kids. Tutors at first picked easy aims and had the same problem getting the kids to work hard at more challenging ones. Hmm.
Round Three: Assigned Khan – Hard
Tutors picked aims right at the upper edge of a kid’s ability. This is the right idea generally. But after too many “right at the edge” problems in a row, our students became frustrated.
I had flashbacks to my days as a distance runner in high school. I’d go out too hard at the start and implode before the end of the race. Fly and die, coach called it. Same thing was happening here.
Round Four: The Khan Fartlek
Fartleks are unfortunately-named interval drills for runners. (It’s apparently a Swedish term for ‘speed play.’) You run at peak effort for a certain period of time, then slow to a jog to recover, then speed up again. Over and over.
The Khan Fartlek went something like this: we started with the goal-setting function. Every account can create goals – selections of 5 exercises or videos that the program saves. Tutors created 2 different sets of 5 of goals: an easy set and a hard set. We made kids switch back and forth – achieve mastery on an easy goal, then go achieve mastery on a hard one.
Worked for a little bit. But then kids would get stuck for 30 or 40 minutes on a hard goal and burn out. You can’t sprint forever.
Round Five: Custom-Leveled Khan
Each kid got 5 different sets of goals at increasing levels of difficulty:
Level 1: Walk. Aims that a kid will very easily master. Highly unlikely to get one wrong. Not so easy as to be a waste of time, but something that won’t take much time to achieve proficiency on. No help required from a tutor.
Level 2: Light jog. A kid will often get a problem or two wrong at first, but then pretty quickly figure it out. No help required from a tutor.
Level 3: Moderate jog. A kid will almost certainly get some of the problems wrong at first. Some minimal tutor intervention may be necessary.
Level 4: Run. An aim a kid does not remember how to do. Will need a few minutes of tutor help before is able to work productively alone.
Level 5: Sprint. Most challenging aim. Something a kid may have never seen before. Is at their upper limit of ability. Will need 5-10 minutes of tutor help or a video to get started.
Tutors created these levels for each of their kids. An advanced kid had much more challenging aims than a struggling kid. So level 1 for one person might be another’s level 4. And it was easy for everyone to know what they needed to do. No searching around on Khan’s ‘concept map’ for the next aim. Instead, a kid would just click on the ‘goals’ button and see this:
Then a kid could pick any goal he or she liked.
With the 5 levels created, we needed to get kids to push themselves without burning out. So we created an incentive program. Every kid got a half-sheet of paper at the beginning of each class. On that sheet were a bunch of boxes with numbers inside, like this:
Each number represents a level. If a kid completed an exercise at that level, they raised their hand and a tutor came over and checked off that number. Once they completed every number inside one box, they would earn a merit. Kids got to choose the aims they worked on (they liked having choice), but they weren’t rewarded unless they pushed themselves and completed at least some higher-level aims.
Once we figured all this out, the room became quiet. Kids worked productively and independently for most of the hour. Once in a while a kid would raise her hand and ask her tutor for help on a problem. Sometimes a kid would pump a fist in the air in victory, or grab his hair when he made a mistake.
Tutors seemed to have a much easier time. In the old tutorial, a tutor sat together with both students and constantly tracked how each kid was doing. With Khan, they stepped back and let the kids struggle much more independently.
We wondered: should we make kids use the videos when stuck, or have tutors just jump in?
We found that, compared to a tutor, the videos were a pretty inefficient way of getting the kids un-stuck. Usually a kid had one small misunderstanding that a tutor could diagnose in 10 seconds and fix in 15. Watching a 5-minute video wasted time.
What they liked:
“I liked that I could choose the topic I could work on.”
“I liked that it was a challenge and you could get merits.”
“It gave me lots of different math things to choose from.”
“I liked that they gave you different ways of showing your work.”
“It gives you hints.”
What they didn’t like:
“I didn’t like that I had to keep doing types of problems until I got a full star.”
“I didn’t like that my tutorial partner and I were split up.”
“It got me mad when I didn’t fill up the star.”
“They didn’t really give you time to talk with your tutor and tutee.”
“Nothing bad. It was a 10 out of 10.”
“Ugh. I never did so much math in my life.”
“My head hurts.”
When I asked kids if they wanted to continue using Khan or go back to the old way, they were split. Half complained that it was much harder using Khan and they missed spending time with their friends in a normal tutorial. They went from having a very interactive tutorial, with little breaks of chatting or waiting for their tutor to come check their work, to a tutorial where they were working very hard for the entire time.
Of course as an educator, I’m discounting their concern to a degree…what I thought I was observing was more efficient math practice. Khan is a huge protector against the natural digression of tutorial. It’s challenging to get kids to work really hard. It’s particularly challenging when you tutor two or three of them at a time (not to mention teach 25 or 30 at a time), all needing attention at once. And when everyone’s sitting around a table it feels less formal, so it’s much easier to take little breaks.
With Khan, there’s an unlimited bank of problems. Fewer logistical bottlenecks. Automatic feedback loops that don’t require a human tutor to be constantly correcting a kid. It creates a more challenging tutorial, and a hugely productive one.
Thinking ahead, if we’re successful at creating 40 to 60 minute bursts of more efficient learning where kids are really racking their brains, we’ll need to look hard at better/longer breaks or downtime for kids during the day than conventional schools.
That’s Khan in our particular context. It took awhile, but eventually we found a reasonably good mix of student choice, incentives, and tutor/teacher interaction that was a net-gain for our kids’ learning. Would love to hear any thoughts or other Khan tips in the comments below.