# The Results

**The Results – How Did the Students Do?**

We know that a lot of people are awaiting the results of the Envision Academy pilot this summer using Khan Academy. Before we get to the results, a quick recap. We set out to pilot a new way to run classrooms via ** blended learning**. At Envision Academy in Oakland, California high school students who had failed algebra were randomly assigned to one of two summer school classes. The “control” classroom received a traditional five-week summer school curriculum for Algebra 1. The “treatment” classroom used Khan Academy for almost all of the period each day, and both classes had the same teacher. We were curious about how the role of the teacher and student might change using blended learning, and we wanted to better understand its challenges and potential.

We took an open source approach to the experiment, welcoming visitors to the classroom and blogging about our learning. There were some immediately clear benefits to the blended learning environment and some equally clear challenges. We did not have a pre-set opinion in the blended learning debate, and we tried to remain objective. The d.school at Stanford helped to observe the classes, interview students, and capture our learning on this blog.

**The Caveats**

Before we discuss the results, a few important caveats are in order. First, no statistician will take our results particularly seriously, and they shouldn’t. The sample size is too small to attribute any real significance to the findings. Secondly, the pilot was very brief, lasting only five weeks, or twenty-four class sessions of two hours each. Thirdly, there is always the risk of the Hawthorne effect, or an observational bias because the students inevitably knew they were part of a study. Finally and perhaps most importantly, it was also difficult to find the right measure by which to evaluate the progress of students in the two classes. After consultation with several researchers, we settled on the University of California Mathematics Diagnostic Testing Program (MDTP) and their Elementary Algebra Diagnostic exam (EA50A90). The exam is designed to measure students’ readiness for an Algebra II course. We settled on this exam in consultation with the team at MDTP as an appropriate means to measure students’ success at the end of an Algebra I course (*see comments for a listing of the topics assessed and the number of questions per topic). A major concern with this assessment, however, was that it would not pick up any gains made on pre-algebra content because it focuses primarily on algebra content.

From the beginning, we knew that the pre and post course test data could not definitively assess the success of the pilot. For all the reasons listed above, we view the data as a single quantitative measure that should be considered alongside the qualitative data captured through the observations and interviews. As such, our hope is that others will not cite this data as proof one way or the other of the effectiveness of Khan or blended learning. It would be dangerous to over-generalize our findings. We see this pilot as providing one small piece of data that suggests reason to be cautiously optimistic, while also clearly showing the need for additional study.

**The Results**

Among the students in the study who had valid scores on the pre and post course assessment, the results were similar for the treatment and the control group. Students in the “control” or traditional summer school course, on average, increased their percentage of correct answers by 5.2% over the five-week period. Students in the “treatment” or Khan class, on average, increased their percentage of correct answers 6.4%. For example, a student who started the summer knowing 60% of the correct answers in the traditional class ended the five weeks knowing 65.2% of the correct answers. The same student in the Khan class would, on average, be able to answer 66.4% of the answers correctly at the end of the same period.

*Increase in Percent of Questions Answered Correctly on the MDTP Algebra II Readiness Exam*

Averages can obviously be deceiving. In terms of distribution, in each class approximately one third of the students saw some significant gains (ten percent or higher gains in percentage of questions answered correctly), whereas two thirds of the students’ scores were essentially flat (less than four percent increase or decrease). There were no particularly strong findings regarding in which content areas the two classes saw concentrated gains. The one exception is that the students in the traditional class saw most of their gains in the areas of “Graphical Representations” and “Polynomials and Polynomial Functions” whereas in the Khan class, students saw gains spread out among almost all the categories.

**Implications**

Remembering the limits of this data’s reliability, it is interesting to note that students in the two groups scored roughly the same, each showing some slight improvement over the five-week course. We wonder whether this trend would hold over a full-year course, and whether the slightly higher gains that the Khan students showed would be multiplied or reduced over the course of a school year.

It would be easy (and wrong) to use this data to conclude that blended learning and Khan are without value. If anything, we find it interesting that the teacher “doing her best” in the control class was roughly equivalent to the gains of the students using the Khan Academy, where students did more of the learning on their own with the teacher as the guide. In the treatment class, the teacher ended up doing mostly one-on-one consulting with pupils, and the students progressed through the assignments at their own pace and sequence. If it is true that Khan centered classes can match or even exceed the traditional teacher-lead pedagogy, there could be interesting implications for the future.

Regarding the more concentrated student gains on “Graphical Representations” and “Polynomials” in the teacher led class, it is plausible that this concentration of gain was due to the teacher focusing more time on these topics. In the Khan classroom, the teacher had less control over which content students devoted the most time to. It therefore makes sense that the gains were more evenly spread across the various topics.

It is also interesting to note that in the Khan classroom, many students spent a significant amount of time working on pre-algebra skills such as fractions, percentages, decimals, and even basic computation. If we could do it over again, we would have used a second measure to also evaluate student progress on these pre-algebra skills. Our hypothesis is that the Khan students would have made significant gains versus the control classroom that did not spend much time on these topics. The data within the Khan software shows that the treatment students were able to correctly answer ten questions in a row on many of the pre-algebra sections.

It is also interesting to consider that students in the treatment group spent approximately half of the summer working on pre-algebra skills. Because the Khan software is individualized, it identified that most of our students had significant pre-algebra skill gaps and delivered instruction and practice problems to address these deficits. Students in the Khan/treatment group therefore spent up to 50% less time than the control group on the algebra content that the MDTP exam measured. The treatment group, however, still performed at a similar level the control group on the algebra measures.

**Questions that Remain**

As with any pilot, we are left with as many questions as answers. We wonder how a Khan type classroom would work with a less skilled teacher. The teacher in this summer pilot had a positive rapport with students, good classroom management, and was a good motivator to both classes. In the hands of a less-than-good teacher, we wonder if the results would hold. That said, with a less-than-good teacher, the traditional classroom would likely suffer significantly too.

We saw strong engagement and interest from the students in the Khan/treatment classroom, as documented throughout this blog. We are curious to know if this is inherently true for such individualized and self-paced content, or whether students would “hit the wall” if the Khan approach were used for a longer period of time. Anecdotally, most of the students told us they preferred the Khan classroom to what they had experienced previously and would prefer to take a “blended” course next year.

Prior to seeing the results of the summer experiment, the teacher predicted that her students would do better on a traditional measure of proficiency such as the California Standards Test or CST if she ran her classroom in the Khan manner versus the traditional classroom approach. Given that she was not a convert prior to this pilot and developed these opinions only through teaching the course, we find this an interesting perspective to consider in the dialogue about how teachers will respond to blended learning.

Finally, there is still much to learn. We hope that this small experiment inspires others to tackle and document their learning in the blended learning space. Clearly we need larger sample sizes and longer trial periods by which to evaluate the approach. Envision Academy has decided to participate in a year long pilot of Khan Academy for this school year for all ninth grade students. They will compare end of year results on the CST compared to the three other Envision high school campuses in the Bay Area as well as to previous year’s ninth grade classes within Envision Academy to evaluate its success.

The qualitative evidence of these past five weeks point to the potential of blended learning. We are curious to hear what others think. Small sample size aside, do the findings that Khan students performed roughly the same as – or even slightly higher than – the traditional classroom students support or undermine the value of blended learning? Weigh in by clicking the comment bubble on the top of the post, and let us know your opinion.

### 30 comments

### Trackbacks/Pingbacks

- Blend My Learning Khan Experiment Results Are In | jackcwest - [...] a promising inference to be made when you drill into the data. Below is the comment I left at…
- links for 2011-09-04 | Kristian Still's Blog - [...] The Results | Blend My Learning Teaching vs blended learning. [...]
- No bells or interruptions to learning times…. | ICT Enhanced Learning and Teaching's Blog - [...] Results from Khan Academy experiment (blendmylearning.com) Eco World Content From Across The Internet.…
- Grading the Digital School: Khan Academy Blends Its YouTube Approach With Classrooms | IMTelkom News Portal - [...] is too early to know whether the Khan Academy software makes a real difference in learning. A limited study…
- Grading the Digital School: Khan Academy Blends Its YouTube Approach With Classrooms | Last Planet News - Daily News Magazine - [...] is too early to know whether the Khan Academy software makes a real difference in learning. A limited study…
- Grading the Digital School: Khan Academy Blends Its YouTube Approach With Classrooms | Inter News Daily - Daily News Magazine - [...] is too early to know whether the Khan Academy software makes a real difference in learning. A limited study…
- Grading the Digital School: Khan Academy Blends Its YouTube Approach With Classrooms | - [...] is too early to know either a Khan Academy module creates a genuine disproportion in learning. A limited study…
- GRADING THE DIGITAL SCHOOL Online Learning, Personalized By SOMINI SENGUPTA | Baby Blue Rose - [...] is too early to know whether the Khan Academy software makes a real difference in learning. A limited study with students…
- Grading the Digital School: Khan Academy Blends Its YouTube Approach With Classrooms | Hour News - Daily News Magazine - [...] is too early to know whether the Khan Academy software makes a real difference in learning. A limited study…
- Online Learning, Personalized | VIKAStm - [...] is too early to know whether the Khan Academy software makes a real difference in learning. A limited study…
- Grading the Digital School: Khan Academy Blends Its YouTube Approach With Classrooms | Finance Explained - [...] is too early to know whether the Khan Academy software makes a real difference in learning. A limited study…
- Grading the Digital School: Khan Academy Blends Its YouTube Approach With Classrooms | G7Finance.com - Finance News & Personal Finance Resources - [...] is too early to know whether the Khan Academy software makes a real difference in learning. A limited study…
- Grading the Digital School: Khan Academy Blends Its YouTube Approach With Classrooms | Articles Blog - [...] is too early to know whether the Khan Academy software makes a real difference in learning. A limited study…
- Grading the Digital School: Khan Academy Blends Its YouTube Approach With Classrooms - TECHNOLOGY GADGETS – TECHNOLOGY GADGETS - [...] is too early to know either a Khan Academy module creates a genuine disproportion in learning. A limited study…
- Grading the Digital School: Khan Academy Blends Its YouTube Approach With Classrooms | Finance Analysis - [...] is too early to know whether the Khan Academy software makes a real difference in learning. A limited study…
- Grading the Digital School: Khan Academy Blends Its YouTube Approach With Classrooms « Breaking News « Theory Report - [...] is too early to know either a Khan Academy module creates a genuine disproportion in learning. A limited study…
- GRADING THE DIGITAL SCHOOL: KHAN ACADEMY BLENDS ITS YOUTUBE APPROACH WITH CLASSROOMS » KHAN, KHANS, ACADEMY, MATH, STANFORD, SHELTON » TECHNOLOGY NEWSPAPER - [...] is too early to know whether the Khan Academy software makes a real difference in learning. A limited study…
- Khan Academy Blends Its YouTube Approach With Classrooms – NYTimes.com « Ye Olde Soapbox - [...] is too early to know whether the Khan Academy software makes a real difference in learning. A limited study with students…
- More classrooms trying Khan Academy, finding it better than group projects | Cost of College - [...] It is too early to know whether the Khan Academy software makes a real difference in learning. A limited study with…
- Online Learning, Personalized « Heptanews * Entertainment * Politics * Opinions * U.S. * Technology * Health * Leisure * World * Sports - [...] is too early to know whether the Khan Academy software makes a real difference in learning. A limited study…
- Grading the Digital School: Khan Academy Blends Its YouTube Approach With Classrooms - World Bad News : World Bad News - [...] is too early to know either a Khan Academy module creates a genuine disproportion in learning. A limited study…
- Innovation | Analytics | Digital Education » Lessons learned from using Khan Academy content in a blended learning pilot - [...] >> The Results – How Did the Students Do [...]
- Khan Academy: Is there any evidence of the Khan Academy significantly boosting scores on standardized math or other subject tests (say in a school where it is used)? - Quora - [...] two separate sections of algebra, one using the Khan Academy, one not using the Khan Acacemy. See http://blendmylearning.com/2011/...Overall, the…
- Where to begin? | Math Differently - [...] came across the blog Blend My Learning and it’s given me a few different ideas. I first found the…
- Analytical Research Paper – Sonja’s Blog - [...] Anonomous, Aug. 31, 2011, The Results, Blog: Blend My Learning, In the trenches of blended learning. http://blendmylearning.com/2011/08/31/the-results/ [...]
- Online Learning, Personalized | nerdenglish - [...] is too early to know whether the Khan Academy software makes a real difference in learning. A limited study…

*MDTP Topics Assessed with number of test questions for each topic in parentheses)

Arithmetic Operations (6)

Exponents and Square Roots; Scientific Notation (6)

Measurement of Geometric Objects (7)

Graphical Representations (6)

Linear Equations & Inequalities, including Absolute Values (9)

Polynomials and Polynomial Functions (7)

Quadratic Equations (4)

Rational Expressions (5)

As you mentioned, I think you would have found much better results for the Khan class if you assessed pre-algebra skills. I would imagine that much of the growth on the MDTP of the treatment group was due to improvements on pre-algebra skills as opposed to improvements on algebra skills since that is one of the big challenges for students in algebra.

My big question for the year-round pilot: Will the teacher supplement the Khan Academy time with PBLs? I think that would be really powerful if done well.

I love the methodology even with the small sample size and the use of summer is a great way to study this.

Given the relative early stage that the Khan Academy is at (I think there is a lot of room to improve their product), I think this study shows promising results. When you add that in the fact that the students were able to keep pace or maybe even outperform (especially considering the pre-algebra skills were not assessed) with the teacher playing a less active role I think there is even more reason to consider the Khan Academy as a very viable tool to be used in the classroom.

Maybe even more importantly, the fact that the kids showed a preference for this style of learning should mean a lot.

It is not hard to imagine that the success of this style of learning will become more potent as Khan improves their product, teachers become more adept at using it and enhancing it through their own efforts, and students become more familiar with it.

Interesting study with potential. Can you share the class sizes for both the control and treatment groups?

Quite small, about 20 students in each.

Thank you for bringing Khan Academy out of Los Altos and into Oakland where the real challenges lie.

I think it’s worth noting that most (and I mean anywhere from 50% to 80%, depending on the school) of Oakland students are below their appropriate grade level in math. So the problem of not having basic skills is a big one.

Rather than trying to replace a good teacher with Khan, it might be more effective to use Khan to build up basic skills during part of class time each day or during intervention classes (called ‘Math Strategic’ in OUSD). Teachers don’t have room in the curriculum to explicitly teach basic skills and every student is on a different level, so Khan could be a differentiation time.

I think these results say thumbs up to blended learning, thumbs down to khan.

“It is also interesting to note that in the Khan classroom, many students spent a significant amount of time working on pre-algebra skills such as fractions, percentages, decimals, and even basic computation.”

This is one of the greatest features of individualized, mastery-based learning. It appears that Khan Academy was able to identify and remediate deficiencies in the treatment group’s math foundations. It would be interesting to see if a better foundation in the lower two-thirds of the treatment group leads to increased achievement in the coming year compared to the lower two-thirds of the control group. Potentially, the better foundation could lead to a deeper, more robust understanding of new concepts.

Thank you for sharing your results.

Interesting study! Is the difference greater if you measure relative gain instead of absolute gain? Relative gain is (post-pre)/(100-pre), or gain as a fraction of the maximum possible gain for each student. Richard Hake in his numerous studies of physics teaching methods argues that it’s a better measure.

What did the Khan students think of the class? Did any students drop out? In our school district the offer of summer school is driven more by ideology, students deserve another chance, rather any history of successfully re mediating students. It is interesting that student achievement in the subject was the same, however, it would be just as interesting to see if a teacher could be effective teaching a larger class using blended learning. Reducing the cost per student while maintaining or improving student achievement is also an important goal.

Thank you for sharing your results and analysis. Even studies with no significant statistical trends add to the field, and this is particularly true in education where such data are lacking.

I will echo what several of the comments before mine have said. The most interesting piece of your analysis is that students in the experimental group matched control group gains despite spending less time on the content measured by the metric.

I like to make inferences when there is insufficient data on which to make judgments because nobody can really hold you accountable when you do 😉 Given that, I think the experimental group would likely pull away from the control in a longer study. If the blended learning class matched gains with half of the exposure to the algebra material because they were making up for lost ground in pre-algebra, it is not too far of a stretch to suggest that they may have developed stronger foundational skills and that they were already on a trajectory that would far surpass the gains of the control.

One question – what were your criteria for “valid scores?” How many from each group did you decide to withdraw?

Thanks again for sharing your process and results.

Thanks Jack, as always, for the thoughtful take. Regarding your question about “valid scores”, we did not disqualify any results. It is just that in each class there were one or two students who missed either the pre or post class assessment due to absences. We thus excluded those few students from the results because there was no data on their gain.

What led the treatment students to spend so much time on on pre-algebra materials?

Who told them where to begin? Were they told to move quickly through the exercises?

They might then make a goofy mistake and have to start all over to get the ten straight

necessary for them to move on. I am still attempting to figure out why they spend so much time on pre-algebra. Only a program like the Khan Academy would allow for

the individualized mop up of weaknesses in pre-algebra.

It would be interesting to know what kind of help the teacher provided in the Khan based group.

Thanks for posting the overview, results and commentary; together they give a thoughtful picture of what happened this summer and leave me eagerly awaiting the results of the coming year’s experiment. I agree with the previous comment that since the blended learning students spent half their time filling in missing pre-algebra skills and still equalled or excelled the control group, the blended learning students would pull ahead of the control group over a longer time frame. Let’s see what happens.

Great report, and good that you ran this experiment. I’m struck by how little the test scores went up for both groups, at least for 2/3 of the sudents. From 60% in the pre to 65-66% in the post is still in the traditional “D” grade range. So does this mean neither approach worked very well? Or is that enough of a gain for a summer school course, and the students are at least in better shape now for the next math level?

Great study and am excited to see results of year-long pilot program in Envision. Most of what I want to say has already been said by former commentators, however I do think that there is perhaps non-quantifiable value derived from an internet-based curriculum i.e. self-paced learning gives students a sense of choice that helps with self-esteem.

Yet the existence of a plethora of online videos featuring amazing performances doesn’t eradicate the compelling need to go to live events. In the same way I believe that there is something meaningful and valuable about teacher-led learning that, for all its flaws, when executed right has a dramatic impact that technology simply cannot replicate.

I think then that tools like Khan Academy can at best supplement teacher-based education, but never supplant it.

I actually like this post. I think I like the reselts, with the treatment classroom. If they could extend the five week period to a year to being like our classrooms, we could learn a lot more and accomplish more. So if every school was like this program we could graduate early, go to college, and even have a better future if they could make the program a little bit more fun. I think when the students are sitting and listening, sometimes they might not be listening because even I get bored and fall asleep. If they could extend the program and put in fun activities to keep the students moving, awake, and make learning fun, I would definitely try it. But, I, myself am impressed with the results. Keep up the work!

I feel that blending the beneficial educational perks resulting from the Traditional and Khan classes would create a successful full year program. This would create a class that could potentially cover all essential Algebra I subjects needed to pass the test. The only way you fail is by not trying so I say press your luck and go for it, students are learning and improving — that’s all that matters!

As a high school student this blog intrests me. Khan Academy is being use in my hybrid personal finance class. This is where we only have to go to class every other day. The day we don’t go we watch videos from Khan Academy, and the day we do go to school our teacher reviews and quizzes us over what we watched. I find it a good way to learn at my own pace and if I have questions I can ask my teacher the next day. Khan Academy’s results are impressive.

Khan Academy is being used in my high school finance class. As a student I find it easier to learn at my own pace and to re-watch the videos I do not understand at first. The results of the summer program prove that this program is capable of working. It is easy for students to ask their teachers questions after they have watched the videos on their own. If Khan Academy helps students improve in all areas taught, then I could find it very helpful. Many students in fact could.

i like the idea of using the Khan Academy in the classroom because it lets students learn at their own pace, and even though it was only a 5 week test it still shows improvement in the kids’ learning. Using this in the classroom could have some negative effects. For instance if the kids got lazy and didn’t watch the video then they couldn’t do the classwork. I agree with Preston that some kids may fall asleep so putting in fun activities would be a good idea!

This seemed to be an interesting experiment. My opinion is that since they only had 5 weeks of study that the results weren’t as accurate as they could have been. The results they received didn’t show much of a difference between the control group and khan academy, but if they could have tested it for a hole school year we could have a better picture. They said that blending learning kids practiced more on certain problems while the regular class practiced a wide variety of different problems. I think it is important to cover many topics. I don’t have a strong enough opinion yet to have a conclusion.

Khan accademy and blended learning could be very effective. It is convienent that Khan can make sure you understand the basics before moving on to a harder topic: such as the students who needed to brush up on their pre-algebra. But it all depends on how you learn. Some students may prefer listening to a teacher’s lecture rather than watching Khan.

Looking far down the road, would students really need teachers? With the internet, all the resources are at their fingertips.

Ok so I think the experiment was very organized, yes, it was a little shaky but that was because there were alot of unanswered questions. This experiment did answer some questions like, the students do learn with Kahn academy and they learn a good percentage more than the traditional class. But there are so many different cases at which could Interfere with this and get completely different results. I personally think that if they kept a couple traditional classes along with some hybrid classes during a full school year and then test each student who participated in each different subject, you’d get better and more adequate results.

I think overall the Khan Academy will be a good way of learning for students. They can learn by themselves at their own pace, and if they dont get it right away they can keep watching it as many times as they need to where as in a classroom the teacher explains it once and the students are expected to understand it. I think that the teacher also contributes a lot to this too. If you have a teacher that isnt very good or doesnt get involved when students cant get answers to questions they might have. Also students might be a little more prepared in the future because they will start learning how to take responsibility to do their assignments and get them turned in.

In my opinion, the idea of using Khan Academy in high school is very beneficial to the students. It may put more responsibility on the students themselves, but it will prepare them for college programs. Using Khan Academy as a high school student is very interesting to me. Hopefully due to taking this hybrid course I will be better prepared for the rest of my learning career. In response to the experiment, I think the testing group was not as well thought out as it could have been. It was a good start, but there is more that needs to be tried out before any assumptions are made.

As great as Khan Academy sounds with all the pros that come along with it people seem to be willing to over look the negative sides to it. Yes, I might have done better in some class due to this program (clearly noted by the rise from 5.2 to 6.4% on the chart about right answers) and it will better prepare us for our future. Many prefer this way of teaching, and I agree with how great it sounds, but in some schools to certain students will this work?

i think it was a smart idea to do this expirement. although they did it at summer school where they only had 5 weeks. they should test it over a full year with different classes to be more sure of what they found. i agreed with olivia when she asked the question ‘will students really need teachers in the future with all this technology?’ if students have all the resources at there finger tips, why would they bother coming to school for 6 and a half hours everyday? students could learn at there own pace. on the down side, some students do learn better by having someone lecture to them. everyone learns different so to say that this would be better for everyone is not true. if a student has a specific question, how can khan academy answer that?

My name is Lindsey Gipson and I am student in EDM 310 at the University of South Alabama. I have been assigned to follow the “Blend My Learning” Blog for two weeks and write a summary of what I have learned.

This is the first time I have heard of the Khan Academy. I would like to research it further so I can form a better opinion. From what I have read so far, the Khan Academy way of teaching/learning gives students the ability to learn at their own speed. This will be beneficial for students who leave class after a lecture and have no idea what the teacher lectured about. Even if they are listening and taking notes sometimes it just doesn’t stick. That one lecture could be the last time they see the material until the test review and sometimes the test its self. A system like this will give students the opportunity to review until they get it. I am curious to see what the results of the year round test will show.

Very interesting. I’d like to know more about how the lowest achieving students responded to the Khan Academy lessons. Bryan and Emily Hassel just published a paper looking at what this sort of hybrid model could mean for the teaching profession:

http://www.edexcellence.net/publications-issues/publications/creating-sound-policy-for-digital-learning.html

My name is Anna Shartzer and I am student in EDM 310 at the University of South Alabama. This is the first time I have heard of the Khan Academy. I would like to research it further so I can form a better opinion. However, what I have read so far, the Khan Academy’s way of teaching/learning gives students the ability to learn at their own speed. The benefits are extremely helpful for students who are still unsure of material that was covered in class. Sometimes students who listen closely and take adequate notes still need further assistance in grasping the information. As a student, I have experienced that problem first hand. Most of the time, that one lecture I seem to struggle on always seems to be the lecture that is covered mostly on the exam. A system like this will give students the opportunity to review until they get it. I am curious to see what the results of the year round test will show. I really wish that there was a program like this for some of my college courses, but as a future educator knowing that this is an issue, maybe I can conquer it before it becomes a problem.