The high-flying online Khan Academy is using contractors to review the accuracy of many of its videos and to sort through complaints about specific videos that come through its Web site. Sal Khan, the academy’s founder, said in an e-mail that it became clear to him that “more process” was needed as his online enterprise has expanded in recent years.

The Khan Academy is essentially a free on-line library of thousands of videos on subjects including math, physics, and history that are designed to allow students to learn at their own pace and for teachers to use as instructional tools.

It was started in 2006 by Khan, a former hedge fund manager, who has been lauded as the savior of education by people including Bill Gates, and blasted by critics who think he is promoting a form of education in which brick and mortar schools and teachers are obsolete. His financial backers include the Bill & Melinda Gates Foundation, the Broad Foundation and Oracle. Khan has said that his academy will never offer a complete education and that it is meant to only be a tool, though the academy’s Web site says:

We’re a not for profit with the goal of changing education for the better by providing a free world-class education for anyone anywhere.

There have been numerous challenges to the accuracy of some of the videos by mathematicians, physicists and others. Last year, two teacher educators critiqued a Khan Academy video on multiplying and dividing positive and negative integers in a video of their own that is intended as a takeoff of the Mystery Science Theater 3000. But the videos are very popular with students and teachers and have the support of mathematicians and other professionals.

I learned about his use of contractors to review videos through an e-mail exchange with Khan, who is starring at the moment in a television ad that highlights his new partnership with Bank of America to promote financial literacy. The conversation began after I received an e-mail from a physicist named Gary Cochran about what he said were fundamental errors he discovered in two physics videos on the Khan AcademyWeb site.

“If this is the future of teaching physics (and maybe math?) we’re in a heap of trouble,” Cochran wrote.

I sent that e-mail and a subsequent one detailing the problems to Khan, who promptly responded that Cochran was right, that the videos had been taken down and that contractors had been working for some time with the academy to review its videos and check complaints that come through a button to the right of the videos that allow for comment. (Below are the emails from Cochran, with Khan’s responses.) He also said content specialists were working to align the videos with the Common Core State Standards.

“As we grow and have more content and more traffic, it’s become clear that we’ve needed to add more process and folks looking at things,” Khan wrote in an apparent acknowledgement that some of the videos have problems.

Here’s the text of the first e-mail I received from physicist Gary Cochran. Note that the two videos to which he refers on the Khan Academy Web site have been taken down.

A baseball friend of mine wondered how hard it was to clear the wall (green monster) in Boston. The physics question might be posed as, “What’s the minimum velocity needed for a baseball to clear the green monster (wall) at Fenway Park in Boston?” I’m a physicist so I I first solved the problem and only then was curious if anyone else had considered this question. I discovered two videos produced by Kahn Academy which calculated the purported answer to the problem:

I was surprised to discover how wrong the physics in these videos was and how their misleading results has been promulgated for two years. I thought I could just contact Kahn Academy and the videos would be corrected. I found no way to contact anyone for corrective action. What do you do when these videos are just plain wrong and teach incorrect information, in my case, on fundamental physics? Apparently there are no peer reviews at Kahn, and no way to be sure the videos are even close to being accurate. I’m wondering if you have any contacts, other than the satirical production of videos, to promulgate corrections. If this is the future of teaching physics (and maybe math?) we’re in a heap of trouble.

Thank you,

Gary Cochran, Ph.D., Physics

BTW, I’m not a professional educator; I’ve been a consulting physicist for my entire career.

I asked him to explain the precise problems in the video and he wrote the following e-mail (which I confess makes me wish I understood physics):

Mr. Kahn begins with the assumption that the ball comes off the bat at a 45deg angle and then calculates the initial velocity needed to clear the wall. He suggests that this velocity is the minimum (he calls the 45deg angle optimal). The descriptive term when Kahn Academy loaded the first video onto YouTube says, “Solving the problem to determine the minimum velocity to hit a ball with to clear the Green Monster”. However, his result is NOT the optimal (minimum) initial velocity the ball can have to clear the wall, and 45deg is NOT the optimal angle at which minimum initial velocity occurs. The limiting case of the general solution to the physics problem, where the minimum initial velocity is at 45deg, occurs only when the relative height of the wall is zero (0), which makes the videos superfluous. Essentially, Mr. Kahn used results from a physics problem not directly applicable to this case. He should have set up the physics equations of motion directly in order to solve the stated problem of finding the minimum velocity to hit a ball with to clear the Green Monster.

The physics equations of motion for this case provide the vertical and horizontal position of the ball as a function of time after leaving the bat. The constraint on these equations for our problem is: at the time the ball has traveled the distance to the wall, the ball’s vertical position must still be high enough to clear the wall. Other conditions, such as air friction, rotation of the baseball, or

how far the ball will travelafter clearing the wall, are not considered. The equations are first solved for the initial velocity in terms of the angle of the ball coming off the bat. A critical angle, which I define as Theta0, is related to the wall parameters by Tan(Theta0) = (h-y0)/L where h and L are the wall height and distance from the batter respectively and y0 is the height of the bat when the ball is hit. The initial velocity equation found for the solution to the equations of motion gives the velocity of the ball at any angle greater than Theta0 needed to clear the wall. The next step is to find if the initial velocity has a minimum (it does) and at what angle.I leave the mathematical details to Mr. Kahn, unless he wants my proof. There is a richness of physics in this question, much more than the simple answer requested. I know Mr. Kahn doesn’t go into more than basic physics, but please make sure the basics are correct! This is not nitpicking; it goes to the heart of developing analytical methods of thought.

I sent the information in the e-mails to Khan, who responded in an e-mail that said:

Thank you and your friend for pointing this out. (please thank him on my behalf) He is correct. The video wrongly assumed that 45 degrees is the optimal angle (it is for distance if there were no wall), but not for actually clearing the wall. We have deleted the video. We are trying our best to keep up with any errors on the site (both through feedback from users and peer-review from educators). I checked into why we didn’t notice this one earlier, either your friend or someone else did point this out in the comments but they did not surface to the top (we currently have contractors with math/science backgrounds reviewing much of the math material and the comments to find other issues like this). We do need to get better at making sure that this type of thing doesn’t happen.

Khan explained in further e-mails that he and the Khan Academy have been “working with content specialists for about a year.” You can see why they are here. He also said that the contractors are looking at submissions sent to a page on the site that is accessible by clicking a button that says “suggest a fix for this video” located below the right side of videos.

And in another e-mail, he responded to a question about which videos were being reviewed:

The ones [content experts] that are listed are most heavily focused on the k-12 math content. They are writing exercises aligned with the Common Core and are helping to make sure that all of our k-12 content is correct and properly aligned. The entire intent is to make sure that we can properly meet the needs of students and teachers.

Above and beyond that, we implemented the clarifications system several months ago to better make sure that we don’t miss errors like the problem your professor friend highlighted. There too, we have had contractors looking through the clarifications and comments below videos to determine if videos had errors.

On top of that, we have recently gotten physics, chemistry and biology experts to peer-review that content (this was started as part of our partnership with the American Association of Medical Colleges —https://www.khanacademy.org/science/mcat).

Even before taking either of these steps, we’ve been working very closely with (mainly local) schools using KA for several years to get feedback on the site and content (including errors or gaps). We’ve also been regularly looking through the error/bug log.

As we grow and have more content and more traffic, it’s become clear that we’ve needed to add more process and folks looking at things .