The Tricky Task of Figuring Out What Makes a MOOC Successful

Why traditional metrics like completion rates aren't a good way to evaluate online courses
John Gress/Reuters

With a few keystrokes, you can register for a HarvardX MOOC on Computer Science, Genomics, Justice, or China.  Hundreds of thousands of people have done so, and in a report that we and our coauthors released this week, we show that only about 5 percent of these registrants go on to earn a certificate of completion these courses.  We could have titled the report: “MOOCs have low completion rates.”

Completion rates in courses, and graduation rates in colleges, have long been important metrics for measuring college success. If students invest time and money into earning college credit and then fail to complete a course, this represents an implicit breach of a commitment made by the students, instructor, and institution alike. If 95 percent of students who enrolled in a residential college course dropped out or failed, that course would rightly be considered a disaster.

After digging deeper into the data, however, we decided that completion rates are at best an incomplete measure, a position that is increasingly shared by many others. We would argue further: at worst, completion rates are a measure that threatens the goals of educational access that motivated the creation of MOOCs. 

Our data show that many who register for HarvardX courses are engaging substantially in courses without earning a certificate. In these course, “dropping out” is not a breach of expectations but the natural result of an open, free, and asynchronous registration process, where students get just as much as they wish out of a course and registering for a course does not imply a commitment to completing it.

Frames of reference change our interpretation of statistics.  A 5 percent completion rate is low for a conventional college course but high for other forms of media with which MOOCs share much in common.  One of the first HarvardX courses, JusticeX, was originally produced as a PBS series by WGBH in Boston. Professor Michael Sandel has 12 video lectures from that series that were posted on YouTube in September of 2009. The first video has nearly 5 million views. The second has 1.2 million views. By the fifth video, views have declined to about 200,000 views for each video. Rather than decry this “5 percent completion rate” as a crisis in public broadcasting,  we find it remarkable that Michael Sandel can post a 45 minute video lecture on moral reasoning and get 5 million views. Anyone who watched one video or all twelve is a little wiser for their efforts.

But completion rates may not only miss the point, they may also discourage foundational principles of open online learning.  For example, consider the impact that the Colbert Bump had on HarvardX this past summer.

On July 24, 2013, Stephen Colbert, of Comedy Central’s Colbert Report, welcomed Anant Agarwal, the president of edX, a MOOC platform that hosts courses from HarvardX and MITx.  Overnight, edX certification rates for daily registrants in Harvard MOOCs dropped noticeably, from a five-day average of 3.2 percent to an average of 2.5 percent.

How do we know that the Colbert Bump was responsible?  Daily registrations in HarvardX MOOCs more than tripled that summer night, from 406 on Wednesday to 1,356 on Thursday.  The number of certificates earned by these students doubled from an average of 12 to an average of 24 per day.

Presented by

Justin Reich & Andrew Ho

Justin Reich is the Richard L. Menschel HarvardX Research Fellow and writes the EdTech Researcher blog for Education Week. Andrew Ho is an Associate Professor at the Harvard Graduate School of Education and Co-Chair of the HarvardX Research Committee.

Would You Live in a Treehouse?

A treehouse can be an ideal office space, vacation rental, and way of reconnecting with your youth.

Join the Discussion

After you comment, click Post. If you’re not already logged in you will be asked to log in or register.

blog comments powered by Disqus

Video

Would You Live in a Treehouse?

A treehouse can be an ideal office space, vacation rental, and way of reconnecting with your youth.

Video

Pittsburgh: 'Better Than You Thought'

How Steel City became a bikeable, walkable paradise

Video

A Four-Dimensional Tour of Boston

In this groundbreaking video, time moves at multiple speeds within a single frame.

Video

Who Made Pop Music So Repetitive? You Did.

If pop music is too homogenous, that's because listeners want it that way.

More in

Just In