A/B Tests in Online Learning

A/B tests are not only beneficial in website optimization, but too have great potential in leveraging online learning. In this very (very) brief post on A/B testing in online learning environments, I share some links to interesting online material that collectively may serve as a primer on the topic.

A/B tests

A/B tests are large-scale online randomized controlled experiments. Meaning, an online environment such as a webpage or app is randomly varied (hence A and B) across users, and after some time it is evaluated which variant is most effective. The randomization and large scale make sure alternative explanations are controlled for. Naturally, an A/B-test can be extended to any complexity one desires (e.g., an A/B/n or multivariate test).

Although superficially the idea is simple and straightforward, proper A/B testing is far from trivial. One great way to learn more about it is through the online course from Udacity. Also, Ronny Kohavi and his team are some of the few that publish many excellent papers on A/B testing.

Others that point out difficulties and propose solutions are for instance Slater Stich, Manojit Nandi, and Will Kurt.

Online learning

Large-scale online learning environments provide an excellent opportunity for experimentally optimizing learning gains. A few online educators are known to run A/B tests, such as DuoLingo and Khan Academy. However, results are shared only sporadically and informally (e.g., here, here, and here), and most of the environments are not available to external academic researchers.

A notable exception is the intelligent tutoring software ASSISTments. ASSISTments uses a crowdsource model that enables external researchers to run experiments on the platform. Moreover, massive open online courses (MOOCs) may as well lend itself to A/B tests (e.g., EdX content experiments), as long as the platform allows and preferably facilitates such tests and one can cooperate with a teacher on the platform. Finally, but unverified, DuoLingo says its crowdsourced course building platform provides A/B testing capabilities.

Interesting work on A/B testing in online learning is for instance done by Joseph Williams; make sure to watch his TEDx talk. I myself am currently setting up A/B tests in Math Garden and a Coursera specialization. If you’re interested in the topic or wish to cooperate (either in setting up experiments or analyzing its data), feel free to send me a message.

Photo: Kevin Dooley / Flickr