31 March 2017

Improving learning

Selecting interventions





What's next

  1. What is online learning?

  2. Improving online learning with A/B tests

  3. What can we learn from A/B tests?

  4. In conclusion

What is online learning?

Online learning

Personal definition




"Any learning activity that is provided online and that scales."

Gamification

Game based learning

MOOCs

Intelligent tutoring

Online learning activities that scale

"… 5.5 million students took at least one online course." (US, 2012)

"… more than 24 million people took Codecademy courses …" (worldwide, up to 2014)

"… pilot included over 150 teachers and 8,000 students in a diverse range of schools from urban, suburban, rural to virtual schools." (US, 2014)

"Ruim 900 scholen zijn op dit moment actieve Rekentuin gebruikers. Er worden zo’n 500.000 sommen per dag gemaakt." (NL, 2014)

"Khan Academy is used in over 30,000 classrooms worldwide." (2014)

Improving online learning
with A/B tests

A/B testing

A/B testing

Benefits

  • Very cheap
  • Effortless randomization
  • Interventions are easy and homogeneous
  • Instantly available data
  • Monitor process and related variables
  • No more B tests
  • Instant ecological validity and valorization with impact

Requirements

  • Many many users and observations

What can we learn from
A/B tests?

A/B testing in perspective

A drug cocktail

Norman: "It is unclear with most curriculum interventions exactly what the therapeutic intervention actually was."

We:

  • A/B tests are used for very precise interventions.

A/B testing in perspective

Illustration: question mark


A/B testing in perspective

Lost in a sea of unexplained variance






Norman: "…, while a controlled trial may well yield unbiased estimates of treatment effects, they will likely be lost in a sea of unexplained variance, and may not even be detectable."

We:

  • Use specific measurements.
  • Target specific users: diagnostics.
  • Lots of users.

A/B testing in perspective

Illustration: question mark

A/B testing in perspective

Where will new knowledge come from?

Norman: "… the accumulation of knowledge about the phenomenon [transfer] came about from many small, tightly controlled studies, …"

I:

  • How do we know whether such a phenomenon generalizes to a natural setting?

We:

  • One could use A/B tests to test such a phenomenon in vivo, that is, test its ecological validity.

A/B testing in perspective

Illustration: ecological validation

Spaced practice


… in Coursera

A/B testing in perspective

A vacuum

A/B testing in perspective

A vacuum

Sure, there are exceptions (MOOCs, ASSISTments), but still.. why?

  • many users required
  • demanding for fast growing companies: priority, payoff, intervention
  • commercial companies

A/B testing in perspective

Illustration: power to the teachers (and researchers)!

In conclusion

Adaptivity


"Ideally, a MOOC* would work like the GPS navigation device in your car. You tell it where you want to go, it figures out where you are, and it guides you along the most optimal route. Keeping with the analogy, current MOOCs are like having all GPS navigation devices instruct every car driver to turn right at 9:15 on Monday morning."
* or any other (online) learning environment


excerpt from Navigating Massive Open Online Courses

Brainstorm



Will (traditional) education benefit?

Brainstorm



Will (traditional) education benefit? And your favorite area of psychology?

Learn more

Savi, A. O., Ruijs, N. M., Maris, G. K. J., & van der Maas, H. L. J. (2017). Online learning solves sturdy problems in education experiments. Manuscript submitted for publication. [preprint]

Savi, A. O. (2017). A tool for MOOC teachers to make evidence-based and domain-specific course improvements. Manuscript submitted for publication. [preprint]

Savi, A. O., Williams, J. J., Maris, G. K. J., & van der Maas, H. L. J. (2016). The role of A/B tests in the study of large-scale online learning. Manuscript in preparation. [preprint]

Savi, A. O., van der Maas, H. L. J., & Maris, G. K. J. (2015). Navigating massive open online courses. Science, 347, 958. [full text, reprint]

Thank you