class: bottom, inverse background-image: url(images/leidsontzet.jpeg) background-size: cover ## Adaptive practice in online learning #### Leiden University, Honours Class, Revolution of Learning in the Digital Age #### Alexander Savi, 7 March 2017 ??? * Before we start, here's a little quiz. * Raise your hand when you think you know the answer. --- ### During what yearly festivity do Leidenaren celebrate the liberation from the Spaniards? -- #### Leidens Ontzet -- ### Who painted "The Relief of Leiden", picturing the Leidens Ontzet? -- #### Otto van Veen -- ### During which years did the siege take place? -- #### 1573 - 1574 -- ### In which century? Or during which war? -- #### 16th century, Eighty Years' War --- ### During what yearly festivity do Leidenaren celebrate the liberation from the Spaniards? <font color="#D55E00">too easy?</font> #### Leidens Ontzet ### Who painted "The Relief of Leiden", picturing the Leidens Ontzet? <font color="#D55E00">too hard?</font> #### Otto van Veen ### During which years did the siege take place? <font color="#009E73">better?</font> #### 1573 - 1574 ### In which century? Or during which war? <font color="#009E73">better?</font> #### 16th century, Eighty Years' War ??? * Last questions aimed at finding the level that is most appropriate for you. * Did we find the right level? --- class: center, middle <img src="images/adaptivity_animation.gif" style="height: 500px;"/> ### Adaptive practice ??? * This is what we call adaptive practice: making sure it fits the needs of a specific student. --- class: center, middle, inverse # <font color="#E69F00">Where?</font> # <font color="#56B4E9">How?</font> # <font color="#CC79A7">Why?</font> ??? * Today I'll tell... --- ## But first, hi! #### Past * UvA Psychology * Honours Bachelor * Research Master * Psychological Methods * Oefenweb -- #### Present * PhD candidate * Online learning * <font color="#CC79A7">A/B tests</font> * <font color="#009E73">Computerized adaptive practice</font> * Cognitive development * Oefenweb / Coursera -- Find out more, contact, and slides @ [alexandersavi.nl](http://www.alexandersavi.nl) ??? * Started as an honours student, just like the students. * Did a few jobs for Oefenweb, a company for computerized adaptive practice, which I'll talk about today. * During PhD I've been working with both Oefenweb and Coursera (MOOC) * Please interupt with questions or ideas. * No idea whether enough or too much to fill 45 minutes, tell me if I need to speed up or slow down. * Help me get out of it what you need. --- class: center, middle, inverse # <font color="#E69F00">Where?</font> ### An application for adaptive practice --- ## Oefenweb .pull-left[ <img src="images/oefenweb_garden.png" style="height: 200px;"/> Math Garden, Language Sea, Words&Birds, Stats Plant ±150.000 active users ±1.000.000 responses p/day ±1600 schools ±45000 items ] -- .pull-right[ <img src="images/oefenweb_item.png" style="height: 200px;"/> Many domains (e.g., +, -, x, /, money, word problems) Choose difficulty level Collect coins Question mark Instructions ] ??? * For primary schools. * Originally developed at the UvA. * We call questions/problems items. Explain * Difficulty levels (60, 75, 90) * Coins (scoring rule, incentive) * Question mark --- ### Adaptive practice by Oefenweb (Dutch) <iframe src="https://player.vimeo.com/video/152146029" width="750" height="500" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe> ??? * Spoken in Dutch, short intuitive explanation of adaptivity. * <font color="#D55E00">Who cannot speak Dutch?</font> * Explain the adaptive algorithm in more detail right after. --- class: center, middle, inverse # <font color="#56B4E9">How?</font> ### The technology for adaptive practice ??? * How is this exactly done? --- class: center, bottom, inverse background-image: url(images/itemchild.jpeg) background-size: cover .pull-left[ # Difficulty? ] .pull-right[ # Ability? ] ??? * Take-away from the video: we want to match students with items. * Faced with a problem. * How to determine ability student and difficulty item? * <font color="#D55E00">Who already understood how it's done from the literature?</font> [Image credit](https://www.nrc.nl/nieuws/2015/11/07/digitaal-is-geen-dogma-1553286-a315962) --- class: right, inverse background-image: url(images/chess.jpeg) background-size: cover # Elo rating system ??? * Important and clever step to make: going from educational practice to sports. * Idea comes from chess, but can be applied to other competitative sports. * Here you see chess competition, want to know ability of each player. * Explain algorithm. * Idea conceived by Prof. Arpad Elo, hence Elo. * <font color="#D55E00">Important realization: opponent of a student is the item.</font> [Image credit](https://commons.wikimedia.org/wiki/File:British_Chess_Championship_2009.jpg) --- class: center, middle ### `$$ability_{p}=ability_{p}+K*\big(S-E\left(S\right)\big)$$` ### `$$difficulty_{i}=difficulty_{i}+K*\big(S-E\left(S\right)\big)$$` where `\(S\)` = observed score {0, 1} `\(E\left(S\right)\)` = expected score, based elo-scores and measurement model `\(K\)` = constant scaling factor (determines the influence of a win or loss) ![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAASAAAADYCAYAAABLNT5AAAAEDWlDQ1BJQ0MgUHJvZmlsZQAAOI2NVV1oHFUUPrtzZyMkzlNsNIV0qD8NJQ2TVjShtLp/3d02bpZJNtoi6GT27s6Yyc44M7v9oU9FUHwx6psUxL+3gCAo9Q/bPrQvlQol2tQgKD60+INQ6Ium65k7M5lpurHeZe58853vnnvuuWfvBei5qliWkRQBFpquLRcy4nOHj4g9K5CEh6AXBqFXUR0rXalMAjZPC3e1W99Dwntf2dXd/p+tt0YdFSBxH2Kz5qgLiI8B8KdVy3YBevqRHz/qWh72Yui3MUDEL3q44WPXw3M+fo1pZuQs4tOIBVVTaoiXEI/MxfhGDPsxsNZfoE1q66ro5aJim3XdoLFw72H+n23BaIXzbcOnz5mfPoTvYVz7KzUl5+FRxEuqkp9G/Ajia219thzg25abkRE/BpDc3pqvphHvRFys2weqvp+krbWKIX7nhDbzLOItiM8358pTwdirqpPFnMF2xLc1WvLyOwTAibpbmvHHcvttU57y5+XqNZrLe3lE/Pq8eUj2fXKfOe3pfOjzhJYtB/yll5SDFcSDiH+hRkH25+L+sdxKEAMZahrlSX8ukqMOWy/jXW2m6M9LDBc31B9LFuv6gVKg/0Szi3KAr1kGq1GMjU/aLbnq6/lRxc4XfJ98hTargX++DbMJBSiYMIe9Ck1YAxFkKEAG3xbYaKmDDgYyFK0UGYpfoWYXG+fAPPI6tJnNwb7ClP7IyF+D+bjOtCpkhz6CFrIa/I6sFtNl8auFXGMTP34sNwI/JhkgEtmDz14ySfaRcTIBInmKPE32kxyyE2Tv+thKbEVePDfW/byMM1Kmm0XdObS7oGD/MypMXFPXrCwOtoYjyyn7BV29/MZfsVzpLDdRtuIZnbpXzvlf+ev8MvYr/Gqk4H/kV/G3csdazLuyTMPsbFhzd1UabQbjFvDRmcWJxR3zcfHkVw9GfpbJmeev9F08WW8uDkaslwX6avlWGU6NRKz0g/SHtCy9J30o/ca9zX3Kfc19zn3BXQKRO8ud477hLnAfc1/G9mrzGlrfexZ5GLdn6ZZrrEohI2wVHhZywjbhUWEy8icMCGNCUdiBlq3r+xafL549HQ5jH+an+1y+LlYBifuxAvRN/lVVVOlwlCkdVm9NOL5BE4wkQ2SMlDZU97hX86EilU/lUmkQUztTE6mx1EEPh7OmdqBtAvv8HdWpbrJS6tJj3n0CWdM6busNzRV3S9KTYhqvNiqWmuroiKgYhshMjmhTh9ptWhsF7970j/SbMrsPE1suR5z7DMC+P/Hs+y7ijrQAlhyAgccjbhjPygfeBTjzhNqy28EdkUh8C+DU9+z2v/oyeH791OncxHOs5y2AtTc7nb/f73TWPkD/qwBnjX8BoJ98VVBg/m8AAB/NSURBVHgB7Z0LvI1V+sef4yScCrlfk4gMTuJDRpQkGkoqlXRKzMhMpSgzDELpKk00LpFJpQ/SuPRPyl1u465cao6aSCiXXCYUB/v//tb07vY++93H2Wdf1lp7/5bP9r77va1nfdc6z17vWs96njSfk4SJBEiABDQQKKQhT2ZJAiRAAooAFRAbAgmQgDYCVEDa0DNjEiABKiC2ARIgAW0EqIC0oWfGJEACVEBsAyRAAtoIUAFpQ8+MSYAEqIDYBkiABLQRoALShp4ZkwAJUAGxDZAACWgjQAWkDT0zJgESoAJiGyABEtBGgApIG3pmTAIkQAXENkACJKCNABWQNvTMmARIgAqIbYAESEAbASogbeiZMQmQABUQ2wAJkIA2AlRA2tAzYxIgASogtgESIAFtBKiAtKFnxiRAAlRAbAMkQALaCFABaUPPjEmABKiA2AZIgAS0EaAC0oaeGZMACVABsQ2QAAloI0AFpA09MyYBEqACYhsgARLQRoAKSBt6ZkwCJEAFxDZAAiSgjQAVkDb0zJgESIAKiG2ABEhAGwEqIG3omTEJkAAVENsACZCANgJUQNrQM2MSIAEqILYBEiABbQSogLShZ8YkQAJUQGwDJEAC2ghQAWlDz4xJgASogNgGSIAEtBGgAtKGnhmTAAlQAbENkAAJaCNABaQNPTMmARKgAmIbIAES0EaACkgbemZMAiRABcQ2QAIkoI0AFZA29MyYBEiACohtgARIQBsBKiBt6JkxCZAAFRDbAAmQgDYCVEDa0DNjEiABKiC2ARIgAW0EqIC0oWfGJEACVEBsAyRAAtoIUAFpQ8+MSYAEqIDYBkiABLQRoALShp4ZkwAJUAGxDZAACWgjQAWkDX1sMu7evXtsHpTgp1DuBAM3NLvzDJWrQGJNnjxZZs6c6XnvDz/8IG3atJFBgwZ5nrf14NGjR60UnXJbWW0xFzqpFNCdd94pt9xyiyek2bNny3//+1/PczxIAslEICcnR7V1tPfjx48LlH21atWkSpUqxhUzqRRQ0aJFBR+vdMEFF8iJEye8Tll9LJzCNb1QlDvyGjp06JDs379fDh48qD74jp49tocPH5Yff/xRjhw5IlBAJUuWlAsvvFAuu+wy+fnnnwU/ziYqoDSfkyJHYd8d7733nhw4cEAeeugh+4SnxClB4NSpU7J3717ZtWuX2u7Zs0dtoXT27dsnJUqUkNKlS0v58uXl4osvlrJlyypFg2NQODiGbZEiRazhlVQ9IGuoU9CUJoDf/G+//Va++uor2bFjh/znP/9RSgeKBr2UypUryyWXXCKXX365XHfddVKpUiWlbAoXLpx03KiAkq5KWSDTCKDnvXXrVvn3v/+tPtu3b1cKBa9HNWvWlHbt2qlXpQoVKkihQqk1MU0FZFprjVCeDz/8UNq3bx/hXfovT2a58bq0YcMG+fTTT2Xz5s1qTOY3v/mN1KtXT2B+gJ5NRkaG/kowQAIqIAMqIRoR3njjDSsVkG1yb5WNMlwGyIbMdfKeNJWRMllKSilVdadPn5aNGzfKmjVrZO3atWqyo2HDhoLP/fffb+TgbzRtLpb3UgHFkiaflZQEvpc90kM6ygHZJ1JVZKUskrvOtpReK4bJpqWfybp169QrVLNmzWTo0KFSo0aNpOQQj0IZq4DOnj2rfkkwlRirhBkGdIvddOWVV0paWpr7VbZs2SJnzpzxf7fhvF9YZ8cm+TFdjMFYG/ij56OUzy+wfeKTr3O2y6Sjo6Xn1X3l8ccfV1Pe4I+pcLeN2dB+AvkHtqVE7RuhgGAwNWHCBPnkk09UZaISf//738vJkyelc+fO8re//U1VcLRQ8F5+7Ngx/2NeeuklCZxZeOedd1Se7gU2nA+0p7FJ/vT0dMGriw389/60W6SY2yp+2RbxSfN2zaRtelv/CZv4Q+jc7dtfkATuGGEHNGDAADUlef3118u4ceNUw/zHP/4htWvXVgoJU5HdunWLCgvtgKLCl3I3oye8bNkygQX9pgbL5EDWV5JT+JSfw3lSWN6Q/5Nm0sp/jDuREzCiB/T++++rwTtYK2MGAZaev/3tb1Vp/vrXvyolFK0CihwN70hFAli6MGfOHJkxY4ayx7n77rtlRNMRcl+htrJdtskp519xZ/i5k9xP5RODBmKEAqpTp44sWLBA0APCr85PP/3kLxqmMTGbwEQC8SSA9VLTp0+XDz74QDCY/OKLL0r16tX9WU6TxbJcFshROSLV5DKpL43857hTcAJGWD098cQTgp5OuXLlpFGjRoLBOygljP/06dNHunbtWvASJvmdsKexMZkiNxTP+PHjJSsrS43/TZw4Ufr37x+kfFy+LeRGSfswg8rHBRKDrRE9ILxuff7552pRHda1YPB53rx5amHdpEmTpFix3COAMSh5kjzCNnsaF7tuubFA891331XuW2688UZ588031TorV75wW91yh5PL1uNGKCDAw3QglA8SFtN16NBB7WdnZ6vp+Kuuukp9z+s/NCh0ob3SN998I/l5hte9PJY8BDD1/9FHHwl6Ok2aNFG9HyyBYNJDwBgFFK74mL2C8nj99dfDXeI//rvf/U6aN2/u/x64g4HFZHTHEVhG7udNAOuxRo4cKRdddJEMHz5crcPK+w6ejTcB4xVQJB4MixcvLvh4pVKlSgUZGXpdY+OxQDsgm+RPpNwwenzttdeUgeAf//hHadmyZYFRJVLuAgtp0Y1G2AEF8oJxGgwR4dsklol2QLGkac+zMJYI5XPTTTcpW7Lzzz/fHuFTQFIjZsHgiAnGiFWrVhU0EPRWYBOE1cMYhGYigUgJwLcOZldhSDhixAjp2bOnaluRPofXx5eAEa9gvXr1ku+//14wNQsfKVA+WJ6BmbHevXsrl5J/+tOf4kuCT08aApiIwA/XXXfdpUw5kqZgSVgQI3pA8+fPV7MRmZmZas0XZsTgfhLT86NGjVK/YknIPiZFMsWeJtLCxENu+Ebu16+f+iF79dVX46J84iF3pOyS6XojFBBetZYsWeLJFbNX8H3L5E0Adik2pljLvWrVKrWAGW1pzJgxcfPBE2u5bay7WMpsxCvY008/LV26dJFXXnlF+VLBTBYsVL/44gu1MHXu3LmxLDOflUQEsGgUCgfOwJ599lmB50EmewgYoYBgILhp0yb517/+JTt37lTjQej1YNzn2muvDfIZYw9aShpvAvDv9NRTTymn7TAspMV8vInH/vlGKCAUC/G8sBiVKTICttqlRCv38uXL1ewW/Ea5VvORkSvY1dHKXbBck/cu4+yA4oWadkDxIpvY58JTJno7cF4H96dw8M5kLwFjekD2IqTkiSIA04whQ4aotYJYwR5Ld72JKgPzCSYQVgHBOBDjMtu2bZOvv/5arZuBqwy4yTjvvLC3BT+d30ggRgQQxO/JJ5+U1q1bq9muGD2Wj9FMIGQaHquFp06dKnXr1lW+eBYvXqwchGEm6t5771XTmy+//LJAQTHpJ2CrXUokci9dulRZNT/yyCPalU8kcutvHeZLEKSA4CPl5ptvVm5RMSMF2wo42obCgbc4eCfE1Diua9WqlYpbbX4Rk1tCW+1S8is3roMnBKxiv+aaa7RXZn7l1i6oJQIEvUvh1WratGnKXUE4+bFIdODAgcpPMwYETUqzZs1Sjsy8ZEL8bRipMdlBAE7p0BYR4gaLSeFCgyn5CIQoILeic3Jy/CFTENMaPnlgk+PaWrhbk5C0aNEirCHaxx9/rELkmiQvZfEm8MMPPygXvYibjpBMHHP05pQMR4MUEAq0Z88eufXWW9UrFpw2Ycrzsccek1q1aqmYWqtXr/Z7LjQNQJkyZQQfr4TXxwMHDnidsvqYrXYp4eTGYDN62B07dpR77rnHuLoJJ7dxgloiUIgdEPymYEX6c889p9wXwFH8P//5T+VPBYoI7lKhmGxLtAMyv8YQVx3trm/fvmE9W5pfCkoYCYGgQWjMgGHgGbYWJUuWFFibYgulhATtv379+kiez2tJIF8E4LcHkToRDiecW918PYgXWUUg6BUMC/vgkdAd31m4cKEgYoCb4NoynMtT9xpuSeBcBFauFHGGeZyFx+KYe4haTIofNiwqRY+bKXUIBPWAMNiHmFwY99m9e7eahbjjjjsUDfSOEC6ZkSXMahy22aV07y7iBBt1bMpOSePGPunUaa0Kyz127FgrlI9tvM1qraHSBCkgnHZtLqpVq6Z6P+3bt1f2PnBzAK+FeD9nMoeATXYp48aJE/IYEx3iTGic7xi4pjkhcuo5A84j/L1uc8h6S2ITb+8SmHU06BUMosFWZteuXcolqvu6hS0MwbBa3Wan3piK//bbb/01AD9EhQsX9n/H2FeghbcN5/3COzumy//yy1867eryQJGdUEkZ4hg6y3XXmS8/+CK0D6L4ItnQPiJp36pQCf4vSAEhJjum2aFoXOUDebDor23btkGiYYAadhoVK1YMOh6rL7C2hjIIlCPaZ6MXF+i6Ibd9CXp7GAdzE88HNQ+Jls9ll10sjj1oUEKQCsf7rkrRPj8R92MVvtuGkq19BFVMor44Yzv+5AxA+wYPHuxzbDB8U6ZM8TmDzv5z2HF6Dz5nSt7nGPz5HGdhPid8TtD5WH5xloD4HF8vMXuks5TE5wxyxux5pjzIcb5uiijnlGP8+LW+IkUO+UScAcWAz9Gj57zVmAts4m0MtDwECbEDguKDpzl0N7G0AcstypcvrwalMSUPR/FYlVy/fv2Y6Uj4dDl48GDQ89D7wYxcRkaGMkpDlINoEu2AoqEX/b0Imz3DGQDKyhopf/lLJccq/X8zYKNHi1xySfTP5xPsJOCpgAKLgoFn1x1HvKZIV6xYId2d6ZGsrCzp2rWryh52IVgQC7sQhOkJZ+EcKGte+1RAedGJ3zn8gMHX9/bt2+X5559XMd/ilxufbBuBkFmwL7/8MmgcpEKFCtKsWTM1RYr1YBj7iXWC4RnsQGCG//jjj/sVDsaeMBsXrfKJtbx8Xv4IYEzxhRdeUL1bhMlBwEkmEggkEKKA8Irlvg6tdCzG/vCHP/ivR4/k73//u/97LHcw2Pz22287NiJ3q0WvMIJkOjcBU+1S9u3bJw899JCaREC0CizhCUymyh0oo9e+rXJ7lcWEYyEKKFCoY8eOqSn5wGPx3kc0SwQqhBJE74spbwIm2qXAewIimmBRM5yIFSoU2sxMlDtv0v87a6vc+SmbjmuC51l1SOCRZ5UqVQThdZGys7MdW5ET+bLAxq8TPDh6JYxB1K5d2+sUj8WQwKJFiwRWzf3795cmTZrE8Ml8VDISMFIBBYLG4DHGnmChfa6EZSLh7JIWLFjA+GLnAhjlefQO8OoMD5qXXnpplE/j7alAwFMB4fUH8dmx+BTT4fv371csEK000WnQoEH5zrJSpUoqSJ3XDfCISH9AXmSiPwbvhbAKRk91nLPeooRrWZjHo231q2Or3HlUhdZTIdPwmHGCR7pw6c4771T+ocOdj/Y4bH8cA0eB69dYJk7Dx5Lmr8+CAzvXLgz+orzGe369mnskEEwgpAeE1x3HcDH4qoBvuc3PA04VeBe9rKFDh8rkyZOVR0bkDwPE6tWrq2gI3bp1K/CzeWP8CKxbt045EMNMKZZBMJFApARCFBCM/hKdevXqpVbaYxAZ3hghA4LQff7559K7d28VhQOzKkzmEHjrrbcEoZowxY41dkwkUBACofOjBXlKlPdg2h2RLjMzM9XCV4w/YRwBNkmjRo0SWEUzeRNItF0Kfhgww7Vx40ZVZwVVPomW25te5EdtlTvykibmDiMUEFyALFmyxLPEc+bMkbJly3qe40GRRNqlIEqus0BYeUHA8gqsDSxoSqTcBZXR6z5b5fYqiwnHQl7BdAiFGZQuXbqoNUM1HD+dsIrGjBuCIGJQGl19Jn0EMCaHGF0YyO/Xr59cffXV+oRhzklFwAgFBPsdxKHHUo+dO3eq8SD0ejDug1hkeCVj0kMAM6LPPPOMyhy2WKVLl9YjCHNNSgJGKCCQLVq0qHKElpSU41ioeNqlwKp8tOMvo1OnTqqHGstixFPuWMqZ+1m2yp27HKZ8D7EDMkWwWMtBO6D8E8VAM1avu0EC4a+JiQTiQcCIQeh4FIzPLBgBuFt54IEH1MD/hAkThMqnYBx5V/4IGPMKlj9xeVW8CBw6dEit4YI3TNj21KlTJ15Z8bkk4CfAHpAfhZ070dqlwGMh3KXCI+UVV1yhYsIlQvlEK7eu2rJVbl28zpUvFdC5CBl+Phq7FBgTwq4HWywive+++yQ9PT0hJY5G7oQIGCYTW+UOUxzth5PqFQwuN8K5jMWyDizzYBIVYABhkBEjDQ7DmjZtSiwkoIVAUikgDJhiOt8rYRFtYJA2r2uS/RhsevALDnsrGH7CvidRPZ5kZ8vyFYxAUikgOMHCxyshukeq+gM6fPiw8rcNux4E1YPXAR2LjgPrxVZ7GlvlDmRv0j7tgEyqjRjLAqXrBJgURPNs166ddO7cOV/OwmIsBh9HAmEJJFUPKGwpk/CEzwlhdPbJp8S3e4+k1a8rhV4eLmnFiqmSYrwLM1ubN2/293hiGeI6CXGySJoIUAFpAh9Ntj7HZufM9W2d0eQ96jG+tevk9MrVsmhwf5nluDZBPK7bbrtNBg4cKOcj+DoTCRhKIKUUEHxc79q1y18Vl+SKCbx7924Vitq9wNTzJZ7oLxno+biCnsqRk46rjCOjx0mHvn38YbNzKx+TyofoGbk9XZokH9B61T9mWm+44QZF3us87KrcZNt5V+5EblNqDAhTz4GNYuLEiUE9hAcffFB5X3QrwNTzj85bLA32H3TF9G/nXJUpM+td4f9uqvwQcOnSpWqtWaCStIH/xx9/LC1btlSMTeYLASOVTxUqwf+llALCLBiiddqYEOwPNk6rVq2SG9dvktuzv5Lzck7/WpRiRaXQ9ClS6OZ2vx4zeO+OO+6QGTNmGCyht2i2yu1dGv1HjXsFi1dUDP2oI5MAa7Ngobx27VpZv369iqsOF7Vwh1qrWjU5U+8qkf0HRH48JlK2jKS1aW2N8omMBK9OZgJGKCBGxRDlhA0uTz/77DPZsmWLCo3UqFEjwQevJgiXFJjSt28T37vvyeoP50rT++6VQm3bBJ42ft9Wexpb5Ta1QRjxCtazZ0/1Bzhs2DDPqBhwDxFtVAyT/AEh8CN87eADt7MIG41Uv359gX/sBg0acNmIqX8xlCumBIxQQIj/heUBFSpUCCnc6tWrZciQITJv3ryQc5Ec0KGAMOaEwH07duxQ668QnRUuZxG8r2bNmipWPZaPILJE7h5OJGXjtSRgKwEjXsHcqBj33HNPCEdTo2JguhXjNAhbjTVWsDr+7rvv1HcoHXy/8MILpWrVqupTzRm3ueaaa1TPJppoEiGAeIAELCZghAIyISrGmTNnBGumjh8/roIiIjw0PjgGF6VQMtiiV4OIHVA+cNCOnku5cuVU761y5crSuHFj/37gFHO82gj809gYlZRyx6tF2PVcIxSQCVExYCOEaW4sWcBCTfRSLrroIvUdcerRgylVqpSKWQ+lg8CJJkTrwOp2GxUQ5bZLUcRLWiMUEAoXLipGdna2nDhxQqCkzpWw6BJjRl4J66ICjRBzX/Poo48KPkwkQAKJI2CMAgpXZAwef/PNN4KYVOdKFStWlCuvvNLzMozFIOghEwmQgDkEjJgFMwEHxm9sjACxdetWNXVvAsNIZKDckdCK/lpMlsBLgmnJuB6QLktoKB+sT7ItYV0S5U5crdnMO3GU8p9TofxfGr8rYQk9YMAANV2NmSMM9mIgGNPzkyZNil/GfDIJkIBWAkb0gHr16qXsZjA1C8fxUD6Y8kaXsXfv3mqFerSW0FopM3MSIAFPAkb0gOY7TrTGjx8vmZmZyngP09uY5sbiy1GjRsns2bM9hedBEiABuwkYoYBcS2gvlKZaQnvJymMkQAKRETDiFcwES+jIsPFqEiCBWBAwQgGZYAkdC5h8BgmQQGQEaAf0Cy8sJIUho22Jcie2xsg7trypgGLLk08jARKIgIARg9ARyMtLSYAEkogAFVASVSaLQgK2EaACsq3GKC8JJBEBKqAkqkwWhQRsI0AFZFuNUV4SSCICVEBJVJksCgnYRoAKyLYao7wkkEQEqICSqDJZFBKwjQAVUK4aw8p8+COyJU2dOlVatWqlXNFmZWWpQIcmyw7nac2bNxfEgrvttttU1BGT5XVls42zK7e7NbZd+5j8BJxQOz4nOqnPiYjhP2byjrMswFe+fHmfE4NMielEmvC1adPGWJGdkEY+Z7mLzwk/7XOc0Pn69Onj69atm7HyuoLZxtmV292a3K7ZA3J/IpwtHKM5fxRGhNsJECvsLoIjTp8+XRwlpK6BQ/5Vq1aFvV73ifXr10udOnWU36fChQsr3jNnztQt1jnzt41z7gKZ3K6NWA2fG5iO74i+gdBAN9xwg47sC5RnpUqVBB83TZgwwegYYbt27Qpa8AvFiSCPJ0+elCJFirjFMG5rG+dAgKa3ayogp7YQRhk+iRCYEK5gbUwTJ06UDz74QNasWWOs+IguC3e7bipWrJjaRdw3kxWQKy+2NnB25bWhXafkK1iPHj0Ezu/xwf7DDz+sBkZXrFghCxcuFDjJhydG/DKblHLL7coGd7aDBg1SslepUsU9bNwWEWUDFTxCX6PXicizNiRbOLssrWjX7kBVKm2dQIe+DRs2qM/OnTt9nTt39jn+p9WnYcOGvvT0dLXv/GIbhSW33BDuzTff9FWoUMG3ZcsWo2T1Embu3Lm+pk2b+k9hMNoJh+T/bvKOTZxdjja0a/oDcn8uftlinKJBgwbizBzkOmPe1x07dogzayeIJoKtm0w1I0CPEuGxp0yZonqcDz74oBrDev75513RjdzaxtkLoqntmmNAXrVlybGxY8fK8ePHBcHyAhOOZWRkBB4yYh/jPKNHj5aOHTuqqCe1atWSMWPGGCFbXkLYxjmvsph2jj0g02okBeTRFf02BdBaV0QqIOuqjAKTQPIQSMlZsOSpPpaEBOwmQAVkd/1RehKwmgAVkNXVR+FJwG4CVEB21x+lJwGrCVABWV19FJ4E7CZABWR3/VF6ErCaABWQ1dVH4UnAbgJUQHbXH6UnAasJUAFZXX0UngTsJkAFZHf9UXoSsJoAFZDV1UfhScBuAlRAdtcfpScBqwlQAVldfRSeBOwmQAVkd/1RehKwmgAVkNXVR+FJwG4CVEB21x+lJwGrCVABWV19FJ4E7CZABWR3/RkrfU5OjrGyUTBzCFABmVMXRknSu3dvGTZsWIFkmjp1qjhhjgp0b143IUJpdnZ2XpeEnIMsrVq1EoStzsrKki+++CLkGh7QR4AKSB/7pMv58OHD8sgjj8hjjz0mTmwq7eVDZNA+ffoIlJATg0yF3YZiZTKHABWQOXWhRZJPPvlE9Q5Kliwpt99+uxw8eDBEjj179kjfvn1VDK/MzExZvHhxyDU4sGjRIhUO6K233vI8n5+DUFzPPPOMIMJr5cqV5dlnn/VUZpAb8iIGGsL8QNnkTmfPnpXp06cLYtAjoRe0atWq3Jfxu0YCVEAa4evO+sCBA3LLLbfIn//8Z/VqUqJECfEKEtilSxdBXPd169apHs6tt94qe/fuDRG/U6dOMnz4cHFjvodckI8DkydPlnfeeUfFuZ89e7bqvaxduzboTgQK7NChg/ps3rxZ5de1a9ega/AFr2zXXnut//iECROkffv2/u/c0U+ACkh/HWiTYObMmVK3bl31h3zBBRfIwIEDxQmfHCTP7t27ZdmyZapXgh4JopnWrFlTZs2aFXRdrL6g99StWzepUaOG1K5dW7p3766UUeDzoZjq1asnDzzwgOopYaxq/vz5sn///sDLgvYnTpyonjNixIig4/yilwAjo+rlrzV3KBcnprz6Qw8UBK9cbnLi0UvVqlXV65B7zInv7tkDcs/ntZ03b57qdeEaKD2MGwUm5P3SSy/JyJEj/YcbNmzo38cOZIIMboJCLF26tJKpXLly7mH/dvz48TJkyBBZsmSJUlj+E9zRToAKSHsV6BOgSZMm0qxZM9V7cKXAq1XFihXdr1K2bFn1h33kyBHBOBHS1q1bxeuVx39THjuYHVu9erW6Ij09PeTKxo0bq9emHj16qHPHjh2TM2fOBF1XpkwZWblypf/Yd999J4cOHZLq1av7j7k76FENHTpUFi5cKHXq1HEPc2sIAb6CGVIROsRo3bq1rFmzRjZt2qSyx9jLTTfdJBi8dRNehfCHjVcYDBB/+umnsmHDhqAeiHttfrbFixcX9GjwwaBw7oTxpUmTJqmeEfLD1Pkrr7wSdFnbtm1l+fLlsm3bNiXr66+/rl4lMYYVmDBW9PDDD8u0adPUeBCUFD5M5hBgD8icuki4JBgsfu6556RFixbqFQvfX3vtNQnsmWD/7bffFgxEv/rqq/Lzzz/LmDFj1BhMPARu166dTJkyRS699FLV+0KvpV+/fkFZoZc0ePBgQQ8Or15QPF5jUmPHjpXjx49Ly5Ytg+7HsYyMjKBj/KKHAGPD6+FuVK54xcErFv6Y80oY5MUrWVpaWl6XxeQclAQSxonCpdOnT8vRo0fPKXe4+3lcPwEqIP11QAlIIGUJcAwoZaueBScB/QSogPTXASUggZQlQAWUslXPgpOAfgJUQPrrgBKQQMoSoAJK2apnwUlAPwEqIP11QAlIIGUJUAGlbNWz4CSgnwAVkP46oAQkkLIEqIBStupZcBLQT4AKSH8dUAISSFkCVEApW/UsOAnoJ0AFpL8OKAEJpCwBKqCUrXoWnAT0E6AC0l8HlIAEUpYAFVDKVj0LTgL6CVAB6a8DSkACKUvg/wF7Oai6sxSaowAAAABJRU5ErkJggg==)<!-- --> ??? * Don't worry, most technical slide in the presentation, I'll guide you through it. * First ability. * Then S * Then - E(S). Show how it comes from Rasch model (difference in rating determines probability correct). * Then K. * Show when ability increases and when decreases. * Same for items! --- class: middle <iframe src="images/elo.mov" width="750" height="500" frameborder="0" webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe> .footnote[S. Klinkenberg, M. Straatemeier, H.L.J. van der Maas (2011). [Computer adaptive practice of Maths ability using a new item response model for on the fly ability and difficulty estimation](http://hvandermaas.socsci.uva.nl/Homepage_Han_van_der_Maas/Publications_files/papers/klinkenberg.pdf).] ??? * Nice illustration. * Starts when abilities and difficulties are not known. * Then practice starts. * Converges to a stable state with known abilities and difficulties. * <font color="#D55E00">Now you can give items at the appropriate level.</font> * This is what Oefenweb does. --- class: center, middle # <font color="#56B4E9">How?</font> ### More opportunities for adaptive practice ??? * So far only considered difficulty. --- ## Multidimensional adaptivity .pull-left[ <img src="images/instruction.png" style="height: 150px;"/> #### Instruction <img src="images/spacing_effect.png" style="height: 150px;"/> #### Repetition ] .pull-right[ <img src="images/misconception.png" style="height: 150px;"/> #### Misconceptions <img src="images/white.png" style="height: 150px;"/> #### And so on... ] ??? Repetition * Also important to consider when to practice an item. * Who knows about practice and spacing effect? * Learning curve and forgetting curve. * Together: retention * You can make anything adaptive! [Image credit](http://mindnudge.com/wp-content/uploads/2015/01/Spacing-Effect.png) --- ## Self-adaptivity <img src="images/self_adaptive.png"/> -- ### A perk and a caveat: can increase engagement, but [do students know best?](https://theconversation.com/students-dont-know-whats-best-for-their-own-learning-33835) ??? * Explain difficulty level in Oefenweb. * <font color="#D55E00">Can anyone think of benefits of or problems with this approach?</font> * Tends to be more engaging (self-control). * But assumes the student knows best! * "That is why many students assume that reading or highlighting passages in their text-book, or merely listening to a lecture, is enough to produce learning. They mistake the ease of the task with greater knowledge. Time-consuming and effortful tasks, like self-testing their knowledge, are consequently seen by students as less efficient for their learning, despite the fact that the more difficult tasks produce the most learning." * <font color="#D55E00">Article is a nice read!</font> --- class: center, middle, inverse # <font color="#CC79A7">Why?</font> ### The rationale for adaptive practice ??? Come to the most important question: why are we doing this? --- ## Personalization is key .pull-left[ Zone of proximal development * 1920s Lev Vygotsky * Developmental psychology <br><br><br><br><br>Scaffolding * 1950s Jerome Bruner * Cognitive psychology ] .pull-right[ 2 sigma problem * 1980s Benjamin Bloom * Educational psychology <br><br><br><br><br>Deliberate practice * 1990s Anders Ericsson * Expertise ] ??? * Here's a few theories that relate to adaptive practice. * First: interesting to notice that the ideas came in very different decades, from scholars in distinct fields. * With current crisis in Psychology, this gives some comfort. --- ## Personalization is key .pull-left[ Zone of proximal development <img src="images/zopd.svg" style="height: 183px;"/> Scaffolding <img src="images/scaffolding.jpeg" style="height: 183px;"/> ] .pull-right[ 2 sigma problem <img src="images/2sigma.png" style="height: 183px;"/> Deliberate practice <img src="images/deliberatepractice.jpeg" style="height: 183px;"/> ] ??? ZOPD: * Explain figure. What learner can do with guidance. Education should be at that level. Scaffolding: * Theory derived from ZOPD. * Each time try to provide a scaffold (see picture), to make sure the learner can reach the next level. 2 sigma: * Explain figure. Children with conventional education show spread. * Children with one-to-one education are on average at the + 2 sigma level of conventional. * Mastery learning: student must achieve level of mastery before proceeding to next level of learning. Deliberate practice: * Goal-directed practice. * Just beyond current ability. * Direct feedback. [Image credit](https://en.wikipedia.org/wiki/Zone_of_proximal_development) [Image credit](https://en.wikipedia.org/wiki/Instructional_scaffolding) [Image credit](https://en.wikipedia.org/wiki/Bloom's_2_Sigma_Problem) [Image credit](https://en.wikipedia.org/wiki/Practice_(learning_method)#Deliberate_practice) --- ## Individual differences are huge <img src="images/individual_differences.png" style="height: 400px;"/> .footnote[M. Straatemeier, H.L.J. van der Maas, B. Jansen (2013). [Combining computerized adaptive practice and monitoring: The possibilities of self-organizing adaptive learning tools](http://www.iaea.info/documents/paper_5b92a6b7.pdf).] ??? * Figure from the literature. Not sure whether immediately clear. * Proportion of students in Math Garden (given domain restrictions), that score 1 or 2 grades higher or lower than expected. * Impossible to cater all students with traditional (frontal, linear) education. --- class: inverse background-image: url(images/microgenetic.png) background-size: cover ## New venues for research ### Microgenetic method .footnote[[M. J. S. Brinkhuis, A. O. Savi, F. Coomans, A. D. Hofman, H. L. J. van der Maas, G. & Maris. Learning as it happens: Advances in computerized adaptive practice.](http://www.alexandersavi.nl/publications/)] ??? * Many opportunities, highlight one. * One year of data for one child (selection). * Explain graph. * Microgenetic method: examine the process of change rather than product of change. * A preprint is available where you can read about many opportunities. --- class: bottom background-image: url(images/monitoring.png) background-size: cover ### Formative / summative ## Monitoring versus testing ??? * Some images from the Oefenweb backend. * Explain images. * Monitor the student's progress. Teacher can intervene at the level of the student. * <font color="#D55E00">Who ever heard of formative and summative testing?</font> * No more summative tests! --- class: center, middle, inverse ### A/Bonus # <font color="#F0E442">A/B testing</font> ### Very short introduction to A/B tests in online learning ??? * Important part of my research. --- class: center <br><br><br><br><br><br> .pull-left[ <img src="images/ab_example_a.gif"/> ] .pull-right[ <img src="images/ab_example_b.gif"/> ] -- <br> [Facebook](https://facebook.github.io/planout/), [Twitter](https://blog.twitter.com/2015/the-what-and-why-of-product-experimentation-at-twitter-0), [Wikipedia](https://www.mediawiki.org/wiki/Wikipedia.org_Portal_A/B_testing), [Etsy](https://apptimize.com/blog/2014/01/etsy-continuous-innovation-ab-testing/), [Netflix](http://techblog.netflix.com/2016/04/its-all-about-testing-netflix.html), [Tinder](http://tech.gotinder.com/smart-photos-2/), [OkCupid](https://blog.okcupid.com/index.php/we-experiment-on-human-beings/), ... [Try this at home.](https://www.behave.org/tests-of-the-week/) ??? * An ad for a sunglass company. * Different versions: one is 'normal', the other 'gamified'. * Which one do you think most people click on? * Gamified one. * Take-away is that many companies do these tests continuously. * On many websites you're in a condition of an experiment. --- class: center, bottom background-image: url(images/interventions.png) background-size: fit -- .pull-left[ <img src="images/qm_active.png"/> ] .pull-right[ <img src="images/qm_inactive.png"/> ] ??? * In online learning we can do the same. * Show ideas where to intervene. * Explain question mark study. --- class: center <img src="images/qm_conditions.jpeg" style="height: 550px;"/> -- More on A/B testing in computerized adaptive practice environments and massive open online courses @ [alexandersavi.nl](http://www.alexandersavi.nl). ??? * Explain results. * Many advantages: double-blind, iterative, non-invasive, high ecological validity, ... --- class: center, middle # A/B testing # ⚔ # Adaptivity --- class: inverse background-image: url(images/bandit.jpeg) background-size: cover ## Multi-armed bandit .footnote[[Tinder](http://tech.gotinder.com/smart-photos-2/) does it!] ??? [Image credit](https://en.wikipedia.org/wiki/File:Las_Vegas_slot_machines.jpg) --- class: center, middle, inverse ### Ideally, a MOOC* would work like the GPS navigation device in your car. You tell it where you want to go, it figures out where you are, and it guides you along the most optimal route. Keeping with the analogy, current MOOCs are like having all GPS navigation devices instruct every car driver to turn right at 9:15 on Monday morning. excerpt from [Navigating Massive Open Online Courses](http://science.sciencemag.org/content/347/6225/958.full) .footnote[[*] Or any other (online) learning environment.] ??? * A final quote from a short comment we published. * As a take-away: ask the other guest-lecturers how they handle individual differences between students.