A/B testing is making waves in the education sector

Education and lean start-up – not words you would usually find together. But in fact, there is huge potential for education providers to harness lean start-up principles to improve the student’s learning experience and with it, their results.

 A great example of a business in the education sector that is using lean start-up principles to great effect is Duolingo – a language learning app, with over 12.5m active users worldwide. The app helps users improve their language ability and learn vocabulary by gamifying the educational experience. It’s so easy and intuitive to use even my mum is has become addicted, using the app to improve her Spanish.

One of the factors that makes Duolingo so effective is that the business is particularly good at making use of the data it gathers from its millions of users to gain a rich understanding of which learning methods work best.

One way the company does this is through A/B testing. This is the process of trialling two different variants with different user groups and then measuring a specific metric to track which performs best. Once the most successful variant has been determined, it is then rolled out to the entire user base. In a recent Fast Company article, Duolingo’s CEO, Luis von Ahn, described how the company is using A/B testing to inform the development of its courses:

“[I]f we want to know how early we should teach plurals, we simply test it. We give a group of 50,000 people plurals a little earlier in the course than everybody else, and we measure: do these people learn faster, do they learn better, do they stick around for longer? And if it’s better, we start teaching plurals earlier to everybody.”

Luis von Ahn is not the only person to advocate using trials and tests within the sector. Ben Goldacre of The Guardian’s Bad Science column makes a good argument for using randomised trials in education.  He argues that trials are the most reliable tool available to us to find out which of two interventions works best and should therefore become part of “the culture of teaching”. 

Wellington College in Berkshire also looks keen to embrace these principles, having appointed a Head of Research this June. Newly appointed Carl Hendrick says that his “vision is for [the school] to become a kind of laboratory”, testing out ideas to gain a more concrete understanding of which learning methods are most effective, rather than relying on “hunch or village wisdom.” There is scope for Hendrick to use A/B testing to achieve this. For example, one set of students could study a maths course intensively in the autumn term, whilst another group could complete the qualification over the course of a year, with the exam results being analysed to determine which method proved to be more successful.

Naturally, there are moral questions about using A/B testing in education: if you were a guinea pig in the least successful group, how would you feel? And is it fair to hamper the learning and grades of one group, even if it does benefit more students in the long-term? Ben addresses these concerns well in his paper. Before trials begin “we generally have no idea which of two interventions is best” – a new intervention may work or it may not, but a trial is what will help decide that and, as a result, improve learning. This does, of course, require parents, teachers and young people alike to be willing and enthusiastic, recognising the long-term benefits of trials.

It feels like there is a wealth of untapped potential to constantly refine the way education is delivered, what is taught, when and to whom by running such tests. I believe A/B testing could play an important role in education in future and we’re already seeing the beginnings of this.

Have you seen any other examples of A/B testing or other lean startup principles in education? We’d love to hear of other ways this methodology is being used to refine teaching.