If choosing your next show on Netflix seems too difficult, the platform has launched a new feature where it will choose for you. Based on viewing and ratings data, among other things, the company is leveraging a predictive algorithm to suggest viewing options it thinks you will enjoy the most.  

And Netflix isn’t alone. Predictive algorithms, the use of machine learning algorithms and big data to generate highly tailored predictions for people, are being used across sectors, including higher education.

“A third of colleges and universities are using predictive analytics to identify whether students are going to succeed or struggle, and to target intervention to students at risk of dropping out,” said Ben Castleman, the Newton and Rita Meyers Associate Professor in the Economics of Education and associate professor of public policy and education at the University of Virginia. “But we have little rigorous evidence that this broad use of algorithms in education is actually leading to better outcomes for students.”

Castleman, who’s Nudge4 research lab is part of the EdPolicyWorks research center at the UVA Batten School of Leadership and Public Policy and School of Education and Human Development, is focusing his current research on critically investigating these algorithms and designing open-source applications that provide students and instructors with tailored recommendations about course and career pathways that position students for success.

We sat down with Castleman to learn more about how these algorithms are being used in higher education and what more we need to know.

Q. How are predictive analytics being used in higher education? What are they aiming to achieve?

Colleges and universities are using predictive analytics in a broad range of applications, from predicting whether admitted students would matriculate, to identifying students who might default on their loans and targeting alumni who are likely to give generously to the institution. Far and away the most common use of predictive analytics, however, is to identify students at risk of failing courses or dropping out of college. In principle these “early alerts” are supposed to help instructors and advisors proactively reach out to students and provide support before students fail a course or leave the institution. But despite their broad use and the large investment colleges make in these tools, we lack rigorous evidence that predictive analytics are leading to better outcomes for students.

Q: Why don’t we currently have much evidence showing those goals are being met?

I think the primary reason is that the majority of predictive analytics applications used in higher education are operated by private companies that are proprietary and often opaque about their underlying prediction models. This is a $100M+ industry and the companies may be hesitant to engage in rigorous evaluation, at risk of undermining future sales and revenue.

Q: In an effort to improve student outcomes, you and your team built your own course-recommendation algorithm. What is it designed to do?

Our algorithm is an open-source recommendation engine that identifies for community college students courses they can take that fulfill degree and transfer requirements and that maximize their predicted probability of academic success. We made the tool both to generate recommendations for students and to support advisors to provide more tailored guidance when they’re meeting with students about course registration. At many community colleges advisors have at least 1,000 students on their caseload and just may not have the time to research course maps that best position each student for success. Our algorithm hopefully saves advisors time and supports them to provide students with more tailored guidance about educational pathways that advance their degree and transfer goals.

Q. How does your longstanding partnership with the Virginia Community College System inform a project like this?

This kind of project wouldn’t be possible without the close and ongoing partnership of the Virginia Community College System and the individual colleges. The System’s incredible data systems are what make it possible for us to bring data science methods to bear, and we rely on close collaborations with college staff to design and implement interventions that are well-suited to the local context and student population.

Q: How will you be testing your design and what do you hope to see?

We test all of our innovations through randomized controlled trials—the same rigorous evaluation strategy used in medicine, for example, to test COVID vaccines. We hope to see whether providing these personalized course recommendations leads students to accumulate more credits that count for transfer and degree and to make additional progress towards their degree or transfer goals.

Q: What makes you hopeful about the future use of these data systems in education?

Both undergrads and graduate students are increasingly taking the kinds of data science courses that equip them with the skills they need to bring to bear Netflix-style algorithms to solve public sector problems. I’m most excited to see how future researchers partner with public agencies and organizations to deploy algorithms and other machine learning strategies in ways that make it easier for historically marginalized populations to achieve educational and workforce success.