How can we harness the power of edtech?

Descriptive transcript

So, thanks, Monica, very much for joining me today. I'm going to begin this conversation by acknowledging that I'm joining you from Aboriginal land, from the land of the Gadigal people of the Eora Nation, and pay my respects to Elders past and present, and my respects to all First Nations people.

So we're here to talk about your study, which explored scaling intensive tuition through the addition of a computer-based learning application. My first question to you is, what prompted you to undertake this study?

Sure. Well, first, thank you so much for having me. I wish I could be there in person, but alas, the world is large, and I am thrilled that technology is facilitating my ability to be there in the room with you today. And it really is at the heart of this, because technology can facilitate so many things in our society, including the advancement of teaching and learning so that all students benefit. And yet, there are some guardrails that I think we want to place along that, and we wanted to explore that as part of the study.

So for the past decade or so, at the University of Chicago Education Lab, we have been looking at the effects of what we call high-dosage tutoring. So this is a tutorial where one tutor works with a few students in the context of the school day as part and parcel of how instruction is delivered during the school day. And we wanted to understand the impacts of that intervention on student learning.

So about 10 years ago now, we did a series of randomised control trials, which is the most rigorous type of study that you can do to understand the causal impact of one sort of intervention or program on outcomes. And we saw that when students meet with a tutor in ratios of two students to one tutor, it is enormously beneficial to them.

In this case, it was a study of ninth grade algebra, which seems to be a gatekeeping course at least here in the United States, and I imagine in Australia as well, to later life outcomes. We saw that it was so beneficial and very cost-effective.

So we went running for the hills and told all of our public sector partners in Chicago, across the country, at the departments of education at various states and districts, and we said, we found the educational equivalent of nuclear fission. This is how we're actually going to address these longstanding disparities in achievement that we see along race and class lines that have been very persistent in the US educational context.

And they said, that's wonderful. It is cost prohibitive because it costs $3,000 per student, which is about one quarter or 25% of the per pupil expenditure of Chicago public schools annually. That's too high to spend on one student and one grade in one subject.

And so they sort of tasked us with trying to find an intervention that was just as effective, but cheaper and less costly. And we said to our partners at the time, Saga Education, who was the program delivery partner—they're sort of a best-in-class nonprofit tutoring provider—we said, how can we do this?

And they went off and did some design work, and they came up with an interventional design that was four students sitting at a table with one tutor, similar sort of structures. It was an elective course called Math Lab. The kids went every day. It was part of their schedule. So none of that changed. But for the same amount of money, for the same tutor, you could serve twice as many students.

And in order to facilitate that larger group, the students spent every other day on an educational technology platform. In this case, it was Alex, but you could sub in any kind of adaptive educational technology platform that helped you differentiate instruction. And we said, does this work?

Now, oftentimes in social science, when you reduce the cost of something, you reduce the efficacy as well. And so we wanted to understand, could you cut the cost? In this case, it cut the cost of this very effective tutoring intervention that doubled or even tripled what a student learned in mathematics in one year compared to their peers down the hall in the same school, in the same grade.

We said, can you cut the cost of that by 30% to 50%? Let's say 30% just to be really conservative here and maintain the effectiveness. And what we found was that it was just as effective.

So I know we'll get into the study findings in a minute, but we were really motivated by whether you could sort of use this design recipe of increasing the ratio of students to tutors just slightly, and then leaning on the educational technology to facilitate that scale and still mimic the differentiated instruction. We wanted to know, could that interventional design work?

Thank you. I love that phrase, leaning on the educational technology, helping it to leverage for equity, which you have described as being at the very heart of this study.

Now, is it fair to say that boiled right down, what your study found was that by introducing the technology platform, you could effectively double the number of students that the program reached, keep the same number of human tutors—which is a really big consideration in a resource-constrained environment—reduce, as you've just said, the per-student cost significantly, and yet not compromise the significant learning outcomes that you'd seen in the previous human-only studies. Is that correct?

That's right.

We saw increases of 0.23 standard deviation—so I apologise for using a term like that—but we saw basically no difference in the amount of learning for a ninth grade student in algebra, whether they had the Saga technology-enabled version or the prior version where it was two students working with one tutor.

So again, we were really thrilled by this because it provided a roadmap or recipe, if you will, of how we could maybe think about taking this very effective intervention delivered by a boutique vendor at modest scale—maybe a few thousand students—and really think about how we could reach millions of students across the country and even across the world.

And you said you were thrilled, which I can absolutely understand. Were you surprised?

I think something that people don't appreciate as much is that there was a lot of design work and iteration that went into the intervention itself done by Saga Education prior to us studying its efficacy. And so personally, Saga had sort of given their stamp of approval and said, okay, this is the version to test. We had done other iterations that ended up on the cutting room floor.

So we had tried a half-dose version where kids met every other day, but in the context of a school building, that's very confusing for kids because then they have a new schedule every other day and principals didn't like that very much, neither did the teachers. But in this version, it was on the students' schedules and it was expected by teachers, tutors and students alike what was going to happen. And so they liked this version better than the sort of alternate day.

So we tried lots of different iterations. And so I don't know that I was surprised that it was effective. I am surprised that it was just as effective, that there was no deterioration in treatment effects, just because that's something we never see in social science.

And it's really heartening because it means that technology can facilitate scale. And now we have to get even more sophisticated in our research questions and our practice and understand under what conditions.

So that segues very nicely. The next thing I'd like you to talk about is, in your view, what were the primary factors that supported the success of this implementation?

Sure. So oftentimes when we think about introducing technology, there are no standards attached to that phrase. So we could mean everything from enabling kids to have a login from the administrative side—all the way to what this Saga technology intervention was, which is a course that had to be scheduled into a student's school day.

That means time was found, dedicated time for the student to go to a classroom where there was a teacher of record and six tutors. So 24 kids were scheduled into this class period. They met with a tutor. The tutors had to be sourced, staffed, hired, managed, trained. And the kids sat at the table with the tutors. So there were still four kids sitting at a table with one tutor.

And so even when they were on the EdTech platform, they would start off by the tutor checking in with the kids, both interpersonally, and they had this thing called the Do Now, where they would all work on a math problem as kids filtered in. They would check in with the students about halfway through the class period when the two students that they were working with were working on a math problem. They would lean over and say, hey, how's it going over there? Are you making sure they're not texting their girlfriend or doodling or just staring at the computer screen and not actually engaging?

By having that sort of motivational piece that the human tutor could do, but allowing the platform to do the differentiated instruction and get kids to practise math problems—which is miraculously what gets kids to learn more math, is that practice—I think that interventional design and those kind of component pieces were really important.

Because the kid wasn't just on a Zoom with a black screen. They had a human tutor who expected them, who was motivating them, and they had access to this technological platform. I think that's just a very different interventional construction than, "I'm going to give you a slip of paper with a login, and you can log in on your own time."

I worry a little bit about that use of technology in that construction, because we know that the kids who are more likely to log on are the kids who are already doing better in school, or at least we have mounting evidence that that's true.

That effect is described in your paper. We sometimes call it the Matthew effect of increasing advantage, building on advantage. Would you describe a little bit more about what you did find about those different patterns of EdTech usage?

Sure. So I should be really clear here that there is a lot more work to be done on this topic. So I'll try to stick to what this study showed and then try to differentiate that from my opinion.

So in this study, we had the technological usage of the kids who logged onto the Alex platform. By definition, these were all the kids who in our random assignment trial were assigned to receive tutoring, and the tutoring was inextricably linked to logging onto the Alex platform.

And even in that constrained design setting, when we looked at the relationship between students' baseline characteristics and Alex usage, we found that kids who had higher test scores and higher GPAs were more likely to log on to Alex and use it more, learn more topics than students who didn't have those characteristics.

Now, is that causal evidence? Absolutely not, because this is just descriptive in that one use case from our study. But I think that is the pattern that is replicated in other studies as well. And we're starting to learn that, left in the wild—and now I'll sort of veer into my hypothesis—we see that kids who log on and do their work either already have the study skills, the motivation, the attachment to schooling, and also the inclination to do that.

And so they're the kids who kind of had higher baseline test scores and GPAs to begin with. And I think we have mounting evidence of that from the COVID-19 pandemic, in which we have a lot of data coming from remote instruction, and you can see who logged on and who dropped off.

And so I would sort of count that evidence from the remote instruction during the pandemic as substantiating this hypothesis.

Thanks very much for that, Monica. And I think that moves us kind of naturally to the next question that I'd like to ask you, which is, were there any aspects of this particular study that you feel warrant now further exploration? You already mentioned the interest in evolving studies. What are they? What would you still like to know in this space?

Sure, I think there's so much to know. So I think I'm heartened by the fact that we know that this pedagogical practice of tutoring, individualising instruction, really is the best way for anyone to learn anything.

So if I were to take piano, I could sit in a class, I could watch online YouTube videos, I might learn the piano. I'm much more likely to learn the piano in a way that I get better faster if I had a coach or a teacher come to my home and teach me one-on-one. Take what I already know from my, I don't know, 30 years ago piano tutorial experience, and then build on that and teach me in a consistent way.

I think what we've never been able to figure out in public education is how do you take that pedagogical practice and make it so that you can fit it into the parameters of what the public education system can afford and how the system is set up, where we typically have one teacher with 30 students.

I think we have some evidence, and I would say that this study provides a proof point that technology can facilitate scale. I think what we don't really have are practical answers to policymakers' questions about how do I do this in my own district? What is the first thing I stand up? How do I find these tutors? How do I make it so that they're allowed to be in a classroom with students, even legally, without a teacher of record there, or maybe with a teacher of record allowing that teacher to be the architect of a classroom where the kids are getting some individualised instruction from tutors, some are getting individualised instruction from platforms, and then the teacher can use their pedagogical expertise to really work on the more difficult problems of practice.

That's the classroom of the future. I think we're still building towards that, and we don't know how to take all of these effective practices in their pockets and put them together so that they create a system that makes sense for students regardless of background.

And so from my vantage point, I want to understand what are those ratios of students to tutors at which you can't rely on technology anymore to facilitate scale? Can you have six kids? Can you have eight kids? Can you have ten kids? Then you start to get into some of those classroom management problems that you had earlier.

And I think a lot of the focus that we're really concentrating on for the next few years is what is the right mix of human capital and technology to facilitate that individualised instruction. So I don't know the answer yet, but maybe I can come back in a few years.

Thank you. I like that phrase, the best mix of human capital and technology.

One thing that did strike me from your paper was the question of is there a sweet spot for the amount of technology used by either students in general or individual students? Could you say something about that?

Sure. I think during the pandemic, we saw there was too much technology. So the sort of log on to Zoom for eight hours a day and have a teacher talk at you is not the kind of technology that seems to facilitate genuine learning opportunities. And what we hear a lot is kids are burnt out on technology.

Now, if you think about a different situation, maybe just prior to the pandemic, where we actually do see that technology helps kids learn if you use the tech platform in the right way, it actually can help kids learn. And so if you're not using it enough, we're actually leaving learning on the table. We're using technology in a suboptimal way. And that's malpractice because we have all these students who are just begging to learn and we have very inequitable outcomes, not just within our individual countries, but globally, that we need to think about.

So we can't afford to leave learning on the table that technology could facilitate. That's opportunities and advancement that we are leaving on the table. And so I think what we're trying to do is find that sweet spot where it's the optimal mix, again, of both the human capital that you have at your disposal and also the technology and the technological modalities and platforms that you might be able to deploy.

And something that I think is likely obvious to everybody in the audience, but should be stated, is that I think that right mix is highly context dependent. So if you're in a rural area, you might have to lean on the modality of technology more, but then you really need to think about how you're going to use your human capital to overcome the Zoom fatigue that we saw during remote instruction.

That might not be as true if you're in a more densely populated area, but you might have other challenges as well in terms of equitable distribution of that human capital and those opportunities.

And so I think we just have to get a lot more granular, not just about what works on average, but what are the overarching design principles as we've been talking about? And then how do you adapt them in your local context and still be confident that they're yielding the treatment effects that we all want on student learning?

Well, thank you so much, Monica. My last question was going to actually be about why is your study and why is technology more broadly important, particularly for students experiencing disadvantage and the schools that serve them? But I think everything you've had to say and share with us has been from that foundation of equity and needing to do better. And I think we all know that we can't leave any of the things on the table when we're trying to do that. The problems are too pressing and too large to overlook any of the potential levers.

So that just leaves me to say thank you so much once again for your insights and for your time.

Thanks.

Of course. Thank you for having me.

Thank you.