TeachLab with Justin Reich

Failure to Disrupt Book Club with Cristina and Neil Heffernan

Episode Summary

For TeachLab’s third Failure to Disrupt Book Club episode, Justin Reich reflects on a live conversation with special guests Cristina and Neil Heffernan. They discuss Failure To Disrupt’s Chapter 2: Algorithm-Guided Learning at Scale: Adaptive Tutors, and discuss the success of their tool ASSISTments. “According to SRI, they thought the reason why it was successful...They were like, "This fit in with what teachers were used to doing." They're used to actually assigning homework, and classwork. They could see before the kids walked in the door, which problems were hard. And so they could do something a little differently. In fact, what they did find is actually teachers didn't go over every item the way they used to. And of course they didn't because all the kids got feedback, but they still went over the stuff that was hard. And particularly in the places where there was common wrong answers because all those kids should be told, "Hey, you weren't all alone." Meaning, actually, you and half of the rest of you all screwed up this problem in the same way. And I think there's a social-emotional component of actually doing that as opposed to just sitting in class and realizing, ‘I got everything wrong’ and not knowing everyone else, or a large number of other kids are in the same boat.” - Neil Heffernan, Professor/Researcher/Program Director

Episode Notes



- Neil Heffernan, Professor/Researcher/Program Director


In this episode we’ll talk about:

Resources and Links

Check out ASSISTments!

Watch the full Book Club webinar here!

Check out Justin Reich’s new book, Failure To Disrupt!





Produced by Aimee Corrigan and Garrett Beazley

Recorded and mixed by Garrett Beazley


Follow TeachLab:




Episode Transcription

Justin Reich:                 From the home studios of the Teaching Systems Lab at MIT, this is TeachLab, a podcast about the art and craft of teaching. I'm Justin Reich. We're back with episode three of our Failure to Disrupt book club. This is a book club discussing my new book, Failure to Disrupt: Why Technology Alone Can't Transform Education. Today we're talking about chapter two, algorithm guided learning at scale. These are technologies for many, many learners with few experts to guide them where a learner is sitting in front of a computer or an app or a tablet and they're completing puzzles, they're solving problems, they're doing activities and the computer is evaluating their performance in these activities.

                                    And as students get things right, the computer assigns harder problems or different kinds of challenges. And as people get things wrong, they get easier problems or a chance to repeat or they get other kinds of remediation in support. But there's an algorithm that's evaluating human performance and then making a guess at what would be the right thing for that student to do next. These kinds of technologies are called adaptive tutors, cognitive tutors, intelligent tutors, a number of different names for them but collectively we call them in the book algorithm guided learning at scale.

                                    Neil and Cristina Heffernan have built a technology platform called ASSISTments, which is about homework help. And they really are two of the most humane, thoughtful people working in education technology. They met as teachers in Baltimore, teaching middle school mathematics. And I think they have never lost their understanding, their empathy for the teachers who have to do this work day after day. The things that they build, they are not for imagined classrooms in some distant transformed future. They are for the classrooms that exist right now. 

                                    And the other thing that I admire so much about them is they are just absolutely determined to bring an evidence-based approach to their work. So they are constantly submitting their tools to evaluation, to further research. When I think about people who are trying to one incremental step at a time make learning technologies that are better and having science really inform, educational science inform that kind of incremental improvement, there's no better people to talk to about that than the Heffernan's. We're going to hear from Neil and Cristina Heffernan about their ed tech stories. But in a sense, it's also a love story. Let's go ahead and listen.

Cristina Heffernan:       Technically our ed story together is really a love story. So we met teaching middle school math but soon after we started dating we all went off to Carnegie Mellon. And then after his first year, he decided to create a tutoring system online which in itself was a big deal. I remember the day we were in the car and he's like, "I can put this on the internet." And I was like, "Okay." And so this sort of idea that he could build a tutor and put it on the internet was super cool in like 96, 97. But then he decided to make the tutor behave like a human tutor. And so he videotaped me tutoring my students on these specific set of problems that he had decided to focus on. And so that's how we got started in the online tutoring system. 

Justin Reich:                 [inaudible 00:03:53] teaching. That's just unbelievably romantic, Cristina.

Cristina Heffernan:       Isn't it?

Justin Reich:                 How could you resist the guy [crosstalk 00:03:58] in your teaching and using it to program a tutor?

Cristina Heffernan:       I know. And then we call ASSISTments our third child. So we've been working on that since 2002, which is between the birth of our two real children, who of course count more. But that's sort of our ed tech story.

Justin Reich:                 So in the book, I try to tell a story about where I think intelligent tutors, adaptive guided learning experience comes from. What would be your version of that story? Kind of where does the feel come from? What has it accomplished? How would you all summarize the state of things? And where are there points of disagreement with Failure to Disrupt [inaudible 00:04:46] continents? Neil, maybe you can get us started.

Neil Heffernan:             Yeah. Because I do think that actually the people that build these intelligent tutoring systems, it starts from a good place, which is thinking, "Hey, if a child has learned something, we should let them move on." And we actually don't believe that. Meaning we believe that actually having some amount of that activity on tonight's homework might be useful but we actually want to keep kids in the same place. Because we want to actually have whatever they're doing on the computer be relatable to what the teacher is doing. If the computer is all by itself and everyone's on a different page of the book, then there's not a joint experience. 

                                    I find that interesting, like the title of your book, right? Failure to Disrupt actually maybe we're less ambitious in that actually we're like we... I never thought we were really going to disrupt the world. We were just going to make it mildly more useful to make sure kids got feedback and teachers can assign these tiny little adaptive things but that's just one night's worth of homework. They're going to sign something that's a little adaptive. And so I don't know. And I think we've kept with this somewhat simple idea and maybe that's why it's been successful. I don't know. What do you think? 

Cristina Heffernan:       I was just going to say that I think that what you said in the last chapter, chapter one about the moves and stuff and how the people who really succeed are the ones who are ready to learn and ready to do it. And you go all the way back to your example with the weaving and the rainbow weavers.

Justin Reich:                 [inaudible 00:06:39].

Cristina Heffernan:       And all those balloons and all those videos. And we've watched videos ON how to put our dishwasher and install the dishwasher and stuff like that. When you really want to learn, offering people a way to learn and a way to catch up and a way to have automation, that's super amazing. And I think that the difference between chapter one and chapter two is you went kind of from university level stuff to K-12. And so K-12, there's certainly a place for a tool that allows kids to catch up. And there's a place for something that's going to do skill practice or get someone excited. 

                                    But there's also this concern and this issue with we've got a batch of students who have to clock themselves through the years. And the goal is by the end of these 12 years, they have some sort of joint understanding and that we have a country, or we have our common core standards. We don't do it perfectly. But that is the goal is to have a society that has the same education in civics or in math or in history, or whatever. And the problem with doing things with an automated tutor is you end up with a lot of kids dropping out of that a little bit.

Justin Reich:                 Because those systems while they in theory allow everyone to go at their own pace, there's a lot of reasons to be concerned that if students get to choose their own pace, there will be some of them who choose an aggressive, rapid pace suitable to their rapid-like minds but there'll be a lot of students who also choose a pace which is too slow which is insufficiently ambitious. I mean, I think it's great the way that you both framed it, which is that the vision of a lot of folks behind algorithm guided instruction is the notion that if we can personalize learning with a computer, that every student can move at their own pace. And that really is a tension with another value that we have in schools of forming communities. 

                                    I think I once observed in a talk that if you're sort of in favor of that more personalized individualized approach, then when you talk about a classroom, you talk about it as a batch or a cohort or something that seems like a sort of suspicious sounding group. And then if you like the idea of students working together and being with each other and staying online, then you refer to it as a community or as a team or something else that has a sort of more positive valence around it. And I used to be a high school history teacher. So I come at this with much more of a community bias. 

                                    I mean for me, it was never important how far we got or how fast kids went through a world history curriculum. The most important thing was that together, we looked at primary sources because we all had different brands and different backgrounds, we would read these sources differently. We would have different ideas. We would come to them with different values. And we talk about those differences. There's no reason to have you race past the Declaration of Independence into the War of 1812, because I want us all together talking about the Declaration of Independence. And it sounds like you have the same feelings about math.

Cristina Heffernan:       Well, I think with math, yes. And in math, the skills that come along the way are maybe more adapted to technology. You sort of mentioned that in your book. But I think in history, there's just as many skills but all that memorizing of the dates and understanding the timeline and knowing where things land, when did people die and when did people live and that you two lived at the same time even though I studied you here. Those things are really important. But I think in math, there's also this really deep problem solving. And we talk in math education about having problems that everybody can enter. And so giving a really rich problem that...

                                    I've got a group of seventh graders. I know all my seventh graders can answer this problem. And then I with all my strategies, get this group to take it to a level where they're actually writing expressions using variables. This group's still counting, and that's the rich group. But this group gets to listen to this group talk and this other group gets listen to that group talk about the same experience. And they really hone their mathematical thinking. So we know that that stuff's really important. And that can't happen if you're learning math just from one of these automated systems. 

Justin Reich:                 What do you all see as sort of the push or the... One of the things about the chapters it says, implementing a new technology is in part around rhetoric. It's part around the stories that teachers tell, departments heads tell, trustees tell, school boards tell. I mean, these things cost money and someone needs to justify that. What are the stories out there that you find problematic? Or what are the stories out there that you find powerful? You've been doing this for 20 years, how do you see some of those stories changing. When you think about the narrative and the rhetoric behind algorithm guided ed tech, what strikes you about that?

Neil Heffernan:             Well, when I hear the claim that actually, "Hey, why are we doing all this tech?" And actually if all the pedagogy needs to change? So let's be clear. The study we did in Maine was really simple, right? Because I just built a really dumb platform. We put all of the answers for every different math textbook used in the state of Maine online. So all of a sudden, kids instead of waiting until tomorrow to find out what problems they got right or wrong, they knew as they went. In fact, they could then try again. And if they still needed trouble, they could actually ask to be told the answer. And so in one sense, we added immediate feedback to the math experience. We also enabled teachers to add [crosstalk 00:12:42].

Justin Reich:                 And that's really the core of ASSISTments. You get assigned whatever it is problems you were going to do before, you just instead of doing one to 31 odd on a piece of paper and a workbook, you put one to 31 odd into ASSISTments. It immediately tells you whether you got them right or wrong. It tells the teacher, everyone got three right but only four kids got seven right. And then it has...

Cristina Heffernan:       [crosstalk 00:13:06] common wrong answer.

Justin Reich:                 It highlights the most common wrong answers. And then it has a few kind of adaptive tutors, other kinds of practice problems that you can use as a supplement to these things.

Neil Heffernan:             And so that was like... According to SRI, they thought the reason why it was successful actually, because SRI is the one that did this study. They were like, "This fit in with what teachers were used to doing." They're used to actually assigning homework and classwork. And now they actually got to... They could see before the kids walked in the door, which problems were hard. And so they could do something a little differently. In fact, what they did find is actually teachers didn't go over every item the way they used to. And of course they didn't because all the kids got feedback but they still went over the stuff that was hard. And particularly in the places where there was common wrong answers. 

                                    Because all those kids should be told, "Hey, you weren't all alone." Meaning you and half of the rest of you all screwed up this problem in the same way. And I think there's a social, emotional component of actually doing that as opposed to just sitting in class and realizing I got everything wrong and not knowing everyone else or large numbers of other kids are in the same boat.

Justin Reich:                 I mean, I think there's also a really valuable contrast to how most individualized adaptive tutors are developed, where when you get something wrong, you're also the only person in the world as far as you know who've gotten it wrong. So there's a... Now that social component doesn't come directly through the ASSISTments platform to the student, right? When I get something wrong, I don't know, "Hey, don't worry, 14 of your 17 colleagues got this wrong." 

Neil Heffernan:             In fact, I've been thinking of building that feature. Meaning if you're the sixth kid that has actually come into class and next thing you all started and you all got number one wrong, should we actually tell you? Right? Like, by the way... We probably should build out something like that, particularly... 

Cristina Heffernan:       Well, we do. At the end, when the kids do their report, it does tell them how they did with respect to the class. And I know Neil's assistant for a year, her son was using ASSISTments at school. And he was one of those who're like I'm always... He was good at math, right? So a good math student. And then he's in sixth grade and he starts getting problems wrong and he gets super upset and he finds out that he can compare himself at least to the class and see that, well, maybe I got some problems wrong. I'm still higher than everybody else. 

                                    So by getting that information for him and every stern student has whatever their own hangups are, that was his. And it was super helpful. We do have a few teachers that really find that... I've only had complaints about that report from teachers who have never used the system. I have never heard from a teacher come back and say, "I've had students crying over this report." But maybe they do. I mean, I don't know. So we do tell them that but I think that's super interesting. 

                                    I will say that, because you mentioned we've been doing this for 20 years. I don't know if I see so much of a change over the last 20 years with respect to the automated tools. But I've seen a huge difference in teacher's confidence in using technology, personally. Even just 10 years ago, it was so high but then about five years ago maybe, everybody started to have a cell phone. And so you went from only a few people who weren't scared to just click around because you really can't break a technology. You just can't break it. But most teachers like a lot of people think they can, so they're really scared. And that's gone away. 

                                    So teachers are way more willing to just jump in, which of course is what we've seen in the last six months. And I think that the disruptor is going to be this pandemic with technology because the teachers are going to come out of the other side of this so much more confident with technology, so much more interested in using it flexibly and with creativity.

Neil Heffernan:             Right. Let me just argue with that. Because here goes the thing. Flexibility with technology is the wrong term because technology means so many things to different people. Some people are thinking technology means graphing calculator or Desmos. When we're thinking about that, we're just thinking about our dumb way of actually doing things that is stupidly simple to adopt, right? There's other technologies that are massively difficult to adopt, right? They massively change all of your pedagogy and everything. But I think there's some simple things based on good cognitive science about immediate feedback that we can adopt relatively cheaply and it doesn't require a lot of training. Which is why when Cristina was just alluding to the fact that during this pandemic, we went from 50,000 users to actually half a million users actually used ASSISTments in the last 30 days.

Justin Reich:                 Tell us about the school system that you just feel like is knocking it out of the park with ASSISTments. Can you tell us [crosstalk 00:18:24].

Cristina Heffernan:       What's interesting about that question is that we are mostly driven now. We've just started. A year and a half ago, we started a foundation. And we're starting to really kick up our sales or our outreach to districts. But really the question is what's a teacher that's been kicking it out. And for us it's really, we are the superstars but we have yet to really become a district thing because we're freely adopted. So it's kind of like, you don't find districts that have adopted Kahoot! But you have lots of teachers who use Kahoot! a lot. And I think it's the same with ASSISTments. 

Neil Heffernan:             Most teachers don't actually go and actually solve every problem they're assigning and actually and write explanations or ever use [inaudible 00:19:05]. But actually I'm kind of excited because I see like Wikipedia, very few people like Wikipedia entries but lots of people can use it. And so if we can benefit from a small number of teachers that are excited to share stuff in a sort of creative comments, open sort of freeway, then we can learn a lot as a field.

Justin Reich:                 I think this conversation brings up some really interesting market dynamics in education and education technology which don't always show up in other kinds of sectors, which is that there are really multiple entry points. There are 13,000 school districts in the country. Very, very few of them are so completely centralized that a district office just decides what technology gets used in the schools and what doesn't. But there are a handful that are that way. There are lots of schools and districts that make technology more readily available in some places than others. Sometimes that's through buying it, but it can also be through... No one has to buy ASSISTments because it's free. 

                                    But you would get different school-wide and district-wide adoption if they brought you in as partners for professional development or did their own internal professional development, or if there's a math department chair who's really enthusiastic about something, that lets you have more teachers be involved and it lets you collaborate and grow as a team in a different way. But they're still, which is unusual in some places in the world. An individual teacher can just be looking around at their district plan and say, "I don't think what we're doing is right. I'm going to adopt these ASSISTment things."

Cristina Heffernan:       But remember, they're still doing what their districts said to do. They're still using the same textbook. The big adoption is right now because of the synergy between these curriculums that Neil was mentioning and our system is that the schools that are really picking them up fast right now, especially during the pandemic were the schools that were using Engage New York. And so that was a school district push. And then they found ASSISTments and said, "I can do what my school has been telling me to do but now you're saving me all the time of gee, I can't even print anything and get it to my students." So I was supposed to what? Send PDFs to my kids?

                                    I had one teacher, she emails and she says, "We went home on Friday and then didn't come back. My students didn't take their books home." And so she's sitting there mandated to assign Engage New York homework and her kids don't have their books. And she somehow found us on the Facebook engaged New York group. And she's like, "Well, now I can... You've saved a pandemic's weary teacher." And so that feels good. But if they're not changing what the school district is saying most of the time when they're using ASSISTments.

Justin Reich:                 Yeah. I think that's fair. But there is a way to adopt your tool independently of a department-wide, school-wide, district-wide decision which is an interesting feature of ed tech. And there's some ed tech companies and organizations that have really tried to lead with the kind of individual teacher centered approach. I'm going to try to get one teacher at a time to subscribe to these kinds of things or something like that. And then there are other parts of that market when she said, "No, we're going to try to sell this thing as a whole curriculum package." 

                                    And you all have some insulation from this because for most of your 20 years, rather than funding this thing through sales, you are funding it by getting the federal government to give you money to research it or other organizations so that you can give it away to everyone on the side. I mean, I think you'd agree that there's quite a bit of sort of freedom and flexibility that comes from having a long runway where you don't have to react to market forces. You can just build the thing that you think will serve the teachers that you're working with best.

                                    I hope people here in the midst of this conversation, I think we've already talked about three things that Neil said, "I can do a study about that." [inaudible 00:23:11] which is I mean, I think very much a feature of your all's approach. I mean, I would say every conversation I've ever had with Neil over the years, has had a feature of that like, none of the... I mean, I think you're an exemplar of the Tinker's mindset that is described throughout the book, which is this idea of like, okay are we going to replace all of K-12 math teaching? No. Can we build something that works better than what we had before? Yes. Every single year, can every single new student I bring in build one thing that makes this system a little bit better? Yes. 

                                    If we can get 100 of those right, will we get to the point where students are learning this math a little bit more happily and a little bit faster? Yes. And that matters. I mean, that seems like it's sort of at the heart of kind of the ASSISTments philosophy.

Neil Heffernan:             And I do believe that actually one of our goals is to actually not just figure out how to actually make the student experience better but actually, how do we help us learn what are the teacher things that are useful? And so we have all these elementary teachers that are starting to use us. And the thing that her advisor, Peg Smith at the university of Pittsburgh really got known for is the five practices for orchestrating conversations. And the first step they actually tell pre-service teachers to get good at is anticipate what kids are going to do wrong. Your average elementary teacher didn't go into teaching elementary sometimes because of math. And so anticipating the different ways kids are going to solve problems is actually not necessarily easy for every elementary teacher. But we could use our system to actually share with them actually the different ways they go wrong and then maybe we can make them to be better teachers because of it. 

Justin Reich:                 So the feedback loop includes not only students improving their math skills but as teachers use the system and see more and more students inputting their answers, what are the patterns of correct and incorrect? Then we start saying that the teachers could be learning too. I mean, I think I mentioned this in a previous week that I have a colleague here, Peter Sangay who talks about learning organizations that when firms of any kind are well-designed, when people are going about doing their daily job, they're not just getting things done. They're learning more about the work that they're doing and ASSISTments has that built into them. 

                                    Now, Cristina, part of the shop that you look at is how you support teachers in developing these skills and integrating it. And I think that that is a theme that I hope a bunch of people will come away from the book with that really good education technology is not just about downloading software on people's computers or pointing them to the right website, but building the capacity of teachers and students to be able to thoughtfully use these tools. In your ideal sort of induction program for teachers, what do they need to learn? What are you doing well now? What do you hope to do more of in the future in terms of supporting teachers?

Cristina Heffernan:       Well, I think that's a good question. So a lot of our iteration of teacher training has been through these randomized controlled trials that we've been funded to do. Otherwise we didn't have any. I taught a class at WPI and through another grant on formative assessment. So formative assessment itself, it's a process that teachers and students partake in and technology like ASSISTments is super helpful in that process. So teaching a class on formative assessment and then use ASSISTments while you're doing that, those teachers became... And during that time was really we were developing the tool. So basically, it was a win-win between me and Neil and those teachers. They were telling us what they wanted. We were iterating the development, making the...

                                    So that was kind of weird. Now that we're a little bit more solid in what the product is, in our first brand up in the state of Maine, I was like, "Neil, we have to do coaching." I had been coaching in Boston, Boston public schools. I really love the idea of teacher coaching. And so what happens with coaching especially back in 2011, you've got teachers with lots of different technology backgrounds. So you have to coach them at wherever they are. And for many of those teachers, it's just computer like they gas, they hold their breath, they don't know what to do. And so getting them past that. The other teachers are like, yeah, yeah, no, I use this and this and this on the computers, what am I supposed to do with your thing? Okay. 

                                    And then with them, you can coach them because it's a one-on-one coaching to start to think about what to do with the data and what are you going to do with the mathematics and all this? So coaching was really good when people are at different stages, especially with technology. But that's pretty expensive, the coaching, one-on-one coaching. And so we did that again in our replication because we were supposed to replicate the study. So it had to be the same. But we are also doing a effectiveness study. And the effectiveness study can be a little bit less hands-on.

                                    And so with that one, we have mentoring, which is like the coaching, but it's done online, with this like a Zoom call. So it's kind of like call in help. But that's still one-on-one. With our most recent funding, which is the EIR grant that we got to scale up ASSISTments but that also has an RCT in it, we're right now creating virtual PLCs. And the PLC part, the professional learning community is super important because assessments is not a thing that just happens. You don't just sit kids down and say, "I got them in their seat. They're going to sit still. It's all set." It's something that teachers have to use and engage with. 

                                    And it can feed into growth mindset stuff. It can feed into space, practice stuff. It could feed into formative assessment stuff. It can feed into all these different already learned things. What's important about that for adoption is that all of these schools have been spending years getting their ducks in a row with one of those things. I don't know which one. And we don't come in and say, "No. Switch that away." Which teachers hate. We say, "What have you been doing? Okay. Now use ASSISTments to help you with that."

Justin Reich:                 Great. So schools... I mean, there are two things that I hear there. One is this notion that one of the things that you're sort of experimenting with is what is the level of support the system needs to provide teachers for them to be able to implement this well and help their students learn. Which is in a sense the parallel question that ASSISTments asks, what is the level of [crosstalk 00:29:59]-

Cristina Heffernan:       I have noticed. Yeah. [crosstalk 00:30:00]. Nice.

Justin Reich:                 ... needs to have in order to learn math? And we want to have these kinds of trade-offs of saying if we can make it a little bit cheaper then that's a good thing because there's an opportunity cost of everything that we're doing. So a virtual PLC where we just have to pay for a little bit of teacher time to connect with each other works as well as hiring an individual person to drive to their classroom and sit in the back and coach them, then there may still be some coaching that we want to have happen. Maybe there's a district, which is really struggling with mathematics in which one-on-one coaching is absolutely still the right thing to do. But there may be other places where this group can sort of figure things out together. You were going to say?

Cristina Heffernan:       Yeah. And it's super important that we're not coming in and saying, "Sweep all of this away." We're not saying take away your curriculum or take... You spent three years on differentiated instruction. I don't know. We're moving away from that. We're going to do this again. It's like, "You're doing differentiated instruction." Well, I bet when you're doing differentiated instruction, you want to have flexible grouping. Okay. How are you going to determine your flexible groups? Well, one thing you can do is use ASSISTments data. And you have to listen first and that's where the PLCs, you're listening to where people are and then you let them percolate up with what they want to use ASSISTments for.

Justin Reich:                 That's great. Although one enormous complexity of that is that 13,000 school districts, 130,000 schools, there are a lot of different things that people could have been working on in math. And so your coaches, your team need to have a familiarity with that range. Now, one thing that's sort of helpful is that there's probably seven big ones that you can sort of really tie into. And you mentioned them, formative assessment, differentiated instruction, response to intervention, really trying to...

Cristina Heffernan:       Growth mindset.

Justin Reich:                 Growth mindset. Having teachers actually really know, especially in elementary school sort of content knowledge about math and so forth. But there's also a long tail of a lot of different ways that schools describe those things. And so what you're trying... And it means that you can't as you said, just sort of plug in, just slot assessment automatically in and press play. You have to have this belief that there's an interface between the human community and the technology supports and that it's really only by engaging the human community, not only in some new stuff but in some stuff they were already at. Like, what are you already good at? What do you already care about struggling with? What are you already committed to? And let's build from and work on that. I mean, to me that's very well aligned with how I think schools get better. 

                                    In part, because teachers are so exhausted by being told every two or three years that there's a brand new thing coming along. So in a common approach, when a new thing comes along, it's like, "This technology is one more thing." And you all get to engage them in a conversation. No. No. This is not one more thing. We're still doing differentiated instruction. We're still doing growth mindset. We're still doing your thing. We're just doing your thing with a tool that's going to help you do your thing better.

Cristina Heffernan:       Right. Yeah. And also teachers are super smart and they're super engaged in their craft. And so you really want to... You need to engage that in schools because in schools is where the children go to be exposed to that craft and to be exposed to the synergy and the excitement. I mean, I think kids across the country these days have been realizing, gee, I want to go to school. 

Justin Reich:                 Right.

Cristina Heffernan:       And so yeah. They want to go to school because the teachers are there.

Justin Reich:                 What can you get... Let's go to where we are now and go to the school. So for the first time you've been distributing ASSISTments not just as kind of a homework helper but increasingly it's got to be slotted into these fully hybrid, fully online kinds of modalities. What are you learning? And you have the surge of users. You have 10 or 20 times as many users as you had before. What are you learning from that experience? How is ASSISTments being stretched?

Cristina Heffernan:       Yeah. So very conveniently in October, we got a big, large grant to scale up. And when we applied for that grant, we started the ASSISTments Foundation. So we're a spinoff of WPI. So what we say is we say that ASSISTments is a joint project of the WPI and the ASSISTments foundation. And so the ASSISTments Foundation is providing that structure. And so we were already working on scaling up this infrastructure and we were already getting our support kind of figured out. And we had new staff members starting to really develop this virtual PLC that we were going to offer next fall in this study. And so we had these things going in place. We did find that during the spring, when we started to do some marketing work, we're like, "Let's stop calling it a homework tool and call it..." 

                                    So it's not about homework. It's about assigning online and making it more generic. We were already going there because we found that a lot of our teachers used ASSISTments in class for do nouns or warmups or exit tickets or... So we're agnostic. And so why just call it homework. But homework is such a good storyteller because everyone knows what homework is. And they also know the deficiencies of homework. Kids are sitting at home alone, and teachers are then infused with students into their classroom. And they have no idea what happened last night until they start to hear the little bits and pieces. And so homework tells our story really well but we have become something... We now say ASSISTments is a tool for assigning online. And we don't. We're agnostic. 

Justin Reich:                 Which people are doing at home, which people are doing synchronously on Zoom. [crosstalk 00:36:08].

Cristina Heffernan:       Everything homework now. Right. So like, what do you do with that? 

Justin Reich:                 Neil, what kind of research are you going to be doing, Neil? What new learning opportunities show up because of all of this?

Neil Heffernan:             So one thing that excites me right now is actually we have a project... So even before the pandemic hit, we had three million open-ended questions that were asked actually. That means three million times some kid wrote some explanation. And by the way if you go look at these modern curriculums, almost half of the math questions are please explain. And it's super important. Teachers care a lot. The common core cares a lot about your ability to communicate and not just compute numbers. And so we think it's really important for that. But we have thousands well no, we had millions of instances where children were not getting anything from their teachers that we could tell. Now by the way, the teacher can pull up kind of the report and talk to their kids about the different ways they solve problems and explained it. And so it doesn't mean every homework question needs to be graded. In fact, we hate that idea. And homework should be formative.

                                    So anyways, we looked at the different, we did a quick look at actually this and then we're like, "Let's actually figure out how to get a bunch of teachers to actually see if we could make suggestions to other teachers of what you might want to say to these kids." And so we wound up building this product actually after. We had about 10 teachers grade, well, like look at their kids open-ended responses each night and then write back feedback so that we could then make suggestions to other teachers saying, "For this kid," who maybe said almost nothing you might want to say, "Could you say a little more." And a child that says a beautiful answer, you might want to say, "Good." And then somewhere in between, you might want to actually help them try to move them along the path. 

                                    And so anyways, this obviously involves a certain type of natural language processing that typically is thought of as an AI sort of topic. And since I teach AI, that's what I do. But I'm also smart enough to realize we will always get this wrong, right? So we need to do what Smart Reply does, which is we just tell the teacher, "You might want to say one of these three things." And we already know that our teachers like this for triage, for the instances when actually there was 18,000 comments that were actually answered with this thing called [Quick Comments 00:38:42]. And they basically use our suggestions for when actually the kid has got a really low score or a really high score.

                                    And so that gives us plenty of room to improve. But at the same time, they're like, "This is a useful tool." And in fact, I was getting this advice, which is like, one of our teacher trainers uses this. And she's like, "This gets more teachers to actually want to even make a comment back to a kid because they see comments." And then maybe even encourages them to ask the open-ended questions which they're sometimes inclined not to ask because they kind of know, if the kids are going to answer it, we should give them feedback. And so anyway, I'm hoping we'll see good stuff coming out of that.

Justin Reich:                 There was a great question up here from Kevin, which I'll answer. Which is the book is titled Failure to Disrupt. And then we spend an hour talking about how great ASSISTments is. Is ASSISTments actually disrupting our approach anyway, or is it successful because it's not disruptive. And I don't know if you have a different answer but my answer to that question is right. It's successful because it's not disruptive because it's not stepping into systems and saying, "We can completely transform the way you teach mathematics. We're going to reorganize the structure of schools. Things will be profoundly different."

                                    And I think those kinds of promises to schools are very, very rarely kept. But I think the promise of we can do things better. And two things will happen when you adopt this technology. First, you'll have some new tools for doing things better. So some things that used to be hard like taking meals, spreadsheet on the wall of where all the students are at and their progress, we're going to make that easier. But the existence of this technology is going to give us a site for having a conversation about mastery learning in gradeless classrooms, a conversation we might not be having otherwise.

                                    So to me, I mean, this is why I'm delighted to have Neil and Cristina join us and why I enjoy following the progress of ASSISTments because the evidence seems to suggest that as math software goes, this seems to be about as helpful as anything anyone else has built at any kind of scale. 

Neil Heffernan:             I guess this gets to like, how do you think actually make things better in education, right? This is like Hamilton and Jefferson, Jefferson was a revolutionary, wants a revolution every 20 years and Hamilton doesn't want that. And I think I'm an Alexander Hamilton, right? Which is, how can we just make this mildly better actually. 

Cristina Heffernan:       And let's just make sure we prevent him from going out and fighting a duel.

Justin Reich:                 [crosstalk 00:41:19] duel with someone from Mafia or something like that. Well, Neil and Cristina, it's been wonderful having you. Thanks to all of you who joined us for this session. Thanks for all the conversation that was happening in the back channel. Appreciate the questions that popped up. And I hope folks continue to make progress in these difficult months during the pandemic but really great to have this conversation be part of our weeks. So thank you. Any final thoughts, Neil or Cristina?

Neil Heffernan:             Thank you for having us. Actually it's a great book and I appreciate the service you've done by actually getting it out.

Cristina Heffernan:       Yeah. And that's great. I'm looking forward to reading chapter three and then watching chapter three's videos. Thanks.

Justin Reich:                 Good. Well, glad to have you join us. All right. Thanks everybody. Have a good afternoon.

Cristina Heffernan:       All right. Bye-bye. 

Justin Reich:                 Thanks for joining us for the Failure to Disrupt book club, we were discussing chapter two, algorithm guided learning at scale about adaptive tutors, intelligent tutors, cognitive tutors. Thanks to Neil and Cristina Heffernan of Worcester Polytechnic Institute and the ASSISTments Foundation for joining us for the conversation. I'm Justin Reich. Thanks for listening to TeachLab. I hope you enjoyed that conversation. Be sure to subscribe to TeachLab to get future episodes on how educators from all walks of life are tackling distance learning during COVID-19. 

                                    You can check out my new book Failure to Disrupt: Why Technology Alone Can't Transform Education available from booksellers everywhere. And you can find out more at failuretodisrupt.com. That's failuretodisrupt.com. This episode of TeachLab was produced by Aimee Corrigan and Garrett Beazley, recorded and sound mixed by Garrett Beazley. Stay safe until next time.