There is a particular kind of failure in educational technology—quiet, well-documented, and almost never discussed—that tells you more about learning than any success story. It is the failure that works. The intervention that achieves exactly what it was designed to achieve and still gets rejected, not because the engineers were wrong about cognition but because they were wrong about people.
Scooter the Tutor is that failure.
Scooter was an animated agent—a small, expressive puppy embedded inside a Cognitive Tutor lesson on scatterplots—designed to address one of the more interesting problems in intelligent tutoring research: students who “game the system.” Gaming, as researchers define it, means attempting to succeed in an educational environment by exploiting properties of the software rather than learning the material. Rapid hint-clicking until the answer appears. Systematic guessing through every possibility until the system accepts one. It is not laziness exactly. It is a form of rational strategy, if your goal is completion rather than comprehension. Baker and colleagues had documented that students who gamed learned roughly two-thirds as much as those who didn’t, and that heavy gaming in middle school mathematics predicted lower college attendance and diminished likelihood of entering STEM fields. The behavior was real, measurable, and costly.
So they built Scooter. When a student engaged honestly, Scooter looked happy. When the system detected harmful gaming—gaming on material the student hadn’t mastered—Scooter progressed through displeasure to anger. More importantly, when a student successfully gamed through a problem step, Scooter served up supplementary exercises targeted precisely to the concept the student had just bypassed. The design logic was elegant: gaming stops being efficient if it generates more work. The social logic was grounded in research: students treat computers as social actors, so an agent’s distress would invoke the same norms that govern human disapproval.
And in controlled experiments, it worked. Students in the experimental condition gamed at half the rate of the control group—18% versus 33%. More striking, the students who received the most supplementary exercises from Scooter—the heaviest gamers, the ones furthest behind—caught up to the rest of the class by the post-test. In past studies without Scooter, those same students fell further behind. The intervention reversed a pattern researchers had observed repeatedly: the students most likely to game were the students least likely to recover.
By every standard metric of educational technology research, Scooter should have scaled.
What the Data Couldn’t See
Here is what the data couldn’t see: a teacher standing at the front of a classroom, watching thirty students work on computers, suddenly noticing a cartoon puppy displaying anger on a dozen screens simultaneously.
Teachers didn’t like Scooter. This is documented. What is less documented—and more important—is why. Scooter was designed to be transparent, to “signal to students and their teachers” when gaming was occurring. Transparency, the designers assumed, was a feature. In practice, it transformed every instance of student misbehavior into a visible, persistent, public event. A teacher who had spent years cultivating a particular classroom climate—one of encouragement, of productive struggle, of failure as part of learning—found a software agent broadcasting student failure in real time, with an angry cartoon face. The agent didn’t distinguish between a student having a difficult moment and a chronic gamer. It didn’t respond to the teacher’s authority. It had its own rules.
This is not a small thing. Teachers are gatekeepers, not just pedagogical guides. They decide what tools stay in their classrooms, and they make those decisions based on whether tools support or complicate the social environment they’ve built. An intervention that is pedagogically sound but socially disruptive will lose that contest almost every time.
The students who needed Scooter most felt similarly. Heavy gamers, the ones who received the most supplementary exercises and showed the greatest learning gains, also registered the sharpest drop in positive attitudes toward the system. On one survey question—”the tutor is smart”—their rating dropped from 5.3 to 2.9 out of 6 after working with Scooter. They reported feeling that the system was “irritable,” that it was being unfair. From their perspective, this assessment was accurate. A system that detects your shortcuts and makes you do more work is not, by any definition they were using, on your side.
Students complained to teachers. Teachers discontinued the feature. The intervention that worked was abandoned by the populations it was designed to serve.
The Philippines Changed Everything
If the American deployment revealed a social problem, the international deployments revealed something more fundamental: the individual model on which Scooter was built was not universal. It was culturally specific.
When researchers deployed similar systems in the Philippines, they found that Scooter increased gaming. Not because Filipino students were more resistant or less motivated, but because the agent’s supplementary exercises were interesting. Students were fascinated. They gamed deliberately to see Scooter’s reactions and access the extra content. The deterrent became a reward, because the “extra work” penalty assumed that all students experience extra exercises as burdensome—an assumption that proved culturally contingent.
In both the Philippines and Costa Rica, a more fundamental problem emerged. The Cognitive Tutor was built on an individual-user model. One student, one computer, one log file reflecting one student’s cognition. In many schools in those countries, students shared computers, shared answers, clustered around screens together. The log file recorded collaborative activity as individual behavior. The gaming detector analyzed time-on-step and error rates that no longer reflected any single student’s learning. The social reality of those classrooms—collective problem-solving as a norm rather than a deviation—had broken the technical assumptions underneath the software.
The model didn’t fail because it was wrong. It failed because it was measuring the wrong unit. Learning, in those classrooms, was not happening inside one brain at one keyboard. The engineers had built a precise instrument for measuring something that wasn’t there.
What Scooter Actually Teaches
I find myself returning to a particular detail in the research: that students who received the most supplementary exercises caught up to the rest of the class. This is remarkable. These were students who had been falling behind in every previous study, in every iteration of the same lesson. Scooter’s exercises—targeted precisely to the material each student had bypassed—created a second chance that the standard tutor design didn’t allow for. The intervention worked as learning theory. It failed as social design.
This is learning engineering’s central problem, and the case of Scooter states it more clearly than any theoretical framework: there is no such thing as a purely technical educational intervention. The moment software enters a classroom, it enters a network of relationships—between students and teachers, between norms and authorities, between a student’s private self-assessment and the social face they maintain in front of peers. An angry cartoon puppy is not just a feedback mechanism. It is a social actor making a public accusation in a room full of other people.
The question the Scooter research forces is not “did the intervention work?” It worked. The question is: worked for whom, measured how, in what context, managed by whom, with what social consequences? These are not secondary questions. They are the primary ones. They determine whether a technically valid intervention becomes a usable tool or a cautionary tale in a graduate seminar.
Baker’s honest documentation of Scooter’s failure is itself a kind of institutional courage. Most educational technology research publishes what succeeded. Publishing what worked and still failed—publishing the mechanism of rejection alongside the evidence of effectiveness—gives the field something more valuable than a success story. It gives the field a map of where the code meets the classroom.
That meeting is where most educational technology goes to die, not because the research was wrong, but because the research didn’t ask the right questions about who would manage the intervention, in what culture, with what norms, in rooms where the social stakes are often higher than the academic ones.
Scooter was right about learning. It was wrong about people.
And it turns out people are the context everything else happens inside.
Tags: Scooter the Tutor, intelligent tutoring systems gaming behavior, Baker Cognitive Tutor research, educational technology classroom deployment, cross-cultural learning engineering