Demonstrations of Understanding
The problem
Once large language models could write competent pandas code on request, a working notebook stopped being good evidence that a student understood the analysis inside it. Traditional take-home assignments measured the wrong thing, and proctored exams measured a narrow slice of it under conditions that look nothing like real analytical work.
What I built
A Demonstration of Understanding (DoU) combines three pieces of evidence: a Jupyter notebook, a short screencast in which the student walks through and justifies their reasoning, and a written metacognitive reflection on what they learned and where they got stuck. I wrote six DoUs for SI 345, each with its own dataset, rubric and solution notebook, including one built on a synthetic dataset of automated test-taking behaviour and one comparing pandas with Polars.
Grading screencasts for roughly 130 students by hand was not feasible, so I built a pipeline that transcribes each screencast with Whisper, scores the transcript against the rubric through the Anthropic batch API, and merges the results into the Canvas gradebook. The same pipeline produces class-wide and individual feedback on the reflections.
What happened
In SI 345 the framework was a qualified success. Together, the screencasts and reflections seemed to discourage students from relying too heavily on AI tools, although they did not stop it. That fits the aim of the framework, because I care less about whether students use AI to write code than about whether they understand the code it wrote. The cost was grading, which swamped the teaching team until the transcription and scoring pipeline took over part of the work.
When I carried the framework into SI 618, I dropped the screencasts, which were too slow to grade, and kept the written reflections.