TAFFIES: Tailored Automated Feedback Framework for Developing Integrated and Extensible Feedback Systems.

ABSTRACT

Delivering high-quality, timely and formative feedback for students' code-based coursework submissions is a problem faced by Computer Science (CS) educators. Automated Feedback Systems (AFSs) can provide immediate feedback on students' work, without requiring students to be physically present in the classroom-an increasingly important consideration for education in the context of COVID-19 lockdowns. There are concerns, however, surrounding the quality of the feedback provided by existing AFSs, with many systems simply presenting a score, a binary classification (pass/fail), or a basic error identification ("The program could not run"). Such feedback, with little guidance for how to rectify the problem, raises doubts as to whether or not these systems can stimulate deep engagement with the related knowledge or learning activities. In this paper, we propose TAFFIES, a framework to scaffold the development of AFSs that promote high-quality, tailored feedback for student's solutions. We tested our framework by applying it to develop an AFS to mark and provide feedback to 160 CS students in an introductory databases class. In contrast to most introductory-level coursework feedback and marking, which typically generate significant student reaction and change requests, our AFS deployment resulted in zero grade challenges. There were also no identified marking errors, or suggested inconsistencies or unfairness. Student feedback on the AFS was universally positive, with comments indicating an AFS-related increase in student motivation. The experience of designing, deploying, and evolving the AFS using TAFFIES is examined through reflective practice, student evaluation, and focus group (involving peer teachers) analysis.