More, better feedback please: are learning analytics dashboards (LAD) the solution to a wicked problem?

There is a long-standing lack of learner satisfaction with quality and quantity of feedback in health professions education (HPE) and training. To address this, university and training programmes are increasingly using technological advancements and data analytic tools to provide feedback. One such educational technology is the Learning Analytic Dashboard (LAD), which holds the promise of a comprehensive view of student performance via partial or fully automated feedback delivered to learners in real time. The possibility of displaying performance data visually, on a single platform, so users can access and process feedback efficiently and constantly, and use this to improve their performance, is very attractive to users, educators and institutions. However, the mainstream literature tends to take an atheoretical and instrumentalist view of LADs, a view that uncritically celebrates the promise of LAD's capacity to provide a 'technical fix' to the 'wicked problem' of feedback in health professions education. This paper seeks to recast the discussion of LADs as something other than a benign material technology using the lenses of Miller and Rose's technologies of government and Barry's theory of Technological Societies, where such technical devices are also inherently agentic and political. An examination of the purpose, design and deployment of LADs from these theoretical perspectives can reveal how these educational devices shape and govern the HPE learner body in different ways, which in turn, may produce a myriad of unintended- and ironic- effects on the feedback process. In this Reflections article we wish to encourage health professions education scholars to examine the practices and consequences thereof of the ever-expanding use of LADs more deeply and with a sense of urgency.

Providing insights into health data science education through artificial intelligence.

BACKGROUND: Health Data Science (HDS) is a novel interdisciplinary field that integrates biological, clinical, and computational sciences with the aim of analysing clinical and biological data through the utilisation of computational methods. Training healthcare specialists who are knowledgeable in both health and data sciences is highly required, important, and challenging. Therefore, it is essential to analyse students' learning experiences through artificial intelligence techniques in order to provide both teachers and learners with insights about effective learning strategies and to improve existing HDS course designs. METHODS: We applied artificial intelligence methods to uncover learning tactics and strategies employed by students in an HDS massive open online course with over 3,000 students enrolled. We also used statistical tests to explore students' engagement with different resources (such as reading materials and lecture videos) and their level of engagement with various HDS topics. RESULTS: We found that students in HDS employed four learning tactics, such as actively connecting new information to their prior knowledge, taking assessments and practising programming to evaluate their understanding, collaborating with their classmates, and repeating information to memorise. Based on the employed tactics, we also found three types of learning strategies, including low engagement (Surface learners), moderate engagement (Strategic learners), and high engagement (Deep learners), which are in line with well-known educational theories. The results indicate that successful students allocate more time to practical topics, such as projects and discussions, make connections among concepts, and employ peer learning. CONCLUSIONS: We applied artificial intelligence techniques to provide new insights into HDS education. Based on the findings, we provide pedagogical suggestions not only for course designers but also for teachers and learners that have the potential to improve the learning experience of HDS students.

Profiling students via clustering in a flipped clinical skills course using learning analytics.

Flipped classrooms have become popular as a student-centered approach in medical education because they allow students to improve higher-order thinking skills and problem-solving applications during in-class activities. However, students are expected to study videos and other class materials before class begins. Learning analytics and unsupervised machine learning algorithms (clustering) can be used to examine the pre-class activities of these students to identify inadequate student preparation before the in-class stage and make appropriate interventions. Furthermore, the students' profiles, which provide their interaction strategies towards online materials, can be used to design appropriate interventions. This study investigates student profiles in a flipped classroom. The learning management system interactions of 375 medical students are collected and preprocessed. The k-means clustering algorithms examined in this study show a two-cluster structure: 'high interaction' and 'low-interaction.' These results can be used to help identify low-engaged students and give appropriate feedback.

Personalized Gamification for Learning: A Reactive Chatbot Architecture Proposal.

A key factor for successfully implementing gamified learning platforms is making students interact with the system from multiple digital platforms. Learning platforms that try to accomplish all their objectives by concentrating all the interactions from users with them are less effective than initially believed. Conversational bots are ideal solutions for cross-platform user interaction. In this paper, an open student-player model is presented. The model includes the use of machine learning techniques for online adaptation. Then, an architecture for the solution is described, including the open model. Finally, the chatbot design is addressed. The chatbot architecture ensures that its reactive nature fits into our defined architecture. The approach's implementation and validation aim to create a tool to encourage kids to practice multiplication tables playfully.

A Real-Time Learning Analytics Dashboard for Automatic Detection of Online Learners' Affective States.

Students' affective states describe their engagement, concentration, attitude, motivation, happiness, sadness, frustration, off-task behavior, and confusion level in learning. In online learning, students' affective states are determinative of the learning quality. However, measuring various affective states and what influences them is exceedingly challenging for the lecturer without having real interaction with the students. Existing studies primarily use self-reported data to understand students' affective states, while this paper presents a novel learning analytics system called MOEMO (Motion and Emotion) that could measure online learners' affective states of engagement and concentration using emotion data. Therefore, the novelty of this research is to visualize online learners' affective states on lecturers' screens in real-time using an automated emotion detection process. In real-time and offline, the system extracts emotion data by analyzing facial features from the lecture videos captured by the typical built-in web camera of a laptop computer. The system determines online learners' five types of engagement ("strong engagement", "high engagement", "medium engagement", "low engagement", and "disengagement") and two types of concentration levels ("focused" and "distracted"). Furthermore, the dashboard is designed to provide insight into students' emotional states, the clusters of engaged and disengaged students', assistance with intervention, create an after-class summary report, and configure the automation parameters to adapt to the study environment.

GradeAid: a framework for automatic short answers grading in educational contexts-design, implementation and evaluation.

Automatic short answer grading (ASAG), a hot field of natural language understanding, is a research area within learning analytics. ASAG solutions are conceived to offload teachers and instructors, especially those in higher education, where classes with hundreds of students are the norm and the task of grading (short)answers to open-ended questionnaires becomes tougher. Their outcomes are precious both for the very grading and for providing students with "ad hoc" feedback. ASAG proposals have also enabled different intelligent tutoring systems. Over the years, a variety of ASAG solutions have been proposed, still there are a series of gaps in the literature that we fill in this paper. The present work proposes GradeAid, a framework for ASAG. It is based on the joint analysis of lexical and semantic features of the students' answers through state-of-the-art regressors; differently from any other previous work, (i) it copes with non-English datasets, (ii) it has undergone a robust validation and benchmarking phase, and (iii) it has been tested on every dataset publicly available and on a new dataset (now available for researchers). GradeAid obtains performance comparable to the systems presented in the literature (root-mean-squared errors down to 0.25 based on the specific tuple ⟨dataset-question⟩). We argue it represents a strong baseline for further developments in the field.

Students' Learning Behaviour in Programming Education Analysis: Insights from Entropy and Community Detection.

The high dropout rates in programming courses emphasise the need for monitoring and understanding student engagement, enabling early interventions. This activity can be supported by insights into students' learning behaviours and their relationship with academic performance, derived from student learning log data in learning management systems. However, the high dimensionality of such data, along with their numerous features, pose challenges to their analysis and interpretability. In this study, we introduce entropy-based metrics as a novel manner to represent students' learning behaviours. Employing these metrics, in conjunction with a proven community detection method, we undertake an analysis of learning behaviours across higher- and lower-performing student communities. Furthermore, we examine the impact of the COVID-19 pandemic on these behaviours. The study is grounded in the analysis of empirical data from 391 Software Engineering students over three academic years. Our findings reveal that students in higher-performing communities typically tend to have lower volatility in entropy values and reach stable learning states earlier than their lower-performing counterparts. Importantly, this study provides evidence of the use of entropy as a simple yet insightful metric for educators to monitor study progress, enhance understanding of student engagement, and enable timely interventions.

Do teaching staff trust stakeholders and tools in learning analytics? A mixed methods study.

Learning analytics (LA) has gained increasing attention for its potential to improve different educational aspects (e.g., students' performance and teaching practice). The existing literature identified some factors that are associated with the adoption of LA in higher education, such as stakeholder engagement and transparency in data use. The broad literature on information systems also emphasizes the importance of trust as a critical predictor of technology adoption. However, the extent to which trust plays a role in the adoption of LA in higher education has not been examined in detail in previous research. To fill this literature gap, we conducted a mixed method (survey and interviews) study aimed to explore how much teaching staff trust LA stakeholders (e.g., higher education institutions or third-parties) and LA technology, as well as the trust factors that could hinder or enable adoption of LA. The findings show that the teaching staff had a high level of trust in the competence of higher education institutions and the usefulness of LA; however, the teaching staff had a low level of trust in third parties that are involved in LA (e.g., external technology vendors) in terms of handling privacy and ethics-related issues. They also had a low level of trust in data accuracy due to issues such as outdated data and lack of data governance. The findings have strategic implications for institutional leaders and third parties in the adoption of LA by providing recommendations to increase trust, such as, improving data accuracy, developing policies for data sharing and ownership, enhancing the consent-seeking process, and establishing data governance guidelines. Therefore, this study contributes to the literature on the adoption of LA in HEIs by integrating trust factors.

Matters of Frequency, Immediacy and Regularity: Engagement in an Online Asynchronous Course.

Many models of online student engagement posit a "more is better" relationship between students' course-related actions and their engagement. However, recent research indicates that the timing of engagement is also an important consideration. In addition to the frequency (how often) of engagement, two other constructs of timing were explored in this study: immediacy (how early) and regularity (in what ordered pattern). These indicators of engagement were applied to three learning assessment types used in an online, undergraduate, competency-based, technology skills course. The study employed advanced data collection and learning analytics techniques to collect continuous behavioral data over seven semesters (n = 438). Results revealed that several indicators of engagement predicted academic success, but significance differed by assessment type. "More" is not always better, as some highly engaged students earn lower grades. Successful students tended to engage earlier with lessons regardless of assessment type.

Untangling connections between challenges in the adoption of learning analytics in higher education.

Potential benefits of learning analytics (LA) for improving students' performance, predicting students' success, and enhancing teaching and learning practice have increasingly been recognized in higher education. However, the adoption of LA in higher education institutions (HEIs) to date remains sporadic and predominantly small in scale due to several socio-technical challenges. To better understand why HEIs struggle to scale LA adoption, it is needed to untangle adoption challenges and their related factors. This paper presents the findings of a study that sought to investigate the associations of adoption factors with challenges HEIs face in the adoption of LA and how these associations are compared among HEIs at different scopes of adoption. The study was based on a series of semi-structured interviews with senior managers in HEIs. The interview data were thematically analysed to identify the main challenges in LA adoption. The connections between challenges and other factors related to LA adoption were analysed using epistemic network analysis (ENA). From senior managers' viewpoints, ethical issues of informed consent and resistance culture had the strongest links with challenges of learning analytic adoption in HEI; this was especially true for those institutions that had not adopted LA or who were in the initial phase of adoption (i.e., preparing for or partially implementing LA). By contrast, among HEIs that had fully adopted LA, the main challenges were found to be associated with centralized leadership, gaps in the analytic capabilities, external stakeholders, and evaluations of technology. Based on the results, we discuss implications for LA strategy that can be useful for institutions at various stages of LA adoption, from early stages of interest to the full adoption phase.

Teachers' data literacy for learning analytics: a central predictor for digital data use in upper secondary schools.

Since schools increasingly use digital platforms that provide educational data in digital formats, teacher data use, and data literacy have become a focus of educational research. One main challenge is whether teachers use digital data for pedagogical purposes, such as informing their teaching. We conducted a survey study with N = 1059 teachers in upper secondary schools in Switzerland to investigate teacher digital data use and related factors such as the available technologies in schools. Descriptive analysis of the survey responses indicated that although more than half of Swiss upper-secondary teachers agreed with having data technologies at their disposal, only one-third showed a clear tendency to use these technologies, and only one-quarter felt positively confident in improving teaching in this way. An in-depth multilevel modeling showed that teachers' use of digital data could be predicted by differences between schools, teachers' positive beliefs towards digital technologies (will), self-assessed data literacy (skill), and access to data technologies (tool) as well as by general factors such as frequency of using digital devices in lessons by students. Teacher characteristics, such as age and teaching experience, were minor predictors. These results show that the provision of data technologies needs to be supplemented with efforts to strengthen teacher data literacy and use in schools.

Increasing student engagement with course content in graduate public health education: A pilot randomized trial of behavioral nudges.

Digital advances in the learning space have changed the contours of student engagement as well as how it is measured. Learning management systems and other learning technologies now provide information about student behaviors with course materials in the form of learning analytics. In the context of a large, integrated and interdisciplinary Core curriculum course in a graduate school of public health, this study undertook a pilot randomized controlled trial testing the effect of providing a "behavioral nudge" in the form of digital images containing specific information derived from learning analytics about past student behaviors and performance. The study found that student engagement varied significantly from week to week, but nudges linking coursework completion to assessment grade performance did not significantly change student engagement. While the a priori hypotheses of this pilot trial were not upheld, this study yielded significant findings that can guide future efforts to increase student engagement. Future work should include a robust qualitative assessment of student motivations, testing of nudges that tap into these motivations and a richer examination of student learning behaviors over time using stochastic analyses of data from the learning management system.

Machine Learning applied to student attentiveness detection: Using emotional and non-emotional measures.

Electronic learning (e-learning) is considered the new norm of learning. One of the significant drawbacks of e-learning in comparison to the traditional classroom is that teachers cannot monitor the students' attentiveness. Previous literature used physical facial features or emotional states in detecting attentiveness. Other studies proposed combining physical and emotional facial features; however, a mixed model that only used a webcam was not tested. The study objective is to develop a machine learning (ML) model that automatically estimates students' attentiveness during e-learning classes using only a webcam. The model would help in evaluating teaching methods for e-learning. This study collected videos from seven students. The webcam of personal computers is used to obtain a video, from which we build a feature set that characterizes a student's physical and emotional state based on their face. This characterization includes eye aspect ratio (EAR), Yawn aspect ratio (YAR), head pose, and emotional states. A total of eleven variables are used in the training and validation of the model. ML algorithms are used to estimate individual students' attention levels. The ML models tested are decision trees, random forests, support vector machines (SVM), and extreme gradient boosting (XGBoost). Human observers' estimation of attention level is used as a reference. Our best attention classifier is the XGBoost, which achieved an average accuracy of 80.52%, with an AUROC OVR of 92.12%. The results indicate that a combination of emotional and non-emotional measures can generate a classifier with an accuracy comparable to other attentiveness studies. The study would also help assess the e-learning lectures through students' attentiveness. Hence will assist in developing the e-learning lectures by generating an attentiveness report for the tested lecture.

The use of video clickstream data to predict university students' test performance: A comprehensive educational data mining approach.

Video clickstream behaviors such as pause, forward, and backward offer great potential for educational data mining and learning analytics since students exhibit a significant amount of these behaviors in online courses. The purpose of this study is to investigate the predictive relationship between video clickstream behaviors and students' test performance with two consecutive experiments. The first experiment was performed as an exploratory study with 22 university students using a single test performance measure and basic statistical techniques. The second experiment was performed as a conclusive study with 16 students using repeated measures and comprehensive data mining techniques. The findings show that a positive correlation exists between the total number of clicks and students' test performance. Those students who performed a high number of clicks, slow backward speed or doing backwards or pauses achieved better test performance than those who performed a lower number of clicks, or who used fast-backward or fast-forward. In addition, students' test performance could be predicted using video clickstream data with a good level of accuracy (Root Mean Squared Error Percentage (%RMSE) ranged between 15 and 20). Furthermore, the mean of backward speed, number of pauses, and number/percentage of backwards were found to be the most important indicators in predicting students' test performance. These findings may help educators or researchers identify students who are at risk of failure. Finally, the study provides design suggestions based on the findings for the preparation of video-based lectures.

A methodology to design, develop, and evaluate machine learning models for predicting dropout in school systems: the case of Chile.

School dropout is a structural problem which permanently penalizes students and society in areas such as low qualification jobs, higher poverty levels and lower life expectancy, lower pensions, and higher economic burden for governments. Given these high consequences and the surge of the problem due to COVID-19 pandemic, in this paper we propose a methodology to design, develop, and evaluate a machine learning model for predicting dropout in school systems. In this methodology, we introduce necessary steps to develop a robust model to estimate the individual risk of each student to drop out of school. As advancement from previous research, this proposal focuses on analyzing individual trajectories of students, incorporating the student situation at school, family, among other levels, changes, and accumulation of events to predict dropout. Following the methodology, we create a model for the Chilean case based on data available mostly through administrative data from the educational system, and according to known factors associated with school dropout. Our results are better than those from previous research with a relevant sample size, with a predictive capability 20% higher for the actual dropout cases. Also, in contrast to previous work, the including non-individual dimensions results in a substantive contribution to the prediction of leaving school. We also illustrate applications of the model for Chilean case to support public policy decision making such as profiling schools for qualitative studies of pedagogic practices, profiling students' dropout trajectories and simulating scenarios.

cleAR: an interoperable architecture for multi-user AR-based school curricula.

Although there are some experiences that demonstrate the validity of the use of augmented reality in schools to help students understand and retain complex concepts, augmented reality has not been widely adopted in the education sector yet. This is in part because it is hard to use augmented reality applications in collaborative learning scenarios and to integrate them in the existing school curricula. In this work, we present an interoperable architecture that simplifies the creation of augmented reality applications, enables multi-user student collaboration and provides advanced mechanisms for data analysis and visualization. A review of the literature together with a survey answered by 47 primary and secondary school teachers allowed us to identify the design objectives of cleAR, an architecture for augmented reality-based collaborative educational applications. cleAR has been validated through the development of three proofs of concept. cleAR provides a more mature technological ecosystem that will foster the emergence of augmented reality applications for education and their inclusion in existing school programs.

Recent advances in Predictive Learning Analytics: A decade systematic review (2012-2022).

The last few years have witnessed an upsurge in the number of studies using Machine and Deep learning models to predict vital academic outcomes based on different kinds and sources of student-related data, with the goal of improving the learning process from all perspectives. This has led to the emergence of predictive modelling as a core practice in Learning Analytics and Educational Data Mining. The aim of this study is to review the most recent research body related to Predictive Analytics in Higher Education. Articles published during the last decade between 2012 and 2022 were systematically reviewed following PRISMA guidelines. We identified the outcomes frequently predicted in the literature as well as the learning features employed in the prediction and investigated their relationship. We also deeply analyzed the process of predictive modelling, including data collection sources and types, data preprocessing methods, Machine Learning models and their categorization, and key performance metrics. Lastly, we discussed the relevant gaps in the current literature and the future research directions in this area. This study is expected to serve as a comprehensive and up-to-date reference for interested researchers intended to quickly grasp the current progress in the Predictive Learning Analytics field. The review results can also inform educational stakeholders and decision-makers about future prospects and potential opportunities.

Modelling online community constructs through interaction data: A learning analytics based Approach.

This study aims to use LMS log data to suggest a way to understand CoI constructs. Students' interactions in Moodle components were weighted for indicators of cognitive, teaching and social presences. Traces reflecting students' online interactions were obtained from the Moodle LMS and analyzed through learning analytics techniques. The data is examined with the Euclidean Distance Model, and Correspondence Analysis methods to evaluate the levels of interactions and presences. The results indicated that, cognitive presence is at the center of the CoI constructs, and student-content interaction, is found is more prominent than other interactions in terms of its relation to cognitive presence. Social presence scores were mostly related with student-student and student-teacher interaction scores. In addition, teaching presence scores were found in parallel with student-system interaction scores.

Student assessment in cybersecurity training automated by pattern mining and clustering.

Hands-on cybersecurity training allows students and professionals to practice various tools and improve their technical skills. The training occurs in an interactive learning environment that enables completing sophisticated tasks in full-fledged operating systems, networks, and applications. During the training, the learning environment allows collecting data about trainees' interactions with the environment, such as their usage of command-line tools. These data contain patterns indicative of trainees' learning processes, and revealing them allows to assess the trainees and provide feedback to help them learn. However, automated analysis of these data is challenging. The training tasks feature complex problem-solving, and many different solution approaches are possible. Moreover, the trainees generate vast amounts of interaction data. This paper explores a dataset from 18 cybersecurity training sessions using data mining and machine learning techniques. We employed pattern mining and clustering to analyze 8834 commands collected from 113 trainees, revealing their typical behavior, mistakes, solution strategies, and difficult training stages. Pattern mining proved suitable in capturing timing information and tool usage frequency. Clustering underlined that many trainees often face the same issues, which can be addressed by targeted scaffolding. Our results show that data mining methods are suitable for analyzing cybersecurity training data. Educational researchers and practitioners can apply these methods in their contexts to assess trainees, support them, and improve the training design. Artifacts associated with this research are publicly available.

Revealing latent traits in the social behavior of distance learning students.

This paper proposes a multilayered methodology for analyzing distance learning students' data to gain insight into the learning progress of the student subjects both in an individual basis and as members of a learning community during the course taking process. The communication aspect is of high importance in educational research. Additionally, it is difficult to assess as it involves multiple relationships and different levels of interaction. Social network analysis (SNA) allows the visualization of this complexity and provides quantified measures for evaluation. Thus, initially, SNA techniques were applied to create one-mode, undirected networks and capture important metrics originating from students' interactions in the fora of the courses offered in the context of distance learning programs. Principal component analysis and clustering were used next to reveal latent students' traits and common patterns in their social interactions with other students and their learning behavior. We selected two different courses to test this methodology and to highlight convergent and divergent features between them. Three major factors that explain over 70% of the variance were identified and four groups of students were found, characterized by common elements in students' learning profile. The results highlight the importance of academic performance, social behavior and online participation as the main criteria for clustering that could be helpful for tutors in distance learning to closely monitor the learning process and promptly interevent when needed.