This study investigated the reading behaviors of learners during critical reading tasks executed on an online e-book reader. The learning logs of interactions in the e-book system and the processed data from the learning analytics dashboard were used to define and describe four different reader profiles: effortful, strategic, wanderer, and check-outs. While effortful and strategic readers attempted to complete the critical reading-based highlighting task, the wanderer and checkouts did not attempt it at all. Visualization of the navigation patterns of learners and their engagement score in the task context is presented in the “Results and interpretation” section. It is used to illustrate the characteristics of different profiles. This is also the first attempt to discuss the quantitative account of reading the specific play, Hayavadana, and to our knowledge initiates a systematic inclusion of a learning analytics system in a humanities course. Based on this study findings, we discuss from the more specific aspect related to the particular critical reading task to a more generic aspect of approach of applying learning analytics research in a specific disciplinary domain.
Critical reading of a play: reflections from data
The play Hayavadana is firmly rooted in the Indian tradition and Hindu mythology and yet uses certain subversive tactics to question some of these deep-seated traditions and beliefs. At the performative level, it draws from Yakshagana, a traditional theatrical form practiced in the coastal belt of Karnataka, a state in southern India. It also uses a narrator, a chorus, and a self-reflexive plot within plot strategy thereby bringing in much talked about alienation effect. Before any critical analysis of the play can commence, students are nudged to engage in a critical reading task—which typically is done before the classroom discussions. In this study, the activity was done fully using an online environment. The log data provides reading and annotation behavior of a group of learners interacting with the content of the play for the first time. Even though the readers were uninitiated about the conventions of Yakshagana, from the highlights drawn during their reading, it is evident that they did identify simple directions for performance. Tagging such simple directorial notes as performative elements can be attributed to their unfamiliarity with the more nuanced elements taken from the Yakshagana style. Further, the navigation graph data is in congruence with the above reasoning. It shows that effortful and strategic readers are particularly engaged in these parts of the text. The data also suggests these readers were more clear and accurate in the tagging of cultural references as compared to performative elements. Such data points give a clear indication to the instructor as to where the focus of discussions should be when she goes to the class—whether face to face or online.
This was a pilot attempt to understand and share some of the observed reading patterns and discuss possible ways the learner interacted with the task at hand. With the given data collection affordances in BookRoll (see Table 1), we can interpret the interactions as the only learner behaviors. Hence, the design of the learning task becomes very crucial to ground the interpretation of the action. In our case of critical reading, the explicit learner action is to navigate the portion of the text and highlight the identified critical elements with markers. Thus, based on the interaction log analysis, the count and the navigation patterns in a given reading episode has construct validity to interpret the critical reading activity (Winne, 2020). However, some of the collected data remains difficult to interpret, for instance, the wanderers, who spent time within the content without attempting the task (indicated by annotation action), it is not possible to distinguish whether they are coping up with comprehending the text before engaging in the critical analysis or just being off task in the system. Such disengaged or distracted behavior is still difficult to detect in the system.
While consolidating the action logs of 22 learners (44% of the registered participants) generated our dataset, it is still from a smaller sample space to fully comment on critical reading behaviors. This might be primarily due to the fact that the activity was ungraded. Further, in this analysis, we did not consider any explicit learner output apart from the highlights as the artefacts. Given our analysis, the model considers only clickstream interactions; we claim that the profiles generated are only of the readers based on their behavioral trace-based interactions in the system and the time in between the action, and cannot distinguish learners for their critical reading skills yet.
As a future work, there remains further analysis of the data from the pilot study itself. We aim to investigate the quality of the highlighted text by the learner with respect to the instructor’s annotation and further compute inferential statistics for the difference of the profiles identified. These would lead to developing learner models specific to critical reading activities.
Reflection from practitioners’ point of view
Critical reading activity and critical analysis have been two crucial components of this course since its inception. During the years in which the course was being offered in offline mode with the students being in the physical classroom, the activity took different shapes. While the activities around the critical reading task—ranging from reading, synthesizing, and responding to questions through written essays or discussion groups—enabled the instructor to observe the learning patterns, using the BookRoll during the said semester in which classes were suddenly shifted to online mode enabled us to ask various research questions about reading profiles of the students during a critical reading activity.
The main consideration while designing the activity was to make students critically read with an annotation task at hand. The task of making students read the text of the play Hayavadana to identify cultural references and performative elements was well-thought-out. It called forth the students’ critical faculty and their knowledge of the cultural context within which the play grounded itself. The instructor, from her prior experience, had noted that a challenging task at hand makes students alert to the richness of any text, and hence, the annotation task was zeroed in on.
Post activity classroom discussion (in the online mode of classroom) enabled a greater level of interaction as compared to earlier classroom scenarios. One of the main differences the instructor observed was that the percentage of students accessing the text in BookRoll before the classroom discussion was marginally higher than that of earlier scenarios where the instructor would have given them the reading as a homework activity. This was evident as a greater level of participation in the classroom was observed after the BookRoll activity. This was observed in spite of the fact that there was still a large chunk of the class that had not accessed the text through the BookRoll.
Developing profiles of reflective reading and implications for technology design
In earlier works, Binder and Lee (2012) proposed four types of adult readers: unskilled readers, resilient readers, good decoders/poor comprehenders, and skilled reader. Later, Putro and Lee (2018) conducted a latent profile analysis of readers across different modes (printed, online, and social media) and for different purposes (academic and recreational) of reading. They classified low-interest readers, traditional readers, moderate readers, and high-interest readers. Still, specifically for critical reading, previous literature lacks any reader’s profile. We attempted to approach and fill that gap using learning logs and computing broader navigation patterns of different readers.
Reading strategies and comprehension strategies are considered as cognitive action and remedial action respectively and both assist the learners in achieving reading success (Yang, 2006). A technology framework like LEAF is capable of supporting these aspects by collecting learning logs from the e-reader and using learning dashboards to visualize the traces. Recent work (Gibson et al., 2017) focused on data-driven technology-supported feedback for reflective writing. However, for reflective reading activities, such data-informed digital services are still lacking. This study conceptualized using the interaction count and time as indicators of different profiles of readers. Such indicators are often included in LA dashboards (Tan et al., 2016) and can assist the teachers to directly check the visualized data and decide the status of reflective reading behavior of the learner.
At another level, technical support can also be developed to automatically evaluate the highlighting actions of learners and to give them feedback. During the data analysis process, the instructor highlighted the portions of the text for reference. Presenting the instructor’s highlighted part to the learners in the learning dashboard can also assist the learners.
Contributions and future work
This pilot study is part of the overarching research project that aims at developing a data-driven narrative of learner behaviors during reflective tasks in humanities and design courses and then support it with technology (Majumdar et al., 2020). Here, we focus on the context of humanities investigating a well-designed activity plan with technology affordances to collectit with technology traces of learning behaviors and then applying learning analytics techniques to highlight indicators of that specific activity. The collaborative work brings in expertise from the domain of humanities and learning analytics and learning tool design. The future work aims to integrate the reflective activity context data and the learner’s interaction data to build models of the learning process and thereby design possible learning feedback and teaching support.