Delivery and evaluation of an e-Learning framework through computer-aided analysis of learners’ reflection text in a teacher development course

Abstract

This study aimed at proposing an e-Learning framework in school education. The proposed framework consisted of technology and pedagogy dimensions. The method of computer-aided analysis of learners’ reflection text was used to evaluate the pedagogical delivery of the proposed framework. A study involving 33 in-service teachers in a teacher development course was conducted to investigate the effectiveness of the pedagogical delivery. The computer-aided analysis of learners’ reflection text on “what is e-Learning” before and after the teaching of the course showed that learners deepened their understanding about the technology use and the importance of pedagogy in e-Learning, to consider more the component on theories/models/principles/strategies in pedagogical design and practices, emphasize more the reflection and discussion in learning and teaching activities, and also realize more the possibility of pedagogical decision-making using data collected from online learning. The questionnaire survey and focus group interview with learners also indicated that the pedagogical delivery of the e-Learning framework with the support from the computer-aided analysis of learners’ reflection text coupled with hierarchical visualization of analysis results was positively perceived. The research findings contribute pioneering insights into the use of a computer-aided approach for an accurate and efficient evaluation in teacher development courses on e-Learning.

Implications for practice or policy

• An e-Learning framework in school education was proposed. The approach of computer-aided analysis of learners’ reflection text coupled with hierarchical visualization of analysis results helped deliver the framework in a teacher development course on e-Learning.

• Teachers’ pre-post-course reflections showed a growth of e-Learning understanding with the support of the e-Learning framework and learners realized more the importance of pedagogy in e-Learning.

Introduction

The goal of e-Learning in school education in the coming 10 years is to foster the development of 21st century skills among learners (Kong et al., 2014; Koh et al., 2017). Teachers in school education need guidance of how to promote e-Learning for developing learners’ 21st century skills. This demands teacher development for empowering in-service teachers to truly understand the core concepts of e-Learning to plan and implement concept-driven e-Learning strategies for learners in the 21st century. An e-Learning framework which addresses both the aspects of technology and pedagogy in defining what is e-Learning is desired. For better preparing teachers to realize such e-Learning framework, the corresponding teacher development course should be designed to address the gaps between teachers’ actual understanding of e-Learning and the desirable framework of e-Learning. The approach of computer-aided analysis of learners’ reflection text which bases on the desirable e-Learning framework can help analyze teachers’ understanding and facilitate its pedagogical delivery in teacher development course. This study made a pioneering effort to propose an e-Learning framework for the delivery and evaluation through computer-aided analysis of learners’ reflection text in the teacher development course on e-Learning in school education.

Background of the study

In this study, e-Learning is defined as a process of implementing pedagogy with the support of digital technology for achieving educational goals (Kong & Song, 2014, 2015). This definition is formulated with the consideration of more and more integration of e-resources, digital communication processes, and use of digital data for learning support in e-Learning context throughout recent decades. From late 1990s to the mid of 2000s, the Internet was popularized around the world and its use was penetrated in every aspect of daily life, including education (Krasnova et al., 2017; McConnell, 2018). The tremendous digital resources available on the Internet widened the choices of resource deployment for teaching, besides the use of traditional textbooks (Krasnova et al., 2017; Leow et al., 2016). With the technological advancement of computing devices as well as software tools and apps since early 2000s, there was an increasing trend of integrating these digital resources into classroom learning and teaching (Leow et al., 2016; Zanjani et al., 2017). The school education at this stage focused e-Learning development on the technological aspect, paying attention to the selection and use of different types of computing devices and digital resources for learning and teaching purposes.

Building on the emphasis of technological integration into learning and teaching for years, practitioners realized the need of more pedagogical considerations for advancing towards quality e-Learning. Since the mid of 2000s, there has been more advocacy of promoting learners to learn with diverse digital resources that encourage active engagement in constructive learning and peer interaction for developing subject knowledge (Blundell et al., 2020; Damnik et al., 2017). This drove an additional emphasis on the pedagogical rationales of e-Learning in school education. In the recent decade, more attention has been paid to pedagogy in e-Learning context for the development of 21st century skills. The realization of this educational goal needs a process of active, constructive, and interactive learning for learners to develop and apply skills such as inquiry, critical thinking, communication, and collaboration in the 21st century (Kong et al., 2014; Chou et al., 2019; Rodríguez et al., 2017). The theoretical perspective of constructivist learning is considered favorable to the design and deployment of effective pedagogy for e-Learning (Mchaney et al., 2018; McKnight et al., 2016). The creation of e-Learning environments which implement a constructivist learning approach has placed an additional opportunity for learners to progressively deepen subject knowledge through a convenient access to appropriate resources and sharing of useful information among learning peers (Blundell et al., 2020; Kong et al., 2014). Similar pedagogical guides such as the promotion of models like inquiry-based learning (Acar & Tuncdogan, 2019; Kong & Song, 2014), principles like learner-centered learning (Kong et al., 2014; Walt & Barker, 2020), and strategies like flipped classroom activity design (Blair et al., 2016; Blau & Shamir-Inbal, 2017) for engaging learners in preparation, learning process, and reflection are advocated.

With this driving force of pedagogical guidance, the additional opportunity of e-Learning in school education is the promotion of online interactions among peers from the constructivist perspectives for knowledge construction. This creates the pedagogical need of digital communication among learners in e-Learning environments. Learners in e-Learning process are often required to use digital communication ways to interact and collaborate with peers to complete learning tasks and share learning outcomes (Rodríguez et al., 2017; Varier et al., 2017). On the one hand, there is an increasing trend of using discussion forums in learning management systems (LMSs) for online discussions and peer assessment in formal learning context. The online interaction features of LMSs enable learners to share learning-related information and resources from multiple sources and support their construction and reflection on subject knowledge in the learning process (Kong, 2015; Zanjani et al., 2017). On the other hand, there is an increasing use of social networking systems for online sharing in non-formal learning context. One of the educational use of social networking systems is sharing bookmarks which can be used for the breakthrough of the traditional mindset and the consideration of alternative viewpoints in online sharing (Eid & Al-Jabri, 2016; Greenhow & Lewin, 2016).

Another emerging opportunity is the pedagogical use of online data generated from digital platforms. Such a digital way of collecting data facilitates convenient ways of recording, analyzing, and providing timely feedback to learners’ learning process and learning outcomes. The tools such as discussion forums used in digital platforms store extensive data of learning of learners and records of interactions among peers in the learning process. These data provide opportunities for teachers and researchers to conduct learning analytics for analyzing data generated from these e-Learning environments. The analyzed data provide evidence of the learning progress, no matter it is effective or not (DiCerbo, 2020; Kong et al., 2014). Teachers therefore can use such evidences to make pedagogical decisions to support learners’ learning in an informed manner. On the one hand, learners can accumulate learner-centered learning experience and create learner-centered learning portfolio using these learning data. On the other hand, teachers can provide timely feedback and support according to learners’ learning performance in the learning process (Littlejohn et al., 2016; Yen et al., 2018).

The developmental trends of e-Learning in the recent decades have emphasized the importance of pedagogy for active, constructive, and interactive learning. These trends drive teacher development on e-Learning in the 21st century to put more emphasis on preparing in-service teachers to transform their pedagogical beliefs and practices through technological integration, of which the quality of learning and teaching is expected to be enhanced by an effective application of digital resources, digital communication, and digital assessment (Albion et al., 2015; Kong et al., 2014; Liao et al., 2017).

Opportunities for self-reflection and peer-sharing are considered essential in such teacher development, in which teacher participants articulate their changes in the pedagogical belief and knowledge of technological integration into curriculum delivery (Jones & Dexter, 2018; Koh et al., 2017; Yurtseven Avci et al., 2020). One of the best ways is to arrange teacher participants to write reflection text iteratively in different time points of teacher development courses for engaging them in the self-reflection opportunities (Bashan et al., 2017; McLean & Price, 2019). The writing of reflection text is considered effective to encourage teacher participants to reflect on their changes achieved in teacher development courses (Bashan et al., 2017; McLean & Price, 2019). The provision of scaffolding for such writing is further considered helpful to support teacher participants to retrieve, familiarize and connect the fundamental concepts in the learning topics (Bashan et al., 2017; Pereira et al., 2016). The reflection text generated by the teacher participants is recognized as promising to be both treatment and measurement of teacher development. Such reflection text is, on the one hand, a learning process that engages teacher participants in developing the target knowledge and pedagogy, and on the other hand, a learning product that records teacher participants’ changes in developing the target knowledge and pedagogy (Bashan et al., 2017; McLean & Price, 2019).

Although the existing teacher development on e-Learning generally includes a reflective approach to support teachers’ professional learning, there is no final conclusion on an accurate and efficient evaluation of teachers’ self-reflection products for a critical analysis of their growth in professional knowledge about e-Learning (Albion et al., 2015; Liao et al., 2017; Tondeur et al., 2016). In recent years, the approach of learning analytics with results-presentation in visual-form is advocated to be a computer-aided approach feasible to be both treatment and measurement of teacher development in a more efficient and less tedious manner (Fahrenbach et al., 2019; McKenna et al., 2019). The process of this approach is receiving a documentation of learner learning (such as a set of learner-generated reflection text) and a dictionary of professional competency (such as a topic-specific keyword framework which serves as a reference point for the evaluation) and then returning a match between them (Fahrenbach et al., 2019; Noroozi et al., 2019). This computer-aided approach commonly uses LMS learning analytics to create visual representations of online data for two main purposes: as a feedback-provision for learners to deepen their topic-specific understanding; and as a scaffolding-support for learners to reflect on their topic-specific learning (McKenna et al., 2019; Vieira et al., 2018).

Researchers such as Bikakis et al. (2017) and Elmqvist and Fekete (2010) recommend the use of hierarchical visualization tools in such learning analytics context to make the visual representations more visually scalable and less cluttered. Hierarchical visualization tools allow teachers and learners to visually explore large sets of data in a multilevel manner, as those tools afford an intuitive and practical way for teachers and learners to look at the overview of a dataset as well as look into specific parts within a dataset. Elmqvist and Fekete (2010) and Vieira et al. (2018) denote that the application of hierarchical visualization in learning analytics commonly involves the techniques for visualizing the structure and summary of the multilevel representations of learning data (such as the count and sum of certain items in the data collected from students’ contributions to course discussion forums online on LMS), and interacting with the multiscale data-display through visual exploration operations (such as zooming in-and-out and drilling up-and-down of bubble plots in the data-display for moving across the different levels in the hierarchy to show more or less of the data details).

Accordingly, this study developed an e-Learning framework to guide in-service teachers in school education to understand e-Learning for fostering knowledge construction; as well as delivered and evaluated it through computer-aided analysis of learners’ reflection text coupled with hierarchical visualization of analysis results in a teacher development course on e-Learning for exposing in-service teachers to a theoretically-guided and evidence-based e-Learning process for knowledge construction, learning interaction, and pedagogical decision-making. This study focused on two research questions: (1) What should be the e-Learning framework for the delivery through computer-aided analysis of learners’ reflection text in the teacher development course on e-Learning in school education? (2) How will be the evaluation of learners’ learning through computer-aided analysis of learners’ reflection text in the teacher development course using the proposed e-Learning framework?

Development of an e-Learning framework of the study

In response to the first research question, this study developed an e-Learning framework to support school education for realizing the educational goal in the 21st century. The design of the proposed e-Learning framework also referred to related e-Learning frameworks. Oktavia et al. (2016) review that the two most concerned elements in e-Learning frameworks since 2000 are content and pedagogy. Aparicio et al. (2016) present an e-Learning framework which concerns three elements: the people, the technology, and the services. Pedagogical models and instructional strategies are discussed in the services. Adekola and Dale (2017) present an e-Learning framework which concerns six elements: physical infrastructure, learning technology support, pedagogy, management and organization, institutional culture, and ethical/legal. The common elements in these frameworks are technology and pedagogy in e-Learning. The proposed e-Learning framework in this study was therefore designed to have two dimensions: technology and pedagogy (see Fig. 1). Referring to the discussion in the background of this study, the technology dimension in the proposed framework composes of three components. They are the use of e-resources, digital ways of communication, and digital ways of collecting data in the e-Learning process. The pedagogy dimension concerns theories/models/principles/strategies in guiding the design of learning and teaching activities as well as the use of data collected to provide evidence-based pedagogical decision-making/supports in the e-Learning process.

Fig. 1

An e-Learning framework in school education consisting of two dimensions and six components

In Hong Kong, English and Chinese are the two languages commonly used in the process of learning and teaching. To capture the understanding of e-Learning of teachers in Hong Kong, a set of bilingual keywords in English and Chinese was derived for each of the six components across the two dimensions in the proposed e-Learning framework. Three approaches were adopted for deriving the sets of bilingual keywords of the proposed e-Learning framework. Firstly, based on the current research and practices of e-Learning, the teacher responsible for the teacher development course in this study designed the keywords including synonyms in both English and Chinese. Secondly, based on learners’ reflection of the teacher development course, a manual search of additional relevant words in learners’ reflection text was conducted to enrich the bilingual keyword sets. Thirdly, the enriched sets of bilingual keywords were further checked with Google Translate for validating the connection between the English and Chinese versions of each keyword.

The keywords of the technology dimension in e-Learning related to the use of e-resources, digital ways of communication, and digital ways of collecting data are summarized in Table 1. The categorization of the keywords related to the use of e-resources in school education is grouped into four categories according to their ways of use in school education. E-textbook is a category in which school curriculum e-resources are putting up together in a structural manner for convenient access and use across a substantial curriculum delivery period. Curriculum-related e-resources are another category and typically are topic-wise e-resources purposely designed for use as part of the school curriculum. Open courseware is a category that may or may not relate to the school curriculum and it is in general for self-paced or self-regulated learning. The fourth category is e-resources in general that are available for learners’ and teachers’ consideration of use as resources in learning and teaching activities which are not purposely designed for school use. The second component is digital ways of communication. Currently, it is grouped into four categories. One category is the classroom use of digital communication tools for engaging learners and enhancing interactions in classrooms. Another category is the use of digital platforms for engaging learners and extending interactions among learners which happen in general outside classrooms. The third category is interactions of learners and teachers in social networks. The fourth category enables learners to share their expression/views/reflection other than the above categories. The third component is digital ways of collecting data. It covers formative and summative types of data collection. Formative data are data collected during the learning process such as text from the discussion forum. Summative data are data collected at the critical point of a learning process that learning outcomes need to be evaluated and reported.

Table 1 Finalized keywords of this study on the technology dimension of the proposed e-Learning framework

It should be noted that an additional component of digital technology was added to the technology dimension of the proposed e-Learning framework to cover technological items, such as hardware devices, software products, and general digital technology, which are commonly found in e-Learning environments but not closely coupled with goals of learning and teaching activities. This additional component was added for analyzing the reflection text produced by learners at the beginning and the end of the course. In general, learners of the teacher development course produce more proportion of keywords of this type at the beginning of the course and less towards the end of the course after a more thorough understanding of e-Learning in school education. Table 1 shows the finalized list of bilingual keywords regarding the technology dimension of the proposed e-Learning framework.

The keywords in the pedagogy dimension concerns learning and teaching activities guided by theories/models/principles/strategies in pedagogical design and practices as well as the use of digital learning data to support pedagogical decision-making. This dimension involves three components. The first component is theories/models/principles/strategies—mainly addressing the epistemic rationales favorable to e-Learning, such as constructivist theories guiding the learning process; models for inquiry-based learning; principles facilitating learner-centered learning; and strategies like flipped classroom design for engaging learners in the learning process. The second component is learning and teaching activities—mainly concerning the tasks in e-Learning environments for learners to make reflection for knowledge construction; to interact with peers and teachers for resource sharing and idea exchange; to receive feedback for learning improvement and to obtain scaffolding support for guiding their learning process. The third component is evidence-based pedagogical decision-making/supports—mainly addressing learning data collection in the learning process, analyzing data collected from learners, and finally, revising or modifying ways of teaching with evidence from learning analytics.

The proposed e-Learning framework has the primary rationale that the use of technology in learning and teaching activities in the e-Learning context should be driven by the pedagogical theories/models/principles/strategies selected by teachers. For example, teachers who select the constructivist theories for e-Learning implementation need to base on their selected theories to design learning and teaching activities for e-Learning lessons, which involve the use of e-resources, digital ways of communications as well as digital ways of collecting data for promoting knowledge construction and evidence-based pedagogical decision-making. Table 2 shows the finalized list of bilingual keywords regarding the pedagogy dimensions of the proposed e-Learning framework. The finalized bilingual keyword sets were used in the course as the basis for interpreting learners’ understanding of e-Learning in their reflection on e-Learning understanding before and after the course, of which learners could write their reflection text in English, Chinese, or a mixture of English and Chinese.

Table 2 Finalized keywords of this study on the pedagogy dimension of the proposed e-Learning framework

Delivering the e-Learning framework and investigating the effectiveness

In response to the second research question, this study delivered the proposed e-Learning framework to a cohort of in-service teachers. The methods of computer-aided analysis of learners’ reflection text (bilingual in English and Chinese), questionnaire survey, and focus group interview were used to investigate the effectiveness of the pedagogical delivery of the proposed e-Learning framework for enhancing learners’ understanding of e-Learning in school education.

Design of the teacher development course for pedagogical delivery

Course participants

The proposed e-Learning framework was delivered to a cohort of 33 (14 male and 19 female) in-service primary school teachers in a course “E-Learning in Primary Schools” in a university in Hong Kong. The participants had an average teaching experience of 16 years (SD = 6.355 years). Most of the participants taught the subjects Information Technology and Mathematics, while the others taught the subjects Chinese, English, and General Studies. The sample size with 33 participants is justified to be adequate for the high-power statistical tests in the evaluation part in this study, with α = 0.05 and power = 0.8.

Course delivery and features

The teacher development course consisted of seven 3-h lessons for the pedagogical delivery of the proposed e-Learning framework (see Table 3). The course covered both theoretical knowledge introduction and hands-on experiences for learners to understand the opportunities and challenges of implementing e-Learning across major subjects in primary schools. The course is targeted at fostering learners to realize the importance of pedagogy in e-Learning. It was expected that after completing the course, learners could have the knowledge to design e-Learning plans for their schools and develop pedagogical practices for e-Learning deployment in major subjects in primary schools. The course delivery was designed with five features for realizing its educational goals.

Table 3 Pedagogical design and practices of the seven lessons of the teacher development course

First, the course emphasized reflection opportunities for learners in course learning. The learners were asked to submit an online reflection on their understanding of e-Learning in the first lesson, as the pre-teaching reflection before the pedagogical delivery of the proposed e-Learning framework. After the sixth lesson, learners were asked to submit their second online reflection, as the post-teaching reflection. This feature enabled teachers to experience the meaning of evidence-based pedagogical decision-making in guiding learners to understand e-Learning in school education if gaps are identified from the pre-teaching reflection with the proposed framework.

Second, the course started to introduce the proposed e-Learning framework in the second lesson through reporting learners’ pre-teaching reflection. Between the second and fifth lessons, the lecturer repeatedly arranged the learners to interact with the hierarchical visualization of the results on computer-aided analysis of their pre-reflection in the individual-student and whole-class manner. The learners had many opportunities to access the topic-specific keywords at the overall (i.e., “pedagogy vs technology”), dimension-specific (e.g., “pedagogy” only), and component-specific (e.g., “pedagogical decision-making/supports” only under the “pedagogy” dimension) levels when they browsed and clicked on the different layers in such hierarchical visualization (see Fig. 2 for an example). With the belief that technological integration in e-Learning environments should be guided by a theoretical framework, related issues in the pedagogy dimension of e-Learning such as theories of constructivist learning for developing 21st century skills, models of inquiry-based learning, principles of learner-centered learning, and evidence-based pedagogical decisions were discussed; apart from the issues of the technology dimension of e-Learning such as e-resources, digital ways of communication and digital ways of collecting data. This feature enabled the learners to realize the importance of pedagogy in practicing e-Learning.

Fig. 2

An example that a learner interacted with different layers of the hierarchical visualization of the results on computer-aided analysis of the pre-reflection. a The learner browsed the results of matching keywords at the overall level of “pedagogy vs technology”. b The learner clicked on the overall-results-layer for further browsing the results at the dimension-specific level of “pedagogy”

Third, the course emphasized the subject-specific e-Learning experience for the learners. With the belief that teachers’ first-person experience in e-Learning best convinces the benefits of e-Learning, the course arranged four 30-minute subject-specific e-Learning teaching experience sections in the second to fifth lessons. Learners in these sections experienced hands-on practice of e-Learning pedagogical examples such as inquiry-based learning for four major subjects in primary school education, namely Chinese, English, Mathematics, and General Studies. This feature enabled the learners to experience how technology and pedagogy are integrated for delivery in a subject-specific area.

Fourth, the course adopted the flipped classroom strategy for pedagogical delivery (Kong, 2014, 2015). In this course, the teaching in the second to sixth lessons asked the learners to read selected articles before class, and then discuss the articles read for idea sharing inside class time. This feature not only enabled the teacher to better arrange course teaching time for learner-centered learning; but also enabled the learners to experience the pedagogical strategy for e-Learning in school education.

Fifth, the course engaged learners in classroom observation for learning reflections. In the fourth lesson of the course, learners visited a classroom with students bringing along their own digital devices for observing e-Learning practice in subject-specific lessons. This feature enabled the learners to deepen their understanding of e-Learning through using the framework to guide reflection on the observed teaching practices in real classroom.

Methods of investigating the effectiveness of the pedagogical delivery

The effectiveness of the pedagogical delivery was evaluated with two focuses: (1) Can the pedagogical delivery of the proposed e-Learning framework support learners to enhance their understanding of e-Learning? (2) How do the learners perceive the helpfulness of the support from the computer-aided analysis of learners’ reflection text coupled with hierarchical visualization of analysis results on promoting their e-Learning understanding? Computer-aided analysis of learners’ reflection text—bilingual in English and Chinese—was used to compare learners’ online reflection on their understanding of e-Learning at the beginning and towards the end of the teacher development course, namely pre-teaching reflection, and post-teaching reflection. A statistical power analysis was conducted for the method of computer-aided analysis of learners’ reflection text. The effect size in this method was 0.756, which was considered as large according to Egberink et al. (2015). In addition, for data triangulation purposes, a questionnaire survey and a focus group interview were conducted to investigate learners’ perceptions of the effectiveness of introducing the e-Learning framework with the support from the computer-aided analysis of learners’ reflection text coupled with hierarchical visualization of analysis results.

Analysis of learners’ understanding of e-Learning by the computer-aided analysis of learners’ reflection text

Learners were asked to reflect on their understanding of e-Learning in the first and the sixth lessons of the course. The computer-aided analysis identified and counted learners’ matching keywords used in their reflections based on the sets of bilingual keywords as shown in Tables 1 and 2 of the e-Learning framework of this study. A hierarchical visualization tool based on the proposed e-Learning framework was designed and adopted in this study for an interactive visualization of the computer-aided analysis results through web browsers. This enabled the teachers and learners of the course to visualize the computer-aided analysis results interactively according to the details of the results at various levels hierarchically on-demand. The results were represented through a ringed circular layout of nodes, in which a node and all its children were placed in a circle. This hierarchical visualization tool represented each dimension and its corresponding components of the proposed e-Learning framework as a circular layer (at the dimension level) encompassing a number of balls (at the component level). The blue color scheme of different layers got deeper according to the layering in the proposed e-Learning framework. A keyword set was denoted by an orange ball if there was at least one keyword of that keyword set mentioned by learners’ reflections; otherwise, it was denoted by a gray ball. The size of an orange ball was proportional to the number of matching keywords of the corresponding keyword set. By viewing the number and size of the orange balls through this hierarchical visualization tool, teachers and learners of the course could understand the dimensions and components covered by the reflection text.

Analysis of learners’ perception of the support from the computer-aided analysis of learners’ reflection text coupled with hierarchical visualization of analysis results by questionnaire survey

To investigate learners’ perception of the support from the computer-aided analysis of learners’ reflection text coupled with hierarchical visualization of analysis results on enhancing their understanding of e-Learning, a questionnaire survey consisting of 10 question items in a 5-Likert scale (1 = strongly disagree to 5 = strongly agree) was conducted. Among the 33 learners in this study, 31 learners completed this survey for indicating how the computer-aided analysis with hierarchical visualization could support them to understand the domain-specific intellectual ideas, link personal learning intentions and expectations, and make social interactions with peers in the learning process (Kong & Song, 2015). The Cronbach’s alpha reliability coefficient of this survey was 0.900. The quantitative data collected were analyzed by SPSS software.

Analysis of learners’ perception of the support from the computer-aided analysis of learners’ reflection text coupled with hierarchical visualization of analysis results by a focus group interview

A focus group interview was conducted with three learners to further investigate learners’ perception of the support from the computer-aided analysis of learners’ reflection text coupled with hierarchical visualization of analysis results. The learners were asked to provide feedback on the questions related to the applicability of the tool such as “How does the hierarchical visualization tool influence my understanding of e-Learning?”; on the questions related to the social context of the tool such as “Do you agree that the hierarchical visualization tool enhances the interaction between learners and teachers? If yes, how does it work out?”. The qualitative feedback collected was summarized and analyzed systematically.

Effectiveness of the pedagogical delivery of the e-Learning framework

This section presents and discusses the results on the effectiveness of the pedagogical delivery of the proposed e-Learning framework to enhance learners’ understanding of e-Learning.

Learners’ change in the understanding of e-Learning after the pedagogical delivery of the proposed e-Learning framework support

Figure 3 shows the hierarchical visualization of the results of computer-aided analysis of learners’ pre-teaching and post-teaching reflections on their understanding of e-Learning. The number of matching keywords in the technology dimension (as denoted by the orange balls in Fig. 3) in the pre-teaching reflection and post-teaching reflection were 81 and 203 respectively, while the corresponding matching keywords in the pedagogy dimension were 69 and 221, respectively. These results showed that there was a significant increase in matching keywords from the pre-teaching reflection to the post-teaching reflection. Learners were enabled to visualize the e-Learning framework interactively through the hierarchical visualization of pre-teaching reflection results after the first lesson. They referred to this visualization support for reflection in the process of the course activities of this study. For the technology dimension, learners produced a higher proportion of keywords in the “digital technology” component which covers items associated with the general use of digital technology in the pre-teaching reflection. This indicated that learners were not so familiar with technological issues specific for e-Learning practices in school education before taking the course. In the post-teaching reflection, learners could use keywords across all the dimension-specific components in technology after the course. Learners had a noticeable increase in the use of keywords across all three dimension-specific components, with the number of matching keywords from 8 to 48 (i.e., six times) for the “digital ways of communication” component; from 4 to 13 (i.e., three times) for the “digital ways of collecting data” component, and 11 to 33 (i.e., three times) for the “e-resources” component.

Fig. 3

Hierarchical visualization of the results on computer-aided analysis of learners’ pre-teaching and post-teaching reflections on their understanding of e-Learning

For the pedagogy dimension, learners had two evident improvements in using keywords across all the dimension-specific components after the course. First, all gray balls (denoting absent keywords) in the pre-teaching reflection changed to orange balls (denoting used keywords) in the post-teaching reflection. Second, learners also had a noticeable increase in using keywords across all the dimension-specific components in the post-teaching reflection, with the number of matching keywords from 50 to 143 (about five times) for the “learning and teaching activities” component; from 19 to 52 (about three times) for the “theories/models/principles/strategies” component and from 0 to 26 (start to use these keywords) for the “pedagogical decision-making” component. Table 4 shows examples of changes on learners’ understanding of e-Learning. These two evident improvements show that the pedagogical delivery of the proposed e-Learning framework can support learners to enhance their comprehensiveness of e-Learning understanding, as this cohort of learners was able to mention all keyword sets of the pedagogy dimension in the post-teaching reflection. The visualization, therefore, can provide empirical data for the teacher to adjust pedagogical decision to address the learning needs of learners to achieve its planned goal.

Table 4 Examples of learners’ pre-teaching and post-teaching reflections on their understanding of e-Learning

Table 5 shows the statistical results of comparing the means of counting matching keywords of learners’ pre-teaching and post-teaching reflections. The results showed that learners’ understanding of e-Learning was enhanced in all components in the technology ((1) + (2) + (3)) and pedagogy ((4) + (5) + (6)) dimensions. It showed that learners had deepened their understanding of e-Learning in school education in both technological and pedagogical dimensions.

Table 5 Comparison of means of counting matching keywords from the results on computer-aided analysis of learners’ pre-teaching and post-teaching reflections on their understanding of e-Learning

To further explain the changes of the counting of matching keywords of the e-Learning framework in the pre-teaching and post-teaching reflections, the percentage change of counting of matching keywords and proportions of matching counts at all levels of the e-Learning framework in the pre-teaching and post-teaching reflections were analyzed and tabulated in Table 6.

Table 6 The percentage change of counting of matching keywords and proportions of matching counts at all levels of the e-Learning framework in the pre-teaching and post-teaching reflections

The percentage changes in matching counts of the technology and pedagogy dimensions between the two reflections were 151% and 220%, respectively. Before taking the course, learners paid more attention to the technology dimension in e-Learning. After taking the course, the proportion of matching counts of the technology dimension decreased from 54.0% to 47.9%; while that of the pedagogy dimension increased from 46.0% to 52.1%. These results provided evidences that the proposed e-Learning framework and its pedagogical delivery assisted learners to realize the importance of pedagogy in e-Learning in school education. There were four additional detailed observations.

Firstly, an observation from Table 6 was that learners in the post-teaching reflection had a significant reduction in the proportion of matching counts in the “digital technology” component decreased from 38.7% to 25.7%, which is an additional technological component of the proposed e-Learning framework for recording common keywords associated with the general use of digital technology generated by learners. This result indicated that learners after taking the course recognized the use of e-resources, digital ways of communication, and digital ways of collecting data more relevant to e-Learning. This was also evidenced by the increase in proportion of matching counts of keywords in the components of “e-resources,” “digital ways of communication,” and “digital ways of collecting data” from 7.3% to 7.8%, 5.3% to 11.3%, and 2.7% to 3.1%, respectively.

Secondly, the results of comparing the proportion of matching counts of categories in the component of “digital ways of communication” in pre-teaching and post-teaching reflections indicated that the categories of “discussion forum” and “social network” increased from 4.0% to 9.2% and 0.7% to 1.9%, respectively, which contributed significantly to the increase in the number of matching keywords of the “digital ways of communication” component. These results reflected that learners considered “digital ways of communication” is important in e-Learning after attending the course. Participants also started to realize the importance of “digital ways of collecting data” for formative assessment as evidenced by the increase in matching keywords from 4 to 11 that is an increase of 175%.

Thirdly, the results of comparing the proportion of matching counts of categories in the component of “learning and teaching activities” in pre-teaching and post-teaching reflections indicated that the categories of “reflection” and “discussion” increased from 0% to 3.5% and 19.3% to 22.6% respectively, which contributed significantly to the increase in the number of matching keywords of the “learning and teaching activities” component. It can be concluded that learners considered “reflection” and “discussion” as important learning and teaching activities in e-Learning in school education.

Fourthly, the results of comparing the proportion of matching counts of categories in the component of “pedagogical decision-making/support” in pre-teaching and post-teaching reflections indicated that the categories of “learning data” and “revise/modify” increased from 0% to 3.3% and 0% to 2.1% respectively, which contributed significantly to the increase in the number of matching keywords of the “pedagogical decision-making/support” component.

Learners’ perception of the helpfulness of the support from the computer-aided analysis of learners’ reflection text coupled with hierarchical visualization of analysis results on promoting their e-Learning understanding

Table 7 showed survey results of learners’ perception of the e-Learning framework with the support from the computer-aided analysis of learners’ reflection text coupled with hierarchical visualization of analysis results. The survey results showed that learners in general positively perceived the e-Learning framework with the support from computer-aided analysis and hierarchical visualization for fostering their understanding of e-Learning, with all item ratings in the range of 4.065 to 4.452 (with 5 as maximum). The learners agreed most the impact of the hierarchical visualization tool for providing them with useful information from all participants in the teacher development course, which stimulated their individual understanding of e-Learning. In addition to enabling them to develop intellectual ideas of e-Learning, the learners also indicated that the functions of the hierarchical visualization support can help them link personal learning intentions and expectations by providing an opportunity for them to reflect on their individual understanding of e-Learning and arousing their attention to the understanding of e-Learning.

Table 7 Survey results of learners’ perception of the e-Learning framework with the support from computer-aided analysis and hierarchical visualization

Table 8 showed focus group interview results of learners’ perception of the e-Learning framework with the support from computer-aided analysis and hierarchical visualization. The interview results showed learners’ positive perception of the support from computer-aided analysis and hierarchical visualization. They indicated that the e-Learning framework presented in hierarchical visualization was effective in enhancing their understanding of e-Learning. The hierarchical visualization of the pre-teaching reflection results allowed them to reflect on their individual understanding as well as discussion among peer of e-Learning early at the beginning of the course. The hierarchical visualization support also allowed them to deepen their learning by comparing their individual understanding with the whole-class understanding of e-Learning. The learners also suggested ways to improve the design of the interface of the hierarchical visualization support.

Table 8 Focus group interview results of learners’ perception of the e-Learning framework with the support of computer-aided analysis and hierarchical visualization

Conclusion and future works

The study proposed an e-Learning framework consisting of the “technology” and “pedagogy” dimensions; and used the approach of computer-aided analysis of learners’ reflection text coupled with hierarchical visualization of analysis results to support the delivery and evaluation of learners’ learning in a cohort of seven-lesson teacher development course on “E-Learning in Primary Schools”, which provided reflection opportunities and experiences to enable learners to understand the need of theoretical guidance in designing e-Learning activities for achieving the educational goal. Learners’ reflection text on e-Learning before and after the teaching of the course indicated that learners enhanced their understanding of e-Learning through the counting of matching keywords of the framework. The computer-aided analysis of the reflections showed that learners mentioned less general digital technology terms such as using a computer for learning and realized a more in-depth understanding of the role of technology in e-Learning. They realized that e-Learning was not just the use of e-resources for learning but also the use of digital ways of communication like the use of discussion forum and social network for learning. They also started to realize the importance of data collection from e-Learning platforms in e-Learning. The computer-aided analysis of the reflections also indicated that learners understood more the importance of pedagogy in e-Learning. Learners particularly showed more consideration on reflection and discussion opportunities in learning and teaching activities in e-Learning and started realizing more the possibility of pedagogical decision-making based on data collected from online learning.

As found by the survey and focus group interview, learners positively perceived the hierarchical visualization support for interaction with the proposed e-Learning framework for a better understanding of e-Learning. They treasured the opportunities in comparing their own individual understanding with the whole-class understanding using the proposed e-Learning framework. They valued the opportunities provided for peer discussion on the concrete details of e-Learning activities supported by the proposed framework. The peer discussion provided opportunities for individual reflection on directions for personal expectations of e-Learning development.

This study pioneers the contributions to research literature on the use of computer-aided analysis for an accurate and efficient evaluation of learning progress and outcome among in-service teachers in teacher development courses on e-Learning. This study made an evident contribution that the proposed e-Learning framework and its pedagogical delivery are effective and well-received for preparing in-service teachers in school education. It encourages teachers to plan and design pedagogically desirable learning and teaching activities as well as to make evidence-based pedagogical decisions in promoting e-Learning in school context.

There are two limitations of this research. First, the e-Learning framework is at the proposed stage. The keywords in each dimension need to be further refined. Second, this study only conducted the evaluation in one teacher development course on e-Learning. There are three directions for the further research of this study. First, the e-Learning framework should be practiced in more cohorts of learners to further refine its keyword and add more validation of its effectiveness for evaluation. Second, the pedagogical delivery of such framework can be extended to enable learners to compare their own individual results with the whole-class results on the computer-aided analysis of pre-teaching as well as post-teaching reflections, and in turn facilitating learners to deepen their reflection. Third, to enable teachers in teacher development to use the result of computer-aided analysis to make pedagogical decisions to improve learners’ understanding of the e-Learning framework, teachers might add one more reflection between the pre-teaching and post-teaching reflection so that teachers can adjust their teaching using result from this reflection.