Design and development of semi-automatic concept map authoring support tool

Abstract

This research, to design and develop a concept map authoring support tool, adopts a semi-automatic concept mapping approach to help teachers create concept maps from English readings. A concept map is widely regarded as a useful teaching and learning tool. It offers many potential advantages apart from representing the students’ knowledge and understanding during learning. Students’ engagement in using and creating concept maps with a computer-enabled concept mapping tool raises concept maps’ potential benefits. It contributes to the learning process and improves the students’ meaningful learning. The Kit-Build concept map framework, which incorporates a technology-enabled concept mapping tool, uses concept map recomposition as its essential learning activity. In learning with Kit-Build, teachers compose concept maps that they want the students to achieve. The teachers’ maps are then decomposed into components from which the students recompose and reflect deeply on their understanding. The difference between teacher’s and students’ concept maps depicts the gap between teachers’ expected understanding and students’ actual understanding. Hence, the teachers’ concept map becomes an essential part of learning with Kit-Build. For some teachers, creating a good concept map for learning is difficult and time-consuming. Hence, support to improve teachers’ productivity in creating concept maps is essential. The findings suggest that the support tool yields better concept mapping efficiency while maintaining concept maps of similar quality. Teachers also found that the support tool was useful. Therefore, semi-automatic concept mapping with the supported Kit-Build concept map authoring tool has been shown to be a better approach.

Introduction

Despite their simplicity in graphical form, concept maps have been put to various uses. A concept map can be regarded as an effective teaching and learning tool as it offers a visual representation of the students’ acquired knowledge and other advantages during learning (Stoica et al. 2011). It shows complex interrelations between different concepts of a particular learning topic. Concept maps can be combined with other learning strategies, such as dialog, collaboration, discussion, and feedback, to enrich the learning environment. Adopting concept mapping activities optimizes the students’ learning as it may influence the students’ self-efficacy and self-regulation during learning (Chularut and DeBacker 2004). Previous research shows the positive effects of using concept maps in various learning environments that yield better learning outcomes than a traditional learning expository (Chiou 2008; Joshi and Vyas 2018; Saeidifard et al. 2014; Taie 2014). Not only are the concept maps useful, but activities involved in making the concept maps also help students enhance their critical thinking skills (Latif et al. 2016; Tseng 2020) even though some students face difficulties in concept mapping during learning (Machado and Carvalho 2020). In addition to facilitating the learning progress and assessment of students’ understanding, concept maps have also been found to develop deeper, higher-level cognitive processing (Wu et al. 2016), and meaningful learning (Carr-Lopez et al. 2014; Prasetya et al. 2020; Vanhear 2012).

In addition to depicting one’s understanding or knowledge, concept maps can serve as a tool to evaluate, self-assess, and track students’ understanding during learning (Hay 2007). Concept maps can also help teachers evaluate their teaching and identify students’ misconceptions and missing concepts (Stoica et al. 2011). Kit-Build is one example of learning frameworks that uses concept map recomposition as its essential learning activity to assess students’ understanding. Studies on the practical implementation of Kit-Build in the elementary school science class (Sugihara et al. 2012) and English as a Foreign Language (EFL) studies (Alkhateeb et al. 2016; Andoko et al. 2020) suggested that Kit-Build concept map have many positive impacts towards learning. Additionally, the Kit-Build concept map can be incorporated into a collaborative learning environment to facilitate students’ discussion in sharing their knowledge and understanding (Sadita et al. 2020b; Sadita et al. 2020a; Wunnasri et al. 2018).

A reading-intensive learning environment, as in learning EFL reading comprehension, uses many reading passages to practice reading skills. When the students use Kit-Build concept maps to comprehend the reading passages better, many kits have to be created to support the activities. Teachers have to create the complete concept maps before decomposing the maps into Kit-Build kits for the students to reconstruct. While concept map recomposition activities with Kit-Build concept maps were fun and useful (Pailai et al. 2017), the creation process of the initial Kit-Build concept maps took a committed time and effort (Alkhateeb et al. 2016) and challenging for some teachers (Harrison and Gibbons 2013; Shian-Shyong et al. 2007; Torre et al. 2007). A computer-generated concept map could be used as an alternative to a manually constructed concept map by teachers. However, it has issues regarding coverage, accuracy, readability, and suitability (Atapattu et al. 2017). Furthermore, semi-automatic approach was reported to be better than automatic approach (Presch 2020). Nevertheless, support for an effective and efficient concept mapping process to produce useful and beneficial concept maps is necessary.

This research focuses on assisting teachers to create concept maps from English texts with Kit-Build concept map authoring tool. As automatically generating concept maps from texts, which presumed to satisfy teachers’ intention in using the resulting concept maps is difficult, this study adopts the semi-automatic concept mapping approach where teachers are involved in the generation process. This research presents the design and development of a semi-automatic support function that extends the previous Kit-Build concept map authoring tool to aid teachers in creating concept maps from English reading materials. Thus, teachers could compose concept maps for use with Kit-Build more efficiently than composing the concept maps manually. The new tool implements the Natural Language Processing (NLP), Concept Map Mining (CMM), and text mining approaches to extract and suggest relevant items of reading material to teachers. Currently, teachers have to create concept and link nodes, connect the nodes to make propositions, and arrange the propositions manually. With the proposed concept map authoring tool, teachers can get suggestions of keywords and propositions to be included in their concept maps. This research puts the tool into a trial with English teachers to evaluate how the tool’s support function affects teachers’ concept mapping activities in terms of efficiency and concept map quality. The following research questions were addressed in this research:

• 1. Will the concept map authoring support tool, which semi-automatically generates concept maps, could improve teachers’ concept mapping efficiency in terms of the yielded number of propositions?

• 2. Will using the concept map authoring support tool, which semi-automatically generates concept maps, could yield concept maps of better quality than a manually composed concept map?

• 3. How will the teachers perceive the usefulness of the concept map authoring support tool to create concept maps of English reading materials for teaching?

The results suggested that the concept map authoring support tool could improve the efficiency in composing larger concept maps and the concept maps remain similar in quality. The teachers who participated in this study also perceived that the concept map authoring support tool was useful in assisting them to create concept maps.

Literature review

Kit-Build concept map

Concept maps can be used and applied in many different ways, from regular note-taking to assisting discussion of a problem-solving activity. Organizing knowledge in a concept map with digital technologies allow teachers and students to easily create, modify, and arrange their concept maps. Furthermore, learning with digital concept map have known to improve student’s engagement in constructing knowledge, hence more dynamic and interactive learning (Daley et al. 2007), and lead to a higher level of cognitive thinking (Koh et al. 2010).

Kit-Build uses digital concept map as a tool for simple assessment of students’ understanding (Hirashima et al. 2015). The general flow of learning activities with Kit-Build concept map is shown in Fig. 1. A set of components of a Kit-Build concept map, which is termed a kit, is created based on the decomposition of a teacher’s concept map. Several kits can be made from either a partial or full decomposition of the teacher’s concept map. With the provided concept mapping tool, the decomposition can either be performed automatically or manually.

Fig. 1

General flow of Kit-Build concept map framework

In Kit-Build, both teachers and students compose their concept maps with a computer-supported concept mapping tool. The students compose their concept maps from a respective teacher’s concept map kit. As the students use the same components as their teacher’s concept map components, their concept maps can be directly compared with the corresponding teacher’s concept map. Also with the provided concept mapping tool, the comparison can be performed quickly and automatically (Hirashima et al. 2015), thus helping teachers to quickly diagnose students’ understanding and evaluate their teaching (Pailai et al. 2017). However, in a close-ended concept mapping such as Kit-Build, students learn by using only kits from the teacher’s concept map. They cannot add more information into their concept map other than what had provided by the kits; hence, teacher’s concept map is essential in Kit-Build.

Developing EFL reading comprehension skills with Kit-Build concept maps

In most practical usage of Kit-Build concept maps, students learn from text-based learning materials. The concept map serves as a learning supplement for the materials. Its use in supporting reading-intensive learning, as in EFL reading comprehension, appears to be highly effective. The visual representation and the recomposition of a teacher concept map’s kit help students focus more on the key topics and ideas of the text, thus aiding them to comprehend the text better (Alkhateeb et al. 2015). The kit, in which the students recompose a complete concept map, should possess all of the critical parts of a reading. Therefore, it is necessary for teachers to create a good concept map since the students focus on the ideas and relationships depicted by the kit. Additionally, this explains why teacher’s concept map is essential in learning with Kit-Build.

In an experimental setting, using Kit-Build concept map to practice EFL reading comprehension helped the students retain their knowledge better than a traditional selective underlining method (Alkhateeb et al. 2016). Another Kit-Build concept map learning strategy with source connection capability helps students comprehend an English reading better than using a traditional text-summarization approach (Andoko et al. 2020). Recomposition and review strategies of EFL reading comprehension with Kit-Build concept map helped students better learn and understand English reading comprehension texts. Hence, the use of the Kit-Build concept map to support the learning process of EFL reading comprehension is promising.

In practicing EFL reading comprehension with Kit-Build concept map, students try to comprehend the readings and express their understanding by reconstructing a concept map kit representing the text. When a teacher compares the maps, the parts that are different depict students’ misunderstanding or misconception about the reading. Hence, the teacher can explain to the students and discuss the problematic parts in more depth using a more comprehensive concept map.

In Kit-Build concept map, teachers are responsible for creating the initial Kit-Build concept map. Teachers have to prepare a concept map that is a sophisticated representation of the reading before the students use the map to learn. Lack of understanding of how to properly create a concept map could result in a concept map that is difficult to interpret and review (Veronese et al. 2013). Therefore, teachers should carefully craft and compose a concept map that helps students comprehend the reading better.

It is common to use many readings for practicing EFL reading comprehension with the Kit-Build concept map. The consequence of using many readings to learn EFL reading comprehension is that many concept maps also have to be created. The concept map creation activity becomes a burden for some teachers and increases the cognitive loads of their current academic workload (Harrison and Gibbons 2013). While it is difficult to carry out, concept mapping takes a significant commitment of time and effort to produce a good concept map (Alkhateeb et al. 2016; Harrison and Gibbons 2013; Shian-Shyong et al. 2007; Torre et al. 2007).

Semi-automatic concept map generation with concept map mining

Despite all of the advantages of using concept maps, there are challenges and difficulties in the creation process. How a concept map is composed is subject to its creator purpose and subjectivity. The mental knowledge model of the creator could be transferred to a manually composed concept map. However, the manual concept map composition activity is often perceived as inefficient and a fully automatic approach is difficult to imitate the transfer (Presch 2020).

Reconstructing a reading’s knowledge structure in a concept map format is difficult since it requires a higher level of thinking to discover key ideas of a learning material (Cañas et al. 2017; Carr-Lopez et al. 2014). However, it is possible to discover facts, relationships, and assertions in textual data with help from text mining and NLP (Hayama and Sato 2020; Hearst 1999) to support concept mapping composition activities within a short time. NLP can analyze the information of a human language and help the computer understand or communicate with humans (Collobert et al2011; Manning and Schutze 1999).

The automatic generation of concept maps with information retrieval techniques is called Concept Map Mining (CMM) (Villalon and Calvo 2008). The process of CMM involves three main tasks, including concept identification, relationship identification, and topology identification (Villalon and Calvo 2008), to extract concepts and relations and summarize the results into a concept map. It also involves NLP and text mining techniques in part of the automated process (Presch 2020).

Several studies about the automatic generation of concept maps from texts has been conducted. Most of them incorporated NLP, machine learning, and other linguistic techniques to process the text to identify the necessary concepts and relations (Nugumanova et al. 2015;Qasim et al. 2013;Wafula 2016;Zubrinic et al. 2012). Nonetheless, a concept map is context-dependent and subject to the author’s purpose and preference. Thus, automatically generating a concept map that satisfies subjective results without human interference is considered difficult (Atapattu et al. 2017). Prior study demonstrated that a semi-automatic approach yielded a better concept map than a full-automatic approach in comparison to a manually composed concept map as gold standard (Presch 2020). Hence, semi-automatic concept map generation approach could bridge the gap between manual and fully-automated concept map composition.

This research extends the previous Kit-Build concept map tool’s functionality with an authoring support function. Instead of fully automating the generation process, the tool adopts a semi-automatic approach with CMM to support teachers. The tool’s support function extracts keywords and relation words or phrases from text as concept and relation candidates. Every proposition in a concept map has fundamental purpose and meaning in contrast to an individual concept or relation node. In addition to providing keywords as suggestions to concepts, the authoring tool being developed in this study also provided assistance at proposition level. The tool suggested subject-relation-object triples to the author as the candidates of concept map propositions that were not explicitly provided in prior studies (Hayama and Sato 2020;Presch 2020). This study targeted English teachers as its users who use Kit-Build in teaching EFL reading comprehension and the experiment was also more robust due to the use of multiple texts and uses. The general concept mapping activity with the designed authoring tool is shown in Fig. 2 and the interface where users interact with the suggestions during the concept map composition is shown in Fig. 3.

Fig. 2

Concept mapping activities and support processing flow

Fig. 3

Example of the concept map authoring tool support dialog; suggesting proposition triples and concepts of a sentence

In addition to the regular concept mapping activities with the current Kit-Build concept map tool, teachers can get suggestions from the tool to quickly identify concepts and propositions of learning material. When a text material is loaded, the NLP toolkit annotates the text and extract keywords from it. An unsupervised method to extract keywords from a text document, Rapid Automatic Keyword Extraction (RAKE), is used in the extraction process. RAKE measures the importance of a keyword to the document based on its composing words’ degree and frequency (Rose et al. 2010). Each identified keyword’s weight is computed with the Term Frequency-Inverse Document Frequency (TF-IDF) technique. The computed weight determines the keyword importance level of the text. Similarities among keywords are calculated to avoid suggesting similar keywords and minimize redundancy. The NLP toolkit used to annotate the text material in this research is the Stanford CoreNLP (Manning et al. 2014).

Concepts of a concept map are commonly represented by nouns or noun phrases relevant to a particular context. Relationships between concepts may be represented by linking verbs or linking phrases between two concepts of a sentence. Both noun phrases and verbs are two useful parts-of-speech (POS) tags to identify concepts and relationships of a sentence. In CMM, the POS tags can be obtained from the NLP toolkit’s POS-tagging annotation. Some other useful NLP annotations provided by the Stanford CoreNLP toolkit include tokenization, sentence splitting, lemmatization, dependency parsing, coreference resolution, and open information extraction (Open IE). Additionally, the Stanford CoreNLP toolkit also offers some customization for the annotations.

Identifying an accurate and meaningful relationship between a link and two concepts plays an essential role in composing a concept map proposition. The Stanford CoreNLP’s Open IE annotation extracts triples representing tuples of subject, relation, and relation-object of a text. The Open IE annotation corresponds to the open domain relation that captures the relation phrases expressed by the combination of verb-nouns patterns (Fader et al. 2011) and a natural logic classifier (Angeli et al. 2015). However, many sentences use a combination of verb-noun phrases instead of a single verb to depict a meaningful relationship between subjects and objects. Merely relying on a single verb to express a meaningful relationship is sometimes inadequate. A simple regular expression pattern can be applied to a sentence’s POS tags sequence to identify verbs and verb-noun phrases as another potential candidate of propositions. The extraction process that uses regular expression patterns to extract relationships from a sentence is called Syntactic Relationship Extraction (Fader et al. 2011). The pattern’s use can also reduce the number of uninformative relationships of a sentence (Fader et al. 2011) and improves the extraction performance (Patel et al. 2018).

Methodology

Context and participants

This research aimed to trial the utilization of Kit-Build concept map authoring tools to assist teachers to create concept maps on Phase I of the Kit-Build concept map framework. The context of this research is to support English teachers in creating concept maps from English reading comprehension texts as concept mapping with Kit-Build is shown to promote the students’ comprehension and understanding of its contents. The reading comprehension texts, which were used in the experiment, were structured English readings.

The participants consisted of 47 English teachers and tutors who voluntarily participated in the experiments. They mainly teach English in high school or university in Malang, East Java, Indonesia and they have experience in teaching English reading comprehension with concept maps. The participants consist of 29 (62%) female teachers and 18 (38%) male teachers. The participants’ ages range from 20 to 39 years old, with 48% of participants between 20 and 29 years old, and the remaining 52% between 30 and 39 years old. The participants were invited with an online registration form to give their consent to participate in the experiments. They also provided their contact information for communication purposes regarding the experiment. They also describe their experiences in English teaching and concept mapping over the past few years.

The reading passages used in the experiments were obtained from the ESL Fast website, an online English learning website that provides free English readings with a certain level of difficulties (ESL Fast 2018). This research uses two reading passages for the experiment. Both of the passages are intended for intermediate English learners. The selected passages are composed of about 300 words or 1800 Latin characters without any graphics or images.

Experiment design

The concept mapping tools require a web browser and a network connection to a serving webserver to run. The tool is designed to run natively on modern web browsers with Javascript and HTML5 capabilities. Another requirement was that the computer screen from which the web browser ran needed to have a minimum resolution of 1024 ×768 pixels. Hence, all the tools’ features and the concept mapping canvas can be shown on a single screen without scrolling the window.

In this research, all participants were English teachers or tutors. Most of them were new to the Kit-Build concept map framework and its authoring tool. A pre-task was given to the participants 2 weeks before the experiments to introduce the tool online. All participants were asked to create a concept map based on a pre-made concept map with the online tool. The online tool has the same functionalities as those used in the experiment. A user manual was also given as a guide for them to try and familiarize themselves with the tool.

All participants necessarily prepared their computers based on the specified requirements to minimize problems during experiments. They were requested to connect to the Internet, install an updated web browser, and enable the browser’s Javascript processing capability. Even though all participants’ computers were connected to the Internet, the experiments were carried out on a local area network.

Before they carried out the experiment activities, the teachers were introduced to the Kit-Build concept map framework and the concept mapping tool used to create a concept map with Kit-Build. There was also a short training session for about one hour, where they created a concept map with the tool. This training session was necessary to discover and solve technical issues with the tool, which they might experience or had encountered previously, before conducting the actual concept mapping activities where measurements were taken into account. A demographic questionnaire was also given to all participants at the end of the training session.

There were two types of tools used in the experiment. The first tool was a concept mapping authoring tool that provides the basic concept map creation functionalities; this tool is further referred to as the Scratch tool (SC). The other concept mapping tool has the same functionality as SC but has additional support and feedback functionalities. The second tool, which has the support function, is referred to as the Supported tool (SP).

This research’s experiment was designed so that all participants experienced both concept mapping methods with the tool, i.e., with or without the support function. However, the order in which they used the tool’s support function may have affected their concept mapping activities and further affected the resulting concept maps. When teachers created their second concept map for a particular text with a different method, they had known the contents, and they had made one concept map from the previous session. A learning effect is presumably affected their second concept maps. Hence, it was necessary to divide the participants into two groups, i.e., group A and group B. The group determined the order in which they used the tools’ support function. However, to ensure that all participants experienced the same order of concept mapping methods, it was necessary to also use another text to create a concept map in the opposite order of concept mapping method. The concept mapping flow of this research is shown in Fig. 4.

Fig. 4

Experiment design

Two English reading texts, which were considered similar in terms of their contents and length, were used to discover whether the texts affected their concept mapping activities, thus affecting the resulting concept maps or not. For the first text, group A began to create the concept map with SC and followed with SP. The other group (group B) began with SP and then followed with SC. For the second text, both groups created a concept map using the tools in the opposite order. Hence, all participants performed the concept mapping activities using the support functions in both different concept mapping orders. In this way, each concept map was made from a different text, with different tools and order.

In the experiment’s concept mapping stage, all participants created concept maps from two English readings, and four concept maps were created in total by each participant. Each concept map had a 25-min time allocation. They read the text, created the concept map, and reviewed their concept map in the 25-min allocated time. During the allocated time they were able to read the text while doing the concept mapping to reduce their load in remembering the text’s contents. A 5-min break was also given between each concept mapping session. Finally, at the end of the experiment, a set of questionnaire was given and short discussion with the teachers about Kit-Build, the Kit-Build concept map authoring tool, and how Kit-Build can benefit learning, was carried out.

Measurement and data analysis

At the end of each concept mapping session, the teachers were requested to finalize and save their concept maps. The metadata of the completed concept maps was recorded in the database, including concept labels, link labels, propositions, and their position over the canvas. Additionally, their activities were logged; hence, the concept map can be reconstructed and shown later for analysis, grading, and evaluation.

This research evaluated the Kit-Build concept map authoring tool based on the teachers’ resulting concept maps. Comparisons with the teachers’ concept maps were carried out to discover whether the tool’s support function affected teachers’ concept mapping efficiency. Three kind of parameters were specified to measure and analyze teachers’ productivity in creating concept maps with the concept mapping tools, i.e., the tool type, the session, and the text. The tool type determines the support function’s availability. The session describes the order of the concept mapping method in which teachers used the support function. They either used the supported tool first, then the scratch tool, or they used the tools in the opposite way. The text parameter represents the texts that teachers used to create the concept maps. Analysis of the resulting concept maps was carried out according to the specified parameters to evaluate the concept mapping tool.

A concept map scoring rubric was used to evaluate and grade the maps. The Mueller’s concept map rubric (Mueller 2017), which has continuous rating scales to measure the legibility, accuracy, completeness, and sophistication of a concept map, was chosen to measure the quality of the concept maps. The rubric does not include concept map size as one of its scoring factors in determining a concept map’s quality. Thus, the overlap with the analysis of concept map size could be avoided.

All created concept maps were rated by three expert teachers who were selected based on their expertise and experience. They needed to have taught English reading comprehension with concept maps for at least 2 years. Before performing the grading process, the raters had a quick overview of all of the resulting concept maps from the experiments. They constructed a rationale of the Mueller’s rubric as objective criteria in grading all the concept maps. The rationale was intended to make the judgment in grading the concept maps among raters consistent. An inter-rater analysis was carried out to the scores before further analysis to evaluate the scores’ reliability among different raters. Some examples of the rationale items, which were made from the Mueller’s concept map rubric of this research, are shown in Table 1.

Table 1 Example of Mueller’s concept map scoring rubric rationale

Results and discussion

Concept mapping efficiency

The number of propositions determines a concept map’s size even though it is arguable whether larger concept maps are able to cover more information and knowledge. Every proposition implies knowledge or information from the learning material that teachers want to convey. Hence, the amount of information and knowledge a concept map contains is represented by the quantity of the propositions. In the experiment, teachers took the same amount of time to create their concept maps in all the concept mapping sessions. Hence, the efficiency of concept mapping activities can be described by the concept maps’ size.

Teachers who participated in the experiments numbered 47 persons; however, five teachers’ data were omitted from analysis because the teachers did not follow the standard procedures, or their data were found to be incomplete or missing. The descriptive statistics of the size of teachers’ concept maps from each concept mapping session are shown in Table 2. The predicted concept maps size is visualized in Fig. 5. The data are grouped based on the texts (T1 and T2), the sessions (S1 and S2), and the tools (Scratch and Supported).

Fig. 5

Predicted concept map size based on the Text, Session, and Tool

Table 2 Descriptive statistics of concept map size

According to Table 2, the average size of a concept map made with the supported tool (SP) is larger than those made with the scratch tool (SC) for all texts and sessions. The gap between the map size of the two different tools becomes more apparent in the sessions that used the second text (T2). Hence, concept mapping activities that used SP yielded larger concept maps.

A comparison analysis was carried out to discover the variables that affected the size of a concept map. This research’s experiment specifies three independent variables, i.e., the tool, the text, and the concept mapping session, for analysis. The tool variable represents whether teachers use the support function (with SP) or not (with SC) during concept mapping sessions. The session variable represents the order in which teachers used the tools to create concept maps.

The experimental data conformed to Levene’s homogeneity test. On the contrary, some proposition data violated Shapiro-Wilk normality tests, thus failing to conform to classical statistical assumptions. Therefore, in addressing the first research question, whether the developed concept map support tool could improve teachers’ concept mapping efficiency in terms of the yielded number of propositions, analysis was carried out with the non-parametric Generalized Linear Model (GLM) instead of using three-way analysis of variance (ANOVA) to discover factors that influence a concept map size. The GLM analysis result of the experiment data is shown in Table 3.

Table 3 Generalized linear model analysis result

According to the GLM analysis results, both tool and session variables significantly affected a concept map size. On the contrary, the text had no significant influence on concept map size. Additionally, interaction between tool and text variables was shown to affect concept map size significantly. To clarify the GLM analysis result, a non-parametric Kruskal-Wallis test was used to evaluate the group data individually. The individual Kruskal-Wallis tests described which factors influence the concept map size and which situations. The concept map size from two different tools and two different texts was compared to clarify the interaction between tool and the text. The result of the individual Kruskal-Wallis analysis with a significance level (α) of 0.05 is shown in Table 4. The predicted concept map size with tool as the predictor and the sessions combined is shown in Fig. 6.

Fig. 6

Concept map size to Tool and Text (with Session combined)

Table 4 Kruskal-Wallis rank sum test results

According to the Kruskal-Wallis test result shown in Table 4, it can be clarified that the tool significantly affected the concept map size in all sessions. The session was found to significantly affect concept map size made for Text 1 (T1). On the contrary, the session insignificantly affected the concept map size for Text 2 (T2). In other words, the order in which teachers used the tool affected their concept maps size only for T1. However, when the text was analyzed as the predictor, it only influenced the concept map size when they first created their concept map with SP.

According to Fig. 6, the concept maps made with SP were larger than the concept maps made with SC. When teachers used SC, there was only a slight difference in the concept map size between T1 and T2. To show the interaction between tool and text that influenced the concept map size, the concept map size between two different tools and two different texts were compared with two Mann-Whitney U tests. The result of the Mann-Whitney U test of the number of propositions between T1 and T2 for SC was not significant (p value = 0.34). On the contrary, when teachers used SP, the concept maps made for T1 were smaller than T2 to some extent. The result of the Mann-Whitney U test of the number of propositions between T1 and T2 when they use SP was significant (p value = 0.0035). While those results could be some special characteristic of T1 when processed by the support function of SP, the difference in said Mann-Whitney U tests explains the interaction between tool and text that influence the concept map size. Changes in the text only affected the concept map size when they use SP but not SC.

One plausible reason that explains the interaction was the teachers might have experienced problems in using the tool as they might still be learning to fully take advantage of SP when they first created their concept maps of T1. The difficulties in using SP to compose concept maps were also reported by another study that evaluated students’ experience with Kit-Build concept map authoring tools. To some extent, the supported tool is more difficult to use than the scratch tool (Pinandito et al. 2019). Hence, the gap between concept map size made with both tools on S1 of T1 shows a lesser difference than the other sessions.

Even though both T1 and T2 were considered similar, the concept map size might differ when teachers created concept maps with SP. Teachers’ unfamiliarity with the support function might also have resulted in the smaller concept map size made with SP. However, according to the data, this phenomenon was happening when teachers used the supported tool to create a concept map for the first time, i.e., in S1. Overall, the results show that concept mapping with SP is more efficient than SC in yielding larger concept maps.

Quality aspect

To address the second research question, whether using the developed concept map support tool could yield concept maps of better quality than a manually composed concept map, three expert raters graded the concept maps with a rationale of Mueller’s concept map scoring rubric. The descriptive statistics of the concept maps’ score and the inter-rater agreement analysis results of the scoring of all three raters are shown in Table 5. According to the extended scale interpretation of Kappa categories to Kendall’s coefficient of concordance W (Levitan et al. 2008), raters’ concordance in scoring the concept maps was not always strong, but also substantial (Kendall W >0.60). The Levene’s test, which evaluated the homogeneity of variance, showed a non-significant result (p value = 0.2245 > 0.05). The Shapiro-Wilk’s normality test of each group setting also showed non-significant results (p values > 0.05). Therefore, the Levene’s and Shapiro-Wilk’s test results to the map score data indicated that the concept maps’ score data conformed with the classical homogeneity and normality assumptions.

Table 5 Descriptive statistics of concept map score

According to Tables 5 and 6, the average score of concept maps made with SP was slightly better than concept maps made with SC. The visual comparison between concept map scores made with SP and SC for each experiment setting is shown in Fig. 7. The star annotation over the plot denotes its significance level of comparison analysis between scores for concept maps that were made with SC and SP with independent T-tests. The result of the T tests is shown in Table 6. Even though SC and SP’s average map scores were statistically different in some situations, the differences were small and they were at the same category level of quality. For instance, when the teachers created their concept maps of T2 on S2, the average score of concept maps when they used SC and SP was 15.24 and 15.75, respectively. Even though the standard deviation is low enough for the results to be significant, the score difference between concept maps that were made with SC and SP were not so interesting as a learning achievement. The results implied that the concept maps at a similar level of quality. Furthermore, it can be said that creating concept maps with SP can result in concept maps of a similar quality with SC.

Fig. 7

Predicted concept map score based on Text, Session, and Tool

Table 6 Independent T test result of concept map score between scratch tool and supported tool

Teachers’ response to the authoring support tool

To address the third research question regarding teachers’ perceived usefulness to the concept map authoring support tool to create concept maps of English reading materials for teaching, a set of questionnaire was given to the teachers. They responded to the questionnaire after they had finished all the concept mapping activities. Part of the questionnaire, which captured their perceived usefulness to the authoring support tool, was adapted from the questionnaire of Technology Acceptance Model (TAM). The questionnaire consisted of five 7-Likert scale questions and the internal consistency (Cronbach’s α) of the questions towards Perceived Usefulness factor was 0.84; indicating good reliability. The summary of their perceived usefulness response to the authoring support tool is shown in Fig. 8. The numbers inside the stacked bar chart denotes the number of teachers who responded their agreement of a particular level to the respective questions.

Fig. 8

Teachers’ perceived usefulness regarding assistance from the authoring support tool

According to the result, there was a small percentage of teachers that think neutrally whether using the authoring support tool is easier than not using it to create a concept map. Small numbers of teachers thought neutrally if using the support tool would increase their productivity in creating a concept map. Even though several teachers did express their disagreement whether using the tool would improve their performance in creating a concept map, the overall impression was positive. Most of the teachers perceived that the authoring support tool was useful in assisting the concept map composition activity of English reading material.

Limitation and future work

The impact of teacher experience and personality influencing the quality of concept maps for Kit-Build has not been investigated, and it will be a target of future research. Teachers who are not familiar with concept maps and do not use concept maps in their teaching might have difficulties in creating concept maps and fail; hence, the limitation of use of the tool. However, based on the authoring support method of this study, a support tool for beginner teachers to compose a concept map will be developed as a new future work.

This research took the EFL reading comprehension as its learning context. It uses two similar EFL reading comprehension passages as input. However, actual passages of a general EFL reading comprehension may vary. It cannot be said that teachers’ concept mapping ability improved when they used the supported tool to create concept maps from English readings. Nevertheless, the trial data analysis shows the tool’s potential to improve teachers’ productivity in concept mapping. Additionally, the tool’s support function can process any English texts even though some limitations exist to get a satisfactory result and the effect of the authoring support tool on teachers purpose and subjectivity is a target for future work of this study.

An analysis regarding the tool’s suggestions should be conducted in the future. Evaluating the quality of the suggestions could discover how the suggestions affected teachers’ concept mapping activities, thus discovering how they relate to the concept maps. The support function performance analysis may contribute towards the study in automatic concept map generation and improve the user experience of computer-enabled concept mapping activities.

Conclusion

This research extends the existing Kit-Build concept map tool’s functionality with an authoring support function that assists teachers in composing concept maps from English readings. The authoring support tool semi-automatically assists the concept map composition activity by adapting Natural Language Processing and text mining techniques to extract concepts and propositions and further suggests the extracted concepts and propositions to be incorporated as part of teachers’ concept maps.

According to the experiment results and analysis, larger concept maps can be attained when teachers use the supported tool. Furthermore, the attained concept maps quality remains at the same level; having a slightly better quality. Positive responses were received; ascertaining the concept map authoring support tool useful in providing assistance to compose concept maps of English reading material for teaching. Therefore, it can be concluded that the concept map authoring support tool can improve teachers’ concept mapping efficiency, retains the concept map quality at a similar level, and be perceived as useful.