- Open Access
Analysis of quality of knowledge structure and students’ perceptions in extension concept mapping
Research and Practice in Technology Enhanced Learning volume 17, Article number: 14 (2022)
Extension concept mapping is a technique to connect prior existing concept maps with new knowledge structures. It offers advantages in each stage of the knowledge-integrating process and encourages learners to improve their performance. While previous studies have confirmed that the extended kit-build concept map outperformed the extended scratch-build approach in terms of comprehension test scores and map size, they have yet to evaluate the quality of concept maps and students' perceptions. Although the size of the concept map components could represent the breadth of personal knowledge, it does not constantly describe the good knowledge structure. In addition, the student's degree of acceptance after the concept mapping demonstrates their intention to use systems in the future. The present study aims to compare the effect of extended scratch-build and extended kit-build on the students' quality of knowledge structures and perceptions. Fifty-five second-year university students were involved and divided into two groups: control and experimental. The control group utilized the extended scratch-build map, while the experimental group used the extended kit-build concept mapping tool. Quality of propositions and structural map scores as learning outcomes were used to measure the students' knowledge structures. The possibility of a relationship between quality scores was expressed using the Spearman correlation. This study involved the Technology Acceptance Model to confirm the students' perceptions of extension concept mapping tools. The perceived ease-of-use, perceived usefulness, and behavioral intention constructs were used to investigate users' intentions. The findings suggest that the quality of propositions and structural map scores in the experimental group were significantly higher than in the control group. This study also found that the extended kit-build method achieved better perceptions scores than the extended scratch-build.
Concept maps are effective student-centered learning approaches that facilitate critical thinking (Novak & Cañas, 2008), creative productions (Novak, 2010), quality products development (Kinchin et al., 2019), and complex problem solving (Kim, 2013), which are the top skills needed in the 21st century digital skills (Valtonen et al., 2021; Voogt & Roblin, 2012). The concept map represents an individual's understanding of a particular topic visually. Hence, it can measure important aspects of the learners' knowledge structure (Ruiz-Primo, 2004). Concept maps show different ideas at nodes with linking words connecting two concepts and indicating their relationship, thus forming a proposition (Cañas & Novak, 2014). Propositions are the smallest semantic units and are fundamental elements in the concept map. Therefore, they can be perceived as knowledge components (Aleven et al., 2016), representing the unit's declarative knowledge to form meaningful statements. Therefore, many studies (Pailai et al., 2017; Pinandito et al., 2021; Sadita et al., 2020; Vanides et al., 2005) have evaluated concept maps based on propositions, both in quantity and quality.
There are two concept map construction styles: open-ended and closed-ended maps (Herl et al., 1999; Taricani and Clariana, 2006; Hirashima, 2019). Open-ended fashion provides no components; learners may use any concepts and linking words in their diagrams. This technique is suitable for reflecting the difference between learners' knowledge structures (McClure et al., 1999; Ruiz-Primo et al., 2001; Vanides et al., 2005; Taricani & Clariana, 2006) and helps them recall basic terms in learning (Clariana, 2010). However, it is challenging to assess (Taricani & Clarina, 2006) and provides feedback to learners in the open-ended approach. In contrast, a closed-ended concept mapping style provides finite concepts, connecting lines, and linking phrases. In this case, learners must use the provided components to construct their maps by connecting one idea to another. The closed-ended style could provide automatic assessment, furnish feedback swiftly (Hirashima, 2019; Pailai et al., 2017), and encourage learners to reach maximum learning understanding (Pinandito et al., 2021; Ruiz-Primo et al., 2001). However, the closed-ended technique is complicated for the teacher to capture the learners' knowledge structure.
Concept maps have been widely used and proven to provide positive results in facilitating meaningful learning. Meaningful learning is a process of linking new information to relevant previous knowledge in a cognitive structure (Ausubel, 1968). In practice, meaningful learning is facilitated when learners create concept maps by linking one concept to another (Cañas & Novak, 2014). Meaningful learning engages learners in building knowledge and cognitive processes needed to transfer previously acquired knowledge in new problems and situations. Previous studies (Foley et al., 2018; Schwendimann & Linn, 2016) suggested the design of the extended concept mapping to facilitate enhanced meaningful learning and improve students' knowledge structures. The expansion of the concept map engages the learners in being more active in the knowledge-integrating stage, which is not perceived in the usual concept mapping.
An initial study proposed the Extended Kit-Build (EKB) concept mapping approach to facilitate students in expressing their understanding (Prasetya et al., 2019). The EKB employs a closed-ended and recomposition Kit-Build (KB) framework and integrates the open-ended technique into a single extension concept mapping activity. Recomposition is an essential learning activity that encourages learners to understand the teacher's understanding through map reconstruction. KB proves that the recomposition concept mapping enables learners to transfer previously acquired knowledge in new problems and situations (Hirashima, 2019). The initial study contributed to revealing that the EKB approach achieved higher comprehension scores and map size compared to the Extended Scratch-Build (ESB) method. ESB is a technique for extending an open-ended concept map using the same approach. However, a large map size does not always represent a good concept map (Cañas et al., 2013; Kinchin, 2016). Ruiz-Primo et al. (1997) argued that the quality of interrelationships between concepts is mainly emphasized by weighting the correctness of propositional statements. The previous study also did not provide information about the structural concept map scores. In addition, the students' perceptions of extension concept mapping have also not been revealed. Hwang et al. (2013) suggested that good students' acceptance degrees in concept mapping activities could help them understand the learning content.
The present study focuses on analyzing the effects of different extension concept mapping tools on students' knowledge structures and perceptions. Students' knowledge structure was assessed using the quality of propositions and structural map scores. This study involved the Technology Acceptance Model (TAM) to confirm the students' perceptions on the extension concept mapping designs. Students' perceptions are particularly important in determining their intention to use technology systems in the future (Grandón et al., 2021). The perceived ease-of-use (PEOU), perceived usefulness (PU), and behavioral intention (BI) constructs were used to investigate users' intentions. The following research questions guide this study:
Does the concept map created using the EKB approach have better structural map scores than the ESB method?
To what extent is the quality of propositions of the EKB concept map compared to the ESB?
Is there a relationship between the structural map scores and the quality of propositions?
How are the students' perceptions regarding the EKB and ESB extension concept mapping tools measured using the TAM model?
Extension concept mapping
Extension concept mapping is a technique to expand the existing concept map by connecting it with a new relevant knowledge structure (Foley et al., 2018; Prasetya et al., 2021; Schwendimann & Linn, 2016). The extension map construction efficiently organizes design work and builds a solid knowledge base (Foley et al., 2018). Moreover, it plays an essential task in each stage of the knowledge-integrating process through reviewing initial ideas and connections, eliciting missing ideas and relationships, and adding new concepts and relationships (Schwendimann & Linn, 2016). Thus, the connected concept mapping approach provides opportunities for individuals to simplify problems and improve knowledge structures. It also facilitates learners to improve the previous map and associate with further related information in embodying enhanced meaningful learning.
Previous research on extension concept mapping posits that this activity could encourage learners to improve their achievement. Schwendimann and Linn (2016) confirmed that the original map expanded through two forms of collaborative critique and revision activity could increase and construct a more coherent knowledge structure. The opportunity engages the learners actively in each phase of the knowledge-integrating process, which is not perceived in the usual concept mapping. Foley et al. (2018) found that the Cogex extended map was more efficient in organizing design work than the paper logbook.
Another extension concept mapping work suggested ESB facilitates improved meaningful learning (Prasetya et al., 2021). ESB uses the open-ended technique in building the original map and extends it to generate the additional map. The concept mapping activity in ESB consists of two parts: Phase 1 and Phase 2. In Phase 1, learners are requested to create a concept map by referring to the first or original material. Then, as the usual open-ended method, learners are allowed to add any concepts and links to express their understanding. Furthermore, in Phase 2, learners are asked to expand their original concept map by adding new components to produce the additional map. The ESB approach is proven to improve students' comprehension as measured by pretest and post-test scores (Prasetya et al., 2020).
The current study investigated the EKB to facilitate students in expressing their understanding while keeping their comprehension (Prasetya et al., 2019). Both EKB and ESB are extension concept mappings that allow learners to expand their concept maps. EKB activities also consist of two parts: Phase 1 and Phase 2. In Phase 1, instead of using the open-ended technique as in ESB, EKB employs the KB framework. Hence, learners are requested to construct the original concept map using the KB approach in the first activity. Furthermore, just like ESB, the learners are asked to extend their prior original concept map. EKB was compared with the ESB method to confirm its effect on learning (Prasetya et al., 2021). An exciting result found that the average map size of the ESB students on the original map was slightly larger than the EKB students who were given limited concepts and links. However, in the additional map, the achievement of ESB's students decreased drastically, while EKB's students remained consistent with high achievement. While the previous study investigated the effects of students' understanding and map size on the ESB and EKB, they have yet to evaluate structural scores, quality of map scores, and students' perceptions.
Kit-Build concept map
Kit-Build (KB) is a re-compositional concept map, a promising subcategory of a closed-ended concept map (Hirashima, 2019). The KB approach provides nodes and links components decomposed from the teacher's map and asks students to recompose them. The practical uses of KB map can be described in four main phases (Hirashima et al., 2015): (1) a teacher creates a concept map as a goal map; (2) the KB system deconstructs the goal map into nodes and links called a "kit"; (3) the students are asked to reconstruct the kit; and (4) the learners' maps are assessed by comparing them with the teacher's map. Furthermore, the available components that compare the teacher's and learners' map could be performed automatically (Hirashima et al., 2015), thus enabling teachers to quickly diagnose students' understanding (Pailai et al., 2017).
Recomposition is an essential characteristic that identifies KB (Hirashima et al., 2015; Pinandito et al., 2021) and distinguishes it from other concept mapping frameworks. It describes an activity where learners are directed to understand the teacher's understanding through the goal map reconstruction. With the available kit, the KB approach allows learners to understand knowledge targets in a well-maintained manner. This condition is entirely different from closed-ended concept mapping, which provides components and asks learners to rebuild concept maps according to their understanding. However, the goal map is a concept map designed by experts to facilitate interconnection between relevant information (Hirashima et al., 2015). Therefore, by understanding the teacher's knowledge, learners will obtain a solid basic structure that easily relates to the next topic. In addition, an excellent initial structure also facilitates improved meaningful learning and makes it easier for learners to solve new problems.
Previous KB studies have revealed many positive effects on improving learning outcomes and supporting automatic assessment. Pailai et al. (2017) examined the practical use of the KB map on formative assessment at the elementary school level. The results found that the KB concept map was proven to be effective in increasing learners' understanding. Studies on reading comprehension for English as Foreign Language (EFL) also confirmed that students who used the KB map with the source connection features showed better performance than those who employed the traditional summarization method (Andoko et al., 2020). Pinandito et al. (2021) emphasized that teachers perceived usefulness on semiautomatic concept mapping with the supported KB map measured using TAM's questionnaires. KB is utilized to support individual learning and is also suitable for facilitating collaborative knowledge building (Sadita et al., 2020). Although several new KB functions have been developed and yielded positive effects, none have yet provided opportunities for learners to express their understanding.
Technology acceptance model
User acceptance is considered a pivotal factor pertaining to the success of information and communication technology products (Davis, 1989). The TAM is today the highly frequently adopted model of user acceptance. It addresses why individuals accept or reject information technology when performing a task (Wallace & Sheetz, 2014). David (1989) posits that PEOU and PU are vital factors influencing intentions and actual computer usage. PEOU is an essential factor influencing user acceptance and usage behavior of information technologies (Venkatesh, 2020). Meanwhile, the degree to which a person believes that employing a given system would improve their job performance is defined as PU (Davis, 1989).
Several studies have been conducted to investigate students' perceptions of concept mapping tools. A recent research stated that TAM is suitable for identifying aspects that influence the learners' intention in using the Kit-Build concept map framework (Pinandito et al., 2021). In particular, the authoring support tool has been perceived as ease-of-use and useful in assisting the teachers while concept mapping of English reading material. Furthermore, according to Almulla & Alamri (2021), the conceptual mapping approach for learning could improve the students' academic achievement, interaction, motivation, behavioral intention, perceived utility of cooperative activities, and attitude toward utilizing the tool. However, while several studies have shown students' positive acceptance of the usual concept mapping tools, no information was found that addresses their perceptions of extension concept mapping.
The present study involved 55 s-year university students from two regular classes (A and B). Participants are informatics engineering students in a public university in Malang, Indonesia. Since they do not have a notable age difference (mean age 18.8 years, SD age 0.6 years), then the data on age were not included in the analysis. Participants are accustomed to using personal computers and Internet services to support their learning activities. However, all participants were novices in the practice of concept mapping. A pretest was conducted before determining the role of the group, and the results stated that both classes were homogeneous (p = 0.445 > 0.05). The pretest included database introduction material and was given one week before the experiment was carried out. Class A was randomly assigned as the control group, and class B was the experimental group. The control group consisted of 27 participants (62.96% male; 37.04% female) and the experimental group had 28 participants (67.86% male; 32.14% female).
This study was conducted in the Database 1 course, which was delivered in Indonesian. The lecturer used a presentation and distributed printed handouts during the teaching activity. Figure 1 illustrates the activity of learners during the concept mapping using the ESB approach. Two sequential experiments were conducted with the Relational Database and Structured Query Language (SQL) topics to obtain more reliable results. Corresponding to the two phases of concept mapping design proposed, each topic material was divided into two parts: the original and additional materials. The original part is the material delivered by the teacher in Phase 1 to create the original map. In contrast, the other part is the advanced material provided in Phase 2 to produce the additional map. Original and additional components are on the same subject and were presented in one lecture session.
In the KB approach, the lecturer first defined the teacher's map that would be given to students to be recomposed. Figure 2 depicts the teacher's activity while creating the teacher's map and learners' activity during the concept mapping using the EKB approach. The sequence of activities in the EKB group was the same as the ESB, except for the original map creation used KB rather than open-ended. Original material in the Relational Database topic consists of nine slides (283 words), and the lecturer provided ten propositions. The content of the additional part consists of eight slides (237 words), and there was no teacher's map provided. In the second experiment with the SQL topic, the original material was composed of 13 slides (532 words), and 14 propositions were defined, while the additional material also consisted of 13 slides (829 words). The volume of material on additional material was more congested as it included examples of SQL statements.
The Relational Database and SQL topics provide learners with the skill to understand the basic fundamental concepts and solve SQL problems. To achieve these learning outcomes, the teacher used an extension concept mapping approach. The students were actively engaged to represent their ideas in the form of a concept map. The main indicator used to measure student achievement was through the quality of the resulting concept map. A senior and experienced teacher who had been teaching for 11 years taught both the control and experiment classes.
The present experiment was conducted in a computer laboratory during two lecture sessions. Both the control and experimental group used the same classroom on a different day. A wired connection on a personal computer was used to provide a stable connection to the Internet network. Each student was given an individual account to access the web-based concept map application and assign it to groups that had been determined. Figure 3a shows the situation of experimental activities in the classroom during the students created concept maps using ESB, while Fig. 3b depicts the same activities but using the EKB approach. Before the experiment was conducted, participants in both groups had been given an introduction to concept maps in a previous course meeting. Furthermore, participants were also instructed to build concept maps on the introduction of a database topic.
Figure 4 shows the sequence of learning activities for both groups during the experiment. In Phase 1, the teacher started the activity by delivering the original material for 25 min. The teacher used presentation slides and made use of classroom learning tools as usual. In this situation, students were engaged in learning and actively participated as usual. Furthermore, participants in the control group were asked to create a concept map using an open-ended approach, while the experimental group used a KB approach. The creation of a concept map referring to the original part material was carried out for 15 min. While creating the concept map, learners in both groups were allowed to read printed handouts that had been distributed. The results of the concept maps in these two groups are called "original maps."
Next, for Phase 2, the lecturer continued giving the additional part material for 25 min. After the explanation was complete, all participants continued to create concept maps using the extended open-ended approach. At this stage, they were asked to extend their concept maps and add new concept maps with existing knowledge based on the additional part material. As in the first stage, the concept map extension in the second stage was done in 15 min. After creating the second stage of the concept map, students were asked to upload the final results to the server. The results of the expansion of concept maps in these two groups are referred to as "additional maps." After the second experiment finished, students expressed their perceptions through the questionnaire sheet.
The experimental procedure between control and experimental groups was almost the same and in a comparable design. Both groups used the computer-based concept mapping tool and obtained lecture material from the same lecturer. The only difference between the conditions was constructing the concept map for the first time after the lecturer explained the original material, where the control group used the open-ended approach, while the experimental group used the closed-ended KB concept map. In the next stage, it can be seen that the two groups used the same extension concept map approach, which extended their prior knowledge structure by adding new relevant information.
The students' achievements for both groups were investigated using three measurements: (1) structural map scores; (2) quality of propositions; and (3) students' perceptions. A structural map score is a measurement that calculates the essential concept map components and provides a certain weight. To corroborate the quantity finding in the previous study, the measurement was involved that considers the quality of a proposition using a rubric. A concept map with well-structural and great content quality is categorized as a good map (Cañas et al., 2015). Finally, students' perceptions were involved in determining the users' intention of extension concept mapping as a technology product.
Novak & Gowin's scoring method (1984) was adopted to measure the concept map structure. The structural quality of a concept map is one of the criteria for a good concept map (Cañas et al., 2015). Four main components were used to give a score: propositions, hierarchies, cross-links, and examples. For each valid part, the following points were given: 1 point per proposition, 5 points per hierarchy, 10 points per cross-link, and 1 point per example. This scoring method shows that the most critical component for evaluating concept map structure is cross-links. The cross-links represent the meaningful relationships among concepts in different map segments (Novak & Cañas, 2008). Assessment of concept maps by examining the number of ideas, focal concepts, propositions, linkages, and hierarchies can show the quality of knowledge and depth of understanding (Kirschner, 2004). This study evaluated all components in the original and additional maps separately, and then they would be compared to find out the difference for both groups.
The second students' knowledge structure assessment was evaluated using the quality of proposition scores. The existence of propositions in the concept map represents students' knowledge and understanding of a particular topic. However, relationships are still not clearly defined regarding whether they are reasonable or scientifically meaningful (Vanides et al., 2005). The quality of propositions is one of the most critical and recommended judgments in the concept mapping assessment (Chen et al., 2021; Raud et al., 2016; Reiska et al., 2018) and states the quality of personal knowledge. The quality of propositions scoring method proposed by Vanides et al. (2005) was adopted to examine the students' knowledge and understanding. Four level scoring were formulated: 0 = incorrect; 1 = partially incorrect; 2 = correct with thin scientific understanding; and 3 = scientifically correct. Similar to the previous calculation, the quality measurements evaluated the original and additional maps in the two groups separately, then analyzed and compared their differences. The map quality measurements on both groups were judged manually by the same lecturer.
The present study investigated the effect of the extension concept mapping tool on students' perceptions using the Technology Acceptance Model (TAM). Perceived ease-of-use (PEOU), perceived usefulness (PU), and behavioral intention (BI) scales were involved in describing users' perceptions after using the extension concept mapping tool. PEOU is an essential factor influencing user acceptance and usage behavior of information technologies (Venkatesh, 2020). Meanwhile, PU is defined as the prospective user's subjective probability that using a specific application system will increase their job performance within an organizational context (Davis, 1989). A six-point Likert scale: 1 “strongly disagree,” 2 “disagree,” 3 “somewhat disagree,” 4 “somewhat agree,” 5 “agree,” and 6 “strongly agree” has been chosen to collect research data. The TAM questionnaire regarding the extension concept mapping activity consisted of 3 PEOU items, 3 PU items, and 1 BI. All measurements of the constructs were adapted from previous literature (Davis, 1989; Davis et al., 1989; Pinandito et al., 2021) to ensure survey content validity.
The dataset for this study was mainly collected from 55 participants who were divided into two groups: control and experimental. The analysis was carried out on the structural map scores, quality of propositions, and TAM questionnaire. The normality distribution and homogeneity of variance were examined to determine whether the data could be analyzed using a parametric test. The normality distribution of data was tested using a Shapiro–Wilk test. The present study used the R programming language to perform the statistical analysis.
Descriptive statistics were presented using the arithmetic minimum, maximum, mean, median, and standard deviation. Data were not normally distributed (p = 0.037 < 0.05); therefore, nonparametric statistical tests were used to analyze study data. A Mann–Whitney U test was used to evaluate the significant difference between the performance of the control and experimental groups. Pearson's r was also used as the effect size (ES) metric to examine the correlation coefficient. According to Cohen (1988), correlation coefficients can be stated as small (0.10), medium (0.30), and large (0.50) in terms of the magnitude of the effect. The Spearman's order correlation coefficient (rs) was also applied to reveal the univariate associations between the achievements. In all these analyses results, a p-value < 0.05 was declared statistically significant.
Analysis of structural map scores
Analysis of structural concept map scores was conducted on the original map and the additional map separately. The four main components of the concept map were used to represent the students' concept map structure scores: valid propositions (VP), valid hierarchies (VH), valid cross-links (VCL), and valid examples (VE). Table 1 shows the descriptive statistics of the structural map scores for both groups in the first and second experiments for the original map.
The results in the first experiment found that both groups could define VP and VH well, while neither did express VCL and VE. Although the teacher can determine VCL and VE on the EKB, she did not provide it to preserve the naturalness of the experiment. The mean VP of the control group was higher than that of the experimental group, which was given limitations through the kit. However, the average scores for the original map achieved by the experimental group in the first experiment were slightly higher than the control group. The experimental group that used KB's kit showed that students' achievement tends to be equal with a low standard deviation. Different conditions occurred for those who used ESB, where some obtained low scores and a few others were high enough, causing a higher standard deviation.
In the second experiment, both groups were seen to define only VP and VH components. As in the first experiment, the teacher did not specify any VCL on the KB map. The average achievement of structural map scores for the experimental group was also higher than that of the control group. Based on descriptive statistics, it can be seen that the achievements of the two groups on the original map were almost equal.
Students' structural map performance on the original map between the control and experimental groups was further analyzed using the Mann–Whitney U test. The results in the first experiment reported no significant differences between the control and experimental groups (Z = − 1.439; p = 0.152 > 0.05), with Pearson's r of -0.193, indicating a small effect size. The mean rank of the experimental group in the first experiment was slightly higher than the control group. The second experiment's results reported significant differences between the control and experimental groups (Z = -2.019; p = 0.044 < 0.05), with Pearson's r of -0.272, showing a small effect size. The average rank of the quality of the map structure for the experimental group in the second experiment was consistently higher than the control group.
The same structural map calculations were carried out on the additional concept map, which resulted from expanding activities. Descriptive statistics of the achievement of additional maps are shown in Table 2. In the first experiment, the students in the experimental group could define all assessment components, while the control group still only restricted the VP and VH components. Nine students in the experimental group could find cross-links, with the average VCL score being 1.07. Regarding the VE component, 15 EKB students created examples of their concept with a group average of 3.29. The most striking thing was that the VP achievement in the experimental group was significant compared to the control group. The difference in the accomplishment of VP, VCL, and VE scores was very pronounced, causing the mean structural map scores in the experimental group to be higher than the control group.
Interesting results were found in the second experiment, where the control group defined all components of the structural map scores. On the other hand, no learners in the experimental group expressed the VE component. However, the achievement of scores for other measurement components was dominated by the experimental group. Thus, the experimental group's performance remained constant and outperformed the control group, 30.00 and 16.44, respectively.
The Mann–Whitney U test determined the difference in structural map scores between both groups for the additional concept map. The results in the first experiment reported significant differences between the control and experimental groups (Z = − 5.413; p = 0.000 < 0.05), with Pearson's r of − 0.729, indicating a large effect size. The average rank of the control group in the first experiment was 8.85, while the students in the experimental group had 30.36. Similar results were found in the second experiment, although it involved a different topic. Students in the experimental group who used the EKB consistently achieved higher mean scores than the control group. There was a significant difference between the two groups (Z = − 4.020; p = 0.001 < 0.05), with Pearson’s r of − 0.542, indicating a large effect size.
Analysis of quality of propositions
In contrast to the quantitative analysis, which examines each existing component with the same weight, the calculation of the quality of propositions considers the scientifically meaningful level. Thus, each valid proposition will probably have a different weight based on the scoring rubric. The assessment was based on propositions that describe the smallest linguistic or semantic units and are fundamental elements in the concept map. Table 3 shows the descriptive statistics of the quality of propositions scores in the first and second experiments for the original map.
The previous analysis stated that the achievement of the average number of propositions for the control group's original map in the first experiment was superior to that of the experimental group. However, interesting findings were revealed in the analysis of the quality of propositions. Although the mean number of propositions of the experimental group was lower, its quality score was almost the same as that of the control group, 39.43 and 39.85, respectively. In the second experiment, it was found that the average proposition achievement of the experimental group was slightly higher than the control group. This condition constantly resulted in the average quality of propositions score of the experimental group being superior to the control group.
Further analysis was conducted using the Mann–Whitney U test to determine the differences in the quality of propositions on the original map between the control and experimental groups. The results in the first experiment indicated there were no significant differences (Z = − 1.038; p = 0.299 > 0.05) between both groups, with Pearson's r of -0.140, showing a small effect size. The mean rank of the control group was almost the same as the achievement of the experimental group. For the second experiment, the results indicated considerable differences between the quality of propositions of the original map for the control and experimental groups (Z = − 2,255; p = 0.024 < 0.05), with Pearson's r of − 0.304, showing a medium effect size. Again, the mean rank of the experimental group was higher than that of the control group. The Mann–Whitney statistical analysis results stated significant differences between interventions using the open-ended and recomposition KB methods.
The same quality of proposition calculations was carried out on the additional concept map. Descriptive statistics of the quality of propositions for additional maps are shown in Table 4. Unlike the previous original map, where the control group achieved maximum quality scores, the experimental group was consistently superior in the additional map. Also, the experimental group outperformed the control group in both experiments in terms of the mean value.
The Mann–Whitney U test was further used for statistical measurements of the additional map results between the control and experimental groups. In line with the descriptive analysis results, it appears that the experimental group outperformed the control group in both experiments. Based on a p-value threshold of 0.05, there were significant differences between the control and experimental groups in the first experiment (Z = − 6.221; p = 0.000 < 0.05), with Pearson's r of 0.839, showing a large effect size. Furthermore, the mean rank of the quality of propositions of the control group in the first experiment was 10.22, while students in the experimental group achieved a higher score of 30.46.
The quality of students' propositions in the second experiment was evaluated using the same procedures and measurements. The second experiment's results also showed a persistent condition. There were significant differences between the control and experimental groups (Z = − 5.770; p = 0.000 < 0.05), with Pearson’s r of − 0.778, also indicating a large effect size. The average rank of the quality of propositions of the control group in the second experiment was 22.70, while the students in the experimental group attained 47.89.
Analysis of the correlation between quality scores
The structural map scores and quality of propositions were further analyzed statistically to determine the extent of the relationship. Correlation analysis was focused on the additional concept map, which realizes the expansion of the concept map. However, both groups used the same approach to extend their previous original concept map. Table 5 shows the results of the correlation coefficient on the first experiment. The study found a positive correlation between the structural map scores and the quality of propositions in both groups. Results in the control group indicated a moderate positive correlation with a p-value < 0.005, while the experimental group demonstrated a strong positive correlation with a p-value < 0.001.
Furthermore, different results were found in the control group in the second experiment, as shown in Table 6. The results in the control group showed a weak positive correlation between structural map scores and quality of propositions with a p-value 0.077 > 0.005. Meanwhile, the experimental group's achievement was consistent with a positive correlation with a p-value < 0.001.
Analysis of students' perceptions
Using the TAM model, this study validated the students' perceptions to determine the extent to which they intend to use the extension concept mapping tool. Table 7 shows the descriptive statistics of the PEOU, PU, and BI for both groups. TAM questionnaire results found higher scores mean for PEOU, PU, and BU for the experimental group. The experimental group's average PEOU, PU, and BI were 5.73, 5.71, and 5.82, respectively. This achievement was at the level of agreement approaching "6: strongly agree." Meanwhile, the control group's average PEOU, PU, and BI were 5.14, 5.33, and 5.59, respectively. Thus, the PEOU and BI achievement in the control group tended to be at the level of agreement "5: agree," while the BI score was closer to "6: strongly agree."
The internal consistency reliability of the instruments was assessed using Cronbach's alpha coefficient. In general, a score of 0.7 is considered an acceptable reliability threshold that should be achieved. The analysis results showed that each construct in both groups has a Cronbach's alpha greater than 0.7. The average PEOU and PU in the control group were 0.819 and 0.806, respectively, while the experimental were 0.887 and 0.868, respectively. Thus, all Cronbach's alpha values showed good internal consistency. The attainment indicates that the items consistently measure the same latent variable and are therefore reliable.
Although the students' perceptions of the two extension concept mapping tools were considered in a high level of agreement, it can be further investigated to identify their differences. Therefore, the Mann–Whitney U test was employed to evaluate the difference between the PEOU, PU, and BI subscales in the two groups. Table 8 shows the results of the TAM constructs comparison for both groups.
The PEOU construct results revealed a significant difference between the control and experimental groups with Pearson's r of -0.594, indicating a large effect size. The perception of the PU subscale still shows consistency stating a significant difference between the two groups with Pearson's r of -0.458, showing a medium effect size. A different condition occurred in the BI construct, where the two groups both gave high responses, which indicated an intention to use each tool in the future. Hence, there was no significant difference in BI perceptions between the control and experimental group with Pearson's r of -0.250, indicating a small effect size.
The effect of extension concept mapping on structural map scores
Referring to Table 1, the original map achievements on the two groups coincidentally have the same conditions. Both of them defined VP and VH, and neither of them expressed VCL and VE. Interestingly, although the mean of propositions in the control group was higher, the structural map score was slightly lower than that of the experimental group. In the second experiment, the VP achievements of the two groups tended to be almost the same, but the experimental group achieved a higher structural map score. The visual comparison between the achievement of ESB and EKB for the original map is shown in Fig. 5.
The main cause affecting the achievement of the original map for both groups is seen in the VH component. In the experimental group, students recompose the kit provided by the teacher. Previously, the teacher has designed a teacher's map containing essential concepts and a hierarchical structure. In line with previous studies, the KB kits have an important role in providing a solid knowledge structure (Hirashima et al., 2015; Pailai et al., 2017; Pinandito et al., 2021). It is proven that students who use the EKB approach are consistently able to recompose the concept map with a better structural map score. On the other hand, although the students in the control group were allowed to add as many components as possible, they could not produce a concept map with an optimal structure.
Both the ESB and EKB approaches facilitate learners to expand on their previous original concept maps. This is shown by expanding the concept map, which is equally able to produce additional maps. Although the two groups in Phase 2 utilized the same approach, the mean structural map scores for the EKB group were higher than that of the ESB group. Figure 6 shows structural map scores for the additional maps for both experiments. Extension concept mapping encourages learners to link initial ideas in solving new problems. This finding is in line with the opinion of several previous researchers (Schwendiman & Linn, 2016; Foley et al., 2018). The EKB approach that asks learners to expand the KB map produces a concept map with a better structure than the ESB method.
The present study found the basic structure of a concept map that has been improved with new ideas is proven to produce complex concept map propositions, gain a deep hierarchical level, and reveal the interconnection between concepts in different sub-segments (cross-links). The EKB approach encourages learners to express more cross-links than the ESB method. According to Novak and Cañas (2008), cross-links are the most critical feature in facilitating creative thinking. Thus, the first and second experiments' results showed that the quality of structural map scores representing students' creative thinking in the experimental group was superior to that of the control group.
The initial study emphasized that students who used the EKB approach could maintain their achievement on the original and additional maps (Prasetya et al., 2021). This condition can be seen in the design of the extension concept mapping, which asks students to link the original map and the additional map. As a result, EKB students tend to recompose the teacher's map optimally and consistently expand the concept map with better structural scores than ESB students.
The effects of extension concept mapping on quality of proposition scores
The current findings highlighted that the EKB design outperformed the ESB design regarding the quality of propositions scores. In constructing the original map in the first experiment, even though the average number of propositions for the EKB group was lower than that of the ESB, the quality score was almost the same as the achievement of the ESB group. Similarly, in the second experiment, when the number of EKB propositions was more remarkable, the proposition's quality score significantly outperformed the ESB group. The visual comparison between the quality of proposition scores of ESB and EKB for the original map is shown in Fig. 7.
In completing the original map, KB's kit plays a vital role in being the leading cause affecting the achievement of the EKB group score. Even though it used a limited number of concept map components, the recomposition activity could produce meaningful propositions with optimal value weights. A different condition occurred in the ESB group, where the number of proposition achievements did not represent the quality of the knowledge structure. Previous studies emphasized that the kit is an important learning activity that encourages students to increase their achievement (Hirashima, 2019; Pailai et al., 2017; Pinandito et al., 2021).
The conspicuous difference was increasingly apparent in the completion of the additional map. Based on the kit recomposition in the original map, EKB students could consistently add more relevant ideas. Thus, the quality score achievement of the experimental group using EKB was higher than the control group. The visual comparison between the quality of propositions scores of ESB and EKB groups for the additional map is shown in Fig. 8.
This finding is in line with the opinion of Foley et al. (2018) that the extension design on EKB can build a solid knowledge structure. In Phase 1, EKB students recomposed the basic knowledge structure according to the teacher's map. Next, in Phase 2, students were encouraged to connect the prior concept map properly with the new additional map. As a result, the quality of propositions of the EKB group in both experiments significantly outperformed the ESB group. Thus, the combination of recomposition and open-ended techniques on EKB offers more excellent enhanced meaningful learning than the extension open-ended method. As Boffey et al. (2010) argued, improved meaningful learning is expressed by linking preexisting maps to new related knowledge.
In Phase 1, students in the experimental group who utilized KB recomposed the concept map related to learning material that the teacher had formulated. As a person with a mastery of the topic, the teacher defined key concepts to guide students to reconstruct them correctly. The concept map recomposition approach not simply facilitates meaningful learning but also encourages individuals to create an integrated and scientifically appropriate relationship between concepts. As Novak (2008) argued, meaningful learning requires individuals to have a well-organized, relevant knowledge, and solid understanding to connect new information with existing knowledge.
In Phase 2, both the groups associated their original map with the additional map using the same open-ended technique. Although students in the control group expanded using the same approach, their previous original maps were less well-structured. This situation greatly affected the results of the concept map expansion. The first and second experiments' results showed consistently that the quality of propositions scores of students in the experimental group was much higher than those of students in the control group. This finding indicated that the expansion of the recomposition KB map significantly impacted group quality of propositions scores.
The correlation between quality scores
There was a strong positive correlation between structural map scores and the quality of propositions for the additional map in the experimental group that used the EKB approach. On the other hand, the control group that used the ESB techniques reported inconsistent results and correlated the two quality scores less. However, structural map scores and quality of propositions describe an individual's depth of understanding and quality of knowledge structure (Kirschner et al., 2004). Thus, the achievement of the EKB map illustrates a good and ideal concept map. Schwendimann and Linn (2016) stated that map expansion could construct a more coherent knowledge structure.
These results suggest that the teacher's map recomposition approach influenced the experimental group's strong correlation between quality scores. However, the teacher's map has been well-designed to cover essential concepts related to material topics. Moreover, the teacher's map also contains selected propositions that have a high-quality score. Therefore, when learners could recompose the concept map according to the teacher's knowledge, they would achieve the maximum quality of propositions.
In Phase 2, the two groups used the same open-ended approach while expanding the concept map. However, students in the experimental group were more adept at adding concepts, links, and forming meaningful propositions. In adding new knowledge, students intuitively follow the teacher's patterns and thoughts. It was proven by the additional map achievement that consistently has high scores and has a strong relationship between structural map scores and quality of propositions.
The students' perceptions on extension concept mapping
Hwang et al. (2013) revealed that the standard concept map approach was felt to be easy to use and helped learners understand the learning material. The present study investigated students' perceptions of ease-of-use, usefulness, and behavioral intention of two extension concept mapping tools using TAM variables. Thus, the questionnaire items focused on perceptions related to concept map expansion activities. The results indicated that participants in both groups positively perceived all TAM subscales with a high level of agreement. However, the students' perceptions of the EKB were seen to be superior to the ESB approach. This finding emphasized that extension concept mapping, particularly EKB, not only provides positive perceptions as measured using the TAM model (Pinandito et al., 2021), but also produces a better quality knowledge structure than ESB.
EKB and ESB are extension concept mapping tools with different activities in Phase 1 but are precisely the same in Phase 2. Although both participants were given the same expansion activities, the perceptions of EKB students toward PEOU, PU, and BI aspects were higher than in ESB students. The concept mapping design in Phase 1 was appraised to have an important role in expanding the concept map. ESB students who used the open-ended technique in Phase 1 perceived that the ease-of-use and usefulness aspects were less than optimal when expanding the concept map. A different condition was found in the experimental group that used the EKB system. They believed that the recomposition of the teacher's map in Phase 1 had a significant impact on expanding the concept map. As a result, EKB students discern that the ease-of-use and usefulness aspects of expansion activities were felt to be very optimal. As for the context of intended use, the two groups agreed that ESB and EKB were feasible to support the learning process, where the perceptions of EKB students were slightly higher than ESB students.
Limitation and future work
The present study had several limitations that should be considered. This study involved two material sequences in investigating the impact of two extension concept mapping tools reliably. Even so, it is still necessary to apply more material topics to find out more optimal results. Second, the number of participants involved in this experiment was relatively small. Thus, future works should consider a larger group of participants to examine the effects of extension concept mapping tools on a broader scale. Furthermore, the number of PEOU, PU, and BI items related to the extension activities was relatively small. In the future, more questionnaire items will be able to reveal a more comprehensive picture of students' perceptions regarding the extension concept mapping tools.
This study attempted to complement the initial work, which revealed that the EKB approach outperformed the ESB in students' comprehension test scores and map size. For future studies, a discussion of the correlation between test scores, map size, map quality, and students' perceptions may provide valuable new information. However, it is also important to reveal information on students' behavior in creating a concept map. Thus, future studies may involve log data and other instruments to explore students' activities.
This research investigated the effect of two extension concept mapping designs on students' perceptions and quality of knowledge structure. The performance of EKB that extends the KB framework by allowing learners to add new concept maps through open-ended techniques was compared to ESB, which extends the open-ended method by adding concept map components using the same way. This study involved three measurements: structural map scores, quality of propositions, and students' perceptions to confirm their achievements for both approaches.
The experiment results and analysis reported that the EKB approach has better structural map scores than the ESB method. With the achievement on the original map that tends to be the same, the expansion results of the concept map on the EKB group were more structured and achieved high-quality maps. The students in the experimental group who used the EKB consistently outperformed those who used the ESB in quality of propositions scores. The EKB group also confirmed a strong positive correlation between structural map scores and the quality of propositions. In addition, students' perceptions of the EKB showed a higher level of agreement compared to ESB. The EKB's recomposition approach was perceived as an important role in realizing ease-of-use and usefulness in extension concept mapping activities.
Availability of data and materials
As a series of subsequent research papers are still in progress, for now, it is temporarily impossible to share research data sets.
Structured query language
Technology acceptance model
Aleven, V., McLaren, B. M., Sewall, J., Van Velsen, M., Popescu, O., Demi, S., & Koedinger, K. R. (2016). Example-tracing tutors: Intelligent tutor development for non-programmers. International Journal of Artificial Intelligence in Education, 26(1), 224–269. https://doi.org/10.1007/s40593-015-0088-2
Andoko, B. S., Hayashi, Y., Hirashima, T., & Asri, A. N. (2020). Improving English reading for EFL readers with reviewing kit-build concept map. Research and Practice in Technology Enhanced Learning, 15(1), 1–19. https://doi.org/10.1186/s41039-020-00126-8
Ausubel, D. P. (1968). Educational psychology: A cognitive view. Holt, Rinehart and Winston.
Cañas, A. J., Bunch, L., Novak, J. D., & Reiska, P. (2013). Cmapanalysis: An extensible concept map analysis tool. Journal for Educators, Teachers, and Trainers. https://jett.labosfor.com/index.php/jett/article/view/435
Cañas, A. J., & Novak, J. D. (2014). Concept mapping using CmapTools to enhance meaningful learning. Knowledge cartography (pp. 23–45). Springer, London.
Cañas, A. J., Novak, J. D., & Reiska, P. (2015). How good is my concept map? Am I a good Cmapper? Knowledge Management & E-Learning: An International Journal, 7(1), 6–19. https://doi.org/10.34105/j.kmel.2015.07.002
Chen, J., Wang, M., Dede, C., & Grotzer, T. A. (2021). Analyzing student thinking reflected in self-constructed cognitive maps and its influence on inquiry task performance. Instructional Science. https://doi.org/10.1007/s11251-021-09543-8
Clariana, R. B. (2010). Deriving individual and group knowledge structure from network diagrams and from essays. Computer-based diagnostics and systematic analysis of knowledge (pp. 117–130). Springer, Boston, MA.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of computer technology: A comparison of two theoretical models. Management Science, 35, 982–1003. https://doi.org/10.1287/mnsc.35.8.982
Foley, D., Charron, F., & Plante, J. S. (2018). Potential of the cogex software platform to replace logbooks in capstone design projects. Advances in Engineering Education, 6(3), 3.
Grandón, E. E., et al. (2021). Technology acceptance model validation in an educational context: a longitudinal study of ERP system use. Journal of Information Systems Engineering and Management. https://doi.org/10.29333/jisem/9582
Herl, H. E., O’Neil, H. F., Jr., Chung, G. K., & Schacter, J. (1999). Reliability and validity of a computer-based knowledge mapping system to measure content understanding. Computers in Human Behavior, 15(3–4), 315–333. https://doi.org/10.1016/s0747-5632(99)00026-6
Hirashima, T. (2019). Reconstructional concept map: Automatic assessment and reciprocal reconstruction. International Journal of Innovation, Creativity and Change, 5, 669–682.
Hirashima, T., Yamasaki, K., Fukuda, H., & Funaoi, H. (2015). Framework of kit-build concept map for automatic diagnosis and its preliminary use. Research and Practice in Technology Enhanced Learning, 10(1), 17. https://doi.org/10.1186/s41039-015-0018-9
Hwang, G. J., Wu, C. H., & Kuo, F. R. (2013). Effects of touch technology-based concept mapping on students’ learning attitudes and perceptions. Journal of Educational Technology & Society, 16(3), 274–285.
Kim, M. (2013). Concept map engineering: Methods and tools based on the semantic relation approach. Educational Technology Research and Development, 61(6), 951–978. https://doi.org/10.1007/s11423-013-9316-3
Kinchin, I.M. (2016). Visualising powerful knowledge to develop the expert student: A knowledge structures perspective on teaching and learning at university. Sense: Rotterdam, 2016; pp. 15, 73.
Kinchin, I. M., Möllits, A., & Reiska, P. (2019). Uncovering types of knowledge in concept maps. Education Sciences, 9(2), 131.
Kirschner, P. A., Martens, R. L., & Strijbos, J. W. (Eds.). (2004). What we know about CSCL and implementing it in higher education. Kluwer Academic Publishers.
McClure, J. R., Sonak, B., & Suen, H. K. (1999). Concept map assessment of classroom learning: Reliability, validity, and logistical practicality. Journal of Research in Science Teaching, 36, 475–492.
Novak, J. D., & Cañas, A. J. (2008). The theory underlying concept maps and how to construct and use them. https://cmap.ihmc.us/publications/researchpapers/theorycmaps/TheoryUnderlyingConceptMaps.bck-11-01-06.htm
Novak, J. D. (2010). Learning, creating, and using knowledge: concept maps as facilitative tools in schools and corporations (2nd ed.). New York: Routledge. 2010
Novak, J. D., & Gowin, D. B. (1984). Learning how to learn. Cambridge University Press.
Pailai, J., Wunnasri, W., Yoshida, K., Hayashi, Y., & Hirashima, T. (2017). The practical use of Kit-Build concept map on formative assessment. Research and Practice in Technology Enhanced Learning, 12(1), 20. https://doi.org/10.1186/s41039-017-0060-x
Pinandito, A., Prasetya, D. D., Hayashi, Y., & Hirashima, T. (2021). Design and development of semi-automatic concept map authoring support tool. Research and Practice in Technology Enhanced Learning, 16(1), 1–19. https://doi.org/10.1186/s41039-021-00155-x
Prasetya, D. D., Hirashima, T., & Hayashi, Y. (2019). KB-mixed: A reconstruction and improvable concept map to enhance meaningful learning and knowledge structure. In Proceedings of The 26th international conference on computers in education (ICCE 2019). December, 809–812.
Prasetya, D. D., Hirashima, T., & Hayashi, Y. (2021). Comparing two extended concept mapping approaches to investigate the distribution of students’ achievements. IEICE TRANSACTIONS on Information and System, 104(2), 337–340. https://doi.org/10.1587/transinf.2020edl8073
Raud, Z., Vodovozov, V., & Lehtla, T. (2016). Teaching, learning, and assessment integration in electronics on the concept map basis. In Innovating with concept mapping proceedings of the seventh international conference on concept mapping, Tallinn, Estonia (pp. 199–207).
Reiska, P., Soika, K., & Cañas, A. J. (2018). Using concept mapping to measure changes in interdisciplinary learning during high school. Knowledge Management & E-Learning: An International Journal, 10(1), 1–24. https://doi.org/10.34105/j.kmel.2018.10.001
Ruiz-Primo, M. A., Schultz, S. E., & Shavelson, R. J. (1997). On the validity of concept map-base assessment interpretations: An experiment testing the assumption of hierarchical concept maps in science. CRESST.
Ruiz-Primo, M. A. (2004). Examining concept maps as an assessment tool. In Proceedings of the first international conference on concept mapping pamplona, Spain.
Ruiz-Primo, M. A., Schultz, S. E., Li, M., & Shavelson, R. J. (2001). Comparison of the reliability and validity of scores from two concept-mapping techniques. Journal of Research in Science Teaching: The Official Journal of the National Association for Research in Science Teaching, 38(2), 260–278. https://doi.org/10.1002/1098-2736(200102)38:2%3C260::aid-tea1005%3E3.0.co;2-f
Sadita, L., Hirashima, T., Hayashi, Y., Wunnasri, W., Pailai, J., Junus, K., & Santoso, H. B. (2020). Collaborative concept mapping with reciprocal kit-build: A practical use in linear algebra course. Research and Practice in Technology Enhanced Learning, 15(1), 1–22. https://doi.org/10.1186/s41039-020-00136-6
Schwendimann, B. A., & Linn, M. C. (2016). Comparing two forms of concept map critique activities to facilitate knowledge integration processes in evolution education. Journal of Research in Science Teaching, 53(1), 70–94. https://doi.org/10.1002/tea.21244
Taricani, E. M., & Clariana, R. B. (2006). A technique for automatically scoring open-ended concept maps. Educational Technology Research and Development, 54(1), 65–82. https://doi.org/10.1007/s11423-006-6497-z
Valtonen, T., Hoang, N., Sointu, E., Näykki, P., Virtanen, A., Pöysä-Tarhonen, J., & Kukkonen, J. (2021). How pre-service teachers perceive their 21st-century skills and dispositions: A longitudinal perspective. Computers in Human Behavior, 116, 106643. https://doi.org/10.1016/j.chb.2020.106643
Vanides, J., Yin, Y., Tomita, M., & Ruiz-Primo, M. A. (2005). Concept maps. Science Scope, 28(8), 27–31.
Venkatesh, V.(2000). Determinants of perceived ease of use: Integrating control, intrinsic motivation, and emotion into the technology acceptance model. Information Systems Research, 11(4), 342–365. https://doi.org/10.1287/isre.11.4.342.11872
Voogt, J., & Roblin, N. P. (2012). A comparative analysis of international frameworks for 21st century competences: Implications for national curriculum policies. Journal of Curriculum Studies, 44(3), 299–321. https://doi.org/10.1080/00220272.2012.668938
Wallace, L. G., & Sheetz, S. D. (2014). The adoption of software measures: A technology acceptance model (TAM) perspective. Information & Management, 51(2), 249–259. https://doi.org/10.1016/j.im.2013.12.003
This work was partially supported by JSPS KAKENHI Grant Number 19H04227. The first author would like to acknowledge the Islamic Development Bank (IsDB) for a PhD scholarship in partnership with Universitas Negeri Malang (UM), Indonesia.
This work was partially supported by the JSPS KAKENHI Grant Numbers 19H04227.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Prasetya, D.D., Pinandito, A., Hayashi, Y. et al. Analysis of quality of knowledge structure and students’ perceptions in extension concept mapping. RPTEL 17, 14 (2022). https://doi.org/10.1186/s41039-022-00189-9
- Extension concept map
- Quality of propositions
- Structural map scores
- Students' perceptions