Learning in digital play: a dual case study of video gamers’ independent play

Abstract

This paper explores the implications of youths’ out-of-school gaming practices for teaching and learning in formal and informal learning contexts. We report on a study where we examined the video game play of two youths using a case study approach. User experience approaches, e.g. the think-aloud protocol and interviews, were grounded in the theoretical framework of social semiotics to analyse the gameplay videos and to discuss the implications for the youths’ learning. The paper contends that youths are demonstrating critical thinking, empathy, and multimodal literacy through their gameplay. We offer suggestions for how adults can use video games for youths’ learning.

Introduction

People learn as they play (Oliver & Carr, 2009). Digital play involves the use of technology, including computers and laptops, tablets, smartphones, electronic toys, and console games for playing and learning activities (Marsh et al., 2016). The boundaries between online and offline spaces have become blurred in digital play as communication and play move across physical and virtual domains and integrate material and immaterial practices (Garcia, 2020; Giddings, 2014; Marsh et al., 2016). Digital play is rooted in children’s everyday experiences and can promote cognitive and imaginative development (Edwards, 2011). Engagement with video games is a type of digital play that can bring about learning (Toh & Lim, 2021; Squire, 2011) as they incorporate sound learning principles, such as “just in time learning” that can be used for teaching various subjects in schools (Gee, 2003).

Digital play motivates learning because it engages with the students’ “lifeworld” (New London Group, 1996). Digital play can be integrated into schools to intrinsically motivate students for their self-directed (Karakas & Manisaligil, 2012) and peer-to-peer learning (Butler, 2017). For example, Beavis et al. (2015) reported that teachers were able to improve students’ learning and engagement by connecting teaching to students’ lifeworlds and redesigning their English and literacy curriculum to leverage the joy of gameplay to build students’ literacy proficiency over time. Digital play can also be an effective way for students to demonstrate new literacies needed for the future workforce (Apperley & Walsh, 2012). Playing video games is a common pastime of many people, especially amongst males (Jansz, 2005). Video game play has also been described as a literacy practice (Steinkuehler, 2010; Steinkuehler & King, 2009). When video games are used in the school curriculum, it can motivate students (Deng et al., 2016) and draw on their funds of knowledge (Risko & Walker‐Dalhouse, 2007) for the learning activities. Digital play has been integrated into formal educational contexts for STEM learning (e.g. Ibáñez & Delgado-Kloos, 2018) and to motivate L2 vocabulary learning (e.g. Butler, 2017), argumentation and persuasion writing (e.g. Dickey, 2011), as well as social skills (e.g. Craig et al., 2015), creativity (e.g. Ott & Pozzi, 2012), critical thinking (e.g. Barab et al., 2012), ethical thinking (e.g. Schrier, 2015), and mathematical thinking and problem-solving skills (e.g. Kiili et al., 2015).

Video games incorporate a whole set of fundamentally sound learning principles that can be used for teaching and learning activities in schools (Gee, 2003). Gee (2005) also explicated the principles of learning that are built into successful games. These include empowering players as active agents who can co-design the learning experience as well as learning through well-ordered problems in which learners are scaffolded to learn how to proceed when they face harder problems later in the game. Squire (2002) also argued that examining how gameplay can be used to support learning in formal learning environments can be productive. Games like SimCity and Civilisation can mediate our understandings of other phenomena, and they can be used to support learning in formal and informal learning contexts. Pedagogical activities can be integrated into the game design or teachers can adapt the game to engage students in meaningful practices and critical thinking by using scaffolding questions for students to think about how games can be used as a tool for answering historical questions.

Shaffer et al. (2005) posited that video games are powerful contexts for learning because they make it possible to create virtual worlds and acting in such worlds help to develop the situated understandings, effective social practices, powerful identities, shared values, and ways of thinking of important communities of practice. For instance, Railroad Tycoon can help players to learn economic and geographic issues faced by railroad engineers in the 1800s by allowing them to engage in design activities when playing the game. However, the authors also highlighted that it was challenging to integrate game-based learning into institutional education because games encourage exploration, personalised meaning-making, individual expression, and playful experimentation with social boundaries, factors which conflict with institutional modes of teaching and learning that emphasise conformity, one size fits all curriculum, and standardised testing and assessment practices.

A game-based curriculum informed by the theoretical framework of digital play has been designed and integrated into classrooms to teach persuasion writing to seventh grade students (Barab et al., 2012). The results indicated statistically significant differences in learning and engagement between the control and experimental groups where the students in the game-based curriculum demonstrated significant improvements in their learning gains. The authors explained that the technological affordances of well-designed videogame play supported the students’ engagement and learning by allowing them to play out new possible selves, such as scientists, historians, or investigative reporters in the virtual world to enact disciplinary expertise.

Studies have also shown that commercial video games, such as The Walking Dead can be integrated into classrooms for teaching domain-specific content, such as ethical theories when teachers are able to orchestrate the learning activities with appropriate teaching approaches to support student learning and engagement (de Sousa et al., 2018). Playing commercial video games has been found to improve the desirable skills and competencies sometimes referred to as “graduate attributes” in higher education students. Using a laboratory-based randomised controlled trial, Barr (2017) reported that students in the gameplaying group obtained highly significant positive changes in their adaptability, resourcefulness, and communication skill.

The studies on digital play described here have been conducted in formal pedagogical contexts. However, there is a knowledge gap in terms of our understanding of youths’ gameplaying activities and the types of learning that are demonstrated in their gameplaying process in the informal space. As such, the goal of this study is to identify the types of learning, such as critical thinking, empathy, and multimodal literacy, that can be demonstrated from the players’ gameplaying activities. In our previous study (Toh & Kirschner, 2020), we have investigated how Participant 1 adopted an inquiry-based learning approach to progress the same game segment examined in this paper. The inquiry-based learning approach has been integrated into a self-directed learning framework in video games for that study. In an upcoming paper (Toh & Kirschner, 2022), we examine how Participant 2 exhibited resilience in his learning of the stealth option to progress the same game segment analysed in this paper. Resilience has been integrated into a social-emotional learning model in that paper. In this study, we adopt a case study approach through a multimodal discourse analysis of the gameplay videos to surface the types of learning that can be identified in two youths’ gameplay recordings. Our paper aims to answer the following research question: What is the learning that can be demonstrated through video gameplay?

Methods

The theoretical orientation of social semiotics informs our lenses on youths’ gaming practices (Kress, 2010; Kress & van Leeuwen, 2001). The participants’ gameplay interactions are described using social semiotic theory, which models meanings according to the sociocultural contexts where the interactions took place. Bezemer and Kress (2016, p. 4) argue it is important to respond “theoretically to the social”, recognise the ways for making meaning, and describe the “semiotic work done”. From this argument, the tenets of multimodal social semiotics have been adopted in the analysis and interpretation of the data to understand youths’ gaming practices.

The first tenet recognises that youths communicate with multimodal ensembles, drawing from language, visual, and the verbal modes to constitute interactivity. The second tenet recognises signs of learning (Kress & Selander, 2012). Making meaning is guided by interest, and learning occurs as the learners’ entire set of resources is augmented and transformed when they integrate new ways of conceiving the world. The third tenet focuses on how semiotic modes and digital technologies facilitate different pedagogic relations in human–computer interaction and offer learning opportunities. From the participants’ gameplay interaction, we discuss the learning demonstrated by the participants in their gameplay, the affordances of the video games that supported their expression of these literacies, and pedagogical implications of digital play and learning.

Research design overview

This paper adopts a qualitative case study approach method (Creswell, 2013) through a multimodal discourse analysis of the gameplay videos produced by two youths. The qualitative case study approach involves the exploration of case(s) over time through detailed, in-depth data collection involving multiple sources of data. The rationale for a qualitative case study approach is that it allows for a holistic understanding of a phenomenon within real-life contexts from the perspective of those involved (Merriam, 1988; Stake, 1995, 2005; Yin, 2009) and allows the researcher to grasp the intricacies of a phenomenon (Stake, 2005).

The think-aloud protocol (Hu & Gao, 2017; Li et al., 2012) was adapted for use in the study by asking participants to verbalise their gameplay experience. Participants were asked to explain how and why (Gillern, 2016) they made a specific choice or perform an action during and after gameplay. They were also instructed to record their natural reactions and comments in the game recordings. When they verbalised their experience as they played the game, the method used was concurrent think-aloud protocol (Kelley et al., 2015) that involved the participants thinking-aloud while simultaneously performing an activity in usability studies. When the participants discussed their experience after gameplay, retrospective think-aloud (Ji & Rau, 2019) was used to reflect on their game experience. The think-aloud protocol is an effective method to elicit the participants’ demonstration of their learning because of the prompt response of the participants when a problem is experienced (Ericsson & Simon, 1993), which shows how they overcome the problem to progress the game. As the participants were able to refer to their gameplay recordings during the interviews, stimulated recall (Dempsey, 2010) was also used to review their gameplay recordings.

The framework of multimodal discourse analysis (O’Halloran & Lim, 2014; Kress, 2012; O’Halloran, 2004) was adopted to analyse the video recordings of the participants. The multimodal discourse analysis approach integrates the tenets from social semiotics, discourse analysis, think-aloud protocol, and the interviews to examine the player experience in their video recordings. The multimodal combination of language, visual, verbal, and action semiotic modes in the gameplay recordings are combined to form the ludonarrative frame for video games, which conceptualises the gameplay, the narrative, and the player as a whole (Toh, 2018). A semiotic mode can be understood as a distinct way for meaning-making (Jewitt, 2009) and the multimodal discourse analysis approach to examining the gameplay recordings is relevant because video games are highly multimodal artefacts. A central tenet of multimodal discourse analysis is that a text is embedded within the sociocultural context (Guo, 2017) in which it is produced and the text’s meaning is never fixed but contingent on different external factors, such as the cultural background and prior gaming experience of the participants (Toh, 2018).

Data collection

Multiple sources of data were used to interpret the lived experience of the participants through the method of data triangulation and method triangulation. Data triangulation (Denzin, 2009) has been used to gather data from different players, and method triangulation (Bauwens, 2010) has been used for data gathering and analysis. The different methods of data collection include the interviews of the participants, the observation of the participants’ gameplay in the laboratory, and the gameplay recordings. Member checking has been adopted during the qualitative interviews to validate the interpretations of the participants’ gameplay recordings (Iivari, 2018).

Prior to the start of the study, the participants filled up a written survey to provide demographic information and gaming habits. During the first gameplay session, the participants played their selected game from the start for one to three hours in a computer laboratory. Their gameplay was streamed and recorded using Fraps (computer) and a PS3 recorder on a nearby computer whereby the researcher was using to observe their gameplay. At the end of the first gameplay session, the first author conducted a one-to-two-hour interview with the participants. The interview questions asked during the first gameplay session were more general and open-ended. Some of the questions that the interviewer asked the participants included whether they liked the scripted game sequences, whether they were clearly able to see a divide between narrative and gameplay in the game, and whether the information provided by the characters helped them to make decisions to progress the game.

The first author then instructed the participants to record their subsequent gameplay sessions at home. The participants incorporated their natural reactions with their think-aloud verbalisations in their gameplay recordings. A total of 15 and 14.8 h of think-aloud video recordings were collected and analysed for Participants 1 and 2, respectively. After they have completed the game, the first author arranged a final interview to review their gameplay recordings together with them and discuss their gaming experiences. A total of 7 and 6.7 h of interview recordings were collected for Participants 1 and 2, respectively. More specific interview questions were asked during the final session with the participants. For instance, the researcher asked Participant 1 to explain whether specific objects in the game world such as the spark plug and the pillow helped him to progress the game and how he figured out the sequential order to progress the game. For Participant 2, an example of a specific question the researcher asked him was whether playing as Joel after Ellie caused him to feel worried for Ellie’s safety and motivated him to progress the game to find out the story.

Participants

The data were obtained from the first author’s study on the player experience. The research aim for the study was to propose an original theoretical framework, the ludonarrative model for studying video games, and this model was grounded on the empirical data of a group of players in Singapore. For the study, participants were recruited through convenience and snowball sampling via advertisements posted on the university’s website. Consent was obtained from the participants to share their video recordings, which included their voices, for research purposes. This study was approved by the university’s Institutional Review Board.

The data from two of the participants were selected for analysis. The two participants were selected because they were experienced players (Soylu & Bruning, 2016). Participant 1 is a 22-year-old male undergraduate student with 11–15 years of gaming experience. He mentioned that he played a variety of games such as shooters, puzzle games, action games, and so on a few times a week. The game he chose for the study was The Walking Dead Season 1, Episodes 1–5. Participant 2 was a 24-year-old male undergraduate student with 11–15 years of gaming experience. He mentioned that he played a variety of games such as role-playing games, strategy games, shooters, and so on a few times a week. He chose The Last of Us.

Description and rationale of game choices

The Walking Dead (PC game) is set in a fictional world in which a zombie apocalypse has occurred. The player controls Lee Everett to explore the story world and is accompanied by several survivors, including Clementine, a young girl. The Last of Us (PS3 game) is set in a post-apocalyptic world in which a mutated strain of the Cordyceps fungus has infected humans and transformed them into zombie-like creatures. The player controls Joel to explore the story world and is accompanied by Ellie, a young girl.

Data analysis

The thematic analysis (Braun & Clarke, 2006) has been used to code the types of learning of Participants 1 and 2. Theoretical thematic analysis involves the use of theoretical frames in a deductive way to search across the data set in the form of the think-aloud verbalisations and players’ reflections in the gameplay recordings and the interviews to discover repeated patterns of meaning. The coding categories were developed a priori from relevant literature (e.g. Toh & Lim, 2021) and included the three main types of learning, namely, critical thinking, empathy, and multimodal literacy. During the thematic analysis, the researchers familiarised themselves with the data by watching the gameplay recordings and listening to the interviews multiple times. During each review of the data, the researchers generated initial codes and then collated these codes into themes. The themes were then reviewed and defined to arrive at the final analytic scheme (Braun & Clarke, 2006). The two authors of the paper coded the data, and they have received prior training to conduct the analysis. The data were independently coded by the two authors, and disagreements were discussed to reach a consensus.

Case 1¹ involved Participant 1, who played The Walking Dead Season 1, Episode 1. The selected gameplay segment was set at the Travelier Motel. In a previous scene, Glenn, the non-player character (NPC), had gone searching for supplies and gas at the motel for Lee’s group. He noticed a woman, Irene, in a boarded-up room on the upper level of the motel. When Glenn tried to talk to her, she asked him to leave. Walkers emerged from the forest, and Glenn was forced to hide in an icebox until Lee and Carley arrived to rescue him. The trio then decided to rescue Irene together.

The analysis of this gameplay segment indicated that Participant 1 adopted a problem-solving and inquiry-based approach (Toh & Kirschner, 2020; cf. Squire & Jan, 2007) to reach Irene. Participant 1 performed the following actions sequentially and iteratively: understand (orientate), plan (conceptualise), act (apply), and reflect (evaluate). First, Participant 1 understood how to proceed by listening to the characters. For example, Lee hinted that stealth was important to proceed by saying that “Noise attracts these things”. At 3:16, Participant 1 demonstrated his understanding of using stealth to reach the objective by repeating that “noise attracts these things”.

At 4:12, the game hinted to Participant 1 that peeking too long will result in a “game over” if the zombies detected him. The linguistic text warned Participant 1: “Peek too long, and you will be seen”. Simultaneously, Participant 1 saw the visual display turning red and heard the heartbeat sound as he peeked over the wall to observe the setting. In his first attempt, Participant 1 peeked too long, so a “game over” occurred. Participant 1 laughed loudly at 4:22 as Lee died comically (Newman, 2016). Participant 1’s unsuccessful first attempt demonstrated how he experimented with the game rules to understand and plan his actions to progress the game. After his first failure, he understood that one specific condition that could result in a “game over” for him was his exposure to the zombies beyond a specific time limit.

Participant 1 continued to understand and plan the correct sequential order to reach the objective. As he crossed the road to go towards the recreational vehicle (5:05), the game hinted to him that he needed a weapon to kill the zombies by having Lee verbalise that he was not sure whether he could take out the zombies even if he had a weapon. This was followed by the camera’s zooming in on the zombies who were obstructing Lee. The game also hinted to Participant 1 by having Lee verbalise, while being shown the car and the zombie, that he could use the car to kill the zombies (6:10). However, before Participant 1 can use the car, he must remove the zombie who was lurking near the front of it.

Participant 1 then noticed an awl inside a locked vehicle, which he assumed can be used to kill the zombie lurking near the car (6:25). Participant 1 chose to hit the glass but was unsuccessful (6:33). Glenn hinted to Participant 1 that his action had attracted the zombie’s attention. At 6:50, Participant 1 reflected that the awl was the weapon that he needed and he must find some way to get it quietly. He planned his next action by commenting that he had to kill the zombie with the awl before using the car to kill something else (7:05).

When Participant 1 returned to the brick wall, he found a pillow on the other side of the wall. He planned to use the pillow to smother the zombie and kill him quietly with some other implement. Carley reaffirmed Participant 1’s plan by hinting, “Good luck smothering them to death”. Returning to the front of the vehicle, he killed the zombie quietly by smothering him with the pillow and asking Carley to shoot him with her gun. Participant 1 then reflected that the action will not work in real life because the gun did not have a silencer.

After removing the zombie, Participant 1 opened the car’s door, retrieved a spark plug, unlocked the gear shift, and opted to push the vehicle towards the wall to immobilise a zombie there (10:34). Returning to the car with the awl inside it, Participant 1 used the porcelain on the spark plug to break the car window to retrieve the awl (11:42). Finally, Participant 1 reflected that he can use the awl to kill the other three zombies (12:12). He then killed these zombies with the awl (13:25, 14:00) and retrieved the axe from the wall (14:27). Finally, Participant 1 reflected that he felt safer with the axe (14:32). He easily killed the last two zombies to reach the trapped survivor (15:22).

Case 1 demonstrates how Participant 1 adopted a problem-solving and inquiry-based approach to reach the game goal. Participant 1’s ability to perform his actions in the correct sequential order in an iterative manner enabled him to remove the zombies obstructing him from reaching the woman trapped on the upper level of the house across the street. Our analysis of his think-aloud shows how he safely removed the zombies through critical thinking and reflection before he succeeded in reaching the trapped woman to save her.

Case 2² involved Participant 2, who played The Last of Us. The segment that we analysed occurred in the winter chapter when Ellie escaped from David, the antagonist, after his failure to recruit her into his group. Joel was searching for Ellie when she escaped from David. This gameplay segment involved Participant 2’s alternating control of Joel and Ellie. This alternating control allowed Participant 2 to demonstrate empathy for both characters, the understanding of their thoughts, feelings, and actions through embodied play and perspective shifting (Toh & Lim, 2021; Elliott et al., 2011).

Ellie killed James (David’s partner) with the chopper on the table when he was distracted while talking with David. She escaped and grabbed a knife from a nearby table for protection as David attempted to shoot her but missed. The cutscene ended as Ellie escaped into the snow, and Participant 2 was given control of Ellie in the snow outside (19:10). He controlled her to take shelter in a nearby building. Ellie’s verbal utterance and subtitles informed Participant 2 that Ellie needed to find a gun to protect herself (19:28). Inside the building, Participant 2 overheard David ordering his people to find Ellie (20:20). He controlled Ellie to sneak into another building and planned to wait for David’s people to come inside for him to kill them individually (21:17). When the first man came in, Participant 2 stealthily stabbed him and obtained a revolver (21:45). He repeated his action on the other man who entered the building (22:09).

As the game progressed, the video showed that he faced difficulties when Ellie had to fight two or more men simultaneously. This led to Ellie’s being surrounded and killed. In a later gameplay segment, he planned to distract and kill David’s men inside another building (34:20). He succeeded (34:48), but Ellie was killed by a nearby enemy who detected Ellie and shot her (35:05). This heightened Participant 2’s feeling of vulnerability when he played as Ellie. He said that he had to retry the game segment (35:07). When he replayed this segment, he was killed again as he controlled Ellie to fight the two men directly instead of stealthily (36:30). He reflected that he had to sneak into that room (36:40). During the third (37:30), fourth (38:24), and fifth attempts (39:47), he failed again when Ellie was surrounded and killed. He reflected: “The idea was there—the execution (39:50)”. Finally, on the sixth try, he cleared this segment by separating David’s men and stealthily killing them one at a time (40:30).

Participant 2 entered the open area in a restaurant and triggered the fight with David (45:58). During the fight, Participant 2 used Ellie’s stealth ability to track David’s location and planned his attacks. Participant 2’s ability to use Ellie’s ability to track David’s movements by sound was key to performing a stealth attack on him.

After Participant 2 stabbed David thrice, the game changed perspective and allowed him to control Joel to search for Ellie (56:25). When Participant 2 entered a nearby building, he heard David’s men searching for Ellie (57:05). It was easier for Participant 2 to play as Joel because he could choose to engage in close combat and kill enemies with a single blow of the axe (57:29, 57:48) or use stealth attacks. With Ellie, he could only use stealth attacks. As Participant 2 cleared several gameplay segments, he entered the slaughterhouse and found Ellie’s backpack (1:09:04). From Joel’s verbalisation, “What is this? Why is Ellie’s stuff here?”, Participant 2 understood that Joel was worried about Ellie (1:09:19). He picked up a meat ledger note that indicated the amount of meat that David’s men, who were cannibals, had collected for supplies (1:09:30). This note, along with the human corpses in the meat locker (1:09:50) and Joel’s utterance “I gotta find her”, (1:09:55) further underscored Participant 2’s understanding of Joel’s concern for Ellie’s safety. Participant 2 had just played as Ellie and understood the danger Ellie faced as she fought against David and his men.

Case 2 shows how Participant 2’s ability to control 2 characters allowed him to demonstrate empathy by understanding their thoughts, feelings, and actions through embodied play and perspective shifting. Our analysis of his think-loud and observation of his actions during the gameplay shows how Participant 2’s empathetic concern for Ellie motivates him to find her when he played as Joel.

Findings

Critical thinking

In the field of education, Bloom’s (1956) taxonomy is the most widely cited source when it comes to teaching and assessing higher-order thinking skills. Bloom’s (1956) taxonomy consists of the lower-order thinking skills, such as remember (recall, recognise), understand, and apply as well as the higher-order thinking skills, such as analyse, evaluate, and synthesis, which are frequently said to represent critical thinking (Kennedy et al., 1991). The analysis of Participant 1’s gameplay recording showed his ability to select information from the game world, reflect on it (e.g. failure to progress the game), and integrate it into a coherent whole in a mental model to gain a deeper understanding of the story, rules, and mechanics (Toh, 2018, Buckingham, 2015). When he played the game, we observed that he was applying his critical thinking: the selection, evaluation, and integration/synthesis of information provided by digital texts (Salmerón et al., 2018) by adopting the inquiry-based learning approach (Toh & Kirschner, 2020) to reach the objective.

Participant 1 demonstrated an iterated approach of using a mixture of lower and higher-order critical thinking skills to progress the game. At the start of this gameplay segment, Participant 1 demonstrated lower-order critical thinking skills, such as recognising through his counting of the number of zombies that must be overcome to reach the objective. He demonstrated understanding by explaining what he had to do for this gameplay segment through his think-aloud “Ya, so basically, we are here rescuing Glenn from [the zombies]”. He also demonstrated recall by commenting on what happened previously in the game “He [Glenn] came to top up petrol from the motel but he got into a bit of trouble and now we have to rescue this survivor upstairs over here, yes.”, which is followed by understanding as he explained what he had to do to achieve the gameplay objective “So, we have to kill all the zombies to rescue her. So, let’s find some way to kill them quietly”.

At 4:10, Participant 1 experienced how the losing condition of the game could happen by peeking over the wall at the zombie eating the dead person on the ground. However, he spent too long peeking and Lee ended up being eaten by the zombie. On his second attempt, he demonstrated his understanding (a lower-order critical thinking skill) of the losing condition of the game by commenting that “Oh, it looks like a timed thing” and controlled his character not to long too long over the wall. At 5:36, Participant 1 demonstrated the use of evaluate, a higher-order critical thinking skill as he reflected on how Lee can see over the wall but not when he was running.

At 7:05, from Participant 1’s verbalisation that he had to kill a zombie in front of the vehicle with an awl (“the screwdriver”) before using the car to kill another zombie, he demonstrated planning and analysis, which is a higher-order critical thinking skill. At 8:54, Participant 1 demonstrated the use of the lower-order critical thinking skills when he applied his plan and chose the action in the dialogue option to kill the zombie quietly by covering him with the pillow and asking Carley to shoot him with her gun. Finally, Participant 1 demonstrated a higher-order critical thinking skill, such as evaluation from his verbalisation that the action would not have worked in real life because the gun did not have a silencer.

Empathy

Likewise, a learning observed from the participants’ gameplay is empathy, that is the ability to understand and share the feelings of others (CASEL, 2015). This was observed in Participant 2’s gameplay. For example, when the game allowed Participant 2 alternate control of Ellie and Joel, he used embodied action to gain a better understanding of both characters’ playstyle through situated and just-in-time learning (Gee, 2005). Another example observed in the gameplay recordings was when Participant 2 discovered Ellie’s backpack, the meat ledger, and the hanging corpses, and he showed his concern for Ellie’s safety. For instance, in 1:09:19, Participant 2 verbalised that Joel felt worried for Ellie when he found her backpack in the slaughterhouse. He appreciated that Joel’s actions had been driven by his fear for Ellie’s safety. Participant 2 demonstrated his empathy in an interview:

Interviewer::

So later on, when you shift to Joel to control him to save Ellie, did you feel an even more urgent need to save her? Because you discovered Ellie’s backpack.

Participant 2::

Because you played what Ellie’s going through right? So, you know that she needs help. So, in that sense, you can empathise more with what Joel is feeling, as in Joel doesn’t know what is happening to Ellie, but you know the emergency. So …

Interviewer::

When you discover all her objects, huh?

Participant 2::

Um-hmm.

Interviewer::

And then the slaughterhouse.

Participant 2::

Um-hmm. So, he knows that she’s not in a good place.

In the above interview, Participant 2 explained how his ability to control two different player characters one after the other during the gameplay allowed him to understand both characters’ thoughts, feelings, and actions. Because Participant 2 was given control of Ellie early in the Winter chapter of The Last of Us, he was able to feel Joel’s sense of fear for Ellie’s safety when the game later offered him control of Joel to look for her. As Participant 2 had been able to play as Ellie, he could understand her precarious situation in the game world when she met the antagonist, David with his group of cannibals while searching for supplies to take care of Joel. David appeared to Ellie as a compassionate man who looked out for her needs and he even fought together with Ellie against waves of Infected enemies during their initial encounter.

However, as the chapter progressed, Ellie discovered that David and his group were cannibals. David captured Ellie and offered her a place in his group. When she rejected his offer, David and his man attempted to slaughter Ellie for food, but she managed to escape. The game then switched control to allow Participant 2 to play as Joel after playing as Ellie. He could then feel a greater connection to both player characters because he possessed more information than either of the player characters possessed on their own in the game world. The perspective shift (Toh & Lim, 2021) in the game therefore allowed Participant 2 to demonstrate empathy when playing the game.

Multimodal literacy

We also observed that the participants demonstrated multimodal literacy in their engagement with the multimodal orchestration in the game, such as the verbal mode of the NPCs, the linguistic mode of the game hints, the visual representations of the characters, and the characters’ actions and gestures to gain an overall idea of how to progress the game. Multimodal literacy (Jewitt & Kress, 2003; van Leeuwen, 2017) involves both critical and creative engagement with multimodal representations that reflect a semiotic awareness (Lim, 2021a; Lim, 2021b). Multimodal literacy also involves an understanding of the affordances of the different meaning-making resources and how they work together to produce a coherent and cohesive multimodal text (Mills, 2016). The player’s multimodal literacy refers to their sensitivity to the design features in the video games, including knowing how to work across different semiotic modes, such as language, visuals, audio, music, and action to perform critical and creative actions in the game.

Participant 1’s multimodal literacy can be observed at the beginning of the gameplay segment. He demonstrated verbal awareness by listening to the characters’ verbal interactions in the cutscene before commenting through his think-aloud to show his understanding that he must rescue the survivor in the house across the street, to achieve the winning condition of this gameplay segment. We see how he showed a visual awareness of the objective, by pointing at the house across the street with the mouse pointer to indicate his understanding that he must reach the objective, the house to progress the game. He showed us his linguistic awareness by following the game hints provided through the linguistic prompts, such as “peek over the wall”, “push car”, “look at truck”, “open door”, “examine window”, amongst others to perform the micro actions of the gameplay so that he can reach the gameplay objective. He also demonstrated an action awareness, which refers to his understanding of the dynamic interplay between game and player (Toh, 2018; Apperley & Beavis, 2013; Beavis, 2014) by selecting the specific actions in the correct sequential manner to interact with the zombies and overcome them to reach the woman in the house across the street. He combined his verbal awareness, visual awareness, linguistic awareness, and action awareness in the gameplay to reach the objective and progress the story.

Participant 1 also demonstrated his multimodal literacy in quickly making sense of the orchestration of multimodal meanings through the semiotic resources. He combined the visual mode of the entire screen turning red, the auditory mode of a pulsating heartbeat, the linguistic mode of a text that warned him not to peek for too long, and the verbal mode of the player character’s announcing that the zombie had discovered him to come to his understanding of the losing condition of this gameplay segment. Participant 1 verbalised his reading of the situation through his think-aloud that “it looks like a timed thing” to reflect his understanding that his peeking over the wall beyond a certain time limit resulted in the losing condition whereby his character was killed when the zombies discovered where he was hiding.

Discussion

The findings of this study affirm the findings from previous studies that playing video games can allow players to demonstrate critical thinking (e.g. Gumulak & Webber, 2011) and empathy (e.g. Hilliard et al., 2018). Additionally, our study adds to the field of knowledge by showing that players can demonstrate multimodal literacy (Lim, 2021b) during gameplay by learning how to combine information from semiotic modes, such as the language, verbal, visual, and aural modes to come to an overall understanding of the game. Their demonstration of their multimodal literacy is evidenced in their think-aloud verbalisations where they articulated how they understood, planned, acted, and reflected on how to progress the game by selecting and integrating information from multiple semiotic modes, such as language and visuals during the gameplay process. From an educational standpoint, we have noted that there is a difference between demonstrating these skills within a game context and having the players transfer skills such as critical thinking in digital play to academic subjects or real-world situations. We recognise that the demonstration of these skills in video gameplay does not mean that players are able to transfer the critical thinking to academic subjects or empathy to real-world situations. With this in mind, we have explored ways in which educators can design for digital play in the classrooms, such as introducing a pedagogic metalanguage for digital play (Toh & Lim, 2021; Lim & Toh, 2022) which teachers can use to guide players in transferring their critical thinking and empathy through reflection questions during classroom activities.

The affordances of video games that can facilitate the expression of these multiliteracies include immediate feedback from semiotic resources in the game world (Nadolny et al., 2020) and a safe space (LaFleur et al., 2017) that allows the player to retry again after failing the gameplay segment. Additionally, scaffolded learning in the form of virtual learning companions (Calvert, 2017) and multilinear pathways (Roswell & Wohlwend, 2016) in the gameplay can allow players to try out different approaches of solving a problem to demonstrate their critical thinking. Finally, the player’s ability to take control of two player characters during gameplay to understand their thoughts, emotions, and actions (Toh & Lim, 2021) can support their demonstration of empathy.

In contrast to empirical studies that suggest learning occurs in stages (e.g. Luft & Buitrago, 2005), our findings showed that learning is a nonlinear and iterated process where sometimes the learner may experience progress in their learning, but at other times, they may experience failure if they uncritically applied one approach that worked in a previous context on a different context. For instance, Participant 2 failed when he tried to play Ellie using Joel’s direct approach of attacking enemies in the open instead of adopting stealth when fighting multiple enemies.

Digital play is arguably attractive to people from all ages but is especially appealing to children and youths because of design features, such as immediate feedback, social affordances, and the presence of multisensory stimuli when using mobile devices and games to interact with others (Marlatt, 2018; Oliver & Carr, 2009). Notwithstanding the value of digital games, concerns have also been raised about the risks, such as exposure to sexual (Tompkins & Lynch, 2018) and violent content (Elson & Ferguson, 2014) as well as potential addiction (Grüsser et al., 2007) and health issues (Saunders & Vallance, 2017) related to the prolonged screen time with mobile devices. While these concerns are understandable, it is also acknowledged that studies have found no evidence of negative effects that playing violent video games has on youths (e.g. Ferguson & Olson, 2014). In addition, to mitigate the concerns on the possible ills of digital play, strategies of integrating digital play and learning into pedagogical context, such as the use of metacognitive scaffolding for inquiry-based learning activities (Ibáñez & Delgado-Kloos, 2018) and reflection questions to allow teachers to guide students’ digital play (Toh & Lim, 2021) have been proposed.

With the increasing ubiquity of mobile devices and gaming activities amongst youths and children, we argue that adults should not disregard youths’ use of video games. Instead, they can design learning experiences with video games using a series of learning activities and scaffolding questions structured by theoretical frameworks, such as Cope and Kalantzis’ (2015) learning by design framework or the metalanguage that we have developed for digital play (Toh & Lim, 2021; Lim & Toh, 2022). The framework allows adults to design learning using guided inquiry techniques, such as reflection and conversational prompts after gameplay for youths’ learning.

Given the resistance typical schools may have towards the use of video games for learning (Bourgonjon et al., 2013; Gee, 2005), we recognise that the degree of integration of video games into formal learning contexts will depend on the receptiveness and attitude of the educators, parents, and students (Mifsud et al., 2013). Nonetheless, video games can be used well in the classrooms for teaching and learning STEM subjects (e.g. Ormsby et al., 2011) or content subjects involving ethical thinking (e.g. Schrier, 2015), decision-making (Toh, 2021), and narrative concepts (Ostenson, 2013) in many schools.

Conclusion

In this study, we have explored the digital play of two players using a qualitative case study approach to examine the display of critical thinking, empathy, and multimodal literacy in their gameplay. These are important life skills that overlap with the skills required in school and the workplace. Through this study, we hope to advance the call for educators to harness video games in teaching, given how it can bring about different types of learning in the youths.

We also acknowledge that the limitations of the study include the male gender bias of the subjects recruited for the study due to the snowball and convenience sampling method used. Additionally, the qualitative research methods used, such as interviews, think-aloud protocol, and observations in the laboratory can be complemented with more objective measures in future studies, such as galvanic skin conductance, eye tracking, and brain scans to triangulate the findings. Future studies can also use a broader range of games to elicit the different types of learning that are demonstrated by gameplayers.