3 Questions: Teaching computational maker skills through gaming

The early stages of teaching maker skills, such as digital fabrication, typically involve simple exercises like laser cutting or 3D printing basic shapes and objects. In our hyperconnected, hyperstimulated world, this learning activity can feel a bit underwhelming — a sentiment that caused Dishita Turakhia, an MIT PhD student in electrical engineering and computer science and an affiliate of the Computer Science and Artificial Intelligence Laboratory (CSAIL), to re-imagine the learning pipeline in the pursuit of keeping students interested, inspired, and empowered. Together with colleagues including MIT Professor Stefanie Mueller, Turakhia has since developed a new system to teach computational making that lets kids fabricate their favorite characters straight from digital games.

Q: What are some of the ways in which we can re-imagine teaching maker skills to young children?

A: One of the key factors in teaching skills to young children is to keep them engaged, interested, and inspired. So we challenged ourselves to re-imagine how digital fabrication can be introduced and taught to young learners in a playful and fun way.

We designed a novel approach where we combined teaching fabrication with playing video games. Kids already play a myriad of video games that have countless digital objects and characters that the players engage with. So we thought to ourselves, what if, while playing these games and interacting with the digital objects and characters, kids can fabricate them to interact with them in the physical world, and learn fabrication and maker skills along the way? 

Consider an example where a young learner plays the game “PokémonLet’sGo” and every time they capture a new Pokémon, they also receive the fabrication files to make a physical Pokémon to add to their collectible. Or imagine when a kid playing the game the “Legend of Zelda” acquires one of the rarest items, the Biggoron's Sword, she can fabricate a physical version of the rare sword and use it as a game controller. Now, these young learners are able to play their favorite engaging games, and fabricate game objects that they may have personal connections with — while also gaining knowledge about digital fabrication, such as tool settings and material specifications, along the way. 

Implementing this vision of teaching fabrication through games, however, comes with its two main challenges. The first challenge is converting existing digital games into fabrication games without open access to the game’s source code. The second challenge is generating fabrication files of the desired game objects without access to the game’s repositories or asset files. In our work, we addressed both these challenges by using computer vision algorithms for object detection, segmentation, and extraction, and built a toolkit called FabO.

The FabO toolkit allows educators to select significant moments within existing games — for example, capturing Pokémons or acquiring the Biggoron's Sword — and tag them as fabrication events. When learners play these games, FabO monitors their gameplay searching for the tagged fabrication events. When they encounter a fabrication event, FabO auto-generates the fabrication files for the game objects within the event and notifies the learners. The learners can fabricate the objects from their personal gameplay during or after their game. 

This idea of designing fabrication games for learning can personalize the maker-skill learning experience for young learners in an engaging and meaningful way by bringing game objects from their gameplay into the physical world. 

Q: What utility does expanding gameplay back into the physical world have?

A: This is a great question! Once we built our FabO toolkit, our next research inquiry focused on exactly this question: Given the opportunity to bring the digital objects into the physical world, in what ways could the experience of both — learning and gaming — be expanded?

To address this question, we conducted an exploratory study where we invited participants to use FabO and convert existing video games of their choice into fabrication games. We then analyzed the characteristics of 47 fabricated objects from 33 different games that the participants chose to modify into fabrication games using FabO. Our analysis indicated that this idea not only allowed us to merge the two worlds of virtual gaming and tangible interaction through fabricated objects but also allowed the creation of objects that the learners have personal associations and meanings attached to. In other words, this idea allowed every learner to bring objects unique to their virtual gaming experience into the tangible world, something like a timestamp of their gameplay movement. 

This personalization provided an opportunity to add another layer of personal storytelling to their learning experience. For example, several learners playing the game of “PokémonLet’sGo” would learn the same skills of digital fabrication, but each fabricated object is unique based on their gameplay! 

Through our analysis, we identified five common categories or ways in which the learners attached meanings and personal associations with the objects fabricated from their gameplay — objects of pride, objects of creative self-expression, objects of resources, objects useful for expanding the gameplay into the physical world, and objects of shared experience. The last category of shared experience is particularly unique for multiplayer games where these objects are associated with the shared moments of gameplay, like collective wins or team losses. So, in the case of multiplayer games, yet another dimension of social connection and shared learning experience gets attached to the objects fabricated from their shared gameplay. 

Q: How could you see a system like FabO being used in the Metaverse or with Roblox?

A: We see augmented reality as a natural extension of our system application. The promise of virtual universes like Metaverse and Omniverse, especially the AR [augmented reality] environments, is that they can support learning through the seamless merging of the digital and physical worlds. This seamless integration of the physical and the virtual is especially game-changing (pun intended!) for learning maker skills because of the physical nature of the skills. 

With a system like FabO, the experience of playing and learning can be integrated further in an immersive way. So imagine a young learner playing the AR game of “PokémonGo” captures a Pokémon in a virtual world and then is transported into a virtual fabrication lab to train on how to use the digital fabrication tools, like laser-cutter and 3D printer, to fabricate their unique Pokémon. Once she completes her training, she can confidently fabricate the physical version of her Pokémon. This physical object can be brought back into the AR world for a more interactive gameplay experience — for example, during the Pokémon battles. 

Furthermore, because FabO allows any user (such as an educator) to design the fabrication events for another user (such as a learner), this feature can be extended for several interesting social gaming experiences, especially for platforms like Roblox. For example, an educator could design co-making puzzles for her class of young learners where each learner fabricates their gameplay object, which is a piece of a larger puzzle that the entire class builds together. 

Another scenario in which social gaming can have an interesting intersection with FabO is where users embed fabrication events for each other inside the same game. So even though players could be playing the same game, depending on whose “FabO version” they play, the fabrication events and objects could widely vary, and thus the experience of learning fabrication could be unique. 

We therefore think that there are several exciting avenues in which applications for a system like FabO could be expanded and we are excited to pursue these directions in our research.