General education courses such as culture, history and oral communication are difficult to teach in classrooms due to the lack of authentic contexts. Virtual reality (VR) can offer immersive environments for a situated learning experience, and it is also well grounded in learning theory, in particular constructivism, which encourages learning from experiences. However, there are few existing guidelines on how to design such VR-based learning systems to support general education. Situated cognition theory suggests three key elements of situated learning: an authentic learning context, social interaction and collaboration, and progressive training. For this thesis, we conducted three empirical studies to investigate the three elements separately in the context of general education...[ Read more ]

General education courses such as culture, history and oral communication are difficult to teach in classrooms due to the lack of authentic contexts. Virtual reality (VR) can offer immersive environments for a situated learning experience, and it is also well grounded in learning theory, in particular constructivism, which encourages learning from experiences. However, there are few existing guidelines on how to design such VR-based learning systems to support general education. Situated cognition theory suggests three key elements of situated learning: an authentic learning context, social interaction and collaboration, and progressive training. For this thesis, we conducted three empirical studies to investigate the three elements separately in the context of general education with VR.

For the first study, we developed ShadowPlay2.5D, a 360-degree video authoring tool for immersive appreciation of classical Chinese poetry. Owing to the lack of authentic contexts, learning and appreciating classical Chinese poetry can be challenging. Using Chinese shadow play as a metaphor, we designed and implemented a sketch-based authoring tool to help novices easily create 360-degree videos about classical Chinese poetry. Through two user studies, we show that ShadowPlay2.5D can help novices make a short 360-degree video in about 10-15 minutes, and the 2.5D stylized illustrations created can bring about a better immersive experience for poetry appreciation.

For the second study, we developed Live Emoji, a live storytelling VR system with programmable cartoon-style emotion embodiment. While existing storytelling systems for democratizing VR technology, such as Google VR Tour, use 360-degree images to immerse users in a lifelike environment, engaging learners in a socially interactive way is not automatic. In fact, it can be quite difficult. Thus, we propose a novel cartoon-style hybrid emotion embodiment model to increase a storyteller’s presence during live performance. We further designed and implemented a system to teleoperate the embodiment model in VR for live storytelling. Based on interviews with three experts and a workshop study with local secondary school students, we show the potential of the emotion embodiment model on VR storytelling for education.

In the third study, we explored whether a VR coach with embodied feedback could foster a situated learning experience for progressive training. To formulate our design, we interviewed experts and observed real elevator pitches. We then designed a VR coaching system with three different embodied feedback strategies. Through a between-subject experiment with 40 participants, we found that receiving embodied feedback can create a strong sense of cognitive apprenticeship, i.e., coaching and helping from experts, and can also help improve the perception of the virtual character and the effect of learning.

Through these three studies, we gained practical insights into VR and situated learning. Thus in this thesis, we summarize important design guidelines of VR systems for supporting general education.

Finally, as a first step towards developing computational intelligence for supporting situated learning, we studied language grounding, which connects language to the real world, because of the importance of language on education and the necessity of an authentic context for situated learning. In particular, we used machine learning methods to study physical common sense learning and emotion recognition from the aspects of model generalization and small sample sizes, respectively. In the first study, we formulated physical common sense learning as a knowledge graph completion problem. We propose a novel pipeline that combines pre-training models and knowledge graph embedding to increase the generalization ability of our model to predict physical common sense. In the second study, we devised an efficient meta-learning approach to learn text emotion distribution from a small training sample.

We conclude this thesis by sketching further plans for building conversational agents to support situated learning, reducing cybersickness through mixing the physical world into the virtual world and human perception-optimized planning, and developing applications for cultural heritage education.