Aerial view from inside the student's Varjo headset of densely populated Hajin, Syria, terrain in the mixed-reality Sand Table of the JTAC trainee's eye-gaze (red and blue cones) and the instructor's view (green frustum).
Student zoomed-in view inside the Varjo headset of a virtual sand table scene displaying DIS-based entities, the JTAC trainee’s eye gaze (red and blue cones) and the instructor’s view (green frustum) with real-world view.
MVRsimulation’s mixed-reality Sand Table provides a shared 3D visualization tool for realistic geospecific mission planning/rehearsal, classroom-deployed tactics training, real-world mission planning, and After Action Review (AAR). The mixed-reality Sand Table allows commanders, instructors, trainees, and students to interactively plan and review training missions in a collaborative, mixed-reality setting.
Traditional military sand tables consist of 2D maps, aerial photographs or sand molded into 3D representations of terrain, upon which tokens or cards are placed to represent friendly and enemy forces ahead of military actions. The MVRsimulation Sand Table replaces this physical system with a mixed-reality 3D ‘table’ upon which the virtual battlespace is rendered by Virtual Reality Scene Generator (VRSG) in its geospecific, round-earth 3D format. This allows users to understand how the terrain, elevation, and physical features of the landscape, such as mountains, valleys, trees and waterways will impact planned maneuvers; along with cultural and man-made features such as buildings, dense urban areas, roads, bridges, dams or other cultural infrastructure. Users can plan, enact, and review training scenarios while directly interacting with cultural objects in VRSG or by the use of any DIS-based Semi-Automated Forces (SAF) commercial or government off-the-shelf (GOTS) software.
Users are immersed in the virtual sand table via the use of Varjo mixed-reality headsets. The Varjo headsets provide the mixed-reality environment that allows the wearer to freely explore the 3D visualization of the scenario and geospecific 3D terrain. Mixed-reality provides real-world pass through for direct interaction with other observers in the real-world. Users can also operate physical emulated military equipment in the real world and simulated military training packages such as ATAK via the Special Warfare Assault Kit (SWAK) without the need to remove the headset. An instructor can interact directly with a student in the mixed-reality sand table while streaming the visuals of both users to an external monitor for classroom instruction. The instructor stays immersed in the sand table while also being able to see the other students in the classroom past the edge of the virtual sand table.
Each user has a Valve Index Controller to control their own unique view within the sand table. The controllers allow each user to independently traverse the terrain to evaluate specific views of the 3D virtual world. Users can zoom in and out with the controllers to give them vantage points from a close up evaluation of a single entity to a top-down view of the entire battlefield. The Valve Index Controllers also provide a virtual pointer used to point to specific elements in mission planning or AAR and to direct other participants’ attention to a particular location or entity.
During Sand Table use, the participant’s eye gaze is captured by the Varjo headset and depicted as a colored frustum allowing each user to see where all participants in a scenario are looking. The colored frustum is valuable for student training as it allows the instructor to verify in real-time if the student is looking at the correct location or entity. Other Varjo headsets connected to the DIS-based network can output their eye gaze tracking from VRSG. Eye-gaze tracking from role players in AAR is critical to remove ambiguity from the question of where a student’s attention was focused throughout the mission.
The network and LVC-ready Sand Table can be used as a stand-alone training and evaluation asset. It can also be integrated with any DIS-based training simulator such as MVRsimulation’s Deployable Joint Fires Trainer (DJFT) or Part Task Mission Trainer (PTMT). VRSG listens for DIS entities on the network and displays them as 3D models in the virtual sand table. DIS recording capabilities facilitate recording and playback of training sessions. During playback the instructor has control of the recording allowing them to pause the recording for all users to evaluate the full 3D virtual world including the geospecific terrain, 3D entities, line of sight, and student eye gaze.
Users are immersed in the virtual Sand Table via the use of the Varjo mixed-reality headset. Users interact with physical devices in the real-world as well as read and write without the need to remove the headset.
Instructor view of the 3D virtual terrain: the student is seen here in the real-world view just beyond the edge of the virtual sand table with correlated blue frustum displaying the current view of the student.
- Mission Planning/Rehearsal: Prior to carrying out a training mission in a simulator such as the DJFT or PTMT, trainees can meet with their training counterparts and instructor to plan the mission, watch a run-through of the scenario, and make adjustments to the mission as required.
- Stand-alone classroom training: In classroom settings, the Sand Table provides a critical learning tool for students and instructors to engage in mission planning and rehearsal to train for tactics and strategic decision making. Students see the results of their decisions in real-time and gain immediate feedback from the instructor. Other students in the room can watch the view of the headsets projected onto screens while the instructor and student discuss their tactics.
- Real-world mission planning: A mixed-reality replacement for current mission planning methods in theatre, the Sand Table allows users to rehearse actual mission plans, giving all parties a crystal-clear understanding of how to complete the mission most effectively. When connected with a DIS-based SAF, users can also implement the mission from their planning to evaluate the success of the mission objectives.
- After Action Review: Following training missions in networked simulators, students and instructors can review a DIS stream playback, allowing them to experience the mission from all angles, review critical outcomes and deepen learning opportunities. AAR of the trainee’s recorded eye-gaze log also allows students and instructors to understand where the trainee was looking throughout the mission, giving insight on whether critical cues or incidents were missed.