Research Gate
`
`See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/277295097
`
`Applications of Mixed Reality
`
`Article · January 2009
`
`CITATIONS
`2
`
`1 author:
`
`Hannes Kaufmann
`TU Wien
`
`139 PUBLICATIONS   3,507 CITATIONS   
`
`SEE PROFILE
`
`READS
`1,028
`
`Some of the authors of this publication are also working on these related projects:
`
`Project
`
`Project
`
`Flexible Spaces View project
`
`Towards User-Adapted Navigation Techniques in Virtual Environments: Study of Factors Influencing Users Behavior in Virtual Reality View project
`
`All content following this page was uploaded by Hannes Kaufmann on 28 January 2016.
`
`The user has requested enhancement of the downloaded file.
`
`Exhibit 1022 page 1 of 95
`DENTAL IMAGING
`
`

`

`
`
`
`
`
`
`Habilitationsschrift
`
`
`Applications of Mixed Reality
`
`
`ausgeführt zum Zwecke der Erlangung der venia docendi
`im Habilitationsfach ”Angewandte Informatik”
`
`
`
`eingereicht im Mai 2009 an der
`Technischen Universität Wien
`Fakultät für Informatik
`
`
`
`von
`
`
`
`Mag.rer.nat. Dr.techn. Hannes Kaufmann
`Ospelgasse 1-9/1/39, 1200 Wien
`kaufmann@construct3d.org
`
`
`
`
`
`
`
`Wien, am 27. Mai 2009
`
`Exhibit 1022 page 2 of 95
`DENTAL IMAGING
`
`

`

`Contents
`
`
`
`Introduction ................................................................................................................. 1
`
`
`1. Kaufmann H., Schmalstieg D.
`Mathematics and geometry education with collaborative augmented reality
`Computers & Graphics, Volume 27, Issue 3, pp. 339-345, June 2003. ................ 20
`
`
`2. Kaufmann H., Dünser A.
`Summary of Usability Evaluations of an Educational Augmented Reality
`Application
`R. Shumaker (Ed.): Virtual Reality, HCI International Conference (HCII 2007),
`Volume 14, LNCS 4563, ISBN: 978-3-540-73334-8, Springer-Verlag Berlin
`Heidelberg, pp. 660–669, July 2007. .................................................................... 27
`
`
`3. Kaufmann H.
`Dynamic Differential Geometry in Education
`to appear in Journal for Geometry and Graphics, vol. 13, 1, 2009.
`(accepted for publication) ..................................................................................... 37
`
`
`4. Kaufmann H., Csisinko M., Totter A.
`Long Distance Distribution of Educational Augmented Reality Applications
`Eurographics’06 (Educational Papers), pp. 23-33, Vienna, Austria, September
`2006. ..................................................................................................................... 50
`
`
`5. Kaufmann H., Csisinko M.
`Multiple Head Mounted Displays in Virtual and Augmented Reality
`Applications
`International Journal of Virtual Reality, vol. 6, no. 2, pp. 43-50, June 2007. ....... 60
`
`
`6. Kaufmann H., Meyer B.
`Simulating Educational Physical Experiments in Augmented Reality
`Proceedings of ACM SIGGRAPH ASIA 2008 Educators Program, Singapore,
`ACM Press, New York, NY, USA, December 2008. ............................................ 68
`
`
`7. Kaufmann H., Csisinko M., Strasser I., Strauss S., Koller I., Glück J.
`Design of a Virtual Reality Supported Test for Spatial Abilities
`in Proceedings of the 13th International Conference on Geometry and Graphics
`(ICGG), Dresden, Germany, pp. 122-123, August 2008. ..................................... 76
`
`
`8. Csisinko M., Kaufmann H.
`Towards a Universal Implementation of 3D User Interaction Techniques
`in Mixed Reality User Interfaces: Specification, Authoring, Adaptation
`(MRUI'07), Workshop Proceedings, Charlotte, North Carolina, USA, pp. 17-25,
`March 2007. .......................................................................................................... 86
`
`Exhibit 1022 page 3 of 95
`DENTAL IMAGING
`
`

`

`
`
`
`Introduction
`
`This thesis gives an overview of the author’s scientific work in previous years. It
`reflects the author’s ambition to develop applications of mixed reality which are
`beneficial to society as a whole or to specific groups of people. Providing and
`deploying high-end mixed reality hardware and software applications to multiple users
`and larger target groups finally raises questions of scalability, robustness, design and
`affordability of the technology involved. They trigger scientific questions and
`developments in return. All of these aspects will be touched in this work.
`
`The first part of the introduction defines the scientific domain and various problems
`therein followed by a discussion of the author’s contribution in this area. The
`individual publications that constitute the remainder of the thesis are briefly discussed
`and put in context.
`
`
`Definitions
`
`Since the area “Applications of Mixed Reality” – chosen as the title of this thesis – is
`very broad it is important to establish common terms in the beginning.
`
`In order to classify virtual reality (VR) research Milgram and Kishino [1] published a
`taxonomy 15 years ago. Although the field widened and diversified over the years
`their work still provides a rough framework which helps to classify any work done in
`this area. We refer to the virtual continuum (Figure 1) as discussed in Milgram and
`Kishino’s paper. The virtual continuum represents a continuous set of (infinite)
`possibilities between real environments and fully virtual environments (VEs). All
`environments within that range (except the extremes of fully real and virtual
`environments) are considered mixed realities.
`Mixed Reality
`
`Real
`Environment
`
`Augmented
`Reality
`
`Augmented
`Virtuality
`
`Virtual
`Environment
`
`Virtuality Continuum
`Figure 1: The Virtuality Continuum
`
`Further on the authors specify and classify hardware and software environments
`within the virtual continuum and define six classes of “hybrid display environments”.
`In a second paper [2] they add a seventh class. Nevertheless given the broad range of
`
`
`
`Exhibit 1022 page 4 of 95
`DENTAL IMAGING
`
`

`

`virtual reality hardware and setup variations available today it is neither always clear
`nor easy to identify into which category a specific setup falls.
`Related to Milgram’s taxonomy many applications presented in this thesis belong to
`class 6 (defined in [1]) which states
`“6. Completely graphic but partially immersive environments (e.g. large screen
`displays) in which real physical objects in the user's environment play a role in
`(or interfere with) the computer generated scene, such as in reaching in and
`"grabbing" something with one's own hand […].”
`They mention further
`“We note in addition that Class 6 displays go beyond Classes 1, 2, 4 and 5, in
`including directly viewed real-world objects also. As discussed below, the
`experience of viewing one's own real hand directly in front of one's self, for
`example, is quite distinct from viewing an image of the same real hand on a
`monitor, and the associated perceptual issues (not discussed in this paper) are
`also rather different. Finally, an interesting alternative solution to the terminology
`problem posed by Class 6 as well as composite Class 5 AR/AV displays might be
`the term"Hybrid Reality" (HR), as a way of encompassing the concept of
`blending many types of distinct display media.”
`
`
`Milgram and Kishino described three additional dimensions that distinguish different
`mixed reality systems: Extent of world knowledge (i.e. degree of knowledge of the
`real world by the application), reproduction fidelity (visual quality) and extent of
`Presence metaphor. Presence in short can be defined as a subjective phenomenon of
`the sensation of being in a virtual environment [3, 4]. It is the most researched
`dimension of the three and of high importance when designing new applications [5].
`Different concepts and interpretations of presence have been discussed [5]. Whereas
`some applications require full presence of users others might require shared and equal
`awareness of the real and virtual e.g. in educational applications where teachers are
`outside the VE guiding students. With different Mixed Reality (MR) setups these
`variations can be achieved while maintaining a high level of presence in all cases.
`Appropriate and corresponding examples of application areas and target groups will
`be mentioned later.
`
`Our work fulfills as well Azuma’s definition of Augmented Reality (AR) [6], who
`defines AR as systems that have the following three characteristics:
`1) Combine real and virtual
`2) Interactive in real time
`3) Registered in 3-D
`The presented research covers a wide range of environments which are always
`interactive in real time but fulfill items 1 and 3 to varying degrees. The variety of
`systems is better encompassed by the term Mixed Reality or even Hybrid Reality – the
`latter term was not in use after Milgram and Kishino coined it.
`
`
`System Architecture
`
` A
`
` wide variety of MR hardware and software setups are imaginable and have been
`built in the past. However all share a common general system architecture. The five
`key elements of an MR system [7] comprise input and output devices whose spatial
`position and orientation might be tracked, a computing platform with a powerful
`
`Exhibit 1022 page 5 of 95
`DENTAL IMAGING
`
`

`

`graphics processor and a VR/MR software framework handling input, output and
`application behavior. The most important part is the user (or multiple, collaborating
`users) working on a certain task and interacting with the system.
`Up to the end of the 20th century high-end graphics workstations dominated the
`VR/MR market. Due to the exponential performance increase of graphics hardware
`(due to the growing gaming market) starting in the mid 1990’s PC graphics hardware
`reached an acceptable level of performance at the beginning of this decade. It is the
`main platform used in MR systems nowadays. High quality real-time rendering
`became state of the art in MR systems [8]. Diverse technologies for tracking are in use
`to determine the location of input, output devices, the user or specific body parts of
`the user up to full body motion capture [7]. Optical tracking evolved into the de-facto
`standard in recent years. In stationary indoor setups infrared-optical tracking based on
`retro-reflective markers is frequently used whereas outdoor applications utilize
`computer vision algorithms to perform natural-feature tracking (in combination with
`high-sensitive GPS or differential GPS, compasses, inertial and other sensors).
`Input from (tracked) devices is typically handled by so called tracking middleware [9,
`10] which supports a wide range of devices, pre-processes input events and passes
`them to the MR application. Most software frameworks are based on scene graph
`libraries for example open source toolkits such as Studierstube [11], VR Juggler [12],
`Avango [13] or commercial ones such as 3DVIA Virtools [14] and provide additional
`support for (stereo) output devices.
`A comprehensive overview of VR technology including input, output devices and
`graphics architectures is given in [7]. Further details on hardware, software and
`application requirements with additional chapters on design and implementation
`approaches and evaluations are to be found in [15]. Looking at an early book on this
`topic [16] gives an insight on how technology changed over the years.
`
`Within Augmented Reality a lot of research focused on mobile devices and
`applications for these devices in recent years. Stationary and mobile systems have a
`few opposing characteristics. While devices can be tracked at very high precision
`(sub-millimeter accuracy) within a stationary setup tracking data is usually imprecise
`(from centimeters to meters) in mobile setups. In general conditions in an indoor
`environment can be controlled well whereas outdoor controllability might be low. Full
`immersion is harder to achieve (or even unwanted) in a mobile setting compared to a
`stationary one. These and other characteristics reflect on the type of application
`scenario that can be realized. The author’s work is limited to application areas which
`require stationary setups only, mainly because of high precision requirements. This
`means that all users interacting with the application are always inside a room or
`connected via a distributed setup in multiple rooms.
`
`Little systematic work has been done on software design and implementation of
`virtual environments. Wilson et al. presented a structured approach called “Virtual
`Environment Development Structure” (VEDS) [17] which can be used as a
`methodology for MR software engineering. The author of this thesis followed a
`similar approach as described by Hix & Gabbard [18] namely usability engineering of
`virtual environments.
`
`There are various approaches for application development. As Bimber [19] states
`“We believe that a rich pallet of different display technologies, mobile and non-
`
`Exhibit 1022 page 6 of 95
`DENTAL IMAGING
`
`

`

`mobile, must be considered and adapted to fit a given application so that one can
`choose the most efficient technology”.
`The author fully agrees with an application-centered and user-centered approach. User
`requirements have always been taken into account while designing and developing
`applications presented in this thesis. The whole range that the virtual continuum offers
`should be considered to find optimal solutions for specific end users with a specific
`task and goal in mind.
`
`
`Application Areas of Interest
`
`Ongoing is the search for so called killer applications of virtual reality, augmented
`reality and mixed realities in general. The defining criterion for a killer application is
`usually commercial success. For many reasons – a topic that filled many articles and
`continuously triggers discussions – such an application or area of application has not
`been found yet.
`Nevertheless as Jaron Lanier, VR pioneer who coined the term “virtual reality”, states
`in [20]:
`“[…] As used in industrial technology, there's no question that virtual reality has
`already been a success. You can't buy a car today that wasn't designed using it.
`And you can't put gas in that car that wasn't made out of oil that was discovered
`using virtual reality through an oil field simulation. Most new drugs are made in
`a process assisted by virtual reality. There are many other examples. […]“
`Mixed reality applications in industrial areas of design, prototyping and marketing
`have been successful since the beginnings as well as applications in architecture and
`naturally data visualization. Applications in entertainment are mainly successful in
`theme parks. Due to high (hardware) costs and a lack of maturity of early devices
`(including some novel “VR devices” that target the gaming market nowadays) they
`did not find broad acceptance in the consumer gaming market yet.
`
`In addition there are other very successful applications areas which are typically not
`considered mainstream. They are of primary interest in the context of the author’s
`work:
`• Psychology
`• Medical Sciences
`• Education
`• and combinations of these.
`
`MR technologies are nowadays used in all major directions of theoretical and practical
`work within psychology [21-23] – in research, education, therapy and rehabilitation,
`and in most of the psychological academic disciplines – cognitive, organizational,
`social, clinical, differential, instructional psychology, as well as in philosophical and
`neuropsychological studies of conscience. This relatively new field is being rapidly
`accelerated in universities, and partly in hospitals and rehabilitation centers, mainly in
`North America and the EU (particularly, in Spain, UK, Germany, the Netherlands and
`Italy), in somehow lesser extent – in Israel and several Pacific countries. The results of
`these projects, either finished or currently in progress, include for example pilot and/or
`professional MR systems which support psychological assessment and treatment of
`anxieties, phobias and post-traumatic stress disorder (exposure therapy) [24];
`
`Exhibit 1022 page 7 of 95
`DENTAL IMAGING
`
`

`

`cognitive behavioral therapy in general; MR environments to conduct psychological
`tests in 3D of e.g. spatial abilities [25, 26]; instructional MR applications in
`experimental psychology – as diverse as for example psychology of visual perception
`and psychology of manipulative persuasion – and many more.
`An area with higher success rates than traditional in vivo therapy is clinical
`neuropsychology where mixed reality applications are used as a therapeutic tool for
`e.g. treatment of anxieties [27] and also post traumatic stress disorder (PTSD), chronic
`pain and many more.
`In some aspects these applications fall into and overlap with the medical domain
`which is another successful and growing MR application area. Examples are MR-
`based rehabilitation systems for e.g. physical rehabilitation or for rehabilitation of
`stroke patients; the assessment and treatment of impairments found in persons with
`central nervous system dysfunction
`including Alzheimer's Disease, Vascular
`Dementia, Parkinson's Disease [28]; assessment after traumatic brain injury; MR
`support for the disabled etc. In addition there is ongoing research regarding MR
`(specifically AR and VR) simulators for medical training e.g. to acquire specific
`surgical skills [29].
`
` large part of this thesis focuses on applications of mixed reality in education
`(overviews in [30], [31], [32](chapter 2)) and training. Therefore the problems and
`challenges related to this application domain will be discussed briefly (a summary is
`given in [33]).
`It is interesting to note that nearly all of the educational projects reported in literature
`reached a certain point where trial studies and evaluations were conducted and then
`the projects ended e.g. [34, 35]. No reports about continuous progress, no iterative
`development process and ongoing tests can be found in literature which go a step
`further. Therefore no development process comparable to usability engineering [18]
`took place to optimize the application regarding usability and effectiveness for end
`users. There are exceptions e.g. work by Adamo-Villani [36, 37] but only few.
`A major challenge of this young application area is that there are no studies proving
`the effectiveness of MR learning yet. In this context “effectiveness” measures the
`learning outcome achieved using an educational MR application in comparison to
`traditional teaching. The difficulty in comparing learning outcome is the comparability
`or rather incomparability of traditional and MR learning scenarios. Since a virtual
`learning environment is typically designed to provide added benefit to learners
`(compared to a traditional setting) it might introduce advanced or new learning
`contents. These might be hard or impossible to do in traditional environments.
`Therefore it is difficult to find learning tasks for evaluation purposes that do not
`penalize one scenario e.g. by over-simplifying an MR learning task to be solvable
`within reasonable time by traditional methods whereby eliminating the strength of the
`MR environment.
`A body of work has been done on the theoretical pedagogical foundations of
`educational VEs [30, 38] whereas pedagogical guidelines about how to teach in VEs
`are rare. Evaluations of MR learning environments with a large number of users (>50)
`are difficult to find as well.
`
`The author’s contribution can be found in the latter areas: psychology in the broad
`sense defined above – with recent work reaching into the medical domain – and
`education which includes training. A specific field opened up between psychology and
`
` A
`
`Exhibit 1022 page 8 of 95
`DENTAL IMAGING
`
`

`

`education motivated by the work on the first application [39] namely the education
`and training of spatial abilities.
`
`
`Motivation, Challenges and Context of the Conducted Research
`
`In order to solve three dimensional mathematical but especially geometrical problems,
`spatial abilities are an important prerequisite [40-42]. During his studies of
`mathematics and geometry with the aim of becoming a teacher, the author gave
`countless private lessons to students of these subjects. A personal observation was that
`many students had difficulties solving tasks that required spatial visualization skills
`and spatial thinking. To get passing grades they used strategies such as learning
`construction steps by heart without fully understanding spatial problems and their
`solutions in 3D space.
`
`Geometry Education
`With the emergence of mixed reality technologies it became possible to immerse users
`in artificial worlds that are impossible or difficult to reproduce in reality. A number of
`training studies have shown the usefulness of VR in training spatial ability [25, 43].
`However, little to no work has been done towards systematic development of VR
`applications for practical educational purposes in this field.
`In our first paper we introduce Augmented Reality to mathematics and geometry
`education. The simultaneous sharing of real and virtual space in AR is an ideal match
`for computer-assisted collaborative educational settings. We have developed an
`application called Construct3D, a three dimensional geometric construction tool
`specifically designed for mathematics and geometry education. The main advantage of
`Construct3D to student learning is that students actually see three dimensional objects
`in 3D space which they until now had to calculate and construct with traditional
`methods. Augmented reality provides them with an almost tangible picture of complex
`three dimensional objects and scenes. It enhances, enriches and complements the
`mental images that students form when working with three dimensional objects.
`
`However, there are a number of requirements and challenges for a mixed reality tool
`with the aim of effectively improving spatial skills. They motivated work on
`Construct3D and have not been addressed by existing systems, nor studied in an
`educational context before:
`• No VR/AR application for actual use in high school or higher education has been
`developed with the main purpose of improving spatial skills.
`• No VR/AR application existed prior to Construct3D that offered the flexibility to
`dynamically construct and modify 3D geometric content directly in 3D space. For a
`definition and explanation of “dynamic geometry” please refer to [44, 45].
`• Hardly any evaluations could be found in literature giving hints to the actual learning
`transfer from a VR/AR learning environment to the real world.
`
`The first four papers in this thesis summarize the author’s ongoing efforts towards
`filling these gaps. The first paper titled “Mathematics and geometry education with
`collaborative augmented reality” [39] is chronologically the earliest from these
`included. It introduces to the area and presents from a mainly technological point of
`
`Exhibit 1022 page 9 of 95
`DENTAL IMAGING
`
`

`

`view the first version of the application as well as multiple hardware setups for
`educational use.
`Regarding the earlier mentioned short duration of related educational projects our
`work goes one step further. Construct3D is one of the longest developed educational
`applications so far. We studied how ongoing technological improvements (over a
`course of 8 years) can be integrated into an MR system and looked at pedagogical
`questions such as how to adapt contents of the current high-school curriculum to the
`new learning environment.
`Construct3D was evaluated multiple times with over 500 users in total (students,
`teachers and experts) over the course of 5 years. A summary of three usability
`evaluations and findings is given in “Summary of Usability Evaluations of an
`Educational Augmented Reality Application” [46].
`Based on the second evaluation Construct3D was redesigned with the help of a
`professional graphics designer to improve usability and effectiveness within the
`Lab@Future [47] EU FP5 IST project. In that context general design guidelines were
`formulated in “Designing Immersive Virtual Reality for Geometry Education” [48]
`which are partly applicable to other (educational) MR applications as well. This
`publication is not included herein but part of the appendix of the thesis. We describe
`improvements in the user interface and visual design and report on practical
`experiences with using our system for actual teaching of high school students, and
`present initial quantitative data on the educational value of such an approach.
`The third paper is chronologically the last published paper in the collection. It closes
`the circle of work done on Construct3D and presents the end of an evolution from a
`merely technological focus to an application-centered focus. In “Dynamic Differential
`Geometry in Education” [45] educational dynamic geometry was introduced to the
`specific domain of differential geometry. Construct3D is the only available tool that
`can be used to study this application area. The focus lies on differential geometry
`which can be explored in a new way using three-dimensional dynamic geometry in
`MR.
`
`Problems and Challenges
`In general Construct3D was praised by teachers and students who had used it and
`questions arose about disseminating it to schools. During evaluations major
`hindrances became obvious which avoided the installation of mixed reality
`environments in schools. Some of these issues are not specific to the educational
`sector but apply to other end user groups as well. According to interviews with
`teachers three main reasons hinder the dissemination of mixed reality technology to
`schools:
`1. Costs of the hardware and software environment.
`2. The need for maintenance of all equipment which requires personnel and again
`generates costs (even if the application is very robust and mature).
`3. The most effective and by users most preferred setup supports only a limited
`number of users (2-3) and questions arose about how to support larger groups
`of users.
`
`
`The author was trying to work on solutions to these problems in order to be able to
`spread MR technology to a bigger community. Some of these problems triggered new
`scientific questions.
`Regarding the support of multiple users (the third problem on the list) the following
`
`Exhibit 1022 page 10 of 95
`DENTAL IMAGING
`
`

`

`two papers “Long Distance Distribution of Educational Augmented Reality
`Applications” [49] and “Multiple Head Mounted Displays in Virtual and Augmented
`Reality Applications” [50] discuss different approaches to provide a virtual
`environment to larger groups of users.
`
` Matter of Costs
`Tracking is a critical part of any MR system and typically the most expensive one
`therefore relates to the first problem on the list. Costs of tracking systems especially of
`optical tracking systems which provide the highest accuracy (as needed by an
`application such as Construct3D) have always been high. They are sold by a small
`number of companies worldwide and due to a small end user market for such systems
`prices were kept constantly high for over a decade. In a configuration tailored to a
`room-sized multi-user environment, all of the existing optical tracking systems have
`price tags in the range of tens of thousands of Euros. While corporate entities and
`well-funded research laboratories will not be deterred by such amounts, it is the
`author’s first-hand experience that many smaller educational institutions, especially
`secondary schools, operate on tightly constrained budgets that leave little, if any, room
`for an expense of this magnitude, even if third-party subsidies are available. The
`urging matter of costs finally led to the development of iotracker - a low-cost infrared
`optical tracking system - which the author initiated, conceptually designed and guided.
`The main goal was to reduce costs of high quality optical tracking systems without
`sacrificing quality i.e. speed or accuracy. Iotracker is commercially available
`(http://www.iotracker.com) and has already been a success in this respect. Since the
`introduction of iotracker which is available at a fraction of the price of other systems
`some vendors have already reduced their prices or introduced new products at lower
`prices.
`Affordable tracking technology in return opens up new end user markets and new
`application areas. High quality tracking technology should not only be limited to
`members of the academic community, but also to artists, game designers, educators,
`small commercial application developers and all with an interest in Mixed Reality.
`The publication about iotracker “Affordable Infrared-Optical Pose-Tracking for
`Virtual and Augmented Reality” [51] is not included in this collection but in the
`appendix that accompanies this work. In the paper we describe the hard- and software
`of a new low-cost infrared-optical pose-tracking system for room-sized virtual
`environments. The system consists of 4-8 shutter-synchronized 1394-cameras with an
`optical bandpass filter and infrared illuminator. All image-processing is done in
`software on an attached workstation. Preliminary results indicate low latency (20-
`40ms), minimal jitter (RMS less than 0.05mm/ 0.02°), sub-millimeter location
`resolution and an absolute accuracy of ±0.5cm. Up to twenty independent 6-DOF
`targets can be tracked in real-time with up to 60Hz.
`
`Costs of a full MR system are still higher than those of computing equipment
`traditionally used in educational institutions and need to be justified well. If an MR
`system can be used in multiple courses and different subjects there will be a higher
`degree of utilization, it will be considered more useful than if there is only one
`application available - therefore it is more likely to be acquired.
`In “Simulating Educational Physical Experiments in Augmented Reality” [52] we
`present an augmented reality application for physics, more specifically mechanics
`education. It is based on the same technological setup as used for Construct3D
`
` A
`
`Exhibit 1022 page 11 of 95
`DENTAL IMAGING
`
`

`

`(including hardware and the software framework) and builds on experiences gathered
`during the development of the geometry application. PhysicsPlayground can be
`perfectly integrated into physics courses. It allows students to actively build own
`experiments and to study them in a three-dimensional virtual world.
`
`Spatial Abilities
`As mentioned in the beginning no mixed reality application for actual use in high
`school or higher education has ever been developed with the main purpose of
`improving spatial skills. The author’s motivation was to help students develop correct
`mental models of three dimensional problems and to improve their spatial thinking -
`to enable them to find solutions to geometric problems themselves.
`Based on psychological studies about mathematics and geometry education, national
`school authorities (e.g. Austria [53]) consider improving spatial abilities one of the
`main goals of geometry education. Spatial abilities present an important component of
`human intelligence. Spatial ability is a heterogeneous construct that comprises a
`number of sub-factors, such as mental rotation, visualization, and environmental
`orientation [54-57]. Many studies have shown that spatial abilities can be improved by
`well-designed training (e.g. [58]). Geometry education has proven to be a powerful
`means of improving these skills [59].
`The hypothesis for our work on Construct3D was that if students see three-
`dimensional objects directly in 3D space and can interactively construct, touch and
`modify abstract geometric objects, they later build mental models of complex
`geometric situations more easily in real life. In order to verify this hypothesis the
`author initiated an interdisciplinary research project “Educating Spatial Intelligence
`with Augmented Reality” (FWF P16803-N12). We evaluated the effects of an AR-
`based geometry training on spatial abilities. A summary of this project is given in
`“Virtual and Augmented Reality as Spatial Ability Training Tools” [60] which is not
`part of the thesis but included in the appendix. In this paper we first review studies
`that used MR technologies to study different aspects of spatial ability. Then results
`and findings are presented from an MR large-scale study with 215 students that
`investigated the potential of an AR application (Construct3D) to train spatial ability.
`Further project results were presented in [61-63].
`Our findings in the project indicate that augmented reality can be used to develop
`useful tools for spatial ability training. However within the training period of six
`weeks we were not able to me