1111111111111111 IIIIII IIIII 11111 1111111111 11111 1111111111 11111 11111 111111111111111 11111111
`US 20060092133Al
`
`c19) United States
`c12) Patent Application Publication
`Touma et al.
`
`c10) Pub. No.: US 2006/0092133 Al
`May 4, 2006
`(43) Pub. Date:
`
`(54) 3D MOUSE AND GAME CONTROLLER
`BASED ON SPHERICAL COORDINATES
`SYSTEM AND SYSTEM FOR USE
`
`(75)
`
`Inventors: Pierre Touma, Austin, TX (US); Hadi
`Murr, Beirut (LB); Elie Bachaalani,
`Beirut (LB); Imad Maalouf, Beirut
`(LB)
`
`Correspondence Address:
`Bert Vermeulen
`c/o Henry Waters and Associates
`4740 Table Mesa Drive
`Boulder, CO 80305 (US)
`
`(73) Assignee: Pierre A. Touma
`
`(21) Appl. No.:
`
`11/263,710
`
`(22) Filed:
`
`Oct. 31, 2005
`
`Related U.S. Application Data
`
`(60) Provisional application No. 60/624,335, filed on Nov.
`2, 2004.
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`G09G 5108
`(2006.01)
`(52) U.S. Cl. .............................................................. 345/158
`
`(57)
`
`ABSTRACT
`
`A computer input device constructed from at least one tilt
`accelerometer and at least one linear input element is
`disclosed. This input device can be used in a computer
`system to specify a position on a display using radial
`coordinates, cylindrical coordinates, or spherical coordi(cid:173)
`nates.
`
`101
`
`106
`
`103
`
`. ..
`... .......
`.... .... (:~ .......... ....
`
`104 ~
`~ 102
`
`108
`
`Align EX1022
`Align v. 3Shape
`IPR2022-00145
`
`

`

`Patent Application Publication May 4, 2006 Sheet 1 of 4
`
`US 2006/0092133 Al
`
`107
`
`,r
`
`o-· -·
`-
`:0
`
`f
`
`\
`
`c:::::i
`
`106
`
`103
`
`. ..
`
`........
`....
`.. ... ···
`·· ... : ............... ,,.
`
`1 0 4~
`~ 102
`108
`
`Fig. 1
`
`

`

`Patent Application Publication May 4, 2006 Sheet 2 of 4
`
`US 2006/0092133 Al
`
`rf
`..
`·····-..... I
`
`..c
`N
`
`0)
`
`u:::
`
`-
`
`'
`ill ..
`Ii--~ ~-
`LA~
`Q
`
`,....
`
`-
`0
`
`0 -
`J
`
`' --
`-
`
`co
`N
`
`0)
`
`u:::
`
`-- 0
`'
`ifllf:k
`~ _,.····
`.
`' -
`...
`) .. f.., .,
`~ -
`
`0
`
`....
`J
`
`"C
`N
`
`0)
`
`u:::
`
`C"
`
`-
`
`-- 0
`' ~
`~ Ii .,., .,·
`' -
`
`0
`
`-
`J
`
`CJ
`N
`
`0)
`
`u:::
`
`.-
`
`~ -
`
`-
`
`r
`
`

`

`Patent Application Publication May 4, 2006 Sheet 3 of 4
`
`US 2006/0092133 Al
`
`Fig. 3
`
`

`

`Patent Application Publication May 4, 2006 Sheet 4 of 4
`
`US 2006/0092133 Al
`
`-
`
`-
`
`......... -
`-
`-
`-
`-
`-
`Graphical 3D
`application
`
`1
`
`416
`
`r -
`I
`I
`I
`I
`I
`I
`
`I
`I
`I
`
`Processing unit
`
`Communications
`Module
`
`4015:
`------
`4~~3 -_-_-_-_-_-____ _
`
`415
`
`414
`
`413
`
`425
`
`I
`I
`I
`I
`
`Communications
`Module
`
`424
`
`423
`
`422
`
`Accelerometer(s)
`
`421
`
`User input
`elements
`
`Controller
`
`Power supply
`I
`L ________________ _
`I
`
`Fig. 4
`
`

`

`US 2006/0092133 Al
`
`May 4, 2006
`
`1
`
`3D MOUSE AND GAME CONTROLLER BASED
`ON SPHERICAL COORDINATES SYSTEM AND
`SYSTEM FOR USE
`
`[0001] This application claims priority based on U.S.
`Provisional Patent Application Ser. No. 60/624,335 entitled
`"3D Mouse/Pointer Based on Spherical Coordinates System
`and System for Use," filed 2 Nov. 2004.
`
`FIELD OF INVENTION
`
`[0002] The present invention relates to the field of com(cid:173)
`puter peripherals and controllers. One embodiment of the
`present invention relates to the control of 3D video games
`characters, home entertainment systems or more industrial
`applications such as robotics and the control of UAVs
`(umnanned aerial vehicles) and UGVs (umnanned ground
`vehicles). Specifically, one embodiment of the present
`invention relates to a method and apparatus for moving and
`controlling a cursor, object, character or mechanical system
`in a virtual or physical 3D environment. One embodiment of
`the present invention uses inertial sensing technology and an
`approach based on a mathematical representation of the 3D
`space with spherical coordinates, instead of the Cartesian
`representation mostly used in 3D applications.
`
`BACKGROUND OF THE INVENTION
`
`[0003] The tremendous computing power available at low
`cost in the early 21 st century has made possible many
`computer applications that previously were unattainable
`because of the computational resources required. A prime
`example is three-dimensional modeling. To compute large
`three-dimensional models and to manipulate them in real(cid:173)
`time requires large computational power, unless the models
`are very primitive. Today many applications, ranging from
`computer games with very high levels or realism to mod(cid:173)
`eling of sub-surface geological formations are possible on
`even relatively mainstream computer systems.
`
`[0004] A related trend is the merging of technologies such
`as televisions, home theatre, computers and game stations to
`produce PC Entertainment Centers. This trend is comple(cid:173)
`mented by the drive towards 3D games and game environ(cid:173)
`ments. One challenge, however, is to make full use of the
`three dimensional environments by giving the users attrac(cid:173)
`tive tools to manipulate objects or characters of these three
`dimensional environments.
`
`[0005]
`In the two-dimensional computing world, the
`mouse has become a ubiquitous feature for allowing a user
`to move a cursor around in the two-dimensional space.
`Moving the cursor with the mouse can be used to find and
`select particular objects. There is a need to be able to move
`a cursor to objects located in three-dimensional space as
`well as the need to move objects or characters in a 3D
`environment. This is much more challenging than moving a
`mouse across a tabletop as is the customary means for
`moving a cursor using a three-dimensional mouse.
`
`[0006]
`In the prior art there are several known methods for
`moving a cursor in three-dimensional space. These include
`moving a receiver with respect to a field established by
`external beacons or emitters/receivers, with respect to
`acoustic, magnetic or optical signals that may be detected by
`the receiver. Problems with such approaches include the
`need for using external devices.
`
`[0007] Other prior art solutions rely on gyroscopes to
`detect the movement of a 3D mouse, allowing the device to
`move a cursor in a 2D plan on the monitor. However, these
`solutions lack the 3D capability that is needed when dealing
`with 3D environments.
`
`[0008] From the foregoing it is apparent that there is a
`hitherto umnet need for a 3D pointing/controlling device
`that is self-contained, lightweight, and which uses low-cost
`components. The need is also apparent for a controlling
`device that could be used to remotely control mechanical
`systems such as Unmanned Air Vehicles (UAVs), UGVs
`Unmanned Ground Vehicles (UGVs), Unmanned Water
`Vehicles (UWVs) and other robotics systems, in a natural
`and efficient manner that is different from the method still
`followed today as represented by the control unit of model
`airplanes and the likes. One embodiment of the present
`invention can be used to address needs such as these.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0009]
`
`In the accompanying drawings:
`
`[0010] FIG. 1 shows a 3D input device used in a 3D
`computer system;
`
`[0011] FIG. 2 shows the detailed movement of the device
`and the related control of a vectorial cursor;
`
`[0012] FIG. 3 shows one embodiment of the 3D Mouse/
`Controller with the knobs and buttons used for interaction
`with a 3D environment; and
`
`[0013] FIG. 4 shows a block diagram of the 3D Mouse/
`Controller system and the way it interacts with a 3D appli(cid:173)
`cation on the computer monitor, through interrelated mod(cid:173)
`ules performing the different functions of: Movement
`Sensing, Sensing data interpretation and conversion to digi(cid:173)
`tal data, Wireless Communication of the data to an interface,
`Graphical rendering of the data in a 3D application.
`
`DESCRIPTION OF THE EMBODIMENTS
`
`[0014] This invention is described in one embodiment in
`the following description with reference to the Figures, in
`which like numbers represent the same or similar elements
`or process steps. While this invention is described in terms
`of the best mode for achieving this invention's objectives in
`a particular application, it will be appreciated by those
`skilled in the art that variations may be accomplished in
`view of these teachings without deviating from the spirit or
`scope of the present invention.
`
`[0015] For example, the present invention may be imple(cid:173)
`mented using any combination of computer programming
`software, firmware, or hardware. As a preparatory step to
`practicing the invention or constructing an apparatus accord(cid:173)
`ing to the invention, the computer programming code
`(whether software or firmware) according to the invention
`will typically be embedded in one or more machine readable
`storage devices such as micro-controllers, Flash memories,
`semiconductor memories such as ROMs, PROMs, etc.,
`thereby making an article of manufacture in accordance with
`the invention.
`
`[0016] The article of manufacture containing the computer
`programming code is used by either executing the code
`directly from the storage device, or by transmitting the code
`according to the present invention with appropriate standard
`
`

`

`US 2006/0092133 Al
`
`May 4, 2006
`
`2
`
`computer hardware to execute the code contained therein.
`An apparatus for practicing the invention could be one or
`more devices having network access to computer program(s)
`coded in accordance with the invention.
`
`[0017] One embodiment of the present technological inno(cid:173)
`vation relates to pointing (I/0) devices used to position or
`manipulate a vectorial object. Vectorial objects can be
`vectorial cursors, graphical symbols, or any pictorial repre(cid:173)
`sentation of physical or virtual object or character having
`one or multiple dimensions that has both a linear component
`( such as magnitude [ or size], or position in a Cartesian
`space) and an angular component (such as orientation). In
`particular one embodiment of the present invention relates to
`handheld devices that can be used to position or manipulate
`a vectorial object such as a vectorial cursor or 3D objects/
`Characters in three-dimensional space. A vectorial cursor in
`3D is the analog of a cursor in 2D. It is shaped like an arrow
`giving the user spatial feedback of the direction and position
`of the cursor. Depending on the application, the length of the
`arrow could be variable or fixed, whereby the arrow would
`be either extending from a spherical coordinates point of
`reference, or virtually moving in the 3D space. Thus, pro(cid:173)
`vided is an inertial sensor-based application related to a 3D
`Mouse that can act as a spatial pointer and can reach objects
`and icons in three dimensional environments and manipulate
`said objects, icons or characters. Such three dimensional
`environments could be generated by 3D graphical rendering
`or 3D GUis with 2D monitors, volumetric monitors or
`stereoscopic and holographic monitors.
`
`[0018] Such an embodiment of the present invention is
`based on inertial technology and methods that determine the
`position of a cursor in a 3D environment. This is achieved
`by mapping the movement of an operator's hand in space
`onto a polar coordinates frame of reference, thus optimizing
`the number of inertial sensors needed and reducing manu(cid:173)
`facturing cost. In such an embodiment, the application of the
`technology uses a single accelerometer in a form factor
`allowing it to be used as a desktop mouse or free-standing
`remote controller or game controller. In addition to its role
`as a mouse for the interaction with 3D environments and 3D
`GUis, the device/technology has the capability-in one
`embodiment-to act as a universal remote controller with
`both 2D and 3D interfaces of entertainment/media centers.
`
`[0019]
`In another embodiment the same approach could be
`used with a glove-like application allowing the user to
`interact with both 2D and 3D environments by limited
`movements of the hand and/or fingers. In a further embodi(cid:173)
`ment, it could also act as an advanced game controller for 3D
`games and could be coupled with haptic feedback. Further(cid:173)
`more, the method/technology could be applied in combina(cid:173)
`tion with portable game consoles (Gameboy, PSP ... )
`allowing players to interact with mobile 3D games through
`movements of the console itself, in combination with trig(cid:173)
`gers. This application is also useful with handheld comput(cid:173)
`ers and portable phones, allowing navigation through 2D or
`3D interface menus by moving the device itself instead of
`using a stylus or the operators fingers.
`
`[0020] Another embodiment of the technology would be
`as an add-on to game-specific sports hardware for a new
`generation of sports games (Baseball bat, Golf drive, Tennis
`racket, Skateboard, Skis, Luge ... ) and body movement
`games. In yet another embodiment, the technology could be
`applied for the control ofUAVs and other remote controlled
`aircrafts and/or their embedded systems such as cameras/
`other detection equipment. The same embodiment is appli(cid:173)
`cable to the control of model toys (aircraft, cars, boats ...
`). A person familiar with the art would also find that the
`technology has also applications in the field of medicine,
`engineering and sciences. It could be a virtual scalpel, a
`controller for a robotic arm, or a pointer for the manipulation
`of 3D molecules among other applications ...
`
`[0021] The present invention can provide a natural and
`ergonomic way to interact with 3D environments and to
`control systems in 3D space. This can be done by means of
`a 3-dimensional computer pointing and input device (3D
`Mouse/Controller) that uses a polar (spherical) coordinates
`approach implemented through the use of inertial technol(cid:173)
`ogy (accelerometer), to reach a point in 3D space and to
`control graphical symbols and animated characters in 3D
`environments.
`
`[0022] The present invention can be implemented using a
`3D Pointer concept. The three-dimensional pointer is
`achieved by using a spherical coordinate system. Its struc(cid:173)
`ture permits the user to access any point in his virtual
`environment by properly changing the device's directions
`and by increasing or decreasing the pointer length. The tilt
`angles, Pitch and Roll, captured from the accelerometer are
`used respectively as Alpha and Beta angles of the spherical
`coordinate system as illustrated in the equations below.
`While directions are captured from the hand movement by
`measuring the projection of the static gravity on the tilted
`accelerometer, the pointer length which is the physical
`analog of the radius R is simulated by using a trigger pair on
`the device. The user can change its pointer in order to reach
`the desired three-dimensional point by pressing the increase
`and decrease triggers. An alternative is to use a time varying
`pointer length. As a result the instantaneous position of the
`pointer in the inertial frame can be expressed as a function
`of the time-varying radius and spherical angles.
`
`X-R(t).Cos(a).Sin(i3)
`
`Y-R(t).Sin( a).Sin(i3)
`
`Z-R(t).Cos(i3)
`
`[0023] Like most 3D interfaces it is important to distin(cid:173)
`guish between the inertial frame and the user frames. The
`inertial frame is considered as a reference and all objects in
`the 3D virtual environment are expressed with respect to it.
`Thus this system is fixed. The x-axis is pointing to any
`convenient direction, the z-axis is pointing vertically upward
`and the y-axis is perpendicular to both. The user frame is the
`mobile system containing the pointer. It is defined by a
`rotation around the z-axis by 1.jJ and by the rotation around
`x and y by 8 and W. Moreover the distance between those
`frames defines the offset of the pointer with respect to the
`
`

`

`US 2006/0092133 Al
`
`May 4, 2006
`
`3
`
`inertial frame. The figure below illustrates those rotations.
`The matrix linking between those two frames is the product
`of the following rotation matrix.
`
`-sin(i/1)
`
`cos(i/1)
`
`0
`
`0
`1
`
`0 j r cos(0)
`
`·
`
`0
`-sin(0)
`
`0
`
`sin(0) j
`
`0
`
`.
`
`0 cos(0)
`
`0
`
`r ~l l COS(\O)
`
`sin(\O)
`
`-si~(\O) 1
`COS(\O) j
`
`[0024]
`
`After developing we get:
`
`microprocessor used. The ADXL202E is capable of mea(cid:173)
`suring both positive and negative accelerations to at least ±2
`g. The accelerometer can measure static acceleration forces
`such as gravity, allowing it to be used as a tilt sensor as used
`in our application. Acceleration will result in an output
`square wave whose amplitude is proportional to accelera(cid:173)
`tion. Phase sensitive demodulation techniques are then used
`to rectify the signal and determine the direction of the
`acceleration.
`
`[0029] One of the most popular applications of the
`ADXL202E is tilt measurement. An accelerometer uses the
`force of gravity as an input vector to determine orientation
`of an object in space. An accelerometer is most sensitive to
`tilt when its sensitive axis is perpendicular to the force of
`gravity, i.e., parallel to the earth's surface. At this orientation
`its sensitivity to changes in tilt is highest. When the accel(cid:173)
`erometer is oriented on axis to gravity, i.e., near its +1 g or
`
`cos(l/l).cos(0) cos(l/l).sin(0).sin(\O) - sin(i/1).cos(\O)
`Rrn = sin(l/l).cos(0) sin(l/l).sin(0).sin(\O) - cos(i/1).cos(\O)
`r
`-sin(0)
`cos(0).sin(\O)
`
`cos(l/l).sin(0).cos(\O) - sin(i/1).sin(\O) j
`
`sin(l/l).sin(0).cos(\O) - cos(i/1).sin(\O)
`
`cos(0).cos(\O)
`
`[0025]
`In one embodiment of the present invention the 3D
`interface is used to create the virtual reality scene needed to
`interact with the 3D pointer. This interface is developed in
`an expandable mode in order to permit any improvement in
`the future. This interface allows the user to interact with the
`3D objects, to change the colors of the ground and the
`pointer, to change the render mode between wire frame,
`hidden, and rendered, to change the view angles and the light
`intensity.
`
`[0026]
`It is important to mention that the yaw angle can be
`changed directly from the pointing device in order to make
`the navigation easier. In order to avoid the use of additional
`sensing components such as a magnetic sensor or Gyro(cid:173)
`scope, we have simulated the yaw dimension by a rotation
`of the field of view. This field of view rotation is a manipu(cid:173)
`lation of the graphical perspective through the interface
`software, by a pair of control buttons on the device itself.
`
`[0027]
`In one embodiment of the present invention we are
`using an inertial sensor to detect tilt accelerations that will
`then be converted into movement. In this particular embodi(cid:173)
`ment, we are using a MEMS accelerometer developed by
`Analog Devices, the ADXL202E MEMS accelerometer.
`Any similar inertial sensor including thermal accelerometers
`could be used. The ADXL202E is a low-cost, low-power,
`complete two-axis accelerometer with a digital output, all on
`a single monolithic IC. The ADXL202E can measure both
`dynamic acceleration ( e.g., vibration) and static acceleration
`(e.g., gravity). The outputs are analog voltage or digital
`signals whose duty cycles (ratio of pulse width to period) are
`proportional to acceleration. A microprocessor counter,
`without an AID converter or glue logic, can directly measure
`the duty cycle outputs. The duty cycle period is adjustable
`from 0.5 ms to 10 ms via external timing resistor.
`
`[0028] The ADXL202E is a complete, dual-axis accelera(cid:173)
`tion measurement system. For each axis, an output circuit
`converts the analog signal to a duty cycle modulated (DCM)
`digital signal that can be decoded with the timer port of the
`
`-1 g reading, the change in output acceleration per degree of
`tilt is negligible. When the accelerometer is perpendicular to
`gravity, its output will change nearly 17 .5 mg per degree of
`tilt, but at 45° degrees it is changing only at 12.2 mg per
`degree and resolution declines. Due to the fact that it is
`sensible to the static gravity, it can be used to measure
`especially Tilt angles (Pitch and Roll) just by measuring the
`projection of the vector g over each axis of the accelerom(cid:173)
`eter.
`
`[0030] When the accelerometer is oriented so both its X
`and Y axes are parallel to the earth's surface it can be used
`as a two axis tilt sensor with a roll and a pitch axis. Once the
`output signal from the accelerometer has been converted to
`an acceleration that varies between -1 g and + 1 g, the output
`tilt in degrees is calculated as follows:
`
`Pitch= ASi~ *)
`
`Roll= ASi~ ;~)
`
`[0031]
`In one embodiment of the present invention the 3D
`Mouse/controller is a hand held device that captures the
`movement of a hand in free space and controls the move(cid:173)
`ment of a vectorial cursor, object or character in an appli(cid:173)
`cation on a monitor, or a system in physical space. It uses
`inertial technology in the form of an accelerometer or any
`similar technology that measures angular acceleration/dis(cid:173)
`placement with great precision. This technology allows the
`3D Mouse/Controller to be self-contained without the need
`for beacons or emitters/receivers to detect generated signals,
`as the case would be with acoustic, magnetic or optical
`approaches.
`
`[0032] Practically, it could be either used as a mouse for
`3D GUis and volumetric monitors, a controller for 3D
`
`

`

`US 2006/0092133 Al
`
`May 4, 2006
`
`4
`
`games, a pointer for interactive presentations or as a remote
`controlling device for the upcoming PC Entertainment Cen(cid:173)
`ters that would combine a TV with a Computer and a Home
`Theatre. Its range only depending of that of the wireless
`technology used. In an alternative embodiment, the 3D
`Mouse/Controller is a wired device connected electrically to
`a computing device.
`
`[0033] This control functionality could be extended to
`controlling more household peripherals such as telecommu(cid:173)
`nications, lighting, irrigation, security system, heating/cool(cid:173)
`ing or even car start-up in the morning. This would be done
`through a software user interface (Windows, Linux etc ...
`) that would appear on a large Plasma ( or other) screen. The
`said screen playing the role of a TV, computer monitor and
`command and control interface.
`
`[0034]
`In this respect, the 3D Mouse/Controller could be
`the future universal remote controller for the next generation
`of consumer appliances that would be controlled through a
`central computer (network of computers), instead of each
`having its own micro-controller and remote controlling
`device. The complexity of remote controllers would then be
`in the software interface that would be made more intuitive
`(and ideally in 3D) than the scroll down menu interface and
`large number of buttons currently available.
`
`[0035] As the 3D Mouse/Controller also has a spatial
`capability with the needed degrees of freedom, it is a
`suitable device for the new generation of 3D monitors ( e.g.,
`Stereographic, Holographic and Volumetric). There is a
`number of companies developing such monitor technologies
`in the US, Europe and Asia and their feedback is that they
`still lack a practical/affordable 3D Mouse/Controller that
`would allow operators to navigate easily in the 3D environ(cid:173)
`ment.
`
`[0036] The 3D capability is achieved through a limited
`amount of hand movements (rotations) that would allow the
`alignment of a feedback vector (vectorial cursor) with the
`object to be reached, on the monitor. Practically, the align(cid:173)
`ment is done by varying the vertical and horizontal angles of
`the ray, in a polar frame of reference. Once the alignment is
`achieved, the 3D Mouse allows the extension of the ray
`whereby it would reach the object, thus enabling it for
`further manipulation. This approach allows an optimization
`of needed electronics whereby only one inertial device
`(accelerometer) is needed for the basic 3D functionality.
`
`[0037] The 3D capability of the device would also enable
`a new generation of Virtual reality applications (in this case
`a haptic feedback might be added), Industrial and military
`simulations, advanced 3D CAD/CAM design, Medicine,
`Molecular Chemistry, Bio-informatics ...
`
`[0038] This 3D capability is also an enabling factor for the
`next generation of game stations and game environments. A
`game controller enabled by this 3D technology will be able
`to control characters m a 3D space with very natural
`movements.
`
`[0039]
`In one particular embodiment, the technology
`could be embedded in a portable/mobile game device/
`system (similar to Gameboy, PSP ... ) adding 3D capability
`and control through hand movements and allowing the
`advent of 3D games controlled through movements of the
`game system itself, thus starting a paradigm shift in portable
`game systems.
`
`[0040]
`In another embodiment, the technology could be
`embedded in game controllers with the shape of sports
`equipment, (non-extensive list including Golf clubs, Tennis
`racquets or Baseball bats), thus allowing the creation of even
`more realistic video games around sports themes.
`
`[0041] Other applications would be a remote controller for
`hobbyists or for military personnel tele-guiding flying enti(cid:173)
`ties such as Unmarmed Air Vehicles (UAVs), Unmarmed
`Ground Vehicles (UGVs), or Unmanned Water Vehicles
`(UWV)s.
`
`[0042] From a marketing perspective, the field seems ripe
`for the technology, especially that it has been designed to be
`manufactured cost-effectively. One embodiment of the
`present invention relies on Bluetooth wireless communica(cid:173)
`tions and RS 232 connectivity. It is also possible to have
`wired USB connectivity and Wi-Fi (wireless) communica(cid:173)
`tions or any other enabling technology capable of being
`understood by anyone skilled.
`
`[0043] FIG. 1 shows a 3D computer system at 100.
`Referring to FIG. 1, a computer is shown at 107, a computer
`monitor is shown 101, and a computer keyboard is shown at
`108. A3D environment 105 and a set of3D applications 106
`are shown within the monitor 101. A 3D input device or
`Mouse/Controller 102 interacts with the 3D environment
`105 by controlling a vectorial cursor 104. In the example
`shown here, the vectorial cursor 104 is shaped like an arrow
`giving the user spatial feedback of the direction and position
`of the cursor. Depending on the application, the length of the
`arrow could be extensible or fixed. In the embodiment
`shown here, the base of the arrow is a fixed origin of a
`spherical coordinate system and changes in the length of the
`vectorial cursor 106 are controlled through a linear input
`element comprising a pair of buttons on the input device
`102, allowing a user to reach any point in the space depicted
`on the monitor 101. In an alternate embodiment, the location
`of the base of the arrow can be controlled through the input
`device allowing the entire arrow, or vectorial cursor 104 to
`move virtually in the 3D space, with the length of the arrow
`being either fixed or responsive to user input through the 3D
`input device. A linear input element used in such an input
`device 102 can be any single or multiple user-responsive
`components understood by anyone skilled in the art.
`Examples of linear input elements include a pair of push
`buttons, a slide switch, a touch pad, and a scroll wheel.
`
`[0044]
`It should be noted that a computer system could be
`any system that includes an information-processing unit.
`Examples of computer systems include, but are not limited
`to personal digital assistants (PDAs), personal computers,
`mini-computers, mainframe computers, electronic games,
`and microprocessor-based systems used to control personal,
`industrial or medical vehicles and appliances.
`
`[0045] The movement and control functions of the 3D
`Mouse/Controller 102 are shown as phantom lines at 103.
`The curved lines and arrows at 103 represent possible
`movements of the device held by the user. An upward or
`downward tilt (pitch) of the device would move the vectorial
`cursor 104 in a similar fashion on the screen, while a lateral
`tilt (roll) in a left-right marmer would move the vectorial
`cursor 104 on the screen to the left or right. The magnitude
`of the vectorial cursor 104 is controlled using a pair of
`control triggers on the device. The combination of pitch,
`roll, and vector magnitude allow the user to reach any point
`
`

`

`US 2006/0092133 Al
`
`May 4, 2006
`
`5
`
`in 3D space using spherical coordinates with a minimal
`amount of physical movement.
`[0046]
`In one embodiment illustrated in FIG. 1, the 3D
`Mouse/Controller 102 is pointing at 3D applications 106 in
`3D graphical user interface (GUI) 105 that are displayed on
`a monitor 101. In another embodiment, the 3D Mouse/
`Controller 102 could control one or more 3D graphical
`objects in a 3D games environment in the same manner. A
`graphical object can be a video game character or any other
`graphical symbol in a 3D environment. In that case, the
`physical embodiment of the controlling device 102 could
`look like a game controller and the 3D character would be
`substituted for the vectorial cursor 103. The vector magni(cid:173)
`tude derived from a linear input element in the Mouse/
`Controller 102 can be used to control the size or orientation
`of the graphical object.
`[0047]
`In another embodiment, the Mouse/Controller 102
`is a 2D input device working in radial coordinates. In this
`case, only one tilt angle and a minimum of one linear input
`are measured in the input device 102 to provide a 2D
`navigational device operating in radial coordinates. In yet
`another embodiment, the Mouse/Controller 102 is an input
`device with two linear input elements capable of changing a
`vector magnitude in perpendicular axes. These two perpen(cid:173)
`dicular axis in conjunction with one tilt axis can generate a
`position in 3D space using cylindrical coordinates.
`[0048] FIGS. 2a, 2b, 2c, and 2d show the detailed move(cid:173)
`ment of the 3D Mouse/Controller 102 and the related control
`of the vectorial cursor 104. FIG. 2a shows the initial state
`of the device 102 and vectorial cursor 104 pointing on one
`application 106. FIG. 2b shows a right rolling tilt of the
`device 102 that causes the vectorial cursor 104 to move right
`and point to another application 106 to the right of the initial
`one in FIG. 2a. FIG. 2c shows an upward tilt of the device
`102 that causes the vectorial cursor 104 to move up and
`point to another application 106 above of the initial one in
`FIG. 2b. FIG. 2d shows the extension function through a
`button on the device 102 that causes the vectorial cursor 104
`to move further inside the 3D GUI 105 and point to an icon
`on the desktop 106 above of the application one in FIG. 2c.
`[0049] FIGS. 2a, 2b, 2c are the actual rendering of the
`device movements and vectorial cursor control as described
`in FIG. 1. Namely, an up-down tilt of the device will move
`the cursor in an upward or downward manner. Similarly, a
`left-right tilt of the device would move the vectorial cursor
`to the left or the right. Finally, the vectorial cursor would
`move forward or backward through the depression of a pair
`of triggers on the device itself that controls its spatial
`extension and retraction.
`[0050] FIG. 3 shows one physical embodiment of the 3D
`Mouse/Controller with the knobs and buttons used for
`interaction with a 3D environment. One pair of buttons
`301/302 is the equivalent of the left and right clicks of a
`regular mouse. They activate similar functions. A second
`pair of buttons (triggers) 303/304 enables the extension and
`retraction of the vectorial cursor to reach different parts of a
`3D environment, by increasing the module of the vectorial
`cursor. The vectorial cursor being the physical analog of a
`spherical vector, the buttons actually increase/decrease the
`module of the vector which is rendered on the screen by a
`movement of the vectorial cursor forward or backward.
`[0051] A third pair of buttons 305/306 allows the user to
`change the field of view or "perspective" of a 3D scene, in
`
`order to simulate the Yaw dimension. This is done by
`graphically changing the field of view through a graphical
`transformation in the interface software. The action is con(cid:173)
`trolled by another pair of triggers on the device.
`
`[0052] FIG. 4 shows a block diagram of one embodiment
`of the 3D Mouse/Controller system. The system comprises
`an input device (which can also be a hand-held pointing
`device or a 3D Mouse/Controller) 402 and a display control
`unit module 401. The input device includes an inertial sensor
`(accelerometer) 424 operable to detect an acceleration as the
`user tilts the pointing device in at least one direction; a
`power supply 422 (which can be a battery, AC power supply,
`solar cell or any other source of electrical power under