Case 6:21-cv-00755-ADA Document 45-3 Filed 02/28/22 Page 1 of 15
`
` Exhibit 3
`
`

`

`(12) United States Patent
`Foxlin et al.
`
`(10) Patent 1%.;
`(45) Date of Patent:
`
`US 8,224,024 B2
`Jul. 17, 2012
`
`US008224024B2
`
`(54) TRACKING OBJECTS WITH MARKERS
`
`(75) Inventors‘ IEJm 131g“? Arliilgéon’ 113A (Li/2i Us
`
`.
`
`-
`
`-
`
`~
`
`.
`
`8/2008 Matsushita et al. ......... .. 398/140
`7,415,212 B2 *
`2001/0043737 A1 11/2001 Rogina et al.
`2002/0049530 A1
`4/2002 Poropat
`2004/0032970 A1
`2/2004 Kiraly
`
`2004/0073360 A1 *
`
`4/2004 Foxlin ......................... .. 701/207
`
`“"1 aim“ ’ r00 me’
`
`(
`
`)
`
`_
`_
`_
`(73) Ass1gnee: InterSense, LLC, B1ller1ca, MA (US)
`
`2004/0201857 A1* 10/2004 FoXlin ....... ..
`356/620
`2005/0256391 A1 * 11/2005 Satoh et al. ................. .. 600/407
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`Paieiit is extended or adjusted under 35
`U-S-C- 154(1)) by 1546 days-
`
`(21) Appl. NO.Z 11/543,008
`
`(22) Filed:
`
`Oct. 4, 2006
`
`(65)
`
`Prior Publication Data
`US 2007/0081695 A1
`Apr. 12, 2007
`
`Related U-s- Application Data
`(60) Provisional application No. 60/723,648, ?led on Oct.
`4, 2005.
`
`(51) Int- Ci-
`(2006-01)
`G06K 9/00
`(52) US. Cl- ...... .. 382/103; 348/135; 348/137; 348/139;
`348/142; 348/146; 348/169; 348/208.8
`(58) Field of Classi?cation Search ................ .. 382/103;
`348/135, 137, 139, 142, 146, 169, 208.2
`See application ?le for complete search history.
`
`(56)
`
`References Cited
`
`US. PATENT DOCUMENTS
`5,615,132 A
`3/1997 Horton et al.
`6,373,047 B1
`4/2002 Adan et al.
`6,474,159 B1
`11/2002 FoXlin et al.
`6,603,865 B1 *
`8/2003 Yagi et al. ................... .. 382/103
`6,611,141 B1* 8/2003 Schulz et al. ............... .. 324/226
`6,681,629 B2
`1/2004 FoXlin et al.
`6,937,255 B2 *
`8/2005 Fukuda et al. .............. .. 345/633
`7,324,663 B2 *
`1/2008 Kiraly ......................... .. 382/103
`
`OTHER PUBLICATIONS
`Gunnam et al., “AVision-Based DSP EmbeddedNavigation Sensor”,
`IEEE Sensors Journal, 2(5):428-442 (2002).
`Kim et al ., “An Optical Tracker for Augmented Reality and Wearable
`Combuter”, IEEE Virtual Reality Annual International Symposium
`pp. 146-150 (1997).
`Search Report for International Application No. PCT/US06/38460
`dated Oct. 16, 2007.
`
`(Continued)
`
`Primary Examiner * Brian Q Le
`Assistant Examiner * Julian Brooks
`(74) Attorney! Agent] 0'’ Firm * Fish & Richardson RC
`
`ABSTRACT
`(57)
`The spatial location and aZimuth of an object are computed
`from the locations, in a single camera image, of exactly tWo
`points on the object and information about an orientation of
`the object.
`One or more groups of four or more collinear markers are
`located in an image, and for each group, ?rst and second outer
`markers are determined, the distances from each outer marker
`to the nearest marker in the same group are compared, and the
`outer marker With a closer nearest marker is identi?ed as the
`?rst outer marker. Based on knoWn distances betWeen the
`outer markers and the marker nearest the ?rst outer marker, an
`amount of perspective distortion of the group of markers in
`the image is estimated. Based on the perspective distortion,
`relative distances from each other point in the group to one of
`the outer markers are determined. Based on the relative dis
`tances, the group is identi?ed.
`
`1 Claim, 6 Drawing Sheets
`
`Case 6:21-cv-00755-ADA Document 45-3 Filed 02/28/22 Page 2 of 15
`
`102
`
`

`

`US 8,224,024 B2
`Page 2
`
`OTHER PUBLICATIONS
`
`William Frey, Michael Zyda, Robert McGhee, Bill Cockayne, Off
`the-Shelf, Real-Time, Human Body Motion Capture for Synthetic
`Environments, 1995, Computer Science Department, Navel Post
`graduate School, Monterey, CA 93943-5118.
`Robert M. Haralick & Linda G. Shapiro, Computer and RobotVision
`v.2, Addison-Wesley Publishing Company, pp. 66-68, 1993.
`Robert van Liere & Jurriaan D. Mulder, “Optical Tracking Using
`Proj ective Invariant Marker Pattern Properties,” IEEE Virtual Reality
`2003 Conference, Mar. 22-26, Los Angeles, 2003.
`
`Kiyohide Satoh, Shinj i Uchiyama, and HiroyukiYamamoto, “A Head
`Tracking Method Using Bird’s-Eye View Camera and Gyroscope,”
`Proceedings of the Third IEEE and ACM International Symposium
`on Mixed and Augmented Reality (ISMAR 2004), pp. 202-211
`Washington, DC, Nov. 2004.
`Daisuke Kotake, Kiyohide Satoh, Shinji Uchiyama, and Hiroyuki
`Yamamoto, “A Hybrid and Linear Registration Method Utilizing
`Inclination Constraint,” Proceedings of the Fourth IEEE and ACM
`International Symposium on Mixed and Augmented Reality (ISMAR
`2005), pp. 140-149 Washington, DC, Oct. 2005.
`
`* cited by examiner
`
`Case 6:21-cv-00755-ADA Document 45-3 Filed 02/28/22 Page 3 of 15
`
`

`

`US. Patent
`
`Jul. 17, 2012
`
`Sheet 1 of6
`
`US 8,224,024 B2
`
`Figure 1B
`
`Figure 1A
`
`Case 6:21-cv-00755-ADA Document 45-3 Filed 02/28/22 Page 4 of 15
`
`

`

`US. Patent
`
`Jul. 17, 2012
`
`Sheet 2 on
`
`US 8,224,024 B2
`
`102
`
`C
`
`1106 Z @ a'Xb
`lg ‘ 208
`
`Case 6:21-cv-00755-ADA Document 45-3 Filed 02/28/22 Page 5 of 15
`
`

`

`US. Patent
`
`Jul. 17, 2012
`
`Sheet 3 on
`
`US 8,224,024 B2
`
`304 >
`
`306 l
`
`308)
`
`‘mp
`+15. +
`
`302
`7
`
`312b5
`
`k-u
`
`Figure 3
`
`Case 6:21-cv-00755-ADA Document 45-3 Filed 02/28/22 Page 6 of 15
`
`

`

`U.S. Patent
`
`Jul. 17, 2012
`
`Sheet 4 on
`
`US 8,224,024 B2
`
`Figure 4A
`
`start
`
`?nd all collinear groups
`of 4 or more markers
`402
`
`
`h 0M4 m t6 “m8 C 8T0 H680 0|
`r. .Mm Wm% H
`6 m8 829.8 .m
`t r tar n l
`.. m mnk C a___
`5 re s ei
`0 acnrv e mans
`
`
`
`r’ e Iv dgM.
`
`O F D- e :l r hen m Cd 8 Ct nn
`e adt C a d e .9"
`..l 90 0 e
`
`f rU4
`
`r ibO
`
`reo
`mud
`S:
`m‘
`
`Case 6:21-cv-00755-ADA Document 45-3 Filed 02/28/22 Page 7 of 15
`
`tn
`8 cm n 86
`
`

`

`US. Patent
`
`Jul. 17, 2012
`
`Sheet 5 on
`
`US 8,224,024 B2
`
`(for each set (cont.)) Q
`
`Y
`
`estimate perspective
`distortion
`420
`
`— — — for each encoding marker:
`
`— — -
`
`calculate perspective-corrected
`ratio of distance from single as
`fraction of N-u
`422
`
`calculate actual distance
`from single end
`424
`
`round distance to nearest
`integer multiple of u
`426
`
`determine code value
`from position
`428
`
`Figure 48
`
`end
`
`Case 6:21-cv-00755-ADA Document 45-3 Filed 02/28/22 Page 8 of 15
`
`

`

`US. Patent
`
`Jul. 17, 2012
`
`Sheet 6 on
`
`US 8,224,024 B2
`
`SOBd/‘CI:
`
`502
`
`50608
`
`51Gb
`
`> 504
`
`Figure 5
`
`Case 6:21-cv-00755-ADA Document 45-3 Filed 02/28/22 Page 9 of 15
`
`

`

`US 8,224,024 B2
`
`1
`TRACKING OBJECTS WITH MARKERS
`
`CLAIM OF PRIORITY
`
`This application claims priority under 35 USC §l 19(e) to
`US. Provisional Patent Application Ser. No. 60/723,648,
`?led on Oct. 4, 2005, the entire contents of Which are hereby
`incorporated by reference.
`
`m
`
`TECHNICAL FIELD
`
`This invention relates to tracking of objects.
`
`BACKGROUND
`
`The position of a point in 3-D space can be described by
`three coordinates x, y, and Z. The orientation of a 3-D object
`is described by three additional coordinates, roll, pitch, and
`yaW. Roll, pitch, and yaW are measured relative to some set of
`axes, frequently North, East, and DoWn along gravity, but any
`?xed axes can be used. A rigidbody in 3-D space thus requires
`six coordinates for full description of its pose (position and
`orientation). Tracking the complete pose of a rigid object is
`referred to as 6-degree-of-freedom (6-DOF) tracking.
`It is Well-knoWn in the ?eld of optical tracking systems that
`the position of a point-like marker (such a light-emitting
`diode (LED) or a re?ective ball or dot) can be obtained by
`triangulating it With tWo or more cameras. The majority of
`optical tracking systems on the market require tWo or more
`cameras installed in the Workspace With overlapping ?elds
`of-vieW in order to track the position of a set of markers. To
`provide full 6-DOF tracking of an object, it is necessary to
`install at least 3 position markers on the object in a triangular
`shape, from Which it is possible to Work out the object’s
`orientation by solving the so-called “exterior orientation
`problem.” This triangle needs to have suf?cient extent in both
`Width and length to achieve the desired orientation accuracy,
`and so it can become cumbersome to mount on a slender
`object such as a pen, a surgical tool, or any other object With
`no convenient large ?at surfaces.
`In tracking systems, a camera is an example of a 2-D
`bearing-angle sensor. When it measures the centroid of a
`target or marker, it returns tWo values, usually called u and v,
`Which relate to the horiZontal and vertical displacement,
`respectively, of the target in the image. These measurements
`are related to the aZimuth and elevation angles, also knoWn as
`bearing angles, from the sensor to the target. The relationship
`is non-linear: for a simple pinhole camera model, u and v are
`proportional to sin (azimuth) and sin (elevation), While for
`cameras With lenses the distortion makes the relationship
`more complex. HoWever, in either case the camera outputs are
`isomorphic to bearing-angles and the camera belongs to the
`class of 2D bearing-angle sensors.
`There are many other bearing-angle sensors that have been
`or could be used in optical motion tracking. Some examples
`include a quadcell, a lateral-effect photodiode, a position
`sensitive device (PSD), a projection sensor (e.g., Hamamatsu
`S9132), or a laser-scanner Which sWeeps a fan of light through
`a space and measures the bearing angle to a photosensor
`target based on the timing of the detected pulse during the
`sWeep. Also, tWo single-axis bearing sensors, for example,
`implemented with 1-D CCD or CMOS array sensors or
`single-axis laser scanners, may be combined in one housing
`to form a 2D bearing-angle sensor. Methods other than optical
`imaging can also be used to form a 2D bearing sensor device.
`Radio frequency (RF) and acoustic techniques, including
`sWept radar or sonar beams, time-difference-of-arrival
`
`Case 6:21-cv-00755-ADA Document 45-3 Filed 02/28/22 Page 10 of 15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`(TDOA), and phased arrays of antennas or microphones have
`all been used to measure bearing angles. For the remainder of
`this description We Will use the term “camera” to mean any
`device capable of measuring tWo bearing angles.
`A tracking system Which uses cameras is referred as an
`“optical tracking system,” While a system using both optical
`and inertial sensors is referred as a “hybrid optical inertial
`tracking system” or just “hybrid tracking system.” Most opti
`cal and hybrid tracking systems require some environmental
`installation, i.e., some sort of markers attached to the tracked
`objects or, if cameras are attached to the tracked objects, then
`markers are installed in the environment.
`A variety of items can be used as markers. Examples
`include printed ?ducial patterns, retrore?ective 2-D and 3-D
`targets, active LEDs in the visible, IR or UV spectrum, col
`ored marks, and natural features on an object such as comers,
`lines or textures Which are recogniZable by computer vision
`or pattern matching techniques. Depending on the type of the
`markers, they may have different physical characteristics
`such as siZe, shape, color, etc. We Will use the terms marker,
`target, ?ducial, point or LED interchangeably to mean any
`type of local feature on an object Which can be detected by a
`camera, and for Which the camera can measure the bearings to
`a speci?c point in the feature Which We shall call the centroid
`(even though, for some features such as corners or textures,
`the measured point is not actually a centroid).
`Previous optical trackers Which are capable of measuring
`the 6-DOF motion of an object require at a minimum either:
`a) tWo cameras vieWing three target points on a rigid tri
`angle (stereo triangulate each point in 3D then solve for
`pose), or
`b) one camera vieWing four or more target points on a rigid
`structure (solve perspective n-point pose recovery algo
`rithm from analytic photogrammetry). It is also possible
`to solve a 3-point pose recovery algorithm, but it pro
`duces four ambiguous solutions.
`
`SUMMARY
`
`In general, in one aspect, the spatial location and aZimuth
`of an object are computed from the locations, in a single
`camera image, of exactly tWo points on the object and infor
`mation about an orientation of the object.
`Implementations may include one or more of the folloW
`ing. The information about an orientation of the object comes
`from a ?rst inertial sensor mounted on the object. Image
`locations in said camera image of one or more additional
`points on the object is used to identify said tWo points. The
`camera image is obtained from a camera Whose optical axis is
`collinear With the axis of gravity. The camera image is
`obtained from a camera Whose optical axis has a knoWn tilt
`With respect to gravity. Information from a second inertial
`sensor measuring orientation of said camera is used to com
`pute said spatial location and aZimuth of said object. The
`information from said second inertial sensor includes infor
`mation about pitch and roll With respect to gravity. Azimuth of
`the camera, if measured, is not used to compute the relative
`location and aZimuth of the object With respect to the camera.
`The information from said ?rst inertial sensor includes pitch
`With respect to gravity. The information from said ?rst inertial
`sensor also includes roll With respect to gravity. The one or
`more additional points includes a third point that is closer to
`one of said tWo points than to the other, and identifying said
`tWo points includes using the location in said image of said
`third point to distinguish said tWo points from each other. The
`one or more additional points includes a fourth point that is
`collinear With the tWo points, and identifying the tWo points
`
`

`

`US 8,224,024 B2
`
`3
`also includes using the location in said image of said fourth
`point to distinguish the linear array of points to Which it
`belongs from other linear arrays of points. Information from
`said ?rst inertial sensor includes pitch, and the pitch of the
`line containing the tWo points is equivalent to the pitch of the
`object. Pitch of the line containing the tWo points is calculated
`using the measured pitch and roll of the object and a knoWn
`orientation of said line on the object also including updating
`the computed spatial location and orientation of the object
`based on the information about the angular velocity and linear
`acceleration of the object from the ?rst inertial sensor. Drift in
`the updated spatial location or orientation of the object is
`corrected for using an updated image from the camera. The
`camera image is provided by a camera mounted on the head of
`a person. The object is a marking apparatus on the person’s
`hand. The object is a tool. The object is a Weapon. Projecting
`a visual display representing a virtual World, and including in
`the visual display a representation of the object, including
`locating and positioning the representation of the object
`based on the computed spatial location and aZimuth of the
`object. In a display visible to the person, projecting images
`that supplement the person’s vieW of their surroundings, in
`Which the images include a representation of the object posi
`tioned in the display based on the computed spatial location
`and aZimuth of the object.
`In general, in one aspect, a set of groups of markers, each
`group including four or more collinear markers, is arranged
`such that the ratio of the distances betWeen three ?xed base
`markers is the same for each group in the set. In some imple
`mentations, each group has a different arrangement of one or
`more encoding markers than each other group. Inverting the
`spatial location and azimuth of the object, and from the
`inverted location and azimuth, determining the spatial loca
`tion and orientation of a camera that produced the image.
`From an updated image from the camera and inertial sensors
`associated With the camera, determining an updated spatial
`location and orientation of the camera.
`In general, in one aspect, a group of markers are arranged
`in a pattern that is uniquely identi?able by a single camera,
`including a linear pattern of four or more markers in Which
`three base markers at the ends have a ?xed geometry and one
`or more additional markers betWeen the ends uniquely dis
`tinguish the group from other groups having the same base
`marker geometry.
`Implementations may include one or more of the folloW
`ing. The markers are LEDs. A circuit board has a plurality of
`locations Where LEDs can be installed. At least some of the
`locations include connections for at least tWo separate circuits
`for energiZing LEDs installed at the locations. The markers
`include a printed pattern. The printed pattern includes White
`dots on a black background. The printed pattern includes
`black dots on a White background. The printed pattern is
`attached to an adhesive strip. The printed pattern is attached to
`a magnetic strip
`In general, in one aspect, one or more groups of four or
`more collinear markers are located in an image, and for each
`group, ?rst and second outer markers are determined, the
`distances from each outer marker to the nearest marker in the
`same group are compared, and the outer marker With a closer
`nearest marker is identi?ed as the ?rst outer marker. Based on
`knoWn distances betWeen the outer markers and the marker
`nearest the ?rst outer marker, an amount of perspective dis
`tortion of the group of markers in the image is estimated.
`Based on the perspective distortion, relative distances from
`each other point in the group to one of the outer markers are
`determined. Based on the relative distances, the group is
`identi?ed.
`
`Case 6:21-cv-00755-ADA Document 45-3 Filed 02/28/22 Page 11 of 15
`
`4
`Implementations may include one or more of the folloW
`ing. Identifying the group includes identifying a code value
`based on the distances from each other point in the group to
`one of the outer markers, and identifying one of a knoWn set
`of groups that most closely corresponds to the code value.
`Based on the identi?ed group, identifying a location of a
`source of the image. Based on a location of the identi?ed
`group in an updated image, updating the location of the
`source of the image. Based on an identi?cation of a second
`group in an updated image, updating the location of the
`source of the image.
`In general, in one aspect, coordinate information is
`received for images, on an imaging device of a camera, of tWo
`points on an object. Pitch information is received from a
`sensor on the object. The coordinate information and the pitch
`information are used to obtain candidate values for the aZi
`muth of the object. One aZimuth value is selected based on an
`evaluation of the candidate aZimuth values in equations relat
`ing the coordinate information and pitch information to dis
`tances of the points from the camera. In some implementa
`tions, the candidate aZimuth values are calculated by solving
`a system of equations relating the coordinate information,
`pitch information, and knoWn geometry of the object to the
`distances of the points from the camera.
`The details of one or more embodiments of the invention
`are set forth in the accompanying draWings and the descrip
`tion beloW. Other features, objects, and advantages of the
`invention Will be apparent from the description and draWings,
`and from the claims.
`
`DESCRIPTION OF DRAWINGS
`
`FIG. 1A shoWs a schematic vieW of and coordinate de?ni
`tions for a single camera tracking an object With tWo target
`points.
`FIG. 1B shoWs a schematic representation of the camera
`and object of ?gure 1A.
`FIG. 2 shoWs a schematic vieW of a single camera tracking
`an object With three target points.
`FIG. 3 shoWs a ?ducial strip.
`FIGS. 4A and 4B are How charts.
`FIG. 5 shoWs an example implementation of an LED-based
`?ducial strip.
`Like reference symbols in the various draWings indicate
`like elements.
`
`DETAILED DESCRIPTION
`
`Pose Acquisition from one Camera using just tWo Points and
`Inertial Inclination
`An accurate closed-form analytical computation for the
`pose of an object in all six degrees of freedom can be obtained
`using only a single camera and only tWo markers if the
`marked object is equipped With inertial sensors to measure at
`least its pitch and roll relative to the camera. In some
`examples, as described later, pitch can be estimated Without
`inertial sensors, but the additional information from inertial
`sensors provides a considerable increase in the accuracy of
`the computed x, y, Z, and yaW coordinates. The pose is com
`puted relative to the camera, but if the camera’s position
`relative to the World is knoWn, the pose is easily translated
`into the World-frame. By World-frame, We mean any refer
`ence frame external to the camera and object.
`In some examples, the inertial sensors comprise three axes
`of angular rate sensing and three axes of accelerometers, but
`only pitch and roll relative to gravity (or to the camera) are
`required. The camera remains stationary at a predetermined
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`

`

`US 8,224,024 B2
`
`5
`orientation in the Workspace, and the tracked object moves
`and is tracked in 6-DOF. Although the tracked object may be
`equipped With any number of markers, only tWo need to be
`visible to the camera at a given time in order to acquire an
`initial pose estimate or maintain tracking.
`FIG. 1A illustrates the geometry of a single camera 102
`With its optical axis aimed along the x-axis in camera space
`(xc in coordinate frame 110c). Symbols 1p, 6, 4) represent the
`Euler angles yaW, pitch and roll, respectively, of a tracked
`object 104. TWo markers 106a and 10619 are visible on the
`object 104. We Will refer to them as points 1 (106a) and 2
`(10619) in the mathematical computations that folloW. Let
`rlw:[xlW ylW 21W] and r2W:[x2W y2W 22W] represent the coor
`dinates of points 1 and 2 in a World-space With its Z axis (ZW in
`coordinate frame 110w) pointing straight doWn in the direc
`tion of gravity, and let L be the distance betWeen the tWo
`points. For convenience and Without loss of generality, We
`assume that the b-frame (internal reference frame) coordi
`nates of the object 104 have an origin at point 1 and x-axis xb
`pointing toWards point 2. Assume that the orientation of the
`camera With respect to the World-frame is knoWn in advance
`and can be described by the 3x3 rotation matrix RCW Which
`transforms vectors represented in camera coordinates (de
`noted by superscript c) to vectors in World coordinates (de
`noted by superscript W). Thus, if the origin of the W-frame
`110w is made to coincide With the origin of c-frame 1100 at
`the camera center of projection, then We have:
`
`IJIMIZW-
`
`<1)
`
`Assuming We have an inertial sensor on the object 104
`Which can accurately determine pitch and roll (6, (1)) by itself,
`and the camera 102 has reported image plane centroids (ul,
`v1) and (u2, v2) (see FIG. 1B), We Wish to ?nd all 6 coordi
`nates (x 1””, y 1””, 21W, 11), 6, (1)) explicitly describing position and
`orientation of the object in World coordinates.
`In World-space, the position of point 2 can be expressed
`relative to point I as folloWs:
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`(Z)
`
`6
`Rewriting equation (2) in c-frame and inserting equation
`(3) We have:
`
`Xi
`LcosOcosl?
`xi
`x;
`M2165 = mx? +Rf, Lcosesimp = mg +
`V226;
`vlxf
`—Lsin0
`v1 xi
`
`(4)
`
`R;
`
`LcosO
`O
`0
`
`O
`LcosO
`O
`
`O
`O
`—Lsin0
`
`coslp
`sinl?
`1
`
`Since the four measurements (ul, v1) and (u2, v2) and the
`nine elements of
`
`LcosO
`
`O
`
`O
`
`A E R; O
`O
`
`LcosO
`O
`
`O
`—Lsin0
`
`are all knoWn, equation (4) represents a system of three equa
`tions in three unknoWns, x26, xlc, and 1p. Eliminating x26 by
`plugging the ?rst line into the second and third yields tWo
`equations in x 16, and 1p:
`
`Subtracting these produces the relationship:
`
`Where for brevity We have de?ned:
`
`Where both L and 6 are knoWn.
`
`Case 6:21-cv-00755-ADA Document 45-3 Filed 02/28/22 Page 12 of 15
`
`As shoWn in FIG. 1B, the camera 102 is modeled as a
`pinhole camera 102m having an aperture 120 and a focal
`length f and points 1 and 2 are projected onto its image plane
`122 at (ul, v1) and (u2, v2). (In some examples, Wide-angle
`lenses With lots of distortion are used, in Which case the
`camera is calibrated and undistortion corrections are applied
`to each 2D image point before carrying out the solution
`described beloW. To further simplify the equations, We Will
`use fIl .) Using the pinhole camera model We can express the
`c-frame positions of points 1 and 2 in terms of the unknoWn
`depth values x 16 and x; and the measured u and v values:
`
`(3)
`
`50
`
`55
`
`60
`
`65
`
`Moving the sine term to the right-hand side and squaring
`and rearranging terms yields a quadratic equation in cos 1p:
`
`Which can be solved to yield tWo solutions for cos 1p, and
`thus four solutions for 11). Plugging all four candidates back
`into equation (5), only tWo of them Will produce agreement,
`and only one of these Will have a positive value of x16 as
`required for a point in front of the camera. This unique correct
`solution for yaW can be plugged back through the equations to
`solve for the remaining three unknoWn pose variables (xlW,
`ylw, and 21W).
`There may be many other mathematical approaches to
`solving for the unknoWn variables, but this example serves to
`illustrate that a priori pitch and roll information from inertial
`sensors can be combined With four scalar measurements ((u,
`v) of tWo points) from computer vision or other bearing
`sensors to solve for the 6-DOF pose. Note that the roll mea
`surement Was not used to compute the other coordinates. In
`
`

`

`US 8,224,024 B2
`
`7
`the absence of a measurement of roll, it is still possible to
`compute a S-DOF pose using only one camera, tWo points,
`and a measurement of pitch.
`Once the initial 6-DOF pose is determined, the pose of the
`object can be tracked using only the information from the
`inertial sensors With periodic updates from the vision system
`to correct for drift.
`In some examples, the inertial sensors are used not just to
`obtain accurate prior pitch and roll in order to solve the
`2-point pose-recovery algorithm outlined above. After an ini
`tial pose estimate is acquired, the inertial sensors are used to
`perform high-update-rate inertial tracking With optical mea
`surements introduced, When available, to correct drift. This
`makes it possible to use small or inexpensive cameras With
`relatively loWer frame rate or resolution and still achieve good
`tracking performance. While the described 2-point pose-re
`covery algorithm demonstrates that a Kalman ?lter fusing
`information from inertial sensors and just tWo optical targets
`provides full-observability, it is also true that an inertial sys
`tem, once properly initialiZed, can track for a time With just
`one target visible. With the proper motions, the full 6-DOF
`pose becomes observable over a time period even using just
`one target.
`Relative Tracking
`As noted above, it Was assumed that the camera’s pose
`relative to the World-frame Was knoWn. The camera’s pose
`can be knoWn simply because its position is tightly con
`trolled. In some examples, the camera may be equipped With
`its oWn set of inertial sensors, for example the InertiaCam
`from InterSense, Inc., of Bedford, Mass. In this case, the
`camera need not be held stationary. The tracking is performed
`by differential inertial tracking betWeen the inertial sensors
`on the tracked object and those on the camera, as described in
`Us. Pat. Nos. 6,681,629 and 6,474,159, incorporated here by
`reference. In this arrangement, the camera may be mounted
`on a moving platform such as a vehicle, a simulator, a robot,
`or a person, and can be used to track the marked object relative
`to the camera. If World-frame mapping of the object’s pose is
`required, the measurements of the camera’s inertial sensors
`relative to gravity can be used to create the required trans
`form. In some examples, inertial sensors do not provide a
`reliable measurement of the yaW of the camera (rotation
`around the axis parallel to gravity). A camera yaW of Zero can
`be assumed in the calculations above Without affecting the
`accuracy of the other calculated coordinates. The resulting
`calculated object position and yaW Will be returned in a
`locally-level World-frame directionally aligned With the cam
`era (having Zero yaW displacement betWeen the camera and
`the World-frame). This is often a convenient tracking refer
`ence frame.
`One application of this technique is virtual reality (V R) or
`augmented reality (AR) simulations. A camera With inertial
`sensors is mounted on a user’s head, e. g., on a hat or helmet,
`and the markers and inertial sensors are located on an object
`held by the user, such as a tool, a gun, a piece of sports
`equipment, or a stand-in for such an object. The relative
`tracking of the object With regard to the user’s head can be
`used to insert an image of the object into a virtual World
`displayed in a head-mounted display Worn by the user.
`Because the direction the user is looking matches that of the
`camera, the relative position of the object to the camera can be
`easily mapped to the appropriate location in the display so
`that it remains in the correct location When the user either
`moves the object or looks in another direction. Alternatively,
`in an augmented reality simulation Where the user sees the
`real World but additional graphics or information are added,
`the measured relative position of the object can be used to
`
`Case 6:21-cv-00755-ADA Document 45-3 Filed 02/28/22 Page 13 of 15
`
`8
`indicate, in the display, What the object is pointed at. This
`could be used to alloW a soldier in the ?eld to rehearse an
`operation, for example, by adding targets to his real environ
`ment and accurately indicating Whether his Weapon, When
`?red, Would have hit those targets. In an actual combat envi
`ronment, a projection could be displayed of Where the
`Weapon is aimed.
`Linear-Array Fiducial Patterns
`In the foregoing discussion it Was assumed that the bearing
`sensor is able to uniquely identify and label points 1 and 2
`While measuring their centroid locations. This may be accom
`plished by many different methods. If only one object is to be
`tracked and it is equipped With only tWo marker points, then
`this can be as simple as making one larger or brighter than the
`other, or making them different colors. HoWever, if multiple
`objects may be tracked, or one object needs to have multiple
`pairs of targets on different sides of the object, then a more
`complex approach is required. With active LED targets, this
`could be accomplished by turning on the different targets in a
`speci?c temporal sequence or encoded blink pattern. For
`multi-camera motion-capture systems using passive retro
`re?ective coated balls, the typical approach to making the
`markers identi?able is to group them together into rigid
`“marker cliques” With pre-knoWn inter-marker distances.
`HoWever, to measure these inter-marker distances requires
`?rst triangulating the 3D position of each marker With mul
`tiple cameras. Tracking multiple objects in our single-camera
`system requires a method of grouping targets into uniquely
`identi?able patterns that can be unambiguously recogniZed
`by a single camera, With an 0t priori unknoWn pose relation
`ship betWeen the object and the camera. To assure it is prac
`tical to attach the ?ducials to small or thin objects, a linear
`pattern of dots is used rather than a planar or spatial array.
`Simple ratios of distances betWeen points may not Work
`Well because these ratios are not preserved under perspective
`distortion. The 4-point cross-ratio
`
`(WNW)
`(ENE)
`
`(Where E is the distance betWeen tWo points m and n, and a,
`b, c, and d are four collinear points in order) is knoWn to be
`invariant under perspective projections and has been used for
`constructing identi?able ?ducial patterns from 4 colinear
`markers (Robert van Liere & Jurriaan D. Mulder, “Optical
`Tracking Using Projective Invariant Marker Pattern Proper
`ties,” IEEE Virtual Reality 2003 Conference, March 22-26,
`Los Angeles 2003).
`Unfortunately, We found this metric to be so sensitive to
`point measurement noise that only a feW different con?gura
`tions could be reliably distinguished for a reasonable-siZed
`?ducial array. To address these shortcomings, We ?rst esti
`mate the “perspective distortion” or degree of slant of the line
`aWay from the camera based on the observed distances
`betWeen three collinear points having a knoWn relationship.
`We then use this pe