I 1111111111111111 1111111111 1111111111 1111111111 1111111111111111 IIII IIII IIII
`US010403051B2
`
`c12) United States Patent
`Soon-Shiong
`
`US 10,403,051 B2
`(IO) Patent No.:
`(45) Date of Patent:
`Sep.3,2019
`
`(54)
`
`INTERFERENCE BASED AUGMENTED
`REALITY HOSTING PLATFORMS
`
`(71) Applicant: Nant Holdings IP, LLC, Culver City,
`CA (US)
`
`(72)
`
`Inventor: Patrick Soon-Shiong, Los Angeles, CA
`(US)
`
`(73) Assignee: Nant Holdings IP, LLC, Culver City,
`CA (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by O days.
`
`EP
`EP
`
`(58) Field of Classification Search
`USPC .......................................................... 345/633
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,625,765 A
`5,682,332 A
`
`4/1997 Ellenby et al.
`10/1997 Ellenby et al.
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`6/2000
`1 012 725
`10/2002
`1 246 080 A2
`(Continued)
`
`OTHER PUBLICATIONS
`
`(21) Appl. No.: 16/186,405
`
`(22) Filed:
`
`Nov. 9, 2018
`
`(65)
`
`Prior Publication Data
`
`US 2019/0080518 Al Mar. 14, 2019
`
`(60)
`
`(51)
`
`(52)
`
`Related U.S. Application Data
`
`Continuation of application No. 15/786,242, filed on
`Oct. 17, 2017, now Pat. No. 10,127,733, which is a
`(Continued)
`
`Int. Cl.
`A63F 13121
`A63F 13/65
`
`(2014.01)
`(2014.01)
`(Continued)
`
`U.S. Cl.
`CPC ............ G06T 191006 (2013.01); A63F 13121
`(2014.09); A63F 131212 (2014.09); A63F
`13132 (2014.09);
`
`International Search Report and Written Opinion issued in Interna(cid:173)
`tional Application No. PCT/US2012/032204 dated Oct. 29, 2012.
`(Continued)
`
`Primary Examiner - Wesner Sajous
`(74) Attorney, Agent, or Firm - Maurie! Kapouytian
`Woods LLP; Lana Akopyan; Andrew A. Noble
`
`ABSTRACT
`(57)
`Interference-based augmented reality hosting platforms are
`presented. Hosting platforms can include networking nodes
`capable of analyzing a digital representation of scene to
`derive interference among elements of the scene. The host(cid:173)
`ing platform utilizes the interference to adjust the presence
`of augmented reality objects within an augmented reality
`experience. Elements of a scene can constructively interfere,
`enhancing presence of augmented reality objects; or destruc(cid:173)
`tively interfere, suppressing presence of augmented reality
`objects.
`
`(Continued)
`
`44 Claims, 6 Drawing Sheets
`
`Mobile Device Interface
`(Port{s); APls; HTTP Server; etc.)
`215
`
`Hosting Platform
`(AR Nodes: Server; Switches:
`Routera; etc.)
`200
`
`I
`
`AR Object Addressing Agent
`
`Ob;ed Recognition Engine
`{Object Recognition-by-Conte>ct
`Service, SIFT, VSLAM, etc.)
`
`Memory
`~
`
`l
`ProceS5or(s) w.
`
`ARObject
`Repository
`240
`
`I
`
`ObJect
`Attributes
`244
`
`I
`
`-;
`.
`ii:
`j
`J : : AR ObJect(s)
`LU
`ill
`I
`
`Context(s) I
`
`232
`
`Niantic's Exhibit No. 1001
`Page 001
`
`

`

`US 10,403,051 B2
`Page 2
`
`Related U.S. Application Data
`
`continuation of application No. 15/213,113, filed on
`Jul. 18, 2016, now Pat. No. 9,824,501, which is a
`continuation of application No. 14/329,882, filed on
`Jul. 11, 2014, now Pat. No. 9,396,589, which is a
`division of application No. 13/173,244, filed on Jun.
`30, 2011, now Pat. No. 8,810,598.
`
`(60) Provisional application No. 61/473,324, filed on Apr.
`8, 2011.
`
`(51)
`
`(2014.01)
`(2014.01)
`(2011.01)
`(2006.01)
`(2006.01)
`(2019.01)
`(2019.01)
`(2014.01)
`(2014.01)
`
`Int. Cl.
`A63F 13/33
`A63F 13132
`G06T 19100
`G06F 17130
`G09G 5100
`G06F 16100
`G06F 1619537
`A63F 131212
`A63F 13/335
`(52) U.S. Cl.
`CPC ............ A63F 13/335 (2014.09); A63F 13/65
`(2014.09); G06F 16100 (2019.01); G06F
`1619537 (2019.01); G09G 5100 (2013.01);
`H05K 999/99 (2013.01); G06T 2219/016
`(2013.01)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,742,521 A
`5,815,411 A
`5,991,827 A
`6,031,545 A
`6,037,936 A
`6,064,398 A
`6,064,749 A
`6,098,118 A
`6,130,673 A
`6,173,239 Bl
`6,278,461 Bl
`6,307,556 Bl
`6,396,475 Bl
`6,414,696 Bl
`6,522,292 Bl
`6,535,210 Bl
`6,690,370 B2
`6,804,726 Bl
`7,016,532 B2
`7,031,875 B2
`7,245,273 B2
`7,301,536 B2
`7,477,780 B2
`7,529,639 B2
`7,564,469 B2
`7,565,008 B2
`7,641,342 B2
`7,680,324 B2
`7,696,905 B2
`7,768,534 B2
`7,847,699 B2
`7,889,193 B2
`7,899,915 B2
`7,904,577 B2
`7,907,128 B2
`7,908,462 B2
`7,916,138 B2
`8,218,873 B2
`8,224,077 B2
`8,224,078 B2
`8,321,527 B2
`
`4/1998 Ellenby et al.
`9/1998 Ellenby et al.
`11/1999 Ellenby et al.
`2/2000 Ellenby et al.
`3/2000 Ellenby et al.
`5/2000 Ellenby et al.
`5/2000 Hirota et al.
`8/2000 Ellenby et al.
`10/2000 Pulli et al.
`1/2001 Ellenby
`8/2001 Ellenby et al.
`10/2001 Ellenby et al.
`5/2002 Ellenby et al.
`7/2002 Ellenby et al.
`2/2003 Ellenby et al.
`3/2003 Ellenby et al.
`2/2004 Ellenby et al.
`10/2004 Ellenby et al.
`3/2006 Boncyk et al.
`4/2006 Ellenby et al.
`7/2007 Eberl et al.
`11/2007 Ellenby et al.
`1/2009 Boncyk et al.
`5/2009 Rasanen et al.
`7/2009 Cohen
`7/2009 Boncyk et al.
`1/2010 Eberl et al.
`3/2010 Boncyk et al.
`4/2010 Ellenby et al.
`8/2010 Pentenrieder et al.
`12/2010 Lee et al.
`2/2011 Platonov et al.
`3/2011 Reisman
`3/2011 Taylor
`3/2011 Bathiche et al.
`3/2011 Sung
`3/2011 John et al.
`7/2012 Boncyk et al.
`7/2012 Boncyk et al.
`7/2012 Boncyk et al.
`11/2012 Martin et al.
`
`4/2013 Mattila et al.
`8,427,508 B2
`4/2013 Mattila et al.
`8,427,598 B2
`6/2013 Nadler et al.
`8,472,972 B2
`8,502,835 Bl * 8/2013 Meehan ................ G06T 19/006
`345/633
`
`8/2013 Richey et al.
`8,519,844 B2
`9/2013 Fisher et al.
`8,527,340 B2
`9/2013 Weising et al.
`8,537,113 B2
`11/2013 Ko et al.
`8,576,756 B2
`12/2013 Dialameh et al.
`8,605,141 B2
`12/2013 Chesnut et al.
`8,606,657 B2
`4/2014 Huang
`8,700,060 B2
`4/2014 Douris et al.
`8,711,176 B2
`8/2014 Soon-Shiong
`8,810,598 B2
`10/2014 Choubassi et al.
`8,872,851 B2
`1/2015 Valaee et al.
`8,933,841 B2
`2/2015 McCulloch et al.
`8,965,741 B2
`9/2015 Geisner et al.
`9,128,520 B2
`9/2015 Gay et al.
`9,129,644 B2
`9/2015 Jin
`9,131,208 B2
`10/2015 Valaee et al.
`9,167,386 B2
`11/2015 McKinnon
`9,177,381 B2
`11/2015 Small et al.
`9,182,815 B2
`1/2016 Stroila
`9,230,367 B2
`4/2016 Meadow et al.
`9,311,397 B2
`7/2016 Soon-Shiong
`9,396,589 B2
`11/2016 Baker et al.
`9,482,528 B2
`11/2016 Visser et al.
`9,495,591 B2
`1/2017 Huang et al.
`9,536,251 B2
`2/2017 McKinnon
`9,582,516 B2
`9,582,561 B2 * 2/2017 Krishnan .......... G06F 17 /30575
`9,817,848 B2
`11/2017 McKinnon et al.
`9,824,501 B2
`11/2017 Soon-Shiong
`10,127,733 B2
`11/2018 Soon-Shiong
`10,140,317 B2
`11/2018 McKinnon et al.
`2002/0163521 Al
`11/2002 Ellenby et al.
`2004/0203380 Al
`10/2004 Hamdi et al.
`2005/0024501 Al
`2/2005 Ellenby et al.
`2005/0208457 Al
`9/2005 Fink et al.
`2005/0285878 Al
`12/2005 Singh et al.
`2005/0289590 Al
`12/2005 Cheok et al.
`2006/0025229 Al
`2/2006 Mahajan et al.
`2006/0038833 Al
`2/2006 Mallinson et al.
`2006/0047704 Al
`3/2006 Gopalakrishnan
`2006/0161379 Al
`7/2006 Ellenby et al.
`2006/0190812 Al
`8/2006 Ellenby et al.
`2007/0109619 Al
`5/2007 Eberl et al.
`2007/0146391 Al
`6/2007 Pentenrieder et al.
`2007/0182739 Al
`8/2007 Platonov et al.
`2008/0024594 Al
`1/2008 Ritchey
`2008/0154538 Al
`6/2008 Stathis
`2008/0157946 Al
`7/2008 Eberl et al.
`2008/0198159 Al
`8/2008 Liu et al.
`2008/0198222 Al
`8/2008 Gowda
`2009/0003662 Al
`1/2009 Joseph et al.
`2009/0081959 Al
`3/2009 Gyorfi et al.
`2009/0102859 Al
`4/2009 Athsani et al.
`2009/0167787 Al
`7/2009 Bathiche et al.
`2009/0193055 Al
`7/2009 Kuberka et al.
`2009/0237 546 Al
`9/2009 Bloebaum et al.
`2009/0271160 Al
`10/2009 Copenhagen et al.
`2009/0271715 Al
`10/2009 Tumuluri
`2010/0017722 Al
`1/2010 Cohen
`2010/0023878 Al
`1/2010 Douris et al.
`2010/0045933 Al
`2/2010 Eberl et al.
`2010/0188638 Al
`7/2010 Eberl et al.
`2010/0208033 Al
`8/2010 Edge et al.
`2010/0257252 Al
`10/2010 Dougherty et al.
`2010/0287485 Al
`11/2010 Bettolami et al.
`2010/0315418 Al
`12/2010 Woo
`2010/0321540 Al
`12/2010 Woo et al.
`2010/0325154 Al
`12/2010 Schloter et al.
`2011/0038634 Al
`2/2011 DeCusatis et al.
`2011/0221771 Al
`9/2011 Cramer et al.
`2011/0279445 Al
`11/2011 Murphy et al.
`2011/0316880 Al
`12/2011 Ojala et al.
`2012/0105474 Al
`5/2012 Cudalbu et al.
`2012/0113141 Al
`5/2012 Zimmerman et al.
`
`Niantic's Exhibit No. 1001
`Page 002
`
`

`

`US 10,403,051 B2
`Page 3
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2012/0122570 Al*
`
`5/2012 Baronoff ............... A63F 13/655
`463/31
`
`5/2012 Kim et al.
`2012/0127201 Al
`2012/0293506 Al* 11/2012 Vertucci .................. G06F 3/012
`345/419
`
`2013/0073988 Al
`2013/0128060 Al
`2013/0176202 Al
`2014/0161323 Al
`2014/0184749 Al
`2015/0172626 Al
`
`3/2013 Groten et al.
`5/2013 Rhoads et al.
`7/2013 Gervautz
`6/2014 Livyatan et al.
`7/2014 Hilliges et al.
`6/2015 Martini
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`EP
`JP
`JP
`JP
`KR
`KR
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`
`1 354 260
`1 119 798 Bl
`2 207 113 Al
`2010-118019 A
`2011-153324 A
`2011-253324 A
`2010-0124947 A
`10-1171264 Bl
`WO 97/44737 Al
`WO 99/42946 A2
`WO 99/42947 A2
`WO 00/20929 Al
`WO 01/63487 Al
`WO 01/71282 Al
`WO 02/03091 A2
`WO 02/059716 A2
`WO 02/073818 Al
`WO 2007/140155 A2
`WO 2010/079876 Al
`WO 2010/138344 A2
`WO 2011/028720 Al
`WO 2013/023705 Al
`
`10/2003
`3/2005
`7/2010
`5/2010
`8/2011
`12/2011
`11/2010
`8/2012
`11/1997
`8/1999
`8/1999
`4/2000
`8/2001
`9/2001
`1/2002
`8/2002
`9/2002
`12/2007
`7/2010
`12/2010
`3/2011
`2/2013
`
`OTHER PUBLICATIONS
`
`Wauters, "Stanford Graduates Release Pulse, A Must-Have News
`App For The iPad", Techcrunch.com, techcrunch.corn/2010/05/31/
`pul se-i pad/.
`Hickins, "A License to Pry", The Wall Street Journal, http://blogs.
`wsj .corn/digits/2011/03/ 1 Ola-license-to-pry/tab/print/.
`Notice of Reasons for Rejection issued in Japanese Patent Appli(cid:173)
`cation No. 2014-503962 dated Sep. 22, 2014.
`Notice of Reasons for Rejection issued in Japanese Patent Appli(cid:173)
`cation No. 2014-503962 dated Jun. 30, 2015.
`
`European Search Report issued in European Patent Application No.
`12767566.8 dated Mar. 20, 2015.
`Vu et al., "High Accuracy and Visibility-Consistent Dense Multiview
`Stereo," IEEE Transactions on Pattern Analysis and Machine Intel(cid:173)
`ligence, 2012, vol. 34, No. 5, 13 pages.
`International Search Report and Written Opinion issued in Interna(cid:173)
`tional Application No. PCT/US2014/061283 dated Aug. 5, 2015, 11
`pages.
`Pang et al., "Development of a Process-Based Model for Dynamic
`Interaction in Spatio-Temporal GIS", Geoinformatica, 2002, vol. 6,
`No. 4, pp. 323-344.
`Zhu et al., "The Geometrical Properties of Irregular 2D Voronoi
`Tessellations," Philosophical Magazine A, 2001, vol. 81, No. 12, pp.
`2765-2783.
`"3D Laser Mapping Launches Mobile Indoor Mapping System," 3D
`Laser Mapping, Dec. 3, 2012, 1 page.
`Banwell et al., "Combining Absolute Positioning and Vision for
`Wide Area Augmented Reality," Proceedings of the International
`Conference on Computer Graphics Theory and Applications. 2010,
`4 pages.
`Li et al., "3-D Motion Estimation and Oniine Temporal Calibration
`for Camera-IMU Systems," Proceedings of the IEEE International
`Conference on Robotics and Automation (ICRA), 2013, 8 pages.
`Li et al., "High-fidelity Sensor Modeling and Self-Calibration in
`Vision-aided Inertial Navigation," Proceedings of the IEEE Inter(cid:173)
`national Conference on Robotics and .Automation (ICRA), 2014, 8
`pages.
`Li et al., "Online Temporal Calibration for Camera-IMU Systems:
`Theory and Algorithms," International Journal of Robotics Research,
`vol. 33, Issue 7, 2014, 16 pages.
`Li et al., "Real-time Motion Tracking or a Cellphone using Inertial
`Sensing and a Rolling-Shutter Camera," Proceedings of the IEEE
`International Conference on Robotics and Automation (ICRA),
`2013, 8 pages.
`Mourikis, "Method for Processing Feature Measurements in Vision(cid:173)
`Aided Inertial Navigation," 3 pages.
`Mourikis et al., "Methods for Motion Estimation With a Rolling(cid:173)
`Shutter Camera," Proceedings of the IEEE International Conference
`or Robotics and Automation (ICRA), Karlsruhe, Germany May
`6-10, 2013, 9 pages.
`Panzarino, "What Exactly WiFiSlam Is, And Why Apple Acquired
`It," http:/ /thenextweb.corn/apple/2013/03/26/what-exactly-wifislam(cid:173)
`is-and-why-apple-acquired-it, Mar. 26, 2013, 10 pages.
`Vondrick et al., "HOGgles: Visualizing Object Detection Features,"
`IEEE International Conference on Computer Vision (ICCV), 2013,
`9 pages.
`U.S. Appl. No. 16/168,419, filed Oct. 23, 2018.
`
`* cited by examiner
`
`Niantic's Exhibit No. 1001
`Page 003
`
`

`

`U.S. Patent
`
`Sep.3,2019
`
`Sheet 1 of 6
`
`US 10,403,051 B2
`
`100
`
`(.
`
`Sensor(s)
`130
`
`AR Object( s)
`
`-~/
`
`-------------
`
`-------
`
`!
`
`142'
`
`~
`
`..---,\'\
`
`Mobile Devices
`
`AR\
`/object(s)\
`
`i42 \
`
`"-..
`
`I
`I
`/
`
`\.'\ (Cell Ph_ones, Appliance,
`"'-
`Kiosks, etc.)
`110
`
`_ __,,
`
`(
`\
`
`'\
`
`\
`
`Elli
`
`"'1
`J
`
`Edge
`Device
`180
`
`I I
`
`Networking Fabric
`(Cloud; Internet; Fabric of Switches,
`Routers, etc.)
`115
`
`\
`
`\ I
`
`I
`J
`
`0
`
`/
`
`Node
`120
`
`l
`
`AR Object
`Repository
`140
`
`\
`\
`
`Node
`120
`
`,,.
`
`\
`\
`
`/
`I / /
`
`120
`
`_.,,:,Y
`k..,,.
`
`-,c..
`...................
`,..._
`'
`
`~-(cid:173)-----
`-
`6 ~----
`
`\
`\
`
`AR Object
`Repository
`140
`
`N~e
`120
`
`Figure 1
`
`Niantic's Exhibit No. 1001
`Page 004
`
`

`

`U.S. Patent
`
`Sep.3,2019
`
`Sheet 2 of 6
`
`US 10,403,051 B2
`
`Mobile Device Interface
`(Port(s); APls; HTTP Server; etc.)
`215
`
`Hosting Platform
`(AR Nodes; Server; Switches;
`Routers; etc.)
`200
`
`Object Recognition Engine
`AR Object Addressing Agent ~ {Object Recognition-by-Context
`Service; SIFT, VSLAM, etc.)
`
`220
`
`+------
`
`260
`
`1
`
`Memory
`230
`
`AR Object
`Repository
`240
`
`Object
`Attributes
`244
`
`AR Object(s)
`242
`
`Context(s)
`232
`
`l
`
`Processor(s)
`250
`
`Core(s)
`255
`
`Figure 2
`
`Niantic's Exhibit No. 1001
`Page 005
`
`

`

`U.S. Patent
`
`Sep.3,2019
`
`Sheet 3 of 6
`
`US 10,403,051 B2
`
`Digital Representation
`(Sensor Data; Processed Data; etc.)
`
`334
`
`j}
`
`Context(s) 332
`
`Element{s) 390
`
`Context 332A
`
`Element 390A
`(Recognized)
`
`• Attributes . - - --
`
`• Name
`• Etc.
`
`• Name
`• Signature
`•
`Interference
`,,..- ~ 0 Attribute Vector (Pv)
`Interference Function (F1)
`0
`C Etc.
`
`f---
`
`I Element 390B
`
`I
`
`I Context 332B
`
`I
`
`Available AR
`Object
`342
`
`Derived Interference 350
`
`F1{Pv) = I: Element Properties
`
`Interference Criteria (Object
`Attributes versus Derived
`Interference)
`• Attrv1 >= I: E;(Pv1;)
`• Attrv2 = I: E;(Pv2;)
`• AttrvJ <= I: E(Pv3;)
`• AttrV4 <> NULL
`• Etc.
`
`Context Relevant
`AR Object(s)
`346
`
`Niantic's Exhibit No. 1001
`Page 006
`
`

`

`U.S. Patent
`
`Sep.3,2019
`
`Sheet 4 of 6
`
`US 10,403,051 B2
`
`Relevant AR Object
`(Enhanced Presence from
`Constructive Interference)
`
`446A r-
`
`\ \
`
`\ / Real World Elements
`{Recognized)
`490
`
`Mobile Device
`410A
`........... "-
`"--.._,'--..
`
`Mobile Device
`410B
`
`~
`
`,,
`Ocf6
`I I
`I
`I I
`I
`
`\ \ \
`\\\j
`' }
`./JI\
`I! \
`I I \
`/
`// I I
`
`/
`
`\
`
`i
`
`Relevant AR Object
`{Suppressed Presence from
`Destructive Interference)
`446B
`
`Figure 4
`
`Niantic's Exhibit No. 1001
`Page 007
`
`

`

`U.S. Patent
`
`Sep.3,2019
`
`Sheet 5 of 6
`
`US 10,403,051 B2
`
`Device
`510
`
`Device
`510
`
`' '
`,
`---------------------------->------------~------
`/ Sensor(s)
`'
`,;----,-----..' 530
`
`Invite two more friends
`to Win!
`
`______ .
`
`Platform
`500
`
`Contexts
`532
`
`AR Object
`Repository
`540
`
`Elements
`590
`
`Mobile Device
`510A
`
`AR Object(s)
`(Member Objects) __ _
`546
`-----
`
`Your Team Wins!
`Coupon Available!
`Click Here to Redeem
`
`,--
`;.....______,... _
`
`Digital Representation
`(Sensor Data, Location,
`Time, Image, etc.)
`534
`
`Set of Relevant AR
`Objects
`(Promotions, Coupons,
`etc.)
`546
`
`Interference
`550
`
`Object Recognition Engine
`560
`
`Figure 5
`
`Niantic's Exhibit No. 1001
`Page 008
`
`

`

`U.S. Patent
`
`Sep.3,2019
`
`Sheet 6 of 6
`
`US 10,403,051 B2
`
`Scene
`695
`~
`
`Device
`610
`
`Device
`610
`
`Mobile
`Device
`610A
`
`AR
`Message Board
`1---M-s_g_: O-ra-ng-,-th-is-is-th_e_t_oy-1 m_e_n_ti_o_ne_d_. --1 Object( s)
`646A
`
`Click here to purchase
`Average review:*****
`
`Click here to configure an alert
`Alert Two others interested in this toy
`
`Platform
`600
`
`Relevant AR Objects
`(Tokens, Prizes, Content, Experience,
`Messages, Reviews, etc.)
`646
`
`Figure 6
`
`Niantic's Exhibit No. 1001
`Page 009
`
`

`

`US 10,403,051 B2
`
`1
`INTERFERENCE BASED AUGMENTED
`REALITY HOSTING PLATFORMS
`
`This application is a continuation of U.S. application Ser.
`No. 15/786,242 filed Oct. 17, 2017, which is a continuation
`of U.S. application Ser. No. 15/213,113 filed Jul. 18, 2016,
`now U.S. Pat. No. 9,824,501, which is a continuation of U.S.
`application Ser. No. 14/329,882 filed Jul. 11, 2014, now U.S.
`Pat. No. 9,396,589, which is a divisional of U.S. patent
`application Ser. No. 13/173,244, filed on Jun. 30, 2011, now
`U.S. Pat. No. 8,810,598, which claims the benefit of priority
`to U.S. Provisional Application having Ser. No. 61/473,324
`filed on Apr. 8, 2011, which are hereby incorporated by
`reference in their entirety. U.S. patents and U.S. patent
`application publications discussed herein are hereby incor(cid:173)
`porated by reference in their entirety. Non-patent publica(cid:173)
`tions discussed herein are hereby incorporated by reference
`to the extent permitted by 37 CFR § 1.57(e). Where a
`definition or use of a term in an incorporated reference is
`inconsistent or contrary to the definition of that term pro(cid:173)
`vided herein, the definition of that term provided herein
`applies and the definition of that term in the reference does
`not apply.
`
`FIELD OF THE INVENTION
`
`The field of the invention is augmented reality technolo(cid:173)
`gies.
`
`BACKGROUND
`
`Augmented reality represents a presentation of virtual
`objects along side real-world elements. Individuals can
`experience or interact with augmented realities according to
`the rules defined by the reality designers. Individuals tap into
`augmented reality content via cell phones, mobile comput(cid:173)
`ing platforms, or other AR-capable devices. Augmented
`reality continues to encroach rapidly on every day life while
`the amount of augmented reality content continues to grow
`at an alarming rate. Individuals are easily overwhelmed by
`the growing excess of available augmented reality content.
`Consider one augmented reality service, BUMP.com.
`BUMP.com offers access to annotations bound to individual
`license plates as described in the Wall Street Journal™ web
`articled titled "License to Pry", published on Mar. 10, 2011.
`BUMP.com allows individuals to send images of license
`plates to the BUMP.com service. The service in turn
`attempts to recognize the license plate and returns annota(cid:173)
`tions left by others for the same plate. Users of the system
`require a dedicated application to interact with the content.
`BUMP.com only supports providing access to their available
`content via their application.
`Layar™ of Amsterdam, The Netherlands, makes further
`strides in presenting augmented reality by offering access to
`multiple augmented reality layers where each layer is dis- 55
`tinct or separate from other layers. A user can select which
`layer where layers are published by one or more third party
`developers. Even though Layar provides an application
`allowing users to select content provided by multiple third
`parties, the user is required choose a layer via the Layar 60
`application. Furthermore, the user is presented with single
`purpose content rather than experiencing augmented reality
`as naturally as one would experience the real-world. In the
`coming world of ever-present augmented reality, users
`should be able to seamlessly access or interact with aug- 65
`mented reality content as naturally as they would interact
`with real-world elements.
`
`5
`
`30
`
`2
`Some progress has been made over the last few years
`toward creating a seamless integration between user and
`augmented reality environments. For example, U.S. patent
`application publication 2006/0047704 to Gopalakrishnan
`titled "Method and System for Providing Information Ser(cid:173)
`vice Relevant to Visual Imagery", filed Aug. 30, 2005,
`discusses presenting embedded information services for an
`augment reality experience based on a context. Yet another
`example includes U.S. patent application publication 2009/
`10 0167787 to Bathiche et al. titled "Augment Reality and
`Filtering", filed Dec. 28, 2007, offers deeper insight in
`providing an enhanced user experience based on a context.
`Bathiche discusses that virtual capabilities can be inter(cid:173)
`spersed with real-world situations where the virtual data can
`15 be filtered, ranked, modified, or ignored based on a context.
`In a similar vein, U.S. patent application publication 2010/
`0257252 to Dougherty titled "Augmented Reality Cloud
`Computing", filed Apr. 1, 2009, also describes providing
`overlay information considered pertinent to a user's sur-
`20 rounding environment. Although useful for providing an
`enriched experience for users based on context, the user still
`must interact with a dedicated augmented reality system.
`U.S. Pat. No. 7,529,639 to Raisnen et al. titled "Location(cid:173)
`Based Novelty Index Value and Recommendation System
`25 and Method", filed Mar. 4, 2008, describes using location
`and an inferred context to generate recommendations for a
`user. The above references also fail to appreciate that objects
`within an environment or scene can interfere with each other
`to give rise to an augmented reality experience.
`From the perspective of presenting augmented reality
`context, to some degree U.S. Pat. No. 7,899,915 to Reisman
`titled "Method and Apparatus for Browsing Using Multiple
`Coordinated Device Sets", filed May 8, 2003, appreciates
`that multiple devices can be utilized by a user. Reisman's
`35 approach allows a user to switch among display or presen(cid:173)
`tation devices when interacting with hypermedia. Unfortu(cid:173)
`nately, Reisman merely handles the user's side of a rich
`media interaction and fails to appreciate that a user's expe(cid:173)
`rience is also impacted by the underlying dedicated aug-
`40 mented reality infrastructure or by interference among ele(cid:173)
`ments of a scene.
`U.S. Pat. No. 7,904,577 to Taylor titled "Data Transmis(cid:173)
`sion Protocol and Visual Display for a Networked Computer
`System", filed Mar. 31, 2008, provides some support for
`45 virtual reality gaming through a protocol supporting mul(cid:173)
`tiple players. Even further, U.S. Pat. No. 7,908,462 to Sung
`titled "Virtual World Simulation Systems and Methods Uti(cid:173)
`lizing Parallel Coprocessors, and Computer Program Prod(cid:173)
`ucts Thereof', filed Jun. 9, 2010, contemplates hosting a
`50 virtual work on parallel processing array of graphic proces(cid:173)
`sors or field-progranimable gate arrays. Although focused on
`providing infrastructure, the contemplated infrastructures
`still requires the user to interact with a dedicated augmented
`reality system.
`These and all other extrinsic materials discussed herein
`are incorporated by reference in their entirety. Where a
`definition or use of a term in an incorporated reference is
`inconsistent or contrary to the definition of that term pro(cid:173)
`vided herein, the definition of that term provided herein
`applies and the definition of that term in the reference does
`not apply.
`Unless the context dictates the contrary, all ranges set
`forth herein should be interpreted as being inclusive of their
`endpoints, and open-ended ranges should be interpreted to
`include commercially practical values. Similarly, all lists of
`values should be considered as inclusive of intermediate
`values unless the context indicates the contrary.
`
`Niantic's Exhibit No. 1001
`Page 0010
`
`

`

`US 10,403,051 B2
`
`4
`device, etc.). The digital representation can include data
`representing one or more elements of the scene. In some
`embodiments, the data includes sensor data captured by the
`mobile device, other sensing devices proximate to the scene,
`or devices capable of capturing data relevant to the scene.
`The platform can further include an object recognition
`engine in communication with the mobile device interface
`and able analyze the digital representation to recognize one
`or more elements of the scene as one or more target objects.
`The object recognition engine can further determine a con(cid:173)
`text related to the scene based on the digital representation
`and pertaining to the target object. Further, the engine can
`identify a set of relevant AR objects from available AR
`objects with respect to the context based on a derived
`15 interference among elements (e.g., real-world elements,
`virtual elements, etc.). In more preferred embodiments the
`derived interference forms criteria through which an AR
`experience is presented to an individual via the mobile
`device. The object recognition engine can also configure one
`20 or more remote devices to allow an interaction with a
`member object of the set of relevant AR objects according
`to the derived interference. In especially preferred embodi(cid:173)
`ments, the interaction involves participating in a commercial
`transaction with a commerce engine. For example, an indi-
`25 vidual can purchase the member object or even a real-world
`object participating within the augmented reality.
`Various objects, features, aspects and advantages of the
`inventive subject matter will become more apparent from
`the following detailed description of preferred embodi-
`3o ments, along with the accompanying drawing figures in
`which like numerals represent like components.
`
`BRIEF DESCRIPTION OF THE DRAWING
`
`FIG. 1 is a schematic of an augmented reality ecosystem.
`FIG. 2 is a schematic of an augmented reality hosting
`platform.
`FIG. 3 illustrates a more detailed view of deriving inter(cid:173)
`ference among elements in a scene.
`FIG. 4 provides examples of presenting augmented reality
`object based on constructive or destructive interference.
`FIG. 5 is an overview of a use case of interacting with
`relevant augmented reality objects based on interference
`within a context involving multiple participants.
`FIG. 6 an overview of a use case of interacting with
`relevant augmented reality objects based on interference
`within a context of a message board.
`
`DETAILED DESCRIPTION
`
`3
`Strangely, known approaches for providing augmented
`reality content treat augment reality platforms as silos of
`virtual worlds or objects where each company develops their
`own hosting infrastructure to provide augmented reality
`services to users. Such approaches fail to allow individuals 5
`to move seamlessly from one augmented reality to another
`as naturally as moving from one room in a building to
`another. Furthermore, existing infrastructures fail to treat
`augmented reality objects as distinct manageable objects in
`an infrastructure agonistic manner, where an augmented 10
`reality infrastructure can also be a pervasive utility. For
`example, in the developed world electricity is ubiquitous or
`more aptly internet connectivity is ubiquitous. Augmented
`realities would benefit from similar treatment.
`In a world of ubiquitous augmented realities or associated
`augmented reality objects where individuals interact with
`the augmented realities in a seamless fashion, individuals
`still require presentation of relevant augmented reality con(cid:173)
`tent especially when features, real or virtual, of an aug(cid:173)
`mented reality can interfere with each other. As discussed
`above with respect to references presenting information
`based on a context, the same references fail to address
`interference among augmented realities or elements, real or
`virtual, participating in an augmented reality experience.
`Interestingly, known art seeks to avoid interference among
`elements of the augmented reality by simply forcing indi(cid:173)
`viduals to select which features to experience. The known
`art fails to appreciate that interference among elements can
`occur based on properties or attributes of the elements.
`Interference is more than mere a filtering mechanism. Inter(cid:173)
`ference represents ambient interplay among present, or
`relevant, elements in a scene that gives rise to an augmented
`reality experience through constructive or destructive inter(cid:173)
`ference.
`What has yet to be appreciated is one or more augmented 35
`realities can be hosted by a common hosting infrastructure,
`the networking infrastructure itself for example, or that
`augmented reality objects can be distinct from the hosting
`platform. For example, the Applicant has appreciated, as
`discussed below, networking nodes within a networking 40
`fabric can provide augmented reality objects or other virtual
`constructs to edge AR-capable devices (e.g., cell phones,
`kiosks, tablet computers, vehicles, etc.). As the edge devices,
`or other devices for that matter, interact with the networking
`fabric by exchanging data, the fabric can determine which 45
`augmented reality objects are most relevant or even which
`augmented reality itself is most relevant for the device based
`on context derived from observed real-world elements.
`Augmented reality context can now be used to determine
`how elements in a scene, a location relevant to an individual, 50
`can interfere with each other to give rise to relevant aug(cid:173)
`mented reality experiences.
`Thus, there is still a need for interference based aug(cid:173)
`mented reality platforms.
`
`SUMMARY OF THE INVENTION
`
`The inventive subject matter provides apparatus, systems
`and methods in which one can utilize an augmented reality
`(AR) hosting platform to give rise to an augmented reality
`experience based on interference among elements of a
`digital representation of a scene. One aspect of the inventive
`subject matter includes an AR hosting platform. Contem(cid:173)
`plated hosting platforms comprise a mobile device interface
`through which the platform can obtained a digital represen(cid:173)
`tation of a scene, possibly local to the mobile device (e.g.,
`cell phone, vehicle, tablet computer, PDA, AR-capable
`
`It should be noted that while the following description is
`drawn to a computer/server based augmented reality plat(cid:173)
`form, various alternative configurations are also deemed
`suitable and may employ various computing devices includ-
`55 ing servers, interfaces, systems, databases, agents, peers,
`engines, controllers, or other types of computing devices
`operating individually or collectively. One should appreciate
`the computing devices comprise a processor configured to
`execute software instructions stored on a tangible, non-
`60 transitory computer readable storage medium ( e.g., hard
`drive, solid state drive, RAM, flash, ROM, etc.). The soft(cid:173)
`ware instructions preferably configure the computing device
`to provide the roles, responsibilities, or other functionality as
`discussed below with respect to the disclosed apparatus. In
`65 especially preferred embodiments, the various servers, sys(cid:173)
`tems, databases, or interfaces exchange data using standard(cid:173)
`ized protocols or algorithms, possibly based on HTTP,
`
`Niantic's Exhibit No. 1001
`Page 0011
`
`

`

`US 10,403,051 B2
`
`5
`HTTPS, AES, public-private key exchanges, web service
`APis, known financial transaction protocols, or other elec(cid:173)
`tronic information exchanging methods. Data exchanges
`preferably are conducted over a packet-switched network,
`the Internet, LAN, WAN, VPN, or other type of packet 5
`switched network.
`One should appreciate that the disclosed techniques pro(cid:173)
`vide many advantageous technical effects including provid(cid:173)
`ing an AR hosting infrastructure capable of configuring
`remote device to interact with AR object objects. For 10
`example, contemplated infrastructures determine a relevant
`augmented reality context from environment data represent(cid:173)
`ing a real-world environment local to an AR-capable device
`and instruct the device to interact with other AR-capable
`devices, AR objects, real-world objects participating in an 15
`augmented reality, or other objects considered to be perti(cid:173)
`nent to the germane augmented reality.
`The
`following discussion provides many example
`embodiments of the inventive subject matter. Although each
`embodiment represents a single combination of inventive 20
`elements, the inventive subject matter is considered to
`include all possible combinations of the disclosed inventive
`elements. Thus if one embodiment comprises inventive
`elements A, B, and C, and a second embodiment comprises
`inventive elements B and D, then the inventive subject 25
`matter is also considered to include other remaining com(cid:173)
`binations of A, B, C, or D, even if not explicitly disclosed.