Case 2:20-cv-07872-MWF-PVC Document 1-7 Filed 08/27/20 Page 1 of 26 Page ID #:200
`Case 2:20-cv-07872-MWF-PVC Document 1-7 Filed 08/27/20 Page 1 of 26 Page ID #:200
`
`
`
`
`
`
`
`
`
`
`EXHIBIT G
`
`EXHIBIT G
`
`

`

`Case 2:20-cv-07872-MWF-PVC Document 1-7 Filed 08/27/20 Page 2 of 26 Page ID #:201
`
`US009031278B2
`
`(12) United States Patent
`Boncyk et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 9,031.278 B2
`*May 12, 2015
`
`(54) IMAGE CAPTURE AND IDENTIFICATION
`SYSTEMAND PROCESS
`
`(71) Applicant: Nant Holdings IP, LLC, Culver City,
`CA (US)
`(72) Inventors: Wayne C. Boncyk, Evergreen, CO (US);
`Ronald H. Cohen, Pasadena, CA (US)
`(73) Assignee: Nant Holdings IP, LLC, Culver City,
`CA (US)
`
`30/0643 (2013.01); G06O40/00 (2013.01);
`G06K9/78 (2013.01); G06O 30/0601
`(2013.01); G06K9/0063 (2013.01); H04N
`5/225 (2013.01); G06O20/40145 (2013.01);
`G06O 30/0257 (2013.01);
`Continued
`(Continued)
`(58) Field of Classification Search
`USPC .................. 382/100, 305, 181, 176; 707/728;
`705/14.23, 14.64, 21, 35, 26.1
`See application file for complete search history.
`
`(*) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`This patent is Subject to a terminal dis-
`claimer.
`
`(21) Appl. No.: 14/194,619
`
`(22) Filed:
`
`Feb. 28, 2014
`
`(65)
`
`Prior Publication Data
`US 2014/0177920 A1
`Jun. 26, 2014
`
`Related U.S. Application Data
`-
`-
`-
`(60) Division of application No. 13/693,983, filed O Dec.
`4, 2012, now Pat. No. 8,712, 193, which is a
`continuation of application No. 13/069,112, filed on
`Mar. 22, 2011, now Pat. No. 8,326,031, which is a
`(Continued)
`
`(51) Int. Cl.
`G06K 9/00
`H04N 5/232
`
`(2006.01)
`ol)
`C
`t
`(Continued)
`
`(52) U.S. Cl.
`CPC. H04N 5/23222 (2013.01); G06F 17/30259
`(2013.01); G06K 9/228 (2013.01); G06K
`9/4652 (2013.01); G06K9/4671 (2013.01);
`G06K9/6203 (2013.01); G06O 30/0277
`(2013.01); G06Q30/0623 (2013.01); G06Q
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`3,800,082 A
`3, 1974 Fish
`4.947,321 A
`8/1990 Spence et al.
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`DE
`EP
`
`4/2002
`10050486 A1
`1/1999
`O614559 B1
`(Continued)
`OTHER PUBLICATIONS
`
`Arai T. et al., “PaperLink: A Technique for Hyperlinking from Real
`Paper to Electronic Content.” CHI 97 Electronic Publications:
`P apers, Conference on Human Factors in Computer Systems,
`Atlanta, Georgia, Mar. 22-27. 1997
`327-334
`Georgia, Mar 24-2, Iy, pp.
`(Continued)
`
`LLP
`Ely Euli it. sy I sh, & T
`(74) Attorney, Agent, or Firm — Fis
`Sang,
`(57)
`ABSTRACT
`A digital image of the object is captured and the object is
`recognized from plurality of objects in a database. An infor
`mation address corresponding to the object is then used to
`access information and initiate communication pertinent to
`the object.
`
`20 Claims, 7 Drawing Sheets
`
`
`
`MAGE
`PROCESSING
`
`TARGET
`OBECT
`NFORMATION
`
`OBJECT
`RECOGNTON
`
`DATABASE
`
`ENTFCATION SERVER
`
`

`

`Case 2:20-cv-07872-MWF-PVC Document 1-7 Filed 08/27/20 Page 3 of 26 Page ID #:202
`
`US 9,031.278 B2
`Page 2
`
`Related U.S. Application Data
`division of application No. 13/037,317, filed on Feb.
`28, 2011, now Pat. No. 8,224,078, which is a division
`of application No. 12/333,630, filed on Dec. 21, 2008,
`now Pat. No. 7,899.243, which is a division of appli
`cation No. 10/492.243, filed as application No. PCT/
`USO2/35.407 on Nov. 5, 2002, now Pat. No. 7,477,780,
`-
`0
`which is a continuation-in-part of application No.
`09/992,942, filed on Nov. 5, 2001, now Pat. No. 7,016,
`532.
`(60) Provisional application No. 60/317,521, filed on Sep.
`5, 2001, provisional application No. 60/246.295, filed
`On Nov. 6, 2000
`• as
`(51) Int. Cl
`(2006.01)
`G06F 7/30
`(2006.01)
`G06K 9/22
`(2006.01)
`G06K 9/46
`(2006.01)
`G06K 9/62
`(2012.01)
`G06O 30/02
`CSO 3838
`G06K 9/78
`(2006.01)
`H04N 5/225
`(2006.01)
`G06O20/40
`(2012.01)
`A6DF 9/08
`(2006.01)
`H04N 7/8
`(2006.01)
`A63F 3/00
`(2014.01)
`H4N2L/8
`(2011.01)
`G06K 9/32
`(2006.01)
`A63F I3/30
`(2014.01)
`H4N2L/254
`(2011.01)
`H4N2L/442
`(2011.01)
`G06O 10/02
`(2012.01)
`G06O20/24
`(2012.01)
`G06O20/34
`(2012.01)
`G06F 3/0482
`(2013.01)
`H04N I/00
`(2006.01)
`H04N 5/91
`(2006.01)
`G06O 90/00
`(2006.01)
`G06K 9/64
`(2006.01)
`(52) U.S. Cl.
`CPC A61F 9/08 (2013.01); H04N 7/183 (2013.01);
`A63F 13/00 (2013.01); H04N 21/8173
`(2013.01); G06K9/3241 (2013.01); G06F
`17/30386 (2013.01); A63F 13/12 (2013.01);
`G06K9/6201 (2013.01); G06O 30/0267
`(2013.01); G06Q30/0268 (2013.01); G06Q
`30/0269 (2013.01); H04N 2 1/254 (2013.01);
`H04N 21/44222 (2013.01); H04N 21/8126
`(2013.01); G06Q30/0253 (2013.01); H04N
`21/812 (2013.01); G06K9/00536 (2013.01);
`G06K 9/6202 (2013.01); G06O 10/02
`(2013.01); G06F 17/30256 (2013.01); G06F
`17/30247 (2013.01); G06O20/24 (2013.01);
`G06O20/3567 (2013.01); G06O 30/0217
`(2013.01); G06F 3/0482 (2013.01); H04N
`I/00244 (2013.01); H04N 5/23229 (2013.01);
`H04N 5/91 (2013.01); H04N220I/3253
`(2013.01); H04N220.1/3254 (2013.01); H04N
`220.1/3274 (2013.01); G06F 17/30882
`(2013.01); G06F 17/30879 (2013.01); G06Q
`
`90/00 (2013.01); G06F 17/3025 (2013.01);
`G06F 17/30253 (2013.01); cost
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2f1991 Nama
`4,991,008 A
`7/1991 Rawlings
`5,034,812 A
`8, 1993 Reed et al.
`5,241,671 A
`5,259,037 A 1 1/1993 Plunk
`5,497.314 A
`3, 1996 Novak
`5,576,950 A 11/1996 Tonomura et al.
`5,579,471. A 1 1/1996 Barber et al.
`5,594,806 A
`1, 1997 Colbert
`5,615,324 A
`3/1997 Kuboyama
`5,625,765 A
`4/1997 Ellenby et al.
`5,682,332 A 10/1997 Ellenby et al.
`5,724,579 A
`3, 1998 Suzuki
`5,742,521 A
`4/1998 Ellenby et al.
`5,742,815 A
`4/1998 Stern
`5,751,286 A
`5/1998 Barber et al.
`5,768,633. A
`6/1998 Allen et al.
`5,768,663 A
`6, 1998 Lin
`5,771,307 A
`6/1998 Lu et al.
`5,815.411 A
`9/1998 Ellenby et al.
`5,893,095 A
`4/1999 Jain et al.
`5,894,323 A
`4/1999 Kain et al.
`5,897,625 A
`4/1999 Gustin et al.
`5,911,827 A
`6, 1999 Heller
`5,915,038 A
`6/1999 Abdel-Mottaleb et al.
`5,917,930 A
`6/1999 Kayani et al.
`5,926, 116 A
`7/1999 Kitano et al.
`5,933,823. A
`8, 1999 Cullen et al.
`5,933,829 A
`8, 1999 Durst et al.
`5,933,923 A
`8, 1999 Catlos et al.
`5.937,079 A
`8/1999 Franke
`5,945,982 A
`8/1999 Higashio et al.
`5,971.277 A 10/1999 Cragun et al.
`5,978,773. A 1 1/1999 Hudetz et al.
`5,982,912 A 1 1/1999 Fukui et al.
`5.991,827. A
`1 1/1999 Ellenby et al.
`5.992,752 A 11/1999 Wilz, Sr. et al.
`6,009.204 A 12/1999 Ahmad
`WW-1
`6,031,545 A
`2/2000 Ellenby et al.
`6,037.936 A
`3/2000 Ellenby et al.
`6,037,963 A
`3/2000 Denton et al.
`6,038,295 A
`3/2000 Mattes
`6,038,333 A
`3/2000 Wang
`6,045,039 A
`4/2000 Stinson et al.
`6,055,536 A
`4/2000 Shimakawa et al.
`6,061.478 A
`5, 2000 Kanoh et al.
`6,064,398 A
`5/2000 Ellenby et al.
`6,072,904 A
`6/2000 Desai et al.
`6,081,612 A
`6/2000 Gutkowicz-Krusin et al.
`6,098,118 A
`8/2000 Ellenby et al.
`6,108,656 A
`8, 2000 Durst et al.
`6,134,548 A 10/2000 Gottsman et al.
`6.144,848 A 11/2000 Walsh et al.
`6,173.239 B1
`1/2001 Ellenby
`8.8% R
`38: (th et al.
`eaS
`6,199,048 B1
`3/2001 Hudetz et al.
`6,202,055 B1
`3/2001 Houvener et al.
`6,208,353 B1
`3/2001 Ayer et al.
`6,208,749 B1
`3/2001 Gutkowicz-Krusin et al.
`E. R.
`33. WG,
`6,278.461 B1
`8/2001 Ellenby et al.
`6,286,036 B1
`9, 2001 Rhoads
`6,307,556 B1 102001 Ellenby et al.
`2. R 1939: Sukowicz-Krusin et al.
`6,393,147 B2
`5, 2002 Danneels et al.
`6,396.475 B1
`5/2002 Ellenby et al.
`6,396,537 B1
`5/2002 Squilla et al.
`6,396.963 B2
`5/2002 Shaffer et al.
`
`

`

`Case 2:20-cv-07872-MWF-PVC Document 1-7 Filed 08/27/20 Page 4 of 26 Page ID #:203
`
`US 9,031.278 B2
`Page 3
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`6/2002 Bopardikar et al.
`6,404.975 B1
`6, 2002 Sankrithi et al.
`6.405,975 B1
`6/2002 Rhoads
`6,411,725 B1
`7/2002 Ellenby et al.
`6,414,696 B1
`8, 2002 Rothschild
`6,430,554 B1
`8/2002 Durst, Jr. et al.
`6,434,561 B1
`9/2002 Burkey et al.
`6,446,076 B1
`9, 2002 Morris
`6,453,361 B1
`12/2002 Freeny, Jr.
`6,490.443 B1
`12/2002 Konishi et al.
`6,501,854 B1
`1/2003 Faahraeus
`6,502,756 B1
`1/2003 Haycock
`6,510,238 B2
`2/2003 Ellenby et al.
`6,522,292 B1
`2/2003 Morrison et al.
`6,522,772 B1
`2/2003 Aarnio
`6,522,889 B1
`2/2003 Goldberg
`6,526,158 B1
`3/2003 Cambier et al.
`6,532,298 B1
`3/2003 Koitabashi
`6,533,392 B1
`3/2003 Ellenby et al.
`6,535,210 B1
`4/2003 Durst, Jr. et al.
`6,542,933 B1
`5/2003 Troyanker
`6,563,959 B1
`5/2003 Anderson et al.
`6,567,122 B1
`6/2003 Ebersole et al.
`6,578,017 B1
`6, 2003 Winner et al.
`6,580,385 B1
`7/2003 Kumar et al.
`6,597,818 B2
`7/2003 Appelt et al.
`6,601,026 B2
`8, 2003 Kolls
`6,609, 103 B1
`1 1/2003 Aoki
`6,650,794 B1
`6,651,053 B1 11/2003 Rothschild
`6,658,389 B1
`12/2003 Alpdemir
`6,674,923 B1
`1/2004 Shih et al.
`6,674,993 B1
`1/2004 Tarbouriech
`6,675,165 B1
`1/2004 Rothschild
`6,689,966 B2
`2, 2004 Wiebe
`6,690,370 B2
`2/2004 Ellenby et al.
`6,691.914 B2
`2/2004 Isherwood et al.
`6,711,278 B1
`3, 2004 Guet al.
`6,714,969 B1
`3, 2004 Klein et al.
`6,724.914 B2
`4/2004 Brundage et al.
`6,738,630 B2
`5, 2004 Ashmore
`6,744.935 B2
`6, 2004 Choi et al.
`6,748,122 B1
`6, 2004 Ihara et al.
`6,765.569 B2
`7/2004 Neumann et al.
`6,766,363 B1
`7/2004 Rothschild
`6,771,294 B1
`8, 2004 Pulli et al.
`6,801,657 B1
`10/2004 Cieplinski
`6,804,726 B1 10/2004 Ellenby et al.
`6,842,181 B2
`1/2005 Acharya
`6,853,750 B2
`2/2005 Aoki
`6,856,965 B1
`2/2005 Stinson et al.
`6,865,608 B2
`3/2005 Hunter
`6,866,196 B1
`3/2005 Rathus et al.
`6,868.415 B2
`3/2005 Kageyama et al.
`6,882,756 B1
`4/2005 Bober
`6,885,771 B2
`4/2005 Takahashi
`6,912.464 B1
`6, 2005 Parker
`6,925, 196 B2
`8, 2005 Kasset al.
`6,950,800 B1
`9/2005 McIntyre et al.
`6.956,593 B1
`10/2005 Gupta et al.
`6,963,656 B1
`1 1/2005 Persaud et al.
`6,968,453 B2 11/2005 Doyle et al.
`6,974,078 B1
`12/2005 Simon
`6,985,240 B2
`1/2006 Benke et al.
`6,993,573 B2
`1/2006 Hunter
`6,996,251 B2
`2/2006 Malone et al.
`7,002,551 B2
`2/2006 AZuma et al.
`7,016,532 B2
`3/2006 Boncyk et al.
`7,016,889 B2
`3, 2006 Bazoon
`7,016,899 B1
`3, 2006 Stern et al.
`7,027,652 B1
`4/2006 IAnson
`7,031.496 B2
`4/2006 Shimano et al.
`7,031,536 B2
`4/2006 Kajiwara
`7,031,875 B2
`4/2006 Ellenby et al.
`7,050,653 B2
`5, 2006 Edso et al.
`7,053,916 B2
`5/2006 Kobayashi et al.
`
`6/2006 Giannini et al.
`7,062,454 B1
`9/2006 Jorgensen et al.
`7,103,772 B2
`9, 2006 Stein
`7,113,867 B1
`7,119,831 B2 10/2006 Ohto et al.
`7,121469 B2 10/2006 Dorai et al.
`7,127,094 B1
`10/2006 Elbaum et al.
`7,143,949 B1
`12/2006 Hannigan
`7,167,164 B2
`1/2007 Ericson et al.
`7, 175,095 B2
`2/2007 Pettersson et al.
`7,224.995 B2
`5/2007 Rhoads
`7,245,273 B2
`7, 2007 Eberlet al.
`7,254,548 B1
`8/2007 Tannenbaum
`7,295,718 B2 11/2007 Parket al.
`7,296,747 B2 11/2007 Rohs
`7,301,536 B2 11/2007 Ellenby et al.
`7.305,354 B2 12/2007 Rodriguez et al.
`7,309,015 B2 12/2007 Frantz et al.
`7,310,605 B2 12/2007 Janakiraman et al.
`7,333,947 B2
`2/2008 Wiebe et al.
`7,334,728 B2
`22008 Williams
`7,345,673 B2
`3/2008 Ericson et al.
`7,353,182 B1
`4/2008 Missinhoun et al.
`7,353,184 B2
`4/2008 Kirshenbaum et al.
`7,353,990 B2
`4/2008 Elliot et al.
`7,356,705 B2
`4/2008 Ting.
`7,362.922 B2
`4/2008 Nishiyama et al.
`7,376,645 B2
`5/2008 Bernard
`7,383.209 B2
`6/2008 Hudetz et al.
`7,410,099 B2
`8/2008 Fukasawa et al.
`7.427,980 B1
`9/2008 Partridge et al.
`7,430,588 B2
`9, 2008 Hunter
`7,477,909 B2
`1/2009 Roth et al.
`7,526,440 B2
`4/2009 Walker et al.
`7,548,915 B2
`6/2009 Ramer et al.
`7.558,595 B2
`7/2009 Angelhag
`7,564,469 B2
`7/2009 Cohen
`7,580,061 B2
`8/2009 Toyoda
`7,595,816 B1
`9/2009 Enright et al.
`7,641,342 B2
`1/2010 Eberlet al.
`7,680,324 B2
`3/2010 Boncyk et al.
`7,696,905 B2
`4/2010 Ellenby et al.
`7,707,218 B2 * 4/2010 Gocht et al. .................. 707/758
`7,711,598 B2
`5, 2010 Perkowski
`7,720,436 B2
`5/2010 Hamynen et al.
`7,734,507 B2
`6, 2010 Ritter
`7,737,965 B2
`6, 2010 Alter et al.
`7,751,805 B2
`7/2010 Neven et al.
`7,756,755 B2
`7/2010 Ghosh et al.
`7,764,808 B2
`7/2010 Zhu et al.
`7,765,126 B2
`7/2010 Hudetz et al.
`7,768,534 B2
`8, 2010 Pentenrieder et al.
`7,769,228 B2
`8/2010 Bahlmann et al.
`7,774,283 B2
`8, 2010 Das et al.
`7,775,437 B2
`8/2010 Cohen
`7,797.204 B2 * 9/2010 Balent ............................. 705/28
`7,830,417 B2 11/2010 Liu et al.
`7,843.488 B2 11/2010 Stapleton
`7,845,558 B2 12/2010 Beemer et al.
`7,872,669 B2
`1/2011 Darrell et al.
`7,889, 193 B2
`2/2011 Platonov et al.
`7,896,235 B2
`3/2011 Ramachandran
`7.916,138 B2
`3/2011 John et al.
`8,090,616 B2
`1/2012 Proctor, Jr. et al.
`8,099.332 B2
`1/2012 Lemay et al.
`8,121,944 B2
`2/2012 Norman et al.
`8, 130,242 B2
`3, 2012 Cohen
`8, 131,118 B1
`3/2012 Jing et al.
`8, 131,595 B2
`3/2012 Lee et al.
`8, 189,964 B2
`5/2012 Flynn et al.
`8,218,874 B2
`7/2012 Boncyk et al.
`8,219,146 B2
`7/2012 Connors et al.
`8.255.291 B1
`8, 2012 Nair
`8,312,168 B2 11/2012 Rhoads et al.
`8,320,615 B2 11/2012 Hamza et al.
`8.326,031 B2 12/2012 Boncyk et al.
`8.335.351 B2 12/2012 Boncyk et al.
`8,386,918 B2
`2/2013 Do et al.
`8.442,500 B2
`5/2013 Gupta et al.
`8.447,066 B2
`5/2013 King et al.
`
`

`

`Case 2:20-cv-07872-MWF-PVC Document 1-7 Filed 08/27/20 Page 5 of 26 Page ID #:204
`
`US 9,031.278 B2
`Page 4
`
`(56)
`
`References Cited
`
`FOREIGN PATENT DOCUMENTS
`
`U.S. PATENT DOCUMENTS
`
`
`
`7/2013 Badharudeen et al.
`8.494.274 B2
`7, 2013 Cuttner
`8.497,939 B2
`9, 2013 Persson et al.
`8,542,906 B1
`8,550,903 B2 10/2013 Lyons et al.
`8,588,527 B2 11/2013 Boncyk et al.
`8,605,141 B2 12/2013 Dialameh et al.
`8,626,602 B2
`1/2014 George
`... TOS/20
`8,751,316 B1* 6/2014 Fletchall et al.
`455,552.1
`8,831,677 B2* 9/2014 Villa-Real ...
`8,903430 B2 * 12/2014 Sands et al. ................... 455,457
`2001 OO11276 A1
`8/2001 Durst, Jr. et al.
`2001/0032252 A1 10, 2001 Durst et al.
`2001/0044824 A1 11/2001 Hunter et al.
`2001/0047426 A1 11, 2001 Hunter
`2001/0053252 Al 12/2001 Creque
`2002/0001398 A1
`1/2002 Shimano et al.
`2002fOOO6602 A1
`1/2002 Masters
`2002fOO19819 A1
`2/2002 Sekiguchi et al.
`2002/004.8403 A1
`4/2002 Guerreri
`2002fOO55957 A1
`5, 2002 Ohsawa
`2002fOO84328 A1
`7, 2002 Kim
`2002fOO895.24 A1
`7/2002 Ikeda
`2002/00901.32 A1
`7/2002 Boncyk et al.
`2002/0102966 A1
`8, 2002 Lev et al.
`2002/0103813 A1
`8/2002 Frigon
`2002/O124188 A1
`9, 2002 Sherman et al.
`2002fO140988 A1 10, 2002 Cheatle et al.
`2002/0150298 A1 10/2002 Rajagopal et al.
`2002/0156866 A1 10, 2002 Schneider
`2002/0163521 A1 1 1/2002 Ellenby et al.
`2002fO167536 A1 11, 2002 Valdes et al.
`2003, OO64705 A1
`4/2003. Desiderio
`2003/0095681 A1
`5/2003 Burg et al.
`2003.01.16478 A1
`6/2003 Laskowski
`2003. O164819 A1
`9, 2003 Waibel
`2004/0208.372 A1 10/2004 Boncyk et al.
`2005/00 15370 A1
`1/2005 Stavely et al.
`2005/0024501 A1
`2/2005 Ellenby et al.
`2005/005.5281 A1
`3, 2005 Williams
`2005, 0102233 A1
`5, 2005 Park et al.
`2005/0162523 A1
`7/2005 Darrell et al.
`2005/0162532 A1
`7/2005 Toyoda
`2005.0185060 A1
`8, 2005 Neven
`2005/0206654 A1
`9, 2005 Vaha-Sipila
`2005/0252966 A1 11, 2005 Kulas
`2006,0008124 A1
`1/2006 Ewe et al.
`2006/0038833 A1
`2/2006 Mallinson et al.
`2006, O161379 A1
`7/2006 Ellenby et al.
`2006, O190812 A1
`8/2006 Ellenby et al.
`2006/0223635 A1 10/2006 Rosenberg
`3.29. A.
`358, Rice, et al.
`2007/0182739 A1
`8, 2007 Platonov et al.
`2008, 0021953 A1
`1, 2008 G1
`2008. O157946 A1
`7/2008 Eberlet al.
`2008/O189185 A1
`8, 2008 Matsuo et al. .................. 705/21
`2008/0243721 A1 10, 2008 Joao
`2008/0279481 A1 11, 2008 Ando
`2009/0027337 A1
`1, 2009 Hildreth
`
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`GB
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`WO
`
`6, 1999
`O920.179 A2
`O967574 A2 12/1999
`1O12725 A1
`6, 2000
`O920.179 A3
`9, 2000
`1354260 A2 10, 2003
`1355.258 A2 10, 2003
`2264669 A2 12/2010
`2407230 A
`4/2005
`S6314297 A
`1, 1988
`HO9231244. A
`9, 1997
`H1091634. A
`4f1998
`H10289243 A 10, 1998
`H11265391 A
`9, 1999
`2001101.191 A
`4/2001
`2001160057 A
`6, 2001
`2001.256500 A
`9, 2001
`2001.265970. A
`9, 2001
`2001282825 A 10, 2001
`20021971.03 A
`T 2002
`2002297648 A 10, 2002
`2003.178067 A
`6, 2003
`2003323440 A 11, 2003
`2004.005314 A
`1, 2004
`200403.0377 A
`1, 2004
`20041 18384 A
`4/2004
`2005O11180 A
`1, 2005
`2005038421 A
`2, 2005
`2005049920 A
`2, 2005
`2005SO9219. A
`4/2005
`2007509392 A
`4/2007
`97.44737 A1 11, 1997
`9749060 A1 12/1997
`9837811 A1
`9, 1998
`9846323 A1 10, 1998
`9916O24 A1
`4, 1999
`9942946 A2
`8, 1999
`9942947 A2
`8, 1999
`9944O1.0 A1
`9, 1999
`9942946 A3 10, 1999
`9942947 A3 12/1999
`9967695 A2 12/1999
`O124050 A1
`4, 2001
`O14905.6 A1
`T 2001
`O163487 A1
`8, 2001
`O171282 A1
`9, 2001
`0.173603 A1 10, 2001
`0201143 A2
`1, 2002
`O2O59716 A2
`8, 2002
`02073818 A1
`9, 2002
`O2O82799 A2 10, 2002
`O3O41000 A1
`5, 2003
`
`OTHER PUBLICATIONS
`Bulman J., et al., “Mixed Reality Applications in Urban Environ
`ments. BT Technology Journal, 2004, vol. 22 (3), pp. 84-94.
`Carswell J.D. et al., An Environment for Mobile Context-Based
`Estie Rival IEEE is of th Sh traig
`Workshop on Database and Expert Systems Applications,
`418.8/02, 2002, 5 pages.
`Chang S.F., et al., “Visual Information Retrieval from Large Distrib
`uted Online Respositories.” Communication of Association for Com
`puting Machinery, ISSN: 0001-0782, 1997, vol. 40 (12), pp. 64-71.
`Chang W., et al., “Efficient Resource Selection in Distributed Visual
`Information Systems.” ACM Multimedia, 1997, pp. 203-213.
`Diverdi S., et al., “ARWin—A Desktop Augmented Reality Window
`Manager.” UCSB Tech Report 2003-12, University of California
`Santa Barbara, May 2003, 7 pages.
`Diverdi S., et al., “Level of Detail Interfaces. Proc. ISMAR 2004,
`IEEE/ACMIHyfSymp on Mixed and Augmented Reality, Arlington,
`Virginia, 2004, 2 pages.
`European Search Report for Application No. EP06018047, mailed on
`Oct. 30, 2008, 2 pages.
`Feiner S., et al., “A Touring Machine: Prototyping 3D Mobile Aug
`mented Reality Systems for Exploring the Urban Environment.”
`Personal Technologies, 1997, vol. 1 (4), pp. 208-217.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`. 707/728
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`s al .
`38885 A. ck 3388
`2010.0188638 A1
`7, 2010 Eberlet al.
`2011/0173100 A1
`7/2011 Boncyk et al.
`2012,0002872 A1
`1/2012 Boncyk et al.
`2012/0011119 A1
`1/2012 Baheti et al.
`2012fOO 11142 A1
`1/2012 Baheti et al.
`2012fOO72353 A1
`3/2012 Boone et al.
`2012/0095857 A1
`4/2012 McKelvey et al.
`2012fO231887 A1
`9, 2012 Lee et al.
`2013/0265450 Al
`10/2013 Barnes, Jr.
`2014/0006165 A1* 1/2014 Grigg et al. ................ TO5/1464
`2014/0129942 A1
`5, 2014 Rathod
`2014/0162598 A1* 6, 2014 Villa-Real ..................... 455,411
`
`

`

`Case 2:20-cv-07872-MWF-PVC Document 1-7 Filed 08/27/20 Page 6 of 26 Page ID #:205
`
`US 9,031.278 B2
`Page 5
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`Fernandez F. “Responsive Environments: Digital Objects in the
`Landscape. Thesis Submitted to Department of Landscape Architec
`ture, University of Manitoba, Winnipeg, Manitoba, Mar. 2004. 124
`pageS.
`Geiger C. et al., “Mobile AR4ALL.” Proceedings of the IEEE and
`ACM Intl Symposium on Augmented Reality (ISAR’01), Oct. 29-30,
`2001, Columbia University, New York, 2 pages.
`Gevers T. et al., “PicToSeek: Combining Color and Shape Invariant
`Features for Image Retrieval.” IEEE Transactions on Image Process
`ing, 2000, vol. 9 (1), pp. 102-119.
`Haritaoglu I., “InfoScope: Link from RealWorld to Digital Informa
`tion Space.” IBM Almaden Research, UbiComp, Lecture Notes in
`Computer Science, 2001, vol. 2201, pp. 247-255.
`Hollerer T. et al., “Chapter Nine: Mobile Augmented Reality,” in:
`Telegeoinformatics: Location Based Computing and Services,
`Karimi H., eds., Taylor & Francis Books, Ltd., 2004, Chapter 9, 39
`pageS.
`International Search Report and Written Opinion for Application No.
`PCT/US2007/02010, mailed on Nov. 16, 2007, 5 pages.
`Iwamoto T. et al., “Digital Safari Guidebook With Image Retrieval.”
`International Conference on Advances in Mathematical Computa
`tions and Statistical Computing, 1999, vol. 2, pp. 1011-1012.
`Iwamoto T. et al., “u-Photo: A Design and Implementation of a
`Snapshot Based Method for Capturing Contextual Information.” The
`Second International Conference on Pervasive Computing Pervasive,
`2004, Advances in Pervasive Computing, LinzNienna, Austria, 6
`pageS.
`Jebara T., et al., “Stochasticks: Augmenting the Billiards Experience
`With Probabilistic Vision and Wearable Computers.” International
`Symposium on Wearable Computers, 1997, IEEE Computer Society,
`pp. 138-145.
`Kangas K., et al., “Using Code Mobility to Create Ubiquitous and
`Active Augmented Reality in Mobile Computing.” Mobicom, 1999,
`Seattle, Washington, pp. 48-58.
`Kato H., et al., “Marker Tracking and HMD Calibration for a Video
`Based Augmented Reality Conferencing System.” Proceedings of the
`2nd IEEE and ACM Intl Workshop on Augmented Reality, San Fran
`cisco, California, 1999, pp. 85-94.
`Klinker G., “Augmented Maintenance of Powerplants: A Prototyping
`Case Study of a Mobile AR System.” International Symposium on
`Augmented Reality, 2001, IEEE Computer Society, pp. 124-136.
`Levine J.M., “Real-Time Target and Pose Recognition for 3-D
`Graphical Overlay,” Master's thesis, 1997, 48 pages.
`Ljungstrand P. et al., “WebStickers: Using Physical Tokens to Access,
`Manage and Share Bookmarks on the Web.” Proceedings of the 2000
`ACM Conference on Designing Augmented Reality Environments
`(DARE 2000), 2000, pp. 23-31.
`Rekimoto.J., et al., “Augment-able Reality: Situated Communication
`Through Physical and Digital Spaces.” Wearable Computers, Second
`International Symposium, 1998, pp. 68-75.
`
`Rekimoto J., et al., “CyberCode: Designing Augmented Reality
`Environments With Visual Tags.” Proceedings of the 2000 ACM
`Conference on Designing Augmented Reality Environments, 2000,
`pp. 1-10.
`Rekimoto J., et al., “The World Through the Computer: Computer
`Augmented Interaction With RealWorld Environments.” ACM Sym
`posium on User Interface Software and Technology, 1995, pp. 29-36.
`Rekimoto J., “NaviCam: A Palmtop Device Approach to Augmented
`Reality.” Fundamentals of Wearable Computers and Augmented
`Reality, 2001, Barfield and Caudell, Eds., pp. 353-377.
`Rohs M., et al., “A Conceptual Framework for Camera Phone-Based
`Interaction Techniques.” Pervasive Computing. Lecture Notes in
`Computer Science, 2005, vol. 3468, pp. 171-189.
`Siltanen S., et al., “ImplementingaNatural User Interface for Camera
`Phones Using Visual Tags.” Proceedings of the 7th Australasian User
`interface conference, 2006, vol. 50, pp. 113-116.
`Smailagic A. et al., “Metronaut: A Wearable Computer With Sensing
`and Global Communication Capabilities.” First International Sym
`posium on Wearable Computers, Oct. 13-14, 1997, Cambridge, Mas
`sachusetts; Digest of Papers, pp. 116-122.
`Smith J.R., et al., “VisualSEEk: A Fully Automated Content Based
`Image Query System.” Proceedings of the fourth ACM international
`conference on Multimedia, ACM New York, 1996, pp. 87-98.
`Starner T. et al., “Augmented Reality Through Wearable Comput
`ing.” Presence: Teleoper. Virtual Environ. 6, 4, Massachusetts Insti
`tute of Technology, 1997, 24 pages.
`Supplementary European Search Report for Application No.
`EP02778730, mailed on May 14, 2007, 3 pages.
`Supplementary European Search Report for Application No.
`EP0680 1326, mailed on Aug. 12, 2008, 8 pages.
`Suzuki G. et al., “u-Photo: Interacting with Pervasive Services Using
`Digital Still Images.” Pervasive Computing. Lecture Notes in Com
`puter Science, vol. 3468, 2005, pp. 190-207.
`Toye E., et al., “Interacting with Mobile Services: An Evaluation of
`Camera-Phones and Visual Tags,” in: Personal and Ubiquitous Com
`puting, vol. 11 (2), Springer-Verlag, London Limited, 2007, pp.
`97-106.
`Wagner D., et al., “First Steps Towards Handheld Augmented Real
`ity.” Vienna University of Technology, Proceedings of Seventh IEEE
`International Symposium on Wearable Computers, Oct. 18-21, 2003,
`9 pages.
`Yang J., et al., “Smart Sight: A Tourist Assistant System.” Digest of
`Papers. Third International Symposium on Wearable Computers,
`Oct. 18-19, 1999, San Francisco, California, pp. 73-78.
`Yeh T. et al., “Searching the Web with Mobile Images for location
`Recognition.” IEEE: Proceedings of the 2004 IEEE Computer Soci
`ety Conference on Computer Vision and Pattern Recognition
`(CVPR'04), 1063-6919/04, 2004, 6 pages.
`Zhang X., et al., “Taking AR Into Large Scale Industrial Environ
`ments: Navigation and Information Access With Mobile Computers.”
`Proceedings of the IEEE and ACM Intl Symposium on Augmented
`Reality, 2001, pp. 179-180.
`
`* cited by examiner
`
`

`

`Case 2:20-cv-07872-MWF-PVC Document 1-7 Filed 08/27/20 Page 7 of 26 Page ID #:206
`
`U.S. Patent
`
`May 12, 2015
`
`Sheet 1 of 7
`
`US 9,031.278 B2
`
`
`
`
`
`
`
`
`
`NPUT IMAGE
`CAPTURE
`
`1 O
`
`OBJECT
`MAGE
`
`ETERMINEY
`MAGE
`TYPE
`
`SYMBOLIC
`MAGE
`
`NPUT IMAGE
`DECOMPOSITON
`
`2 6
`
`3 4
`
`DECODE
`
`DATABASE
`MATCHING
`
`
`
`F.G. 1
`
`SELECT BEST
`MATCH
`
`4 0
`
`4 2.
`
`URL RETURN
`
`

`

`Case 2:20-cv-07872-MWF-PVC Document 1-7 Filed 08/27/20 Page 8 of 26 Page ID #:207
`
`U.S. Patent
`
`May 12, 2015
`
`Sheet 2 of 7
`
`US 9,031.278 B2
`
`
`
`

`

`Case 2:20-cv-07872-MWF-PVC Document 1-7 Filed 08/27/20 Page 9 of 26 Page ID #:208
`
`U.S. Patent
`
`May 12, 2015
`
`Sheet 3 of 7
`
`US 9,031.278 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FOREACH INPUT IMAGE
`SEGMENT GROUP
`
`FOREACH OBJECT IN
`DATABASE
`
`
`
`FOREACHSEGMENT
`GROUP N THIS VEW
`
`
`
`GREYSCALE
`COMPARISON
`
`
`
`COLORCUBE
`COMPARSON
`
`SHAPE
`COMPARSON
`
`
`
`
`
`WAVELET
`COMPARSON
`
`FIG 3A
`
`

`

`Case 2:20-cv-07872-MWF-PVC Document 1-7 Filed 08/27/20 Page 10 of 26 Page ID #:209
`
`U.S. Patent
`
`May 12, 2015
`
`Sheet 4 of 7
`
`US 9,031.278 B2
`
`CALCULATE COMBINED
`MATCH SCORE
`
`NEXT SEGMENT GROUPN
`THIS DATABASE VIEW
`
`NEXT VEW OF THIS
`DATABASE OBJECT
`
`NEXT OBJECTIN
`DATABASE
`
`NEXT INPUT MAGE
`SEGMENT GROUP
`
`FIG. 3B
`
`FINISH
`
`

`

`Case 2:20-cv-07872-MWF-PVC Document 1-7 Filed 08/27/20 Page 11 of 26 Page ID #:210
`
`U.S. Patent
`
`May 12, 2015
`
`Sheet 5 of 7
`
`US 9,031.278 B2
`
`TARGET
`OBJECT
`
`CONTENT
`SERVER
`
`1 O2
`
`10 O
`
`1 11
`
`FIG. 4
`
`CAMERA
`
`1 0 3
`
`BROWSER
`
`IMAGE
`PROCESSING
`
`104
`
`1 1 0
`
`
`
`TE
`AL
`Y. A
`t A.
`STVTN
`
`IMAGE DATA
`
`10
`S
`
`1.
`
`\
`OBJE CT
`RECOGNITION
`
`TARGET
`OBJECT
`NFORMATION
`
`DATABASE
`
`IDENTIFICATION SERVER
`
`

`

`Case 2:20-cv-07872-MWF-PVC Document 1-7 Filed 08/27/20 Page 12 of 26 Page ID #:211
`
`U.S. Patent
`
`May 12, 2015
`
`Sheet 6 of 7
`
`US 9,031.278 B2
`
`TARGET
`OBJECT
`
`CONTENT
`SERVER
`
`2 O 2
`
`20 0
`
`2 11
`
`F.G. S
`
`CAMERA
`
`20 3
`
`BROWSER
`
`IMAGE
`PROCESSING
`
`204
`
`210
`
`TERMINAL
`
`MAGE DATA
`
`TARGET
`OBJECT
`INFORMATION
`
`2 OS
`
`
`
`2 O 9
`
`OBJECT
`RECOGNITION
`
`DATABASE
`
`DENTIFICATION SERVER
`
`

`

`Case 2:20-cv-07872-MWF-PVC Document 1-7 Filed 08/27/20 Page 13 of 26 Page ID #:212
`
`U.S. Patent
`
`May 12, 2015
`
`Sheet 7 Of 7
`
`US 9,031.278 B2
`
`IMAGE DATA
`
`
`
`TARGET
`OBJECT
`INFORMATION
`
`3 O 5
`
`
`
`OBJECT
`RECOGNITION
`
`\
`
`DATABASE
`
`DENTIFICATION SERVER
`
`

`

`Case 2:20-cv-07872-MWF-PVC Document 1-7 Filed 08/27/20 Page 14 of 26 Page ID #:213
`
`1.
`IMAGE CAPTURE AND IDENTIFICATION
`SYSTEMAND PROCESS
`
`US 9,031,278 B2
`
`This application is a divisional of Ser. No. 13/693,983, filed
`Dec. 4, 2012, which is a continuation of Ser. No. 137069,112,
`filed Mar. 22, 2011 and issued on Dec. 4, 2012 as U.S. Pat. No.
`8,326,031, which is a divisional of Ser. No. 137037,317 filed
`Feb. 28, 2011 and issued Jul. 17, 2012 as U.S. Pat. No.
`8.224,078, which is a divisional of Ser. No. 12/333,630 filed
`Dec. 12, 2008 and issued Mar. 1, 2011 as U.S. Pat. No.
`7,899,243, which is a divisional of Ser. No. 10/492,243 filed
`May 20, 2004 and issued Jan. 13, 2009 as U.S. Pat. No.
`7,477,780, which is a National Phase of PCT/US02/35.407
`filed Nov. 5, 2002, which is an International Patent applica
`tion and a continuation-in-part of Ser. No. 09/992,942 filed
`Nov. 5, 2001 and issued Mar. 21, 2006 as U.S. Pat. No.
`7,016,532, which claims priority to provisional application
`No. 60/317,521 filed Sep. 5, 2001 and provisional application
`No. 60/246.295 filed Nov. 6, 2000. These and all other refer
`enced patents and applications are incorporated herein by
`reference in their entirety. Where a definition or use of a term
`in a reference that is incorporated by reference is inconsistent
`or contrary to the definition of that term provided herein, the
`definition of that term provided herein is deemed to be con
`trolling.
`
`5
`
`10
`
`15
`
`25
`
`TECHNICAL FIELD
`
`The invention relates an identification method and process
`for objects from digitally captured images thereof that uses
`image characteristics to identify an object from a plurality of
`objects in a database.
`
`30
`
`BACKGROUND ART
`
`35
`
`40
`
`55
`
`There is a need to provide hyperlink functionality in known
`objects without modification to the objects, through reliably
`detecting and identifying the objects based only on the
`appearance of the object, and then locating and Supplying
`information pertinent to the object or initiating communica
`tions pertinent to the object by Supplying an information
`address, such as a Uniform Resource Locator (URL), perti
`nent to the object.
`45
`There is a need to determine the position and orientation of
`known objects based only on imagery of the objects.
`The detection, identification, determination of position and
`orientation, and Subsequent information provision and com
`50
`munication must occur without modification or disfigure
`ment of the object, without the need for any marks, symbols,
`codes, barcodes, or characters on the object, without the need
`to touch or disturb the object, without the need for special
`lighting other than that required for normal human vision,
`without the need for any communication device (radio fre
`quency, infrared, etc.) to be attached to or nearby the object,
`and without human assistance in the identification process.
`The objects to be detected and identified may be 3-dimen
`sional objects, 2-dimensional images (e.g., on paper), or 2-di
`mensional images of 3-dimensional objects, or human
`beings.
`There is a need to provide such identification and hyperlink
`services to persons using mobile computing devices, such as
`Personal Digital Assistants (PDAs) and cellular telephones.
`There is a need to provide such identification and hyperlink
`services to machines, such as factory robots and spacecraft.
`
`60
`
`65
`
`2
`
`Examples include:
`identifying pictures or other art in a museum, where it is
`desired to provide additional information about such art
`objects to museum visitors via mobile wireless devices;
`provision of content (information, text, graphics, music,
`Video, etc.), communications, and transaction mechanisms
`between companies and individuals, via networks (wireless
`or otherwise) initiated by the individuals “pointing and click
`ing with camera-equipped mobile devices on magazine
`advertisements, posters, billboards, consumer products,
`music or video disks or tapes, buildings, vehicles, etc.;
`establishment of a communications link with a machine,
`Such a vending machine or information kiosk, by pointing
`and clicking on the machine with a camera-equipped mobile
`wireless device and then execution of communications or
`transactions between the mobile wireless device and the
`machine;
`identification of objects or parts in a factory, Such as on an
`assembly line, by capturing an image of the objects or parts,
`and then providing information pertinent to the identified
`objects or parts;
`identification of a part of a machine. Such as an aircraft part,
`by a technician pointing and clicking on the part with a
`camera-equipped mobile wireless device, and then Supplying
`pertinent content to the technician, such maintenance instruc
`tions or history for the identified part;
`identification or screening of individual(s) by a security
`officer pointing and clicking a camera-equipped mobile
`wireless device at the individual(s) and then receiving iden
`tification information pertinent to the individuals after the
`individuals have been identified by face recognition software:
`identification, Screening, or validation of documents, such
`as passports, by a security officer pointing and clicking a
`camera-equipped device at the document and receiving a
`response from a remote computer;
`determination of the position and orientation of an object in
`space by a spacecraft nearby the object, based on imagery of
`the object, so that the spacecraft can maneuver relative to the
`object or execute a rendezvous with the object;
`identification of objects from aircraft or spacecraft by cap
`turing imagery of the objects and t