`Case 2:20-cv-07872-MWF-PVC Document 1-2 Filed 08/27/20 Page 1 of 21 Page ID #:92
`
`
`
`
`
`
`
`
`
`
`EXHIBIT B
`
`EXHIBIT B
`
`
`
`Case 2:20-cv-07872-MWF-PVC Document 1-2 Filed 08/27/20 Page 2 of 21 Page ID #:93
`
`US007899252B2
`
`(12) United States Patent
`Boncyk et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,899,252 B2
`*Mar. 1, 2011
`
`(54)
`
`(75)
`
`(73)
`
`(*)
`
`(21)
`(22)
`(65)
`
`(60)
`
`(60)
`
`OBJECT INFORMATION DERVED FROM
`OBJECT IMAGES
`
`Inventors: Wayne C. Boncyk, Evergreen, CO (US);
`Ronald H. Cohen, Pasadena, CA (US)
`Assignee: Evryx Technologies, Inc., Los Angeles,
`CA (US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 43 days.
`This patent is Subject to a terminal dis
`claimer.
`
`Notice:
`
`Appl. No.: 12/568,130
`Filed:
`Sep. 28, 2009
`
`Prior Publication Data
`US 2010/0034468 A1
`Feb. 11, 2010
`
`Related U.S. Application Data
`Division of application No. 1 1/204,901, filed on Aug.
`15, 2005, now Pat. No. 7,680,324, which is a
`continuation-in-part of application No. 09/992,942,
`filed on Nov. 5, 2001, now Pat. No. 7,016,532.
`Provisional application No. 60/317,521, filed on Sep.
`5, 2001, provisional application No. 60/246.295, filed
`on Nov. 6, 2000, provisional application No.
`60/630,524, filed on Nov. 22, 2004, provisional
`application No. 60/625,526, filed on Nov. 4, 2004.
`
`(51)
`
`(52)
`(58)
`
`Int. C.
`(2006.01)
`G06K 9/00
`(2006.01)
`GO6K 9/54
`U.S. Cl. ......................................... 382/181: 382/305
`Field of Classification Search .......... 382/181-182,
`382/100, 305-306, 224, 170, 209, 218, 233;
`
`707/17, 104.1: 709/201-203, 217 219,
`250; 705/26–27, 23, 64; 348/239, 460,552,
`2112-211.6, 207.1; 340/989,988: 455/550.1,
`414.2, 456.3, 556.1, 412.1, 419,556.2:
`235/375–386, 462.09, 462.46
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`5,579,471 A 11/1996 Barber
`5,615,324 A
`3/1997 Kuboyama
`5,724,579 A
`3, 1998 Suzuki
`5,768,633. A
`6/1998 Allen et al.
`5,926, 116 A
`7/1999 Kitano et al.
`6,055,536 A
`4/2000 Shimakawa
`6,181,817 B1
`1/2001 Zabih
`6,286,036 B1
`9, 2001 Rhodes
`6,393,147 B2
`5, 2002 Danneels et al.
`6,522,889 B1
`2/2003 Aarnio
`6,674,993 B1
`1/2004 Tarbouriech
`6,993,573 B2 *
`1/2006 Hunter .......................... TO9.218
`7.305,354 B2 * 12/2007 Rodriguez et al. .............. 705/26
`7.309,015 B2 * 12/2007 Frantz et al. ............. 235,462.46
`(Continued)
`FOREIGN PATENT DOCUMENTS
`92O179
`9, 2000
`(Continued)
`Primary Examiner — Sherali Ishrat
`(74) Attorney, Agent, or Firm — Fish & Associates, PC
`(57)
`ABSTRACT
`Search terms are derived automatically from images captured
`by a camera equipped cell phone, PDA, or other image cap
`turing device, Submitted to a search engine to obtain infor
`mation of interest, and at least a portion of the resulting
`information is transmitted back locally to, or nearby, the
`device that captured the image.
`
`EP
`
`34 Claims, 8 Drawing Sheets
`
`
`
`NPUT MAGE
`
`0
`
`
`
`Case 2:20-cv-07872-MWF-PVC Document 1-2 Filed 08/27/20 Page 3 of 21 Page ID #:94
`
`US 7,899,252 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`7,410,099
`B2
`8, 2008 Fukasawa et al. ....... 235,462,09
`B2
`9, 2008 Hunter ...........
`TO9.218
`7,430,588
`7,558,595
`B2 * 7/2009 Angelhag
`455,550.1
`B2
`8/2010 Das et al. ........................ TO5/64
`7,774.283
`B2 12/2010 Beemer et al. ........... 235,462.01
`7,845,558
`20O2/OO.55957
`A1
`5, 2002 Ohsawa
`A1
`7, 2002 Ikeda
`2002fOO895.24
`A1
`8/2002 Lev et al.
`2002/0102966
`
`
`
`2002/0103813 A1
`2002fO140988 A1
`2002fO156866 A1
`2005/0162523 A1
`2008, 0021953 A1
`
`8, 2002
`10, 2002
`10, 2002
`7/2005
`1, 2008
`
`Frigon
`Cheatle
`Schneider
`Darrell et al.
`Gil
`
`FOREIGN PATENT DOCUMENTS
`WOO2O82799 A2 10, 2002
`WO
`* cited by examiner
`
`
`
`Case 2:20-cv-07872-MWF-PVC Document 1-2 Filed 08/27/20 Page 4 of 21 Page ID #:95
`
`U.S. Patent
`
`Mar. 1, 2011
`
`Sheet 1 of 8
`
`US 7,899,252 B2
`
`
`
`INPUT IMAGE
`CAPTURE
`
`1 0
`
`OBJECT
`MAGE
`
`
`
`
`
`SYMBOLIC
`MAGE
`
`
`
`
`
`
`
`
`
`NPUT IMAGE
`DECOMPOSITON
`
`2 6
`
`DATABASE
`MATCHING
`
`
`
`SELECT BEST
`MATCH
`
`F.G. 1
`
`4 0
`
`4 2.
`
`URL RETURN
`
`
`
`Case 2:20-cv-07872-MWF-PVC Document 1-2 Filed 08/27/20 Page 5 of 21 Page ID #:96
`
`U.S. Patent
`
`Mar. 1, 2011
`
`Sheet 2 of 8
`
`US 7,899,252 B2
`
`so
`
`
`
`
`
`Case 2:20-cv-07872-MWF-PVC Document 1-2 Filed 08/27/20 Page 6 of 21 Page ID #:97
`
`U.S. Patent
`
`Mar. 1, 2011
`
`Sheet 3 of 8
`
`US 7,899,252 B2
`
`
`
`
`
`FOREACH INPUT IMAGE
`SEGMENT GROUP
`
`FOREACH OBJECT IN
`DATABASE
`
`FOREACHSEGMENT
`GROUP N THIS VIEW
`
`
`
`
`
`GREYSCALE
`COMPARSON
`
`
`
`COLORCUBE
`COMPARSON
`
`
`
`SHAPE
`COMPARSON
`
`WAVELET
`COMPARSON
`
`FG. 3A
`
`
`
`Case 2:20-cv-07872-MWF-PVC Document 1-2 Filed 08/27/20 Page 7 of 21 Page ID #:98
`
`U.S. Patent
`
`Mar. 1, 2011
`
`Sheet 4 of 8
`
`US 7,899,252 B2
`
`CALCULATE COMBINED
`MATCH SCORE
`
`NEXT SEGMENT GROUPN
`THIS DATABASE VIEW
`
`NEXT VEW OF THIS
`DATABASE OBJECT
`
`NEXT OBJECT IN
`DATABASE
`
`NEXT INPUT IMAGE
`SEGMENT GROUP
`
`FIG 3B
`
`FINISH
`
`
`
`Case 2:20-cv-07872-MWF-PVC Document 1-2 Filed 08/27/20 Page 8 of 21 Page ID #:99
`
`U.S. Patent
`
`Mar. 1, 2011
`
`Sheet 5 of 8
`
`US 7,899,252 B2
`
`TARGET
`OBJECT
`
`CONTENT
`SERVER
`
`O2
`
`10 O
`
`1 11
`
`FIG. 4
`
`CAMERA
`
`1 0 3
`
`BROWSER
`
`MAGE
`PROCESSING
`
`104
`
`1 1 0
`
`TERMN AL
`
`
`
`MAGE DATA
`
`TARGET
`OBJECT
`NFORMATION
`
`1 OS
`
`1 09
`
`OBJECT
`DATABASE
`RENTION
`IDENTIFICATION SERVER
`
`
`
`Case 2:20-cv-07872-MWF-PVC Document 1-2 Filed 08/27/20 Page 9 of 21 Page ID #:100
`
`U.S. Patent
`
`Mar. 1, 2011
`
`Sheet 6 of 8
`
`US 7,899,252 B2
`
`TARGET
`OBJECT
`
`CONTENT
`SERVER
`
`2 O2
`
`2 O O
`
`2 11
`
`FIG 5
`
`CAMERA
`
`2 03
`
`BROWSER
`
`IMAGE
`PROCESSING
`
`2 04
`
`2 1 0
`
`
`
`TERMINAL
`
`MAGE DATA
`
`TARGET
`OBJECT
`INFORMATION
`
`\
`OBJE CT
`RECOGNITION
`
`\
`DATABASE
`
`IDENTIFICATION SERVER
`
`
`
`Case 2:20-cv-07872-MWF-PVC Document 1-2 Filed 08/27/20 Page 10 of 21 Page ID #:101
`
`U.S. Patent
`
`Mar. 1, 2011
`
`Sheet 7 of 8
`
`US 7,899,252 B2
`
`
`
`TARGET
`OBJECT
`
`SPACECRAFT
`DATA SYSTEM
`
`O 2
`
`3 O O
`
`31 O
`
`FIG. 6
`
`TERMINAL
`
`IMAGE DATA
`
`INFORMATION
`
`
`
`3
`
`3 O 5
`
`3 O 7
`
`
`
`
`
`OBJECT
`RECOGNITION
`
`DATABASE
`
`DENTIFICATION SERVER
`
`
`
`Case 2:20-cv-07872-MWF-PVC Document 1-2 Filed 08/27/20 Page 11 of 21 Page ID #:102
`
`U.S. Patent
`
`Mar. 1, 2011
`
`Sheet 8 of 8
`
`US 7,899,252 B2
`
`400
`
`425
`
`
`
`internet
`
`FIG. 7
`
`
`
`432
`
`Search
`Term
`
`421B
`
`434
`
`Results Set
`
`Search
`Engine
`
`430
`
`
`
`Case 2:20-cv-07872-MWF-PVC Document 1-2 Filed 08/27/20 Page 12 of 21 Page ID #:103
`
`US 7,899,252 B2
`
`1.
`OBJECT INFORMATION DERVED FROM
`OBJECT IMAGES
`
`This application is a divisional of 11/204,901 filed Aug. 15,
`2005 now U.S. Pat. No. 7,680,324, which is a continuation
`in-part of 09/992,942 filed Nov. 5, 2001 now U.S. Pat. No.
`7,061,532, which claims priority to provisional application
`No. 60/317,521 filed Sep. 5, 2001 and provisional application
`No. 60/246.295 filed Nov. 6, 2000; and also claims priority to
`provisional application No. 60/630,524 filed Nov. 22, 2004
`and provisional application No. 60/625,526 filed Nov. 4,
`2004, and utility application Ser. No. 10/492.243 filed May
`20, 2004 now U.S. Pat. No. 7,477,780, which claims priority
`to PCT/US02/35.407 filed Nov. 5, 2002, which claims priority
`to utility application Ser. No. 09/992,942 filed Nov. 5, 2001
`now U.S. Pat. No. 7,061,532. These and all other referenced
`patents and applications are incorporated herein by reference
`in their entirety. Where a definition or use of a term in a
`reference that is incorporated by reference is inconsistent or
`contrary to the definition of that term provided herein, the
`definition of that term provided herein is deemed to be con
`trolling.
`
`10
`
`15
`
`FIELD OF THE INVENTION
`
`The field of the invention is digital imaging.
`
`BACKGROUND
`
`Several years ago the present inventors pioneered the con
`cept of using digitally captured images to identify objects
`within the images, and then using such identifications to
`retrieve information from various databases. Examples
`include:
`Using a local device (cell phone, digital camera, PDA or
`other device) to capture an image of an object in an art
`museum, identifying the object from the image data, and
`then providing the user with information regarding the
`object (i.e., about or relating to the object);
`Using a local device (cell phone, digital camera, PDA or
`other device) to capture an image of an automobile as it
`drives along a road, identifying the make and model
`from the image data, and then providing a user with a
`link to a website relating to that particular make and
`model;
`Using a local device (cell phone, digital camera, PDA or
`other device) to capture an image of a bar code, logo, or
`other indicia in a magazine, using information contained
`in the indicia to identify a product, and providing a
`telephone number or other contact information relating
`to that product;
`Using a local device (cell phone, digital camera, PDA or
`other device) to photograph a billboard of a restaurant,
`identifying the restaurant from a barcode, special target,
`written language, or other information contained in the
`photograph, and using that information to access a data
`base to provide the user with restaurants location,
`menu, or telephone number, and
`Using a local device (cell phone, digital camera, PDA or
`other device) to capture an image of a sign at a sports
`stadium, using information extracted from the image to
`automatically purchase an entry ticket for the user, and
`providing the user with an entry code that can be used to
`bypass the long lines of ordinary ticket purchasers.
`In Such embodiments it was specifically contemplated that
`analysis of the images could be performed locally (i.e. on the
`cellphone, PDA or other device capturing the image), distally
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`at a server, or more preferably using some combination of the
`two. It was also contemplated that any available database
`could be accessed to provide the returned information,
`including publicly accessible databases on the Internet. It was
`not appreciated, however, that one could integrate these con
`cepts with the searching capabilities of standard Search
`Engines.
`In the 1990s Yahoo!TM introduced the idea of indexing web
`pages accessible on Internet, and providing a Search Engine
`that to access the index. Since that time dozens of other
`searching systems have been developed, which use all man
`ner of various search methods, algorithms, hardware and/or
`Software. All Such systems and methods that accept user
`inputs of Key Information, and then utilize such Key Infor
`mation to provide the user with information of interest, are
`referred to herein as Search Engines. The user, of course, can
`be a natural person, as well as a device (computing or other
`wise), algorithm, system, organization, or any other entity. In
`searching for information, a Search Engine can utilize any
`Suitable search domain, including for example:
`A database (including for example a relational database, an
`object database, or an XML database).
`A network of resources including for example web pages
`accessible within the Internet; and
`A public or private collection of documents or information
`(e.g., documents, information, and/or messages of a
`company or other organization(s)) such as that main
`tained by LEXISTM.
`In a typical search, Key Information is provided to the
`Search Engine in the form of key words comprising text,
`numbers, strings, or other machine-readable information
`types. The Search Engine then searches its indices of web
`pages for matches, and returns to the user a hyperlinked
`listing of Internet Uniform Resource Locators (“URLs”), as
`well as some brief display of context in which the key word(s)
`are used. The information of interest can sometimes be found
`in the hyperlinked listing, but is more frequently found by
`linking directly to the listed web pages.
`Providing Key Information to Search Engines in the form
`of text strings has inherent difficulties. It involves strategy in
`the selection of the text to be entered, and even with respect to
`the format of the keywords (for example using wildcards).
`Another difficulty is that Small computing and/or telephony
`devices (e.g. telephones, both mobile and non-mobile), have
`small and/or limited keyboards, thus making text entry diffi
`cult.
`
`SUMMARY OF THE INVENTION
`
`The present invention provides apparatus, Systems and
`methods in which: (a) a digital photograph, video, MPEG,
`AVI, or other image is captured using a camera equipped cell
`phone, PDA, or other image capturing device; (b) key words
`or other search criteria are automatically extracted or derived
`from image; (c) the search criteria are Submitted to a Search
`Engine to obtain information of interest; and (d) at least a
`portion of the resulting information is transmitted back
`locally to, or nearby, the device that captured the image.
`Some images so utilized will include symbolic content that
`is sufficient in and of itself to be relatively non-ambiguous.
`Such symbolic content, for example, can be a telephone num
`ber or a web-site address. In such instances the symbolic
`content search criteria can advantageously be utilized as a
`literal in the search criteria. In other instances significant
`additional processing can be needed. For example, an image
`ofan automobile will likely need to be processed to determine
`the make and model, and that information (e.g. MercedesTM
`
`
`
`Case 2:20-cv-07872-MWF-PVC Document 1-2 Filed 08/27/20 Page 13 of 21 Page ID #:104
`
`US 7,899,252 B2
`
`4
`the invention, along with the accompanying drawings in
`which like numerals represent like components.
`
`BRIEF DESCRIPTION OF THE DRAWING
`
`10
`
`15
`
`FIG. 1 is a schematic block diagram top-level algorithm
`flowchart;
`FIG. 2 is an idealized view of image capture;
`FIGS. 3A and 3B are a schematic block diagram of process
`details of the present invention;
`FIG. 4 is a schematic block diagram of a different expla
`nation of invention;
`FIG. 5 is a schematic block diagram similar to FIG. 4 for
`cellular telephone and personal data assistant (PDA) applica
`tions; and
`FIG. 6 is a schematic block diagram for spacecraft appli
`cations.
`FIG. 7 is a schematic of a system in which a local device
`captures and image, a search term is automatically derived
`from an image, is Submitted to a search engine to produce a
`results set, and information from the results set is sent back to
`the device.
`
`3
`S500TM) can then be transmitted to the Search Engine to be
`used as key words for a search. It is also contemplated that
`processing of Some images will result in only best guesses.
`Thus, a side view of an automobile can not be analyzable into
`a particular make and model, and in that case the system can
`provide more generic terms such as SUV or automobile.
`In general, the present invention provides technology and
`processes that can accommodate linking objects and images
`to information via a network such as the Internet, which
`require no modification to the linked object. Traditional meth
`ods for linking objects to digital information, including
`applying a barcode, radio or optical transceiver or transmitter,
`or some other means of identification to the object, or modi
`fying the image or object so as to encode detectable informa
`tion in it, are not required because the image or object can be
`identified solely by its visual appearance. The users or devices
`can even interact with objects by “linking to them. For
`example, a user can link to a vending machine by pointing
`and clicking on it. His device would be connected over the
`Internet to the company that owns the vending machine. The
`company would in turn establish a connection to the vending
`machine, and thus the user would have a communication
`channel established with the vending machine and could
`interact with it.
`The present invention contemplates any Suitable decompo
`sition algorithms. Clearly, faster and more accurate algo
`rithms are preferred over slower and less accurate algorithms.
`It is especially preferred that algorithms are chosen Such that
`at least some processing can take place locally to the device
`that captures the image. Such processing can in many
`instances eliminate the need to wirelessly transmit detailed
`images, and can eliminate reliance on a distal server that
`might be oversubscribed. Thus, some or all of the image
`processing, including image/object detection and/or decod
`ing of symbols detected in the image can be distributed arbi
`trarily between the mobile (client) device and the server. In
`other words, some processing can be performed in the client
`device and some in the server, without specification of which
`particular processing is performed in each, or all processing
`can be performed on one platform or the other, or the plat
`forms can be combined so that there is only one platform. The
`image processing can be implemented in a parallel computing
`manner, thus facilitating scaling of the system with respect to
`database size and input traffic loading.
`It is further contemplated that some suitable algorithms
`will take into account the position and orientation of an object
`with respect to the user at the time the image was captured,
`which can be determined based on the appearance of the
`object in an image. This can be the location and/or identity of
`people scanned by multiple cameras in a security system, a
`passive locator system more accurate than GPS or usable in
`areas where GPS signals cannot be received, the location of
`specific vehicles without requiring a transmission from the
`vehicle, and many other uses.
`Therefore, it is an object of the present invention to provide
`a system and process for identifying digitally captured
`images without requiring modification to the object.
`Another object is to use digital capture devices in ways
`never contemplated by their manufacturer.
`Another object is to allow identification of objects from
`partial views of the object.
`Another object is to provide communication means with
`operative devices without requiring a public connection
`therewith.
`Various other objects, features, aspects and advantages of
`the present invention will become more apparent from the
`following detailed description of preferred embodiments of
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`DETAILED DESCRIPTION
`
`FIGS. 1-6 are copied from the priority PCT application,
`PCT/US02/35.407 filed Nov. 5, 2002. Discussion of that those
`figures is set forth later in the application.
`Search Engine-Related Embodiments
`In FIG. 7 a system 400 generally comprises a portable
`imaging device 410, a distal server 420, an electronic com
`munications network 425, and a search engine 430.
`In general, the portable device 410 captures an image 412
`ofan object 415; and transmits information 413 regarding the
`image to the server 420. At least one of the device 410 and the
`server 420 derives a search term 421A, 421B from at least one
`of the image 412 and the transmitted information 413, respec
`tively. At least one of the device 410 and the server 420 cause
`the search term 421A, 421B to be submitted via a network 425
`to a search engine 430 that uses an index 432 of web pages or
`other information. The search engine then uses the search
`term 421A, 421B to produce a results set 434, and causes at
`least a portion of the results set 434 to be transmitted back to
`the portable device 410. In the above discussion it should be
`appreciated that information regarding the image can include
`the entire image, one or more Subsets of the image, as well as
`a name or other information derived from the image, but not
`contained within the image. It should also be appreciated that
`one could use a proxy server between his/her portable device
`and the server. In short, the present application contemplates
`using any complexity of circuitous communication between
`the mobile client and server—not necessarily a direct con
`nection.
`Device 410 can be a cell phone, PDA, laptop computer, or
`any other portable device that optically captures an image. By
`“optically captures” is meant Some sort of light sensitive
`array, the output of which can be processed to comprise a
`visually perceptible image. Viewed from another perspective,
`device 410 can be any camera having telephony capability,
`and especially having cell phone capability. With current
`technology, device 410 would usually have a lens or other
`light focusing mechanism, although it is contemplated that
`advances in electronics can eliminate the need for any physi
`cal focusing mechanism. The term "optically captures' is not
`satisfied by a device that has not optical components, and is
`merely capable of downloading images from the Internet or
`other sources.
`
`
`
`Case 2:20-cv-07872-MWF-PVC Document 1-2 Filed 08/27/20 Page 14 of 21 Page ID #:105
`
`US 7,899,252 B2
`
`5
`It is certainly contemplated that the cell phone or other
`device providing the services discussed herein would operate
`software permitting it to do so. That software could be resi
`dent on the device, in external memory (memory card), or
`paged in as needed.
`Object 415 (referred to as a Thing of Interest” in one or
`more of the priority applications) can be any visually percep
`tible object, regardless of dimension. Contemplated “two
`dimensional objects include objects in which the relevant
`information is Substantially in two dimensional format,
`which includes advertisements and articles in magazine or
`other print media, as well as photographs or designs on bill
`boards, street signs, restaurant or other business signs, user
`manuals, paintings at a museum, and so forth.
`Contemplated three dimensional objects include Substan
`tially all physical objects in which the relevant information is
`derived from the shape of the object and/or the appearance of
`the surface of the object. Thus, an automobile is considered
`herein to have three dimensions of relevance where the shape
`or other dimensions convey information about the make and
`model. Similarly, a window in a building can be considered to
`have three dimensions of relevance where the identity of the
`manufacturer or distributor can be gleaned from the overall
`physical dimensions, detail, and so forth. As another
`example, a beverage container can be considered to have three
`dimensions; information can be obtained from the shape of
`the container but further information can also be obtained
`from the label, printing, logos, text, or other Such visible
`markings on the container (obtaining information from vis
`ible markings on the container enables discrimination
`between different containers that have identical physical
`shape). Contemplated three dimensional objects include sub
`stantially all physical objects in which the relevant informa
`tion is derived from changes over time. For example, the
`speed of a bird or its flight patterns, or a gesture of a person,
`can be captured in multiple images over a period of time, and
`can be relevant information, and can be reduced to search
`terms (referred to as Key Information in one or more of the
`priority documents) for Submission to a search engine. Of
`course, many objects will be considered to have two, three or
`40
`four dimensions of relevance herein. Thus, relevant informa
`tion for an automobile can be provided by each of a two
`dimensional logo on the side of the vehicle, the three dimen
`sional shape of the vehicle, and its four dimensional
`acceleration or handling features.
`It is especially contemplated that objects can include ani
`mate and inanimate objects. Among animate objects are
`included faces of people, and biometric information Such as
`the fingerprint pattern on a human finger, an iris of a person,
`and so forth.
`Image 412 is contemplated to be any array of pixels. In
`most cases the pixels will be regularly arranged, but that is not
`absolutely necessary. In most cases the pixels also will num
`bergreater than 19,200 (160x120), such as 78,800 (320x240)
`but they can number few than that. More preferred images
`have greater pixel counts, including for example, 256,000
`(640x400), more preferably at least 2 million, and even more
`preferably at least 4 million. It is not necessary that the image
`be actually constructed at the portable device. Thus, a state
`ment that “the portable device captures an image of an object’
`includes situations where the device receives and derives data
`from light emitted or reflected from the object, even if the data
`is never presented to a user as a visually perceptible image,
`and even if the data is sent to a distal server without ever being
`collected into an image by the device.
`The information transmitted to the server can comprise any
`relevant information regarding the contents of the image.
`
`50
`
`6
`Thus, information 413 could comprise the entire image, or a
`portion of the image. For example, where a user takes a
`picture of a bar code (whether 2D, 3D or any other configu
`ration, the device 410 could process the image 412 to remove
`color and all background except the bar code itself, and then
`merely send the portion of the image containing the bar code
`as the transmitted information 413. In other cases it is con
`templated that the device 410 could sufficiently process the
`image 413 to derive one or more keywords, and then send
`only the keyword(s) as the transmitted information 413. All
`possible combinations are also contemplated. Thus, a user
`might take a photograph of a GucciTM handbag, the device
`412 might derive the word "Gucci” from the image, subtract
`out background except for the handbag, and then transmit: (a)
`the word "Gucci'; and (b) the image of the handbag as the
`transmitted information 413. In Such instances the process
`can be iterative. Thus, the device might initially transmit the
`word "Gucci” as the first transmitted information, receive a
`results set from the search engine indicating clothing acces
`sories, and then Subtract out background except for the hand
`bag, and transmit the image of the handbag as the second
`transmitted information. As discussed above, it is specifically
`contemplated that the device 410 could send the server 420
`numerical/digital data that is mathematically derived from the
`image. Examples include image features and characteristics
`that the server 420 could use in the server recognition process,
`without transmitting the original image.
`As should be apparent by now, the transmitted information
`need not be limited to image information. Sights, sounds, text,
`and all sorts of other information can be included in the
`transmitted information, some of which can be derived
`directly from the image, and some of which can be derived
`indirectly from the image. In addition, the device 410 can also
`capture non-visual information Such as sounds, and that infor
`mation can also be transmitted. Thus, it is contemplated that
`the device could capture the Sounds of a frog, capture an
`image of a lake or forest, and send both to be used as (or
`further analyzed into) search terms.
`Distal server 420 is distal in the sense that it has no hard
`wired link to device 410. Server 420 can be a single device, as
`well as any number of devices coupled together, as for
`example in a server farm. All manner of suitable servers are
`contemplated. Thus, servers can use any reasonable hard
`ware, operate using any reasonable Software, communica
`tions protocols, and so forth.
`In terms of interaction with the device, the various analyti
`cal tasks discussed above can allocated in any suitable man
`ner between server 420 and device 410. For example, in the
`iterative operation discussed above with respect to the
`GucciTM handbag, it is contemplated that the device 410 could
`analyze the image sufficiently to transmit the term "Gucci” as
`an initial search term to the search engine 430, and the server
`420 could then undertake the tasks of subtracting out back
`ground of the image except for the handbag, and transmitting
`the image of the handbag as a second search term.
`In another example, the server 420 could determine that the
`original image provided insufficient information, and send a
`message to the user through the device 410, directing the user
`to take another image (such as from another angle, closer, or
`with greater detail.). Indeed, the server 420 could direct the
`user to take an image of another object entirely, in order to
`help determine identity of the first object. Thus, the user could
`take a first image of a payment display at a ball game, provide
`that image to the server for identification, and then instruct the
`user to take an image of a credit card against which the user
`wants to be billed for entrance into the ball game. The server
`could then process the payment against that credit card, and
`
`10
`
`15
`
`25
`
`30
`
`35
`
`45
`
`55
`
`60
`
`65
`
`
`
`Case 2:20-cv-07872-MWF-PVC Document 1-2 Filed 08/27/20 Page 15 of 21 Page ID #:106
`
`US 7,899,252 B2
`
`25
`
`7
`provide an entry code that the user could type to pass through
`an electronically controlled gate.
`In still another example, a user could use his cellphone to
`capture an image of a screwdriverset at a hardware store, and
`the cell phone could transmit the information derived from
`the image to GoogleTM or some other search engine to find
`comparison prices. The server 420 could then instruct the user
`to turn over the packaging and take another image of the set,
`this time from the back side of the packaging. In this way
`there is iterative interaction among the user's device, the
`server, and the search engine.
`It should also be appreciated that there are embodiments in
`which the search engine never communicates with the por
`table device. For example, the server might do the search
`query, get results, and provide them to the portable device, or
`even to a television or other device besides the portable
`device.
`The phase “search engine' is contemplated herein to
`include any system dedicated to the indexing, searching and
`retrieval of information. The most familiar search engines
`such as GoogleTM, Yahoo!TM, MSNTM, and AltaVistaTM focus
`mostly or entirely on indexing web pages from the World
`WideWeb portion of the Internet. Other search engines, such
`as Lexis/NexisTM focus on indexing proprietary collections of
`data, which can include links to Internet Web pages. The
`phase "search term' is contemplated herein to include any
`keys or other information used by the search engines to access
`their indexing system. In the case of most web based search
`engines, the keys are currently text. In Such instances a user
`typically enters one or more key words, where the term "key
`word is used in an extremely broad sense to include: (a)
`words that would likely be found in a dictionary; (b) proper
`names, number strings and other terms that are not found in
`any dictionary; as well as (c) characters that are interpreted as
`wildcards, truncations and so forth. Such search engines are
`already starting to experiment with use of non-text keys,
`including for example images and/or sounds. All Such pos
`sible keys fall within the scope of contemplated search terms.
`Thus, contemplated search terms include key words, a
`portion of an image, as well as a logo, bar code or other
`symbol. It is specifically contemplated that in some instances
`an image will contain a literal of a search terms (e.g. the name
`of a movie on a movie poster), in some instances an image
`will not contain Such a literal (e.g. a picture of a tree or other
`plant, where the search term is the name of the plant). In either
`case the device and/or the server in any combination can
`perform one or more of the tasks of deriving the search term
`and Submitting it to one or more search engines.
`Network 425 can be any workable electronic network,
`including public and private access networks, and combina
`tions of the two. Preferred networks include the Internet, the
`upcoming Internet II, cell phone networks, and so forth.
`Although not expressly shown, the communication lines in
`FIG. 7 are all contemplated to be one- or two-way commu
`nications as appropriate. Moreover, it is contemplated that
`55
`multiple networks will usually be involved. Thus, for
`example, communications between device 410 and server
`420 will very likely take place over some combination of cell
`phone