llllllll's'n£060; 00/90/11llllll
`
`
`
`llllllllllllllll Old
`
`
`
`H’ag”©o
`
`fl/flrol/
`
`'W
`mm ~ D '
`‘ %\00
`
`RID fir.
`0‘0
`
`PTO/SB/16(8-00)
`Please lip; .31 31-5 Slgn (v; mslde lhls bcx ——-——; D
`Acvroveo for use lhlougll10’31l2002 OMB DEM-W32
`Underll-e Paper.-
`U S Pale-1: and Trademark Office; U.S DEPARTMENT OF COMMERCE
`'1'. Rs:.:!1:n Act cl 1995. no rLrscns are reamed lo respond lo a collec’mn m Intermellofl unless h‘ ulsolays a valor) OMB control number
`0
`n
`Pfifiifi
`irfii’fi i
`(1 fl
`v n :vun.
`. LiC/l TiOi‘.’ FC‘R PA TE: ‘7 COVER SHEET
`This is a request for filing a PROV'SiONAL APPLiCATiON FOR PATENT under 37 CFR 1.53416}.
`,
`,
`,,
`A
`11111594101215)
`T— Reacence
`(CW and ell‘ner Slate or Forelon COurRW
`
`'—
`
`iamuisvme, co 80027
`
`asadena, CA 91107
`
`
` irecl all correspondence to. CORRESPOH
`
`
`@ CustomerNumber
`24046
`___*,
`
`
`
`
`l] “’"W
`EGEORGE w. FINCH
`mm; W m:
`
`_§
`5I
`II
`
`. SA,
`
`515 e
`Te'enhone
`
`ENCLOSED APPL!CAT!0N PARTS check all ma: 3
`
`HS
`FlLiNG FEE
`CFR1 27
`
`AMOUNT (a)
`A check a! money orde' as enclosed lo cove! the lllmg lees
`The Comrnlsszoner I5 herebA
`1 $75 00
`ulhonzed in Charge fiimg
`|
`l
`l
`-
`fees or credit any overpayment to Deposzi Account Number
`|_,__l
`Payment by C(sdll card Form PTOQOEE :5 attached
`,I
`l
`The xmenllcn wens made by an agency a! he Unued States Governmenl or under a contracl wilh an agency or the
`Unlled Slates Government
`a No.
`Yes. in: name of the U S. Cmemmer: "gens, 2",: :ha Gave
`
`r—'~_—__'—l
`I
`l
`
`
`
`
`SlGNAT '.\5
`REGISTRATION NO
`25 1 13
`
`.
`.
`(.u’ap'rrsprzale)
`
`TYPED 0. PR.
`Dockgl Number
`1 BONC—OOO1
`. _
`..
`.
`....
`.
`.
`a...
`,_
`..
`....
`-—.—..
`TELEPHONE NEE—8234*
`I
`v
`USE ONLYFOR huf‘G A “DO":‘n'OM 'L "H ' V‘A H0 1 rC-R P law?
`an: by me PTO {a process) a
`This cultecllon of um‘ormatlon ls. requnreo‘ by 37 CFR 1 5
`The mformallon is used by the DubllC 10 file 1
`provisonal apphcalion Confidenllahly l5 governed by 35 U S C 1 2 and 37 CFR 1.14 frhls collecfilcn 1
`g estimgzed to lake 8 houm to
`me wm varv dependlng u on
`complete. mcludxng gathenng, prenanng‘ and submlmng the complete provrsrowal a pllcahon :0 me
`the lndeu’ual case. Any commenls on lne amomz ol llme you requlre to complete i. is form and/or 5
`ans for redhcmg this bur en.
`should be sent 10 lhe Chief lnformaton Officen U S, Pazenl and Trademark 010cc. U
`D
`
`, Weshmgtcn, D.C
`2023‘. DO NOT SEND FEES OR CO» DLJ’ED FORMS TO THlS ADDRESS SEND
`plrcallom Asslstanl
` {UCJI
`Commlsssonerr'orpetems. vashingmn,
`“""‘
`
`I
`
`
`
`BANK OF AMERICA
`BANK OF AMERICA
`BANK OF AMERICA
`
`IPR2021-01080
`IPR2021-01080
`IPR2021-01080
`
`Ex. 1008, p. 1 of 11
`Ex. 1008, p. 1 of 11
`Ex. 1008, p. 1 of 11
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1008, p. 1 of 11
`
`

`

`
`
`Attorney‘s Docket No. BONC—OOOl
`
`PATENT
`
`IN THE UNITED STATES PATENT AND TRADEMARK OFFICE
`
`Application of: BONCYK ET AL
`
`Application No. TBD
`
`Filing Date: 6 November 2000
`
`Title: TARGET IDENTIFICATION AND IMAGE-BASED INFORMATION
`LINKING
`
`Box PATENT APPLICATION
`Assistant Commissioner for Patents
`Washington, DC. 20231
`
`CERTIFICATE OF EXPRESS MAILING UNDER 37 CFR 1.10
`
`I hereby certify that the provisional patent application referred to as enclosed therein are
`being deposited with the United States Postal Service on 6 November 2000, in an envelope
`as "Express Mail Post Office to Addressee“ Mailing Label Number EF107195763US,
`addressed to the: Assistant Commissioner for Patents, Washington, DC. 20231.
`
`George W. Finch
`(type or print name of person mailing paper)
`1/ .
`
`
`
`
`
`Signatur-
`
`a son mailing paper
`
`
`
`
`
`
`BANK OF AMERICA
`BANK OF AMERICA
`BANK OF AMERICA
`
`IPR2021-01080
`IPR2021-01080
`IPR2021-01080
`
`Ex. 1008, p. 2 of 11
`Ex. 1008, p. 2 of 11
`Ex. 1008, p. 2 of 11
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1008, p. 2 of 11
`
`

`

`Provisional Patent Application
`for
`
`Target Identification
`and
`
`Image-Based Information Linking
`
`Wayne Boncyk
`Ronald H. Cohen
`
`October 30, 2000
`
`
`
`Background
`
`General
`
`A process is presented by which information, such as
`Internet content, can be presented to a User based on
`remotely linking to physical objects. No additional
`information is encoded in the linked object, nor is any
`device, radio or otherwise, embedded in or affixed to the
`object. The only modification to the object is the addition
`
`of a logo “Tag,” for objects linked via pure image linking,
`or a bar code, for objects linked via bar code.
`Image—
`
`linked objects are identified via digital image processing
`and the address of pertinent information is returned to the
`device performing the link. Bar—code linked objects are
`linked by reading the bar code, using the same sensor used
`for the image linking, and similarly returning the
`appropriate information address.
`
`Image Processing
`
`Numerous image processing techniques exist in the current
`art to perform specific “corrections” to raw input
`imagery.
`These include techniques for digital image rectification by
`resampling, as well as various color processing techniques
`designed to transform raw imagery into “corrected” images
`that appear normal to the human eye. However, machine
`image classification based upon the unique chroma content
`of an entire image has not been practiced successfully
`except in a narrow range of applications.
`It currently
`only exists in applications where either the image incident
`illumination is spectrally well known or where the subset
`of possible input targets is reasonably small and well—
`defined, allowing assumptions about
`the changes in target
`
`BANK OF AMERICA
`BANK OF AMERICA
`BANK OF AMERICA
`
`IPR2021-01080
`IPR2021-01080
`IPR2021-01080
`
`Ex. 1008, p. 3 of 11
`Ex. 1008, p. 3 of 11
`Ex. 1008, p. 3 of 11
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1008, p. 3 of 11
`
`

`

`-2-
`
`response to varying illumination intensities or spectral
`characteristics to be made as part of the process.
`Further,
`the current art is generally confined to either
`human vision—like processing designed to adjust input
`
`imagery to look more “pleasing" to the eye, or in feature
`extraction to be used to recognize specific objects in the
`input
`image space for machine target recognition.
`The
`presented method removes the first—order effect of
`illumination variability.
`The presented method further
`utilizes the spatial chroma content of an entire image to
`classify or ID unique targets from a database of desired
`target images.
`
`Definitions
`
`“Target”
`
`A “Target” is herein defined as an object or image which we
`desire to identify via machine algorithm, and in many cases
`for which we wish to subsequently access or provide related
`information to the person or machine which observes the
`target. A Target may a physical object, such as a building,
`a vehicle, or a consumer product. It may also be an image
`in printed or digital media, such as in a magazine, on a
`billboard, or on a computer screen.
`
`“Tag/I
`
`A “Tag” is herein defined as a symbol which serves as a
`
`reference for identifying the position and orientation of
`an image area of interest. The Tag is generally a small
`
`(relative to the Target)
`logo with features sufficient to
`uniquely define its position, orientiation, and chromatic
`response to incident illumination. The Tag is affixed,
`printed, or otherwise embedded in or on the target so as to
`be clearly visible.
`
`“User”
`
`A “User” is herein defined as a human being or machine
`process which initiates the process of obtaining
`information pertinent to the Target.
`
`Process
`
`The following is a summary of the process by which objects
`are linked to information.
`
`
`
`BANK OF AMERICA
`BANK OF AMERICA
`BANK OF AMERICA
`
`IPR2021-01080
`IPR2021-01080
`IPR2021-01080
`
`Ex. 1008, p. 4 of 11
`Ex. 1008, p. 4 of 11
`Ex. 1008, p. 4 of 11
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1008, p. 4 of 11
`
`

`

`l. PREPARATION
`
`1.1 A Tag is included in the Target, as follows. If the
`Target is a graphic image, such as one to appear in
`print media,
`then the Tag is included in the image. If
`Target is a physical object such a building,
`then a Tag
`is made visible on the target, by affixing a Tag to the
`target,
`in the form of a sign or adhesive sticker, or by
`building the Tag into the Target, or by any convenient
`means which result in the Tag being clearly visible.
`
`1.2 A Target Reference Image is prepared as follows. A
`digital image of the Target is first created in an
`aspect ratio which is common among other reference
`images. If the Target is inherently an image, such as an
`magazine advertisement or poster,
`then the Reference
`Image may be extracted directly from the Target if the
`Target exists in digital format. If the Target is a
`physical object such as an automobile or storefront,
`then a digital photograph is made of the Target and the
`Reference Image is extracted from the photograph in
`digital form. The Reference Image is prepared in such a
`manner so as to insure that the Tag appears in a
`specific location, orientation, and size relative to the
`overall Reference Image, and that the overall Reference
`Image has a standard aspect ratio in common with other
`Reference Images. Some References Images do not
`include
`Tags.
`In such cases,
`the Tag is visible in or on the
`Target object but it is adjacent to the Reference Image.
`
`1.3 The Reference Image is then subsampled to a relatively
`low resolution and the resulting data values for each
`pixel are stored in a computer system, hereafter
`referred to as the “Directory.”
`
`2.
`
`IMAGE CAPTURE
`
`2.1 The User utilizes a computer, mobile telephone,
`personal digital assistant, or other similar device,
`equipped with an image sensor
`(such as a CCD or CMOS
`digital camera). This device is hereafter termed the
`“Digital Device.”
`
`
`2.2 The User aligns the sensor of the Digital Device with
`the Tag on the Target of interest. The linking process
`is then initiated by the User pressing a button on the
`
`
`
`
`«newIn
`
`é
`
`BANK OF AMERICA
`BANK OF AMERICA
`BANK OF AMERICA
`
`IPR2021-01080
`IPR2021-01080
`IPR2021-01080
`
`Ex. 1008, p. 5 of 11
`Ex. 1008, p. 5 of 11
`Ex. 1008, p. 5 of 11
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1008, p. 5 of 11
`
`

`

`-4-
`
`Digital Device or sensor, or by the software in the
`Digital Device automatically recognizing the Tag or bar
`code at which it is aimed, or by User voice command, or
`by any other appropriate means.
`
`the Digital
`2.3 Via a combination of software and hardware,
`Device captures a digital image of the scene at which it
`is pointed. This image is represented as 3 separate 2—D
`matrices of pixels, corresponding to the raw RGB (Red,
`Green, Blue) representation of the input
`image.
`
`2.4 A discriminator algorithm operates on the pixel arrays
`to determine if the image contains a bar code. If the
`image is determined to contain a bar code,
`the process
`continues with step 3.1. If image is determined to not
`contain a bar code,
`the process continues with step 4.1.
`
`3. BAR CODE PROCESSING
`
`
`
`
`‘9!:—
`
`(This process is performed on images which are determined
`to contain images of bar codes.)
`
`3.1 An algorithm operates on the pixel arrays to locate the
`bar code within the image. The position and orientation
`of the bar code “subimage” are determined relative to
`the overall
`image.
`
`3.2 Using the position and orientation information from the
`previous step,
`the bar code “subimage” is cropped from
`the total image and spatially rectified via digital
`resampling. This step,
`in essence, rotates the image in
`3—space, resulting in an image in a plane perpendicular
`to the line—of—sight.
`
`3.3 The image is converted into a gray—scale contrast—
`enhanced image transforming the red, green and blue
`pixel arrays to a normalized total Intensity
`representationand performing a contrast stretch
`operation.
`
`3.4 The bar code number is extracted by applying a 2—
`dimensional moving—window filter algorithm to the image.
`
`4.
`
`IMAGE LINK PROCESSING
`
`(This process is performed on images which are determined
`to contain images of Tags, rather than bar codes.)
`
`BANK OF AMERICA
`BANK OF AMERICA
`BANK OF AMERICA
`
`IPR2021-01080
`IPR2021-01080
`IPR2021-01080
`
`Ex. 1008, p. 6 of 11
`Ex. 1008, p. 6 of 11
`Ex. 1008, p. 6 of 11
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1008, p. 6 of 11
`
`

`

`4.1 An algorithm operates on the pixel arrays to locate the
`Tag within the captured image. The position and
`orientation of the Tag, relative to the Digital Device,
`are determined by comparing the image of the Tag with an
`ideal orthogonal representation of the Tag.
`
`4.5 Using the position and orientation information, an
`algorithm spatially rectifies the image via digital
`resampling. This step,
`in essence, rotates the image in
`3—space, resulting in an image in a plane perpendicular
`to the line—of—sight.
`
`4.6 An algorithm determines the boundaries of the desired
`“area of interest” of the image, where this area of
`interest corresponds to the geometry of the Reference
`Image.
`In other words,
`the algorithm crops that section
`of the original image which matches the extent of the
`reference image. This extent is determined by using the
`scale and position (both linear and angular)
`information
`extracted from the image of the Tag. The cropped imaged
`is henceforth referred to as the “Area of Interest
`
`Image” or “AOI Image.” Note that the Tag may or may not
`lie within the AOI
`Image.
`In such cases,
`the Tag is is
`visible in or on the Target object but it is adjacent to
`the Reference Image.
`
`4.7 Chromatic referencing is performed to compensate for
`color temperature variations between the actual
`illumination of the Target at the time of linking and
`“standard” reference illumination. This is done as
`
`the ACT Image is spectrally classified.
`follows. First,
`In cases wherein the Tag is not inside the AOI
`Image,
`the Tag portion is also included though in this step.
`Next, a comparison of the apparent visual color response
`(RmGhIfn) of the Tag portion of the image to a standard
`normalized chroma (RW?B“V response of the Tag is
`determined; a normalized difference vector is determined
`thusly:
`
`(RdeBd)
`
`:
`
`(RinGinBin)
`
`_
`
`(RnGan)
`
`The normalized color difference vector is then used in a
`
`chroma filter function to operate on the ACT image to
`create a referenced color image.
`
`
`
`
`BANK OF AMERICA
`BANK OF AMERICA
`BANK OF AMERICA
`
`IPR2021-01080
`IPR2021-01080
`IPR2021-01080
`
`Ex. 1008, p. 7 of 11
`Ex. 1008, p. 7 of 11
`Ex. 1008, p. 7 of 11
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1008, p. 7 of 11
`
`

`

`—6—
`
`4.8 The referenced color image is then spatially averaged
`to create a reduced resolution 2—D image.
`The reduced
`
`image is transformed to a standard normalized Intensity,
`In—Phase, Quadrature Phase (YIQ)
`image.
`
`4.9 Then the I and Q components of each of the pixels are
`extracted from the normalized YIQ image and sent
`to the
`
`Directory (see below). If the directory resides on a
`remote computer system,
`then the I and Q data may be
`compressed so as to facilitate this transmission. Other
`ancillary information is also sent to the Directory,
`such as:
`
`OData identifying the type of Digital Device used for
`the linking.
`
`OData identifying organizations which may have
`sponsored, manufactured, or otherwise be connected
`with or have interest in the Digital Device used for
`the image linking.
`
`OData identifing the User.
`
`5. DIRECTORY LOOKUP
`
`in the case
`5.1 The Directory receives the bar code value,
`of images containing bar codes, or image (I,Q) data in
`the case of images containing Tags. The directory also
`receives the other ancillary information sent with the
`bar code or image.
`
`5.2 The Directory maintains a database of of bar codes,
`reference image (I,Q) data, and the ancillary
`information (such as device and sponsor identification).
`In addition,
`the Directory includes lists of addresses
`of information corresponding to each bar code or image
`entry. These information addresses are generally (but
`are not limited to) Internet Uniform Resource Locators
`
`(URLs). If the Directory receives a bar code, it looks
`up the input bar code and returns the corresponding
`information adddress. If the Direc:ory receives image
`data, it looks up the input
`image data and returns the
`information address corresponding :0 the closest match
`among the stored image data. If there is no match within
`a specified tolerance,
`then a message is returned
`indicating this situation, and the user is advised to
`try to re—image under better conditions (better
`lighting, closer range, etc.).
`
`
`
`
`
`
`BANK OF AMERICA
`BANK OF AMERICA
`BANK OF AMERICA
`
`IPR2021-01080
`IPR2021-01080
`IPR2021-01080
`
`Ex. 1008, p. 8 of 11
`Ex. 1008, p. 8 of 11
`Ex. 1008, p. 8 of 11
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1008, p. 8 of 11
`
`

`

`_7_
`
`The information address returned for a bar code or image
`
`may also be based on the ancillary information. For
`example, a specific bar code or reference image may
`correlate to different information addresses depending
`on the type of Digital Device performing the link,
`the
`User, and/or an organization sponsoring the link or
`Digital Device. This allows for customization of the
`resulting information based on sponsorship,
`type of
`device, and the particular user, etc.. This furthermore
`allows for sale or rent of links to particular objects,
`e.g., a company may rent the link to a particular music
`recording so as to be a primary sales channel for that
`recording. Several types of such links can be
`implemented, such as shopping, detailed information
`about the product, etc.
`(see REDIRECTION, below)
`
`the choice of reference image with which to
`Furthermore,
`compare the image data may be a function of the
`ancillary data. For the same link, different reference
`images may be used, for example,
`for different types of
`Digital Devices in order to acccomodate differences in
`image data from different Digital Devices.
`
`5.3 In order to minimize the time required for the lookup
`process,
`the lists of bar codes, reference image data,
`and information addresses may be stored in memory
`(Random Access Memory)
`to avoid hard disk access.
`
`6. REDIRECTION
`
`6.1 The software and/or hardware with which the User is
`interacting receives the information address from the
`Directory, and then proceeds to access this information
`via redirection to that address. This access may be via
`the Internet, radio frequency communication, optical
`communication, celluar network, or any other appropriate
`means of information transmission. The address is often
`
`in the form of an Internet URL, but any form of
`information address can be used.
`
`6.2 The supplier of the information sends this information
`to the User upon receiving this request. Typically this
`is done by via the Internet and World Wide Web.
`In this
`case, a request is sent to the information supplier as a
`request from a computer or mobile Digital Device, and
`
`
`
` o-fit‘i:
`
`BANK OF AMERICA
`BANK OF AMERICA
`BANK OF AMERICA
`
`IPR2021-01080
`IPR2021-01080
`IPR2021-01080
`
`Ex. 1008, p. 9 of 11
`Ex. 1008, p. 9 of 11
`Ex. 1008, p. 9 of 11
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1008, p. 9 of 11
`
`

`

`—8—
`
`the Information Supplier’s computer system then responds
`by transmitting information back to the user’s system,
`typically as HTML, CHTML, XML, WAP, etc., although any
`type of information can be sent. Because the information
`address (e.g. URL)
`isbased on a specific Target
`(e.g.
`magazine advertisement, poster, billboard, vehicle,
`building, etc.),
`the Information Supplier sends
`information appropriate to the specific Target with
`which the User is linking. For example,
`the Information
`Supplier may send information specifically appropriate
`to the location or building in which the User is
`currently located.
`
`6.3 Certain types of links are directed to intermediate
`information content. Links can result in the user being
`
`presented with a menu of information and actions
`appropriate to the link. For example, a link from a
`recording (compabt disc, videotape, DVD, etc.) can
`result in the user being presented with a menu with
`options for shopping,
`information on the item, Internet
`searching, etc. Such menus can be implemented as
`Internet content or directly in software on the user’s
`Digital Device.
`
`Notes
`
`l.Links to physical objects which may be viewed over a wide
`viewing geometry are accomplished via spatial
`rectification of the captured image. It is not necessary
`for a user, when wishing to link to an object,
`to be
`positioned directly in front of the object. The user’s
`device may view the object from an angle, and the user’s
`device can also be rotated relative to the object. These
`
`spatial errors are removed by the spatial rectification
`algorithm.
`
`2. Links to physical objects which may be viewed from
`
`different sides are accommodated by having multiple links
`on the same object, with the links being distributed
`around the object on different sides. For example, an
`automobile can be linked via links on the left side,
`
`right side, front, and rear of the vehicle. This is
`accomplished by applying Tags to these 4 sides of the
`vehicle and taking digital reference images (digital
`photographs) of each of the 4 four sides. When performing
`the link,
`the user aims his Digital Device at the
`
`
`
`
`BANK OF AMERICA
`BANK OF AMERICA
`BANK OF AMERICA
`
`IPR2021-01080
`IPR2021-01080
`IPR2021-01080
`
`Ex. 1008, p. 10 of 11
`Ex. 1008, p. 10 of 11
`Ex. 1008, p. 10 of 11
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1008, p. 10 of 11
`
`

`

`-9-
`
`vehicle, and the system extracts that target image which
`is most prominent from the user’s Viewpoint.
`
`
`
`BANK OF AMERICA
`BANK OF AMERICA
`BANK OF AMERICA
`
`IPR2021-01080
`IPR2021-01080
`IPR2021-01080
`
`Ex. 1008, p. 11 of 11
`Ex. 1008, p. 11 of 11
`Ex. 1008, p. 11 of 11
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1008, p. 11 of 11
`
`