W0 99/112947
`
`{’CT/US99/03688
`
`
`
`11/13
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 822 of 1115
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 822 of 1115
`
`

`

`WO 99542947
`
`PCT/U899/03688
`
`
`
`12/
`
`13
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 823 of 1115
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 823 of 1115
`
`

`

`WO 99/42947
`
`PCT/US99/03688
`
`
`
`13/
`
`13
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 824 of 1115
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 824 of 1115
`
`

`

`Electronic Acknowledgement Receipt
`
`10582700
`
`Confirmation Number:
`
`
`
`
`Title of Invention:
`
`Image Capture and Identification System and Process
`
`
`
`First Named Inventor/Applicant Name:
`
`Wayne C. Boncyk
`
`Customer Number:
`
`24392
`
`— martin Fessenmaler/Llndsey Ripley
`
`
`
`Filer Authorized By: Martin Fessenmaier
`
`Attorney Docket Number:
`
`101044.0001US14
`
`Filing Date:
`
`22-MAR-2011
`
`19:00:38
`Time Stamp:
`
`
`
`
`Application Type: Utility under 35 USC 111(a)
`
`Payment information:
`
`Document
`Number
`
`Document Descri
`
`tion
`
`p
`
`.
`.
`Information Disclosure Statement (IDS)
`Form (SBOB)
`
`IDS_2.pdf
`
`Pages
`Multi
`Part /.zip (ifappl.)
`
`FileSize(Bytes)/
`Message Digest
`612829
`
`225i460438ib3c4244l0791S3Scc8ead4658
`3029
`
`
`
`Information:
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 825 of 1115
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 825 of 1115
`
`

`

`.
`.
`Information Disclosure Statement (IDS)
`Form (SB08)
`
`611913
`
`|DS_3.pdf
`
`16afin9Fh§dF337nhn6r3d4df97fief§fa24
`77(d
`
`Information:
`
`Foreign Reference
`
`EP1012725A1.pdf
`
`1464938
`
`bd671tl4e13f0d7397663d19df8725d35927
`4f29
`
`Information:
`
`Foreign Reference
`
`EP1354260A2.pdf
`
`136691
`
`6a4dadf02c23a806a84982f35b7f1de727d7
`91132
`
`Warnings:
`Information:
`
`Warnings:
`Information:
`
`Information:
`
`Foreign Reference
`
`W00163487A1.pdf
`
`Foreign Reference
`
`W00171282A1.pdf
`
`Foreign Reference
`
`WOO2OS9716A2.pdf
`
`Foreign Reference
`
`W002073818A1.pdf
`
`542812
`
`c7bb3iac8451227829cbf258d0c777478b1
`3:608
`
`476737
`
`(61137917fb2823dSOSea434bZaa4d36cb7f
`48548
`
`3693227
`
`1efD99f7IEdZ75bbb§bg4UaSZSCfl143145(
`8:35
`
`1094933
`
`84fer069baa1baaOe47497ai I98570C7da I
`
`Information:
`
`Foreign Reference
`
`WO9942946A2.pdf
`
`453839
`
`ac9d13b71d351ib66b4200718bbb975$b
`076$
`
`
`
`
`
`Warnings:
`Information:
`i
`no
`
`Foreign Reference
`
`WO9942947A2.pdf
`
`850214
`
`35de63a68cc9533834654bc5b47eb78c15c
`iibi7
`
`Warnings:
`Information:
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 826 of 1115
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 826 of 1115
`
`

`

`
`
`Total Files Size (in bytes) 9938133
`
`This Acknowledgement Receipt evidences receipt on the noted date by the USPTO of the indicated documents,
`characterized by the applicant, and including page counts, where applicable. It serves as evidence of receipt similar to a
`Post Card, as described in MPEP 503.
`
`New Applications Under 35 U.S.C. 111
`lfa new application is being filed and the application includes the necessary components for a filing date (see 37 CFR
`1.53(b)—(d) and MPEP 506), a Filing Receipt (37 CFR 1.54) will be issued in due course and the date shown on this
`Acknowledgement Receipt will establish the filing date of the application.
`
`National Stage of an International Application under 35 U.S.C. 371
`Ifa timely submission to enter the national stage of an international application is compliant with the conditions of 35
`U.S.C. 371 and other applicable requirements a Form PCT/DO/EO/903 indicating acceptance of the application as a
`national stage submission under 35 U.S.C. 371 will be issued in addition to the Filing Receipt, in due course.
`
`New International Application Filed with the USPTO as a Receiving Office
`lfa new international application is being filed and the international application includes the necessary components for
`an international filing date (see PCT Article 11 and MPEP 1810), a Notification of the International Application Number
`and of the International Filing Date (Form PCT/R0/105) will be issued in due course, subject to prescriptions concerning
`national security, and the date shown on this Acknowledgement Receipt will establish the international filing date of
`the application.
`
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 827 of 1115
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 827 of 1115
`
`

`

`UVTTET‘I STATES DEPARTVIF‘N'I‘ 0F COM'MFIRCFI
`United States Patent and Trademark Office
`Addless. COMB/ITSSTOVFR FOR PATENTS
`PO Box 1450
`Alexandriay‘hignia 22313-1450
`www.uspto.gov
`
`
`
` APPLICATION NUMBER FILING OR 371(C) DATE FIRST NANTED APPLICANT ATTY. DOCKET NOJTITLE
`13/069,124
`03/22/2011
`Wayne C. Boncyk
`101044.0001US 14
`CONFIRMATION NO. 9532
`PUBLICATION NOTICE
`
`
`
`IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
`00000004997
`98
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`
`
`24392
`FISH & ASSOCIATES, PC
`ROBERT D. FISH
`2603 Main Street
`Suite 1000
`Irvine, CA 92614-6232
`
`Title:lmage Capture and Identification System and Process
`Publication No.US-2011-0228126-A1
`Publication Date:09/22/2011
`
`NOTICE OF PUBLICATION OF APPLICATION
`
`The above—identified application will be electronically published as a patent application publication pursuant to 37
`CFR 1.211, et seq. The patent application publication number and publication date are set forth above.
`
`The publication may be accessed through the USPTO‘s publically available Searchable Databases via the
`Internet at www.uspto.gov. The direct link to access the publication is currently http://www.uspto.gov/patft/.
`
`The publication process established by the Office does not provide for mailing a copy of the publication to
`applicant. A copy of the publication may be obtained from the Office upon payment of the appropriate fee set forth
`in 37 CFR 1.19(a)(1). Orders for copies of patent application publications are handled by the USPTO‘s Office of
`Public Records. The Office of Public Records can be reached by telephone at (703) 308-9726 or (800) 972-6382,
`by facsimile at (703) 305-8759, by mail addressed to the United States Patent and Trademark Office, Office of
`Public Records, Alexandria, VA 22313-1450 or via the Internet.
`
`In addition, information on the status of the application, including the mailing date of Office actions and the
`dates of receipt of correspondence filed in the Office, may also be accessed via the Internet through the Patent
`Electronic Business Center at www.uspto.gov using the public side of the Patent Application Information and
`Retrieval (PAIR) system. The direct link to access this status information is currently http://pair.uspto.gov/. Prior to
`publication, such status information is confidential and may only be obtained by applicant using the private side of
`PAIR.
`
`Further assistance in electronically accessing the publication, or about PAIR, is available by calling the Patent
`Electronic Business Center at 1-866-217-9197.
`
`
`
`Office of Data Managmcnt, Application Assistance Unit (571) 272-4000, or (571) 272-4200, or 1-888-786-0101
`
`page 1 of 1
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 828 of 1115
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 828 of 1115
`
`

`

` 13069124
`Application Number
`
`201 1-03-22
`
`
`Filing Date
`First Named Inventor Wayne C. Boncyk
`
`
`
`
`to be assigned Examiner Name
`
`
` Attorney Docket Number
`101044.0001US14
`
`STATEMENT BY APPLICANT
`
`(Not for submission under 37 CFR1.99)
`
`
`
`PTO/saioaa (01-10)
`Doc code' IDS
`Approved for use through 071311'2012. OMB 0651-0031
`'
`US. Patent and Trademalk Office; US. DEPARTMENT OF COMMERCE
`Doc description: Information Disclosure Statement (IDS) Filed
`Under the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it contains a valid OMB control number.
`
`INFORMATION DISCLOSURE
`
`
`
`Gutkowicz-Krusin et al.
`
`
` Figures Appear
`
`
`
`
`
`U.S.PATENTS
`
`Remove
`
`Pages,Cqumns,Lines where
`.
`.
`.
`Name Of Patentee or Applicant Relevant Passages or Relevant
`Issue Date
`Patent Number
`E’Ft‘mjner Cite
`of Cited Document
`No
`Initial
`Figures Appear
`
`1
`
`2
`
`6081612
`
`2000-06-27
`
`Gutkowicz-Krusin et al.
`
`6208749
`
`2001 —03—27
`
`Gutkowicz—Krusin et al.
`
`
`
`3
`
`4
`
`6307957
`
`2001-10-23
`
`7127094
`
`2006—10—24
`
`Elbaum et al.
`
`If you wish to add additional U.S. Patent citation information please click the Add button.
`
`U.S.PATENT APPLICATION PUBLICATIONS
`
`Add
`
`Remove
`
`Examiner
`.
`.
`,,
`Initial
`
`.
`Cite No
`
`Publication
`Number
`
`Publication
`Date
`
`Name of Patentee or Applicant Pages,Columns,Lines where
`.
`Relevant Passages or Relevant
`of Cited Document
`
`If you wish to add additional U.S. Published Application citation information please click the Add button. Add
`
`FOREIGN PATENT DOCUMENTS Remove
`
`Pages,Columns,Lines
`Name of Patentee or
`where Relevant
`A licant of cited
`Publication
`Kind
`Country
`Examiner Cite Foreign Document
`Passages or Relevant
`pp
`Code4 Date
`Code2 i
`lnitial*
`No
`Number3
`.
`Document
`Figures Appear
`
`T5
`
`EFS Web 2.1.17
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 829 of 1115
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 829 of 1115
`
`

`

`
`
`1
`
`98/3781 1
`
`2
`
`3
`
`99/44010
`
`02/01143
`
`1998—09—03
`
`Electro—Optical Sciences,
`Inc.
`
`1999-09-02
`
`Gutkowicz—Krusin et al.
`
`2002—01—03
`
`Electra—Optical Sciences,
`Inc.
`
`
`
`
`
`4
`
`2264669
`
`2010-12-22
`
`MELA Sciences, Inc.
`
`No
`
`
`
`
`
`
`
` Application Number
`13069124
` Filing Date
`
`2011-03-22
`First Named Inventor Wayne C. Boncyk
`
`
`
`
`
`INFORMATION DISCLOSURE
`
`STATEMENT BY APPLICANT
`
`(Not for submission under 37 CFR1.99)
`
`Art Unit
`
`Examiner Name
`I to be assigned
`
`101044.0001US14
`Attorney Docket Number
`
`
`
`
`
`
`If you wish to add additional Foreign Patent Document citation information please click the Add button
`Add
`NON-PATENT LITERATURE DOCUMENTS
`Remove
`
`
`Examiner Cite
`Initials"
`
`Include name of the author (in CAPITAL LETTERS), title of the article (when appropriate), title of the item
`(book, magazine, journal, serial, symposium, catalog, etc), date, pages(s), volume-issue number(s),
`publisher, city and/or country where published.
`
`T5
`
`
`
`If you wish to add additional non-patent literature document citation information please click the Add button Add
`EXAMINER SIGNATURE
`
`*EXAMINER: Initial if reference considered, whether or not citation is in conformance with MPEP 609. Draw line through a
`
`citation if not in conformance and not considered. Include copy of this form with next communication to applicant.
`
`2 Enter office that Issued the document, by the two—letter code (WIPO
`1 See Kind Codes of USPTO Patent Documents at www.USPTO.GOV or MPEP 901.04.
`Standard ST.3). 3 For Japanese patent documents, the Indication of the year of the reign of the Emperor must precede the serial number of the patent document.
`4 Kind ofdocument by the appropriate symbols as indicated on the document under WIPO Standard ST.16 if possible.
`5 Applicant is to place a check mark here If
`English language translation is attached.
`
`EFS Web 2.1.17
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 830 of 1115
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 830 of 1115
`
`

`

`Please see 37 CFR 1.97 and 1.98 to make the appropriate selection(s):
`
`CERTIFICATION STATEMENT
`
`That each item of information contained in the information disclosure statement was first cited in any communication
`D from a foreign patent office in a counterpart foreign application not more than three months prior to the filing of the
`information disclosure statement. See 37 CFR 1.97(e)(1).
`
`OR
`
`That no item of information contained in the information disclosure statement was cited in a communication from a
`
`foreign patent office in a counterpart foreign application, and, to the knowledge of the person signing the certification
`
`after making reasonable inquiry, no item of information contained in the information disclosure statement was known to
`
`X any individual designated in 37 CFR 1.56(c) more than three months prior to the filing of the information disclosure
`
`
`statement. See 37 CFR 1.97(e)(2).
`
`
`
`
`
`
`
`
`See attached certification statement.
`
`The fee set forth in 37 CFR 1.17 (p) has been submitted herewith.
`
`SIGNATURE
`
`X A certification statement is not submitted herewith.
`
`A signature of the applicant or representative is required in accordance with CFR 1.33, 10.18. Please see CFR 1.4(d) for the
`form of the signature.
`
`Signature
`
`2011—11—04
`
`
`[Nicholas J. Witcheyl
`Date (YYYY-MM—DD)
`63481
`Registration Number
`Nicholas J. Witchey
`Name/Print
`
`
`
`
`
` 13069124
`
`Application Number
`201 1-03-22
`
` Filing Date
`First Named Inventor
`Wayne C. Boncyk
`
`
`Art Unit
`
`
`
`Examiner Name
`I to be assigned
`
`
`101044.0001US14
`Attorney Docket Number
`
`INFORMATION DISCLOSURE
`
`STATEMENT BY APPLICANT
`
`(Not for submission under 37 CFR1.99)
`
`This collection of information is required by 37 CFR 1.97 and 1.98. The information is required to obtain or retain a benefit by the
`public which is to file (and by the USPTO to process) an application. Confidentiality is governed by 35 U.S.C. 122 and 37 CFR
`1.14. This collection is estimated to take 1 hour to complete, including gathering, preparing and submitting the completed
`application form to the USPTO. Time will vary depending upon the individual case. Any comments on the amount of time you
`require to complete this form and/or suggestions for reducing this burden, should be sent to the Chief Information Officer, U.S.
`Patent and Trademark Office, US. Department of Commerce, PO. Box 1450, Alexandria, VA 22313-1450. DO NOT SEND
`FEES OR COMPLETED FORMS TO THIS ADDRESS. SEND TO: Commissioner for Patents, P.O. Box 1450, Alexandria,
`VA 22313-1450.
`
`
`EFS Web 2.1.17
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 831 of 1115
`
`
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 831 of 1115
`
`

`

`Privacy Act Statement
`
`The Privacy Act of 1974 (P.L. 93-579) requires that you be given certain information in connection with your submission of the
`attached form related to a patent application or patent. Accordingly, pursuant to the requirements of the Act, please be advised
`that:
`(1) the general authority for the collection of this information is 35 U.S.C. 2(b)(2); (2) furnishing of the information solicited
`is voluntary; and (3) the principal purpose for which the information is used by the U.S. Patent and Trademark Office is to
`process and/or examine your submission related to a patent application or patent.
`If you do not furnish the requested
`information, the U.S. Patent and Trademark Office may not be able to process andlor examine your submission, which may
`result in termination of proceedings or abandonment of the application or expiration of the patent.
`
`The information provided by you in this form will be subject to the following routine uses:
`
`1.
`
`The information on this form will be treated confidentially to the extent allowed under the Freedom of Information Act
`(5 U.S.C. 552) and the Privacy Act (5 U.S.C. 552a). Records from this system of records may be disclosed to the
`Department of Justice to determine whether the Freedom of Information Act requires disclosure of these record s.
`
`A record from this system of records may be disclosed, as a routine use, in the course of presenting evidence to a
`court, magistrate, or administrative tribunal, including disclosures to opposing counsel in the course of settlement
`negotiations.
`
`A record in this system of records may be disclosed, as a routine use, to a Member of Congress submitting a
`request involving an individual, to whom the record pertains, when the individual has requested assistance from the
`Member with respect to the subject matter of the record.
`
`A record in this system of records may be disclosed, as a routine use, to a contractor of the Agency having need for
`the information in order to perform a contract. Recipients of information shall be required to comply with the
`requirements of the Privacy Act of 1974, as amended, pursuant to 5 U.S.C. 552a(m).
`
`A record related to an International Application filed under the Patent Cooperation Treaty in this system of records
`may be disclosed, as a routine use, to the International Bureau of the World Intellectual Property Organization, pursuant
`to the Patent Cooperation Treaty.
`
`A record in this system of records may be disclosed, as a routine use, to another federal agency for purposes of
`National Security review (35 U.S.C. 181) and for review pursuant to the Atomic Energy Act (42 U.S.C. 218(c)).
`
`A record from this system of records may be disclosed, as a routine use, to the Administrator, General Services, or
`his/her designee, during an inspection of records conducted by GSA as part of that agency's responsibility to
`recommend improvements in records management practices and programs, under authority of 44 U.S.C. 2904 and
`2906. Such disclosure shall be made in accordance with the GSA regulations governing inspection of records for this
`purpose, and any other relevant (i.e., GSA or Commerce) directive. Such disclosure shall not be used to make
`determinations about individuals.
`
`A record from this system of records may be disclosed, as a routine use, to the public after either publication of
`the application pursuant to 35 U.S.C. 122(b) or issuance of a patent pursuant to 35 U.S.C. 151. Further, a record
`may be disclosed, subject to the limitations of 37 CFR 1.14, as a routine use, to the public if the record was filed in
`an application which became abandoned or in which the proceedings were terminated and which application is
`referenced by either a published application, an application open to public inspections or an issued patent.
`
`A record from this system of records may be disclosed, as a routine use, to a Federal, State, or local law
`enforcement agency, if the USPTO becomes aware of a violation or potential violation of law or regulation.
`
`
`
`
`
`EFS Web 2.1.17
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 832 of 1115
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 832 of 1115
`
`

`

`(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT)
`
`(19) World Intellectual Property Organization
`International Bureau
`
`(43) International Publication Date
`3 January 2002 (03.01.2002)
`
` (10) International Publication Number
`
`WO 02/01143 A2
`
`(51) International Patent Classificationl:
`
`GOIB
`
`(74) Agent: SKLAR, Brandon, N.; Lyon & Lyon LLP, Suite
`4700, 633 West Fifth Street, Ins Angeles, CA 90071 (US).
`
`(21) lntcrnationalApplication Number:
`
`PC’I‘IUS01/20524
`
`(22) International Filing Date:
`
`27 June 2001 (27.06.2001)
`
`(25) Filing Language:
`
`(26) Publication Language:
`
`English
`
`English
`
`(81) Designated States (national): AE, AG, AL, AM, AT, AU,
`AZ, BA, BB, BG, BR, BY, BZ, CA, CH, CN, CO, CR, CU,
`CZ, DE. DK, DM, DZ, EE, ES, FI, GB, GD, GE, GH, GM,
`HR, HU, ID, IL, IN, IS, JP, KE, KG, KP, KR, KZ, LC, LK,
`LR, LS, LT, LII, LV, MA, MD, MG, MK, MN, MW, MX,
`MZ, NO, NZ, PL, PT, RO, RU, SD, SE, SG, SI, SK, SL,
`T], TM, TR, TT, TZ, UA, UG, UZ, VN, YU, ZA, ZW.
`
`(30) Priority Data:
`09/604,645
`
`27 June 2000 (27.06.2000)
`
`US
`
`(71) Applicant: ELECTRO-OPTICAL SCIENCES, INC.
`[US/US]; 1 Bridge Street, lrvington, NY 10533 (US).
`
`(72) Inventors: GUTKOVVICZ-KRUSIN, Dina; 229 Shady,
`brook Lane, Princeton, NJ 08540 (US). ELBAUM,
`Marek; 79 Beechdale Road, Dobbs Ferry, NY 10522
`(US). GREENEBAUM, Michael; 1177 East 19th Street,
`Brooklyn, NY 11230 (US). JACOBS, Adam; 212 Bald—
`win Street, Glen Ridge, NJ 07028 (US). BOGDAN,
`Alexandru; 502 West 113th Street, New York, NY 10025
`(US).
`
`(84) Designated States (regional): ARIPO patent (GH, GM,
`KE, LS, MW, MZ, SD, SL, SZ, TZ, UG, ZW), Eurasian
`patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TIVI), European
`patent (AT, BE, CH. CY, DE, DK, ES, FI, FR, GB, GR, IE,
`IT, LU, MC, NL, PT, SE, TR), OAPI patent (BF, BJ, CF,
`CG. CI, CM, GA, GN, GW, ML, MR. NE, SN, TD, TG).
`
`Published:
`without international search report and to be republished
`upon receipt of that report
`
`For two-letter codes and other abbreviations, refer to the " Juid-
`ance Notes on Codes andAbbreviations " appearing at the begin-
`ning ofeach regular issue ofthe PCT Gazette.
`
`(54) Title: MULTISPECTR AL IMAGING AND CHARACTERIZATION OF INTERNAL BIOLOGICAL TISSUE
`
`25"
`
`26'
`
`(57) Abstract: A light image is conveyed from
`internal biological tissue through a flexible op—
`tical system to an image receiver, where it is
`converted to a form which may be entered into
`a computer. The computer segments the im—
`age by generating a segmentation mask defin—
`ing the boundary of a region of interest in at
`least one spectral band, estimates at least one
`rotationally and translationally invariant statise
`tical measure of coefficient distributions of the
`niultiscale wavelet niaxinla representations of
`the digital images in at least one spectral band,
`characterizes the condition of the tissue based
`on the estimated values and outputs the charace
`tcrization of the condition of the tissue.
`
`
`
`22
`
`WO02/01143A2
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 833 of 1115
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 833 of 1115
`
`

`

`WO 02/01143
`
`PCT/USOI/20524
`
`-1-
`
`MULTISPECTRAL IMAGING AND CHARACTERIZATION
`OF INTERNAL BIOLOGICAL TISSUE
`
`5
`
`FIELD OF THE INVENTION
`
`The field of the invention is automatic machine vision and classification, and, in
`
`particular, the classification of biological tissue based on images of the biological tissue.
`
`BACKGROUND OF THE INVENTION
`
`10
`
`U.S. Patent No. 6208749 B1,
`
`issued March 27, 200], discloses systems and
`
`methods
`
`for
`
`the multispectral
`
`imaging of
`
`skin tissue enabling the
`
`automatic
`
`characterization of the condition of a region of interest of skin based on direct digital
`
`imaging of the region of interest or the digitization of color photographic slides of the
`
`region of interest, illuminated by appropriately filtered light. According to that patent, a
`
`15
`
`digital image is automatically segmented in at least one spectral band. A digital processor
`
`segments the other images based on the segmentation mask. Parameters related to the
`texture, asymmetry, blotchiness and border irregularities are also automatically estimated.
`
`The region of interest is automatically characterized by the processor, based on the
`
`parameters. Skin lesions may be characterized as malignant or benign.
`
`20
`
`U.S. Patent No. 6,081,612, issued June 27, 2000, discloses systems and methods
`
`for characterizing the condition of a region of interest of the skin, wherein the absorption
`
`and scattering of light in different spectral bands by the region of interest is a function of
`
`the condition of the skin. The method comprises illuminating the region of interest of the
`
`skin by light in at least three spectral bands and digitally imaging the region of interest at
`
`25
`
`the at least three spectral bands with the light rc-cmitted by the skin to generate digital
`
`images comprising digital signals whose values are a function of the condition of the skin.
`
`The digital images are provided to a processor which segments the digital images by
`
`generating a segmentation mask from a digital image in any one of the at least three
`
`spectral bands.
`
`The processor estimates at
`
`least one rotationally and translationally
`
`3o
`
`invariant statistical measure of coefficient distributions of the multiscale Wavelet maxima
`
`representations of the digital images in each Spectral band, which are functions of the
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 834 of 1115
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 834 of 1115
`
`

`

`WO 02/01143
`
`PCT/USOI/20524
`
`-2-
`
`texture of the region of interest determined by the segmentation mask. The processor
`
`characterizes the condition of the skin based on the estimated values, and outputs the
`
`characterization of the condition of the skin. Preferably, the segmenting, estimating and
`
`characterizing steps are conducted without the intervention of an operator. Additional
`
`5
`
`parameters include measures of the texture, asymmetry, blotchiness and border irregularity
`
`of the portion of the region of interest. The digital images may be obtained by directly
`
`imaging the region of interest with a digital camera, or digitally imaging color slides of the
`
`region of interest, through appropriately filtered light.
`
`Not included in the above references are references to conducting light from place
`
`10
`
`to place so that the image is conserved.
`
`OBJECTS OF THE INVENTION
`
`It is an object of the invention to illuminate and image biological tissue, principally
`
`in vivo, and to convey the image in the fonn of light to an image receiver for conversion to
`
`p 15
`
`electrical signals and for automatic recognition of biological features.
`
`SUMMARY OF THE INVENTION
`
`A light image is conveyed from internal biological tissue through flexible optical
`
`system to an image receiver, Where it is converted to a form that may be entered into a
`
`20
`
`computer. The computer segments the image by generating a segmentation mask defining
`
`the boundary of a region of interest in at least one spectral band, estimates at least one
`
`rotationally and translationally invariant statistical measure of coefficient distributions of
`
`the multiscale wavelet maxima representations of the digital images in at least one spectral
`
`band, characterizes the condition of the tissue based on the estimated values, and outputs
`
`2 5
`
`the characterization of the condition of the tissue.
`
`Internal biological tissue encompasses tissue internal to a human or animal body,
`
`such as the stomach, the esophagus, the colon or the nasal cavities, for example.
`
`In particular, in accordance with the present invention, a method of characterizing
`
`the condition of a region of interest of internal biological tissue, wherein the absorption
`
`3 o
`
`and scattering of light in different spectral bands by the region of interest is a function of
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 835 of 1115
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 835 of 1115
`
`

`

`WO 02/01143
`
`PCT/USOI/20524
`
`-3-
`
`the condition of the tissue, is disclosed. The method comprises inserting a flexible optical
`
`system into a body, proximate the region of interest. A portion of the tissue is illuminated
`
`including the region of interest by light in at least one spectral band. The image is
`
`conveyed through the flexible optical system to an image receiver. A portion of the tissue
`
`5
`
`including the region of interest is digitally imaged at the at least one spectral band with the
`
`light re—emitted by the portion of the tissue. Digital images are generated comprising
`
`digital signals Whose values are a function of the condition of the region of interest of the
`
`tissue. The digital images are provided to a processor. The processor segments the digital
`
`images by generating a segmentation mask defining the boundary of the region of interest
`
`10
`
`from a digital image in any one of the at least one spectral bands, preferably without the
`
`intervention of an operator. The processor also estimates at least one rotationally and
`
`translationally invariant statistical measure of coefficient distributions of the multiscale
`
`wavelet maxima representations of the digital
`
`images in each spectral band and
`
`characterizes the condition of the tissue based on the estimated values, preferably without
`
`15
`
`the intervention of an operator. The processor then outputs the characterization of the
`
`condition of the tissue.
`
`The at least one statistical measure may be calculated separately within a border
`
`region or an interior region of the digital
`
`image. The border region is such that it
`
`encompasses the envelope of circles of fixed radius centered on the boundary of the
`
`2 o
`
`segmentation mask. The inside region comprises all points of the image that are Within the
`
`segmentation mask boundary but not included in the border region.
`
`The computing step may comprise estimating at an individual level at least one
`
`value which is a statistical measure of texture of the portion of the region of interest within
`
`the border region and interior region. The statistical measure is chosen from the group
`
`25
`
`consisting of:
`
`the number of wavelet maxima per unit area;
`
`the ratio of the mean coefficient magnitude to the absolute deviation of the
`
`coefficient magm'tudes from the mean value;
`
`the ratio of the mean coefficient magnitude to the standard deviation of the
`
`3 o
`
`coefficient magnitude; and
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 836 of 1115
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 836 of 1115
`
`

`

`WO 02/01143
`
`PCT/USOI/20524
`
`-4-
`
`the skewness of the coefficient magnitude, normalized to the cube of the standard
`
`deviation.
`
`The estimating step may comprise estimating either the degree of change of a
`
`statistic of the wavelet coefficient distribution with increment of wavelet level, or the
`
`5
`
`degree of deviation of such change from linearity.
`
`The estimating step may also comprise estimating the average rate of change, with
`
`respect to level, of the number of wavelet maxima per unit area.
`
`The estimated texture values may be compared to a threshold derived from
`
`statistical analysis of a multiscale wavelet
`
`transformation of the digital
`
`image,
`
`to
`
`1 o
`
`characterize the condition of the tissue.
`
`In accordance with another embodiment of the invention,
`
`a
`
`system for
`
`characterizing the condition of a region of interest of internal biological tissue is disclosed,
`
`comprising a source of light for illuminating the region of interest in at least one spectral.
`
`band. A flexible optical system is provided for conveying the image of the region of
`
`15
`
`interest to an image receiver. An image receiver is provided for acquiring digital images
`
`of the region of interest based on the light re—emitted from the illuminated region of
`
`interest at each of the spectral bands. The digital image comprises digital signals whose
`
`values are a fimction of the condition of the region of interest. Memory is provided for
`
`storing the digital images provided by the image receiver. A digital processor is provided,
`
`2 o
`
`programmed to perform the steps of:
`
`segmenting the digital images stored in memory by generating a segmentation
`
`mask fi‘om a digital image in any one of the at least one spectral band;
`
`estimating at least one rotationally and translationally invariant statistical measure '
`
`of coefficient distributions for the mulfiscale wavelet maxima representations of the digital
`
`25
`
`images in each spectral band, which are fiinctions of the texture of the region of interest
`
`determined by the segmentation mask;
`
`characterizing the condition of the tissue based on the estimated values; and
`
`outputting the characterization of the region of interest.
`
`The flexible optical system may be an endoscope, a fiber optic bundle or an
`
`3 o
`
`articulated arm.
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 837 of 1115
`
`BANK OF AMERICA
`
`IPR2021-01080
`
`Ex. 1002, p. 837 of 1115
`
`

`

`WO 02/01143
`
`PCT/USOI/20524
`
`-5-
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`Fig. 1 is a sketch of the system of the invention; and
`
`Fig. 2 is a sketch of an articulated arm of a system of the invention.
`
`5
`
`DETAILED DESCRIPTION OF THE INVENTION
`
`Fig 1 shows a sketch of the system of the invention. A patient 10 has a flexible
`
`endoscope 11 inserted into a natural or surgically provided orifice of the body. As is very
`
`well known in the art of endoscopy, light is provided for illuminating the tissue inside the
`
`body, and the light reflected from the tissue and scattered from under the surface of the
`
`10
`
`tissue falls on the end 12 of the endoscope. The first end of the endoscope 12 generally
`
`has a lens (not shown) which images the tissue on to the end of a coherent fiber bundle
`
`(not shown). The light imaged on to the fiber bundle is carried to the other end of the
`
`endoscope 13, where it may be projected on to a film or an electronic image receiver 14
`
`such as are found in video or digital cameras. The light for illuminating the tissue may be
`
`15
`
`provided through the endoscope.
`
`The output of the image receiver is carried to a processor or computer 15 where the
`
`computer segments the image by generating a segmentation mask defining the boundary of
`
`a region of interest in at least one spectral band, estimates at least one rotationally and
`
`translationally invariant statistical measure of coefficient distributions of the multiscale
`
`2o
`
`wavelet maxima representations of the digital
`
`images in at
`
`least one spectral band,
`
`characterizes the condition of the tissue based on the estimated values, and outputs the
`
`characterization of the condition of the tissue. The computer processes are described in
`
`great detail in U.S. Patent No. 6,208,749, B1, issued March 27, 2001, U.S. Patent No.
`
`6,081,612, issued June 27, 2000, International Publication No. WO99/44010, published 2
`
`25
`
`September 1999; and International Publication No. WO98/3781l, published 3 September
`
`1998, which are incorporated by reference herein, in their entireties.
`
`Fig. 2 shows a sketch of an articulated arm 20 for conveying the images. The
`
`articulated