US008379914B2
`
`(12) United States Patent
`Nepomniachtchi et a].
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 8,379,914 B2
`Feb. 19, 2013
`
`(54)
`
`(71)
`
`(72)
`
`SYSTEMS AND METHODS FOR MOBILE
`IMAGE CAPTURE AND REMITTANCE
`PROCESSING
`
`ApplicantszGrigori Nepomniachtchi, San Diego,
`CA (US); Josh Roach, San Diego, CA
`(Us)
`Inventors: Grigori Nepomniachtchi, San Diego,
`CA (US); Josh Roach, San Diego, CA
`(Us)
`
`(73)
`
`Assignee:
`
`Mitek Systems, Inc., San Diego, CA
`(Us)
`
`(*)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21)
`
`(22)
`
`(65)
`
`(63)
`
`(60)
`
`(51)
`
`Appl. N0.: 13/622,329
`
`Filed:
`
`Sep. 18, 2012
`
`Prior Publication Data
`
`US 2013/0022231 A1
`
`Jan. 24, 2013
`Related US. Application Data
`Continuation-in-part of application No. 12/906,036,
`?led on Oct. 15, 2010, Which is a continuation-in-part
`ofapplication No. 12/778,943, ?led on May 12, 2010,
`and a continuation-in-part of application No.
`12/346,026, ?led on Dec. 30, 2008, noW Pat. No.
`7,978,900.
`Provisional application No. 61/561,772, ?led on Nov.
`18, 2011, provisional
`application No. 61/022,279,
`?led on Jan. 18, 2008.
`
`Int. Cl.
`G06K 9/00
`G06Q 40/00
`HO4B 1/034
`
`(2006.01)
`(2006.01)
`(2006.01)
`
`(52) US. Cl. ......................... .. 382/102; 235/379; 455/95
`(58) Field of Classi?cation Search ................ .. 382/ 102,
`382/135, 137, 138, 139, 237, 276; 235/379,
`235/462.24, 462.25, 462.26; 455/11.1, 95,
`455/406, 437, 439, 456.6; 705/1.1, 14.36,
`705/35; 707/950; 715/234; 725/23, 62
`See application ?le for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,326,959 A *
`5,920,847 A *
`
`7/1994 PeraZZa ....................... .. 235/379
`7/1999 Kolling et a1. .
`705/40
`
`6,807,294 B2* 10/2004 YamaZaki . . . . . .
`
`. . . . . .. 382/135
`
`.. 235/46225
`7,331,523 B2* 2/2008 Meier et a1.
`7,377,425 B1 *
`5/2008 Ma et a1. ..................... .. 235/379
`
`* cited by examiner
`
`Primary Examiner * AbolfaZl Tabatabai
`(74) Attorney, Agent, orFirm * Procopio, Cory, Hargreaves
`& Savitch LLP; Noel C. Gillespie
`
`ABSTRACT
`(57)
`Systems and methods are provided for capturing and process
`ing images of remittance coupons using a mobile device and
`obtaining data from the captured image Which is used to set up
`or carry out payment of a bill that corresponds to the remit
`tance coupon. Optimization and enhancement of image cap
`ture and image processing are provided on the mobile device
`to improve the initial quality of the captured image and pro
`vide a user With real time feedback. The image is then sent
`from the mobile device to a remote server, Where additional
`image processing is performed to improve the quality of the
`image and then extract data from the image that is relevant to
`paying the bill. The extracted data may be veri?ed through
`comparisons With databases Which store information on bill
`ers, bill formats and other relevant content that Will appear on
`the bill.
`
`10 Claims, 49 Drawing Sheets
`
`m
`
`Mobile 2
`
`Device
`
`2//102
`
`104 \?)
`
`Remote
`(‘rt-M Server
`i
`MEETS/S
`Database
`
`110
`
`\/\
`\\
`i_:: I
`Content
`gtabalsi
`
`106
`
`FlgrFalyq
`5
`
`Database
`\9
`f
`
`Banking
`
`/ Server
`
`112
`
`108
`
`Page 1 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US. Patent
`
`Feb. 19, 2013
`
`Sheet 1 0149
`
`US 8,379,914 B2
`
`Q
`
`Mobile
`Device @102
`
`104
`
`Remote
`
`Server
`
`110
`
`\/\
`
`Content
`Database
`
`Address
`Database
`
`106
`
`Biller/Payee
`Database
`
`Banking
`Server
`
`112
`
`108
`
`FIG. 1
`
`Page 2 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US. Patent
`
`Feb. 19, 2013
`
`Sheet 2 0f 49
`
`US 8,379,914 B2
`
`SZOZQ Initialize mobile
`billing application
`
`V
`
`(Automatic)
`Image Capture
`
`V
`
`Pre-Processing
`of Image/IQA
`
`S208®
`
`Y
`
`Convert Image
`for Transmission
`to Server
`
`FIG. 2
`
`7
`
`Convert to 2D 4/ S210
`and Crop Image
`
`JGIUSS atowaa
`Mobile Device
`
`Read code line @ $212
`
`Identify Image
`Form
`
`8214
`4/
`
`First OCR Pass /l/ 3216
`
`Read Barcodes /i/ 5218
`
`Fuzzy Address
`Search
`
`8220
`4/
`
`Biller Lookup /i/ 8222
`
`Second OCR
`Pass
`
`e224
`4/
`
`Output 110
`Database
`
`3226
`4/
`
`Page 3 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U S. Patent
`
`Feb. 19, 2013
`
`Sheet 3 0f 49
`
`US 8,379,914 B2
`
`FIG
`.3A
`
`FIG
`.38
`
`
`
`.30 FIG
`
`314
`
`320
`
`3
`
`6
`
`22
`
`324
`
`FIG. 3D
`
`FIG. 3E
`
`FIG. 3F
`
`Page 4 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US. Patent
`
`Feb. 19, 2013
`
`Sheet 4 0f 49
`
`US 8,379,914 B2
`
`
`
`5% wag a3,“
`
`3% ‘w
`
`Now
`
`Page 5 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US. Patent
`
`Feb. 19, 2013
`
`Sheet 5 0f 49
`
`US 8,379,914 B2
`
`MIP Science
`
`Phase 1
`
`Bi?er DB
`
`)
`
`Phase 2
`
`Maid?‘ >0
`
`Phase 3
`
`FIG. 5
`
`Page 6 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US. Patent
`
`Feb. 19, 2013
`
`Sheet 6 0f 49
`
`US 8,379,914 B2
`
`Second Content Recognition Process
`
`560%? Verify Second
`Pass Needed
`
`Load biller
`S604 ® settings
`
`S606@ Update RegEx
`for Fields
`
`S608& Perform Second
`Pass
`
`SMOQ
`
`Evaluate
`Extracted Data
`
`FIG. 6
`
`Page 7 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US. Patent
`
`Feb. 19, 2013
`
`Sheet 7 0f 49
`
`US 8,379,914 B2
`
`702
`
`704
`
`700
`
`704
`
`FIG. 7A
`
`700
`
`702
`
`706
`
`704
`
`FIG. 7B
`
`Page 8 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US. Patent
`
`Feb. 19, 2013
`
`Sheet 8 0f 49
`
`US 8,379,914 B2
`
`800
`
`90Q/\
`
`FIG. 8
`
`915
`
`925
`
`905
`
`FIG. 9
`
`Page 9 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US. Patent
`
`Feb. 19, 2013
`
`Sheet 9 0f 49
`
`US 8,379,914 B2
`
`2500
`
`u“,
`
`FIG. 10
`
`
`
`i K - - f v Y K - - - . 4 v w h I - - f f Y h H - - M w ~ ~ * , , v ¥ _ A ,\ .\ ................................................. , z "
`
`
`
`z
`x
`>
`:-
`*
`1-
`i
`j
`
`k."
`>’
`‘
`’
`
`>
`-‘
`
`i
`K
`
`j
`
`,
`
`.
`
`:
`
`I.
`
`1;
`i
`
`s
`‘
`§
`}
`x
`s
`
`2
`a
`*
`'
`.
`*
`z
`{
`¥
`l
`i
`‘1
`inhwvvwv?n?. A...
`
`~LH
`
`‘
`
`z
`>
`a
`\
`c
`i
`a
`a
`)
`4
`
`4
`.1
`
`a
`3
`1
`1
`<
`a
`a
`a
`
`.
`_,
`_,
`;,
`;,
`.
`
`{I
`;,
`e
`,
`z
`>
`4
`4
`c
`I.
`»
`\
`‘
`c
`3
`
`Page 10 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US. Patent
`
`Feb. 19, 2013
`
`Sheet 10 0f 49
`
`US 8,379,914 B2
`
`Detect document &
`identify corners of
`remittance coupon in
`mobiie image
`12$
`
`Build perspective
`transformation
`1210
`
`V
`
`Perform geometrical
`transformation on
`coupon subimage
`M
`
`V
`
`Perform de~warping 0i
`remittance coupon
`subimage
`12.22
`
`V
`Perform adaptive
`binarization of
`remittance coupon
`subirnage
`@
`
`FIG. 12
`image
`Correction
`Module
`
`identify and read code
`tine from binarized
`subimage
`Q9
`
`V
`Perform second
`geometricat
`transformation to
`generate scanner
`quality image
`1215
`
`V
`
`Perform adaptive
`binarization of
`scanner quality image
`1A
`
`Page 11 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US. Patent
`
`Feb. 19, 2013
`
`Sheet 11 0f 49
`
`US 8,379,914 B2
`
`Receive original color
`image from mobile
`device
`M
`
`Convert original color
`image into smaller
`color image (color
`“icon" image)
`M
`
`Perform color depth
`reduction
`1506
`
`V
`
`Find document
`corners
`1508
`
`Output document
`corner positions
`1510
`
`FIG. 13
`Detect
`Document in
`Image
`
`Page 12 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US. Patent
`
`Feb. 19, 2013
`
`Sheet 12 0f 49
`
`US 8,379,914 B2
`
`Receive gray-scaie
`document image
`1402
`
`Choose pixel on gray
`scale image
`1404
`
`More pixels?
`141§
`
`Output bi-tonai image
`of document
`1420
`
`V
`Compute average
`value and standard
`> deviation of pixei
`intensity
`111%
`
`Std dev too
`smaii?
`ma
`
`Convert low-contrast
`pixets to white
`1410
`
`‘
`
`Convert pixel to btack
`or white based on
`intensity
`L2.
`
`V
`
`Choose next pixel
`1m
`
`FIG. 14
`
`Page 13 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US. Patent
`
`Feb. 19, 2013
`
`Sheet 13 0f 49
`
`US 8,379,914 B2
`
`2
`
`15
`Payment
`' C o u p o n
`l
`
`acoounmroamm
`NAME:
`meow-r.
`W '
`ssm'ics mass
`mm“
`Bum DATE.
`
`am: r-QRHON ALONG wim mun PAYMEN:
`' Prmwi...
`PLEASE MAKE YOUR CHECK PAYABLE T0 cm! 01‘ LEWISVILLE.
`mourn nus
`
`WW Tom Duo
`8580001050001
`CURRENT MAW"? DUE DM'E:
`mount :ucmszo
`‘mm- _
`
`$58.55
`029013097
`
`KM Om)
`
`MasterCard
`
`Vn
`
`i
`
`CITYOFLEIMSWUE
`POBOXBSMGB
`DALLASTX 75265-0568
`
`osqouulusnonx
`
`uounnousa'is
`
`1
`
`FIG. 1 5
`
`Receive original color
`image from mobile
`device
`1601
`
`V
`
`Eliminate over—
`sharpening
`1 602
`
`V
`
`Eliminate small “dark"
`objects
`1604
`
`‘
`'
`
`Suppression of High
`Local Contrast
`Objects
`1606
`
`v
`
`Output color “icon"
`image
`1608
`
`FIG. 16
`Convert original
`color image into
`smaller color image
`(color “icon” image)
`
`Page 14 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US. Patent
`
`Feb. 19, 2013
`
`Sheet 14 0f 49
`
`US 8,379,914 B2
`
`W
`
`FIG. 17A
`
`Page 15 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 15 of 49
`
`US 8,379,914 B2
`
`Reaeiva Caiar “i:::c>n“ image
`3§.§,1
`
`-,..............................,...................;
`
`iifiiasfide Geek}? “ism” image ima; Grid
`.3332
`
`.............................a
`
`Ccsrzzpute i&v»e2ragsa”“i.”3¥;§§}§f§’"‘%"%
`at “fievniraii Part” :1»? “i<:a:m”
`
`i
`
`,,..,,,,,,,,,,.,,..,,.,,,.,,.,,,,,,,,,,..-.,,,,,,,Y,.-,...,..,.-,
`
`3.§§&
`L,MW_W
` ,......
`’ Comrmte Average Cain: “<3:3t:side‘” cf
`«
`“Cenira¥ Part” Q? “ism”
`
`,
`
`.:¥...;i*.».«.£3..2’.:.3.
`
`......
`...................
`5............................................................
`Find Linear Transformatim
`{Map “Centrai £3331?" flmiar an White}-
`{¥s.*¥a;:1“€}uts§de"
`{3z:3§s:::r on fifaci-z}
`
`5
`
`E
`
`)
`
`
`
`gw-.w.—2
`
`Transféfiggaiian czf ¥%G8«vaiu£:5
`{3ray~S%:acies
`3 §'§ Q
`........._.....,..__..............._....,...................._......,....,.,...,.,.,......,...........,...........,..... w....,...,.....w,,m...............'»
`
`.- l1l.lP.F.-.‘.l1l.‘F --
`
`0-H-.4!‘-.‘AP.hAA#.~AA-4AA#fiAA-49A-fl.hAAn-fiA~A~A'»~»~.*0
`
`Guigaui {Bray “Egan” Image
`§§jg
`
`’
`
`FIG. 18
`
`Page 16 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 16 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US. Patent
`
`Feb. 19, 2013
`
`Sheet 16 0149
`
`US 8,379,914 B2
`
`FIG. 19A
`
`FIG. 19B
`
`Page 17 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US. Patent
`
`Feb. 19, 2013
`
`Sheet 17 0f 49
`
`US 8,379,914 B2
`
`Receive Gray “icon” image
`2001
`
`i Y
`Find “voting” Points on Gray “icon”
`Image for Each Side of Document
`2002
`
`i
`
`i
`
`Find Side Candidates
`2004
`
`Choose Best Side Candidates
`2006
`
`l
`
`De?ne Document Corners
`2008
`
`Output Position of Document Corners
`2009
`
`FIG. 20
`
`Page 18 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US. Patent
`
`Feb. 19, 2013
`
`Sheet 18 0f 49
`
`US 8,379,914 B2
`
`'1'
`?owed Landsmpe Qrientaiim
`v@536
`
`Receive Originai {3mm image
`M
`
`"
`Redum {39km {39pm
`é??g
`
`Bzz‘z'id Preéectéve ‘Hansfmmatian Mada;
`
`Gutput Original? Gxawsmaie 393E332
`3m’:
`
`Apgsiy Pmjenf?m “m; n-sformation to
`interim 0? ,WCQ‘ Quadrangie
`3 51 5'
`
`.............................................
`
`Gutpui Gmy~3caia image a?‘ Documeni
`
`FIG. 21
`
`Page 19 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US. Patent
`
`Feb. 19, 2013
`
`Sheet 19 0149
`
`US 8,379,914 B2
`
`FIG. 22A
`The check in this image is intentionally
`depicted in landscape orientation.
`
`FIG. 22B
`
`Page 20 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 20 of 49
`
`US 8,379,914 B2
`
`Rsmeive Ebéorzai ¥ma:;;e of Check
`fig;
`
`.......................................
`........................................
`Razszrx M1€}¥?.:»Lé€3£s an 1323 Bmtom and *
`
`Generate M$CR~€:»;mfidence Vakze
`
`
`
`.
`‘fhresafzoisfi
`‘
`
`..
`
`......................................................................................_
`I
`v:
`5
`.
`,
`Qutpm 8»e~t<ma: image of {Z?1eC§€
`{Originai §3i~ir.ma3% image
`is FEs'ght~S§<ir: Up}
`
`"3
`
`»»»»»»>4
`
`”“~-«.._A___________________
`/‘
`
`E
`-«.*"'» ‘
`f,.w”’" M£C¥3:““-~x,M_~
`itnnfidence #31
`
`‘W/»'*‘
`
`
`
`..................................,.3:,_§_..:............... ...............
`..................................
`Rceiaie image ‘£30 fiegrees,
`‘ Read M¥C¥7¢»3.Jme an the 803mm and ‘V
`
`Generate Mfizfficonfideace Vaiiua
`
`4............................................................................... .
`'5
`
`. ...:
`
`:«::e;m;»::re Two:»§:cR»£:mfi«;sencas
`W
`
`1
`
`
`
`
`
`<::w””Confidenee #2 .~» :sa::::»<2~“-~.%.......
`\""-.., flonfideznoa #1 + ::«2:za‘s,,,w”
`«H
`,
`‘.
`,.,«»
`\~..‘‘''‘''-.
`WW”
`‘*~\,v_¢,,«*’
`E
`E
`
`Q“. ‘§}:‘f§:‘W';f§Q
`39*’
`‘
`3, , _
`5
`zs=R:gh1~Ss::3a~. Up}
`=
`~
`. r
`1
`3:736
`---------------------------------------------------------------------------------------------------.-
`w 1&5 .,.,...,M.m,,,,,,,,,,,,,M....W.
`‘J,
`
`;
`3
`2’.
`1%
`
`E
`
`Page 21 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 21 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 21 of 49
`
`US 8,379,914 B2
`
`Receive Right $é::i:=2 Up
`B§~icsna.%
`of came
`"--.-.-.-.-.-.-.-.-.-.-......-.-...1............1..w........u,1-....k-.........,..u.~
`.‘..... M... ..r..,....... . ... . r... . ... . ...J-
`,1§.Q£W
`
`E
`
`Read ¥u¥1C}R~Line an we Elottam
`
`3§».§$
`
`
`
`Cam=;m%£3- Awarage Width cs?
`M :83?-Characism
`...................................................................................................
`
`‘-
`
`«*5;
`
`1 «I» 86113
`
`““»~__\.‘
`
`_.»-'
`
`1
`
`we
`image: of Chaak
`€3.ray»8c:a§e
`“‘,‘:>«~~~~~~~~~4-------------aw;
`.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,._._....._._.,,,_.,,,,.......A....1.._....,.
`
`Sc
`
`‘K83
`
`§
`
`Reggae‘: Gimxrrreiricat Correztition &
`§ir1ari;‘-.azia~:1 Using Recomputeé {Shack
`\\\\\
`,
`Désnensitrng
`z I
`.....)
`'1 M:
`'
`
`3:
`
`FIG. 24
`
`Page 22 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 22 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 22 of 49
`
`US 8,379,914 B2
`
`Mobile Wage
`E
`
`MDIPE module
`'2‘10—0
`
`Processing
`Parameters
`
`11$
`
`Messages
`
`Selected Tests
`and Parameters
`3-11-2-
`
`prepmoessing
`Module
`gi
`
`Document
`
`Snippet
`2129
`
`Test Execution
`Module
`2130
`
`FIG. 25
`
`Test Results I
`
`Messages
`2140
`
`Page 23 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 23 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 23 of 49
`
`US 8,379,914 B2
`
`Document Processing
`221g
`
`FIG. 26
`
`Mark transaction for
`review
`2265
`
`Receive mobile image
`and processing
`parameters
`
`3315
`
`Process mobile image
`to generate snippet
`
`Select tests from data
`store
`2215
`
`
`
`
`
`
`Test Affect
`Load detailed test
`Overall
`messages
`status?
`2255
` Retrieve test
`Hi
`parameters
`222Q
`
`
`
`
`
`
`Execute test on
`Threshold
`Add test messages to
`exceeded?
`snl ppet
`test results
`2225
`2235
`2260
`
`
`
`
`
`
`Compare test results
`to test threshold
`22351
`More tests?
`OUtp:1tet::;;::uits I
`
`2270
`33-4-9
`
`
`
`
`
`
`
`
`Page 24 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 24 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 24 of 49
`
`US 8,379,914 B2
`
`image(s) and Receive2g3(l)17eck Data
`
`Receive mobile
`
`processing
`parameters
`2305
`
`Z
`
`FIG. 27
`
`Check Processing
`
`Process mobile image
`to generate snippet(s)
`2310
`
`Select tests from data
`store
`231 5
`
`Retrieve test
`parameters
`2320
`
`Execute test on
`snippet(s)
`2325
`
`Compare test results
`to test thresholds
`2330
`
`
`
`
`
`Provide results to user
`to re—take picture of
`check
`2365
`:
`
`Image
`quality
`acceptable?
`2375
`
`
`
`
`Add test messages to
`test results
`E
`
`Execute Check 21
`
`test on snippet(s)
`2340
`
`Compare test results
`to test thresholds
`2345
`
`L°ad detailed test
`messages
`E
`
`Pass snippet(s) to
`mobile application for
`further processing and
`depositing
`E
`
`Image
`quality
`acceptable?
`E
`
`
`
`Yes
`
`Page 25 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 25 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 25 of 49
`
`US 8,379,914 B2
`
`mam.0_..._
`
`Page 26 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 26 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 26 of 49
`
`US 8,379,914 B2
`
`
`
`
`FIG. 29A
`
`FIG. 29B
`
`This figure is intentionally blurry.
`
`Page 27 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 27 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 27 of 49
`
`US 8,379,914 B2
`
`
`
`FIG. 30
`
`This figure is intentionally blurry.
`
`Page 28 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 28 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 28 of 49
`
`US 8,379,914 B2
`
`
`
`FIG. 31
`
`This figure is intentionally blurry.
`
`Page 29 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 29 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 29 of 49
`
`US 8,379,914 B2
`
`Receive mobiie image
`2.§|l§
`
`
`
`Process mobile image
`to generate grayscaie
`snippet
`E
`
`
`
`Buiid histogram using
`snippet
`2815
`
`FIG. 32
`
`Test Document Contrast
`
`
`
`Determine median
`black vaiue
`2829
`
`
`
`
`Normaiize difference
`Lifi
`
`
`
`Determine median
`white vaiue
`2§Z§
`
`Caicuiate difference
`between median black
`& median white
`Z.§§.Q
`
`
`
`
`Return test resuit
`
`2840
`
`
`
`
`
`Page 30 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 30 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 30 of 49
`
`US 8,379,914 B2
`
`FlG. 33A
`
`This figure is intentionally blurry.
`
`Page 31 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 31 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 31 of 49
`
`US 8,379,914 B2
`
`FIG. 33B
`
`Page 32 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 32 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 32 of 49
`
`US 8,379,914 B2
`
`Receive mobile image
`§QQ§
`
`Identify corners of
`sub-image in mobiie
`image
`3.4139
`
`Calculate skew score
`3%
`
`Return test result
`3040
`
`FIG. 34
`
`Test Document Skew
`
`Page 33 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 33 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U S. Patent
`
`Feb. 19, 2013
`
`Sheet 33 of 49
`
`US 8,379,914 B2
`
`FIG. 35
`
`Page 34 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 34 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 34 of 49
`
`US 8,379,914 B2
`
`identify corners of
`sub~image in mobile
`image
`3210
`
`3205
`
`Receive mobile image
`& width / height
`parameters
`
`FIG. 36
`Cut-Off Corner Test
`
`
`
`Select Corner
`
`3215
`
`
`
`Determine whether
`.
`.
`M
`corneririrsaigemo re
`@2251
`
`
`
`Corner in
` Determine comer cut—
`Update maximum cut-
`off measure
`off measure
`image?
`3225
`
`3230
`3235
`
`
`
`
`
` More
`comers?
`
`3225
` No
`
`
`
`Compute test results
`using maximum cut-
`off measure
`3240
`
`3245
`
`Retum test results
`
`Page 35 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 35 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U S. Patent
`
`Feb. 19, 2013
`
`Sheet 35 of 49
`
`US 8,379,914 B2
`
`FIG. 37
`
`Page 36 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 36 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 36 of 49
`
`US 8,379,914 B2
`
`
`
`Receive mobiie image
`8. width / height
`parameters
`§é9§.
`
`identify corners of
`sub~image in mobiie
`image
`3919
`
`Select Side
`3415
`
`
`
`
`
`
`
`
`Determine whether
`side is in mobile
`image
`3.4212
`
`
`
`More sides?
`flfi
`
`Set test resuizs to
`“pass"
`§4_.4.§
`
`Return test resuits
`3.4%:
`
`Page 37 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`
`FIG. 38
`
`Cut~Off Comer Test
`
`Set test resuits to Wail”
`3_‘§.3£
`
`
`
`
`
`Page 37 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 37 of 49
`
`US 8,379,914 B2
`
`
`
`FIG. 39
`
`This figure is intentionally blurry.
`
`Page 38 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 38 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 38 of 49
`
`US 8,379,914 B2
`
`identify corners of
`sub—image in mobile
`image
`3.6.19.
`
`.3I§Q§
`
`Receive mobile image
`& width I height
`parameters
`
`FIG. 40
`
`Warped Image Test
`
`
`
`Select Side
`3615
`
`
`
`
`Calculate piecewise—
`linear approximation
`for side
`E
`
`
`
`Calculate maximum
`deviation between
`
`straight line and
`approximation
`§§Z§
`
`Normalize maximum
`deviation value
`3630
`
`
`
`Update overall
`maximum deviation
`value
`E
`
`
`
`
`
`More sides?
`
`3640
`
`
`
`No
`
`Compute test results
`fifiéé
`
`QLQEQ
`
`Return test results
`
`Page 39 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 39 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 39 of 49
`
`US 8,379,914 B2
`
`
`
`FIG. 41
`
`Page 40 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 40 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 40 of 49
`
`US 8,379,914 B2
`
`
`
`
`Receive mobile image
`and height and width
`parameters
`fiili
`
`
`
`identify corners of
`sub-image
`flfl
`
`FIG. 42
`
`lmage Size Test
`
`
`
`Compute sub—image
`average width
`%
`
`
`
`
`
`Normalize the
`
`
`
`
`
`Compute sub—image
`average width and
`average height
`average height values
`1&9
`§.§Z.Z
`
`
`Determine minimum
`
`
`
`average value
`31$
`
`
`
`Return test resuits
`
`flii
`
`
`
`Page 41 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 41 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 41 of 49
`
`US 8,379,914 B2
`
`Receive mobile image
`3955
`
`Generate bitonai
`
`image
`3960
`
`identify code line in
`bitonai image
`3965
`
`Determine code iine
`confidences
`3212
`
`Return test results
`3975
`
`FIG. 43
`
`Code line Test
`
`Page 42 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 42 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 42 of 49
`
`US 8,379,914 B2
`
`Receive mobile front
`image
`
`4005 Receive mobile rear
`
`image
`4010
`
`Identify front sub-
`image in mobile front
`image
`%
`
`FIG. 44
`
`Aspect Ratio Test
`For Two-Side Documents
`
`Identify rear sub-
`image in mobile rear
`image
`E
`
`Determine relative
`difference between
`aspect ratios
`fl
`
`Determine aspect
`ratio of front image
`4025
`
`Calculate test results
`4040
`
`Determine aspect
`ratio of rear image
`4030
`
`Return test results
`4045
`
`Page 43 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 43 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 43 of 49
`
`US 8,379,914 B2
`
`Receive bi—tonal
`
`
`image of remittance
`coupon
`A295
`
`
`
`
`Execute matching
`4210
`
`Extract data from
`
`coupon using
`
`template information
`& associate data with
`
`field values
`
`5222
`
`
`
`Perform dynamic data
`capture to extract
`information from
`coupon
`4225
`
`FIG. 45
`
`Form Identification
`and Data Extraction
`
`Page 44 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 44 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 44 of 49
`
`US 8,379,914 B2
`
`Receive bi-tonal
`
`image of remittance
`coupon
`E
`
`Locate data fields
`4310
`
`Extract data from
`fields
`4315
`
`Analyze extracted
`data
`
`4320
`
`FIG. 46
`
`Dynamic Data
`Capture
`
`Page 45 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 45 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 45 of 49
`
`US 8,379,914 B2
`
`N
`
`Finished
`updating
`data fields?
`4722
`
`
`
`
`
`
`Y
`
`Launch remittance
`
`application
`4702
`
`Display splash
`screen
`4704
`
`4706
`
`Display login screen
`
`
`
`Display main menu
`4708
`
`4720
`
`
`
`Display data fields
`
`Load data for Recurring
`payment option
`preexisting
`Seiectew,
`recurring payment
`'
`4718
`4710
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Preexisting
`recurring paymen
`
`,
`Display list of
`New reC”m”9
`recurring payments
`payment
`4712
`selected?
`
`4716
`
`
`
`FIG. 47
`
`Page 46 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 46 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 46 of 49
`
`US 8,379,914 B2
`
`
`
`4802
`
` Listen for user input
`
`
`
`“immediately”
`selected?
`4810
`
`
`
`Configure
`application
`4812
`
`
`Configure
`application
`4822
`
`
`
` “By Schedule‘
`
`selected?
`4830
`
`Configure
`application
`4832
`
`
`
`
`
`Return to previous
`menu
`4842
`
`
`
`
`FIG. 48
`
`Page 47 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 47 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 47 of 49
`
`US 8,379,914 B2
`
`
`
`Display Menu
`4902
`
`
`
`Reminder
`
`opfions
`selected?
`4904
`
`Y
`
`Display sub-menu
`4906
`
`
`
`
`ser—finished
`
`specifying
`reminder
`
`opflons?
`4908
`
`Y
`
`Save settings
`4910
`
`FIG. 49
`
`Page 48 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 48 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 48 of 49
`
`US 8,379,914 B2
`
`Mobile Device
`4400
`
`Processor
`4410
`
`Image Capture
`4430
`
`Data Storage
`4440
`
`Wireless Interface
`4450
`
`I/O Interface
`4460
`
`FIG. 50
`
`Page 49 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 49 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`U.S. Patent
`
`Feb. 19, 2013
`
`Sheet 49 of 49
`
`US 8,379,914 B2
`
`PRO£.”.€ESSOR §‘ 3333
`’§'£¥i}8
` STGRAGE i’/‘E"J§{3ES
`33.1%
` ‘ §Ai+::m
`w:az:>:A mass
`
`J/.,... ’§§?f3{}
`
`1
`
`4
`
`' Mwosw
`
`$?'{}R!>«{§£€
`i$t='\3§T
`36:
`*3
`
`
`
`
`smea
`um‘ :29"
`
`19:.a<>
` 4:
`
`
`CH/’>.NNE£ i'§,:}§E_37
`
`>
`
`CGMM ,3fF
`.§,_},.:
`1
`
`FIG. 51
`
`Page 50 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 50 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US 8,379,914 B2
`
`1
`SYSTEMS AND METHODS FOR MOBILE
`IMAGE CAPTURE AND REMITTANCE
`PROCESSING
`
`RELATED APPLICATIONS INFORMATION
`
`This application claims priority to U.S. Provisional Patent
`Application No. 61/561,772, filed Nov. 18, 2011, now pend-
`ing, and is a continuation in part of copending U.S. patent
`application Ser. No. 12/906,036 filed on Oct. 15, 2010, now
`pending, which itself is a continuation in part of copending
`U.S. patent application Ser. No. 12/778,943 filed on May 12,
`2010, now pending, as well as a continuation in part of U.S.
`patent application Ser. No. 12/346,026 filed Dec. 30, 2008,
`now U.S. Pat. No. 7,978,900, which in turn claims the benefit
`ofU.S. Provisional Application Ser. No. 61/022,279 filed Jan.
`18, 2008, now expired, all ofwhich are incorporated herein by
`reference in their entirety as if set forth in full. This applica-
`tion is also related to U.S. patent application Ser. No. 12/717,
`080 filed Mar. 3, 2010, which is now U.S. Pat. No. 7,778,457,
`which is incorporated herein by reference in its entirety as if
`set forth in full.
`
`BACKGROUND
`
`1. Technical Field
`
`The embodiments described herein generally relate to
`automated processing of an image of a financial document
`captured by a mobile device, and more particularly to captur-
`ing, processing and obtaining data from an image of a remit-
`tance coupon captured by a mobile device in order to set up or
`pay a bill associated with the remittance coupon.
`2. Related Art
`
`Remittance processing services provide payment process-
`ing services for other businesses. A remittance processing
`service may be set up by a bank. Often these services set up
`one or more post office boxes for receiving payments from
`customers in the form ofpaper checks. Typically the user will
`include a remittance slip or remittance coupon with the pay-
`ment.
`
`Banks and other businesses often provide remittance slips
`or coupons that a customer can include with a payment. These
`coupons may be included with an invoice or provided to a
`customer in advance. For example, some mortgage lenders or
`automobile lenders provide a book of remittance coupons to
`customers who then tear out a mail a remittance coupon with
`each scheduled payment. The remittance coupons generally
`include customer account information, an amount due, and a
`due date for the payment. The customer account information
`might include an account holder name, mailing address, and
`a customer account number. Other information, such as the
`mailing address of the bank or business may also be included
`on the remittance coupon. Some remittance coupons also
`include computer-readable bar codes or code lines that
`include text or other computer-readable symbols that can be
`used to encode account-related information that can be used
`
`to reconcile a payment received with the account for which
`the payment is being made. The bar code or code line can be
`detected and decoded by a computer system to extract the
`information encoded therein.
`
`In general, the remittance coupon is mailed with a check or
`other form of payment to the bank or other business or to a
`payment processing company designated by the bank or other
`business. In some systems, data entry clerks may manually
`enter payment information into a payment processing system.
`In other systems, the remittance coupon and/or the check may
`be scarmed by a computer system for automated processing.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`
`Other means of paying bills include using online banking,
`paying a bill directly with the biller by accessing the biller’s
`website, or calling the biller to pay the bill over the phone.
`Online banking provides convenience to some customers, but
`the process is cumbersome. Typically, the consumer needs to
`first set up the biller on the consumer’s bar1k’s website, which
`means manually entering the biller information, including
`account number and mailing address. Then, the consumer
`must enter the payment
`information (e.g.,
`the payment
`amount, payment date, etc. .
`.
`. ). This manual data entry is
`susceptible to user error, can be time consuming, and incon-
`venient for consumers, which can inhibit the adoption rate of
`online banking bill payment.
`What is needed is a convenient, easy-to-use method for
`paying a bill which also maintains or increases the accuracy
`of the information gathered from the remittance coupon.
`
`SUMMARY
`
`Systems and methods are provided for capturing and pro-
`cessing images of remittance coupons using a mobile device,
`and obtaining data from the captured image which is used to
`set up or carry out payment of a bill that corresponds to the
`remittance coupon. Optimization and enhancement of image
`capture and image processing are provided on the mobile
`device to improve the initial quality ofthe captured image and
`provide a user with real time feedback regarding the quality of
`the captured image. The image is then transferred from the
`mobile device to a remote server, which performs additional
`image processing before extracting data from the remittance
`coupon to determine one or more parts of the bill, including
`the biller, payment address, user account number, amount
`due, etc. This information is then used to initiate payment of
`the bill to a biller or to initiate creation of a new biller account
`
`for payment of the bill.
`In some embodiments, an image of a check can also be
`captured to be processed as a payment associated with the
`remittance coupon. Some embodiments described herein
`involve a mobile communication device capturing an image
`of a document and transmitting the captured image to a server
`for image optimization and enhancement. Techniques for
`assessing the quality of images of documents captured using
`the mobile device are also provided. The tests can be selected
`based on the type of document that was imaged, the type of
`mobile application for which the image quality of the mobile
`image is being assessed, and/or other parameters such as the
`type ofmobile device and/or the characteristics of the camera
`of the mobile device that was used to capture the image. In
`some embodiments, the image quality assurance techniques
`can be implemented on a remote server, such as a mobile
`phone carrier’s server or a web server, and the mobile device
`routes the mobile image to be assessed and optional process-
`ing parameters to the remote server processing and the test
`results can be passed from the remote server to the mobile
`device.
`
`In on embodiment, a method of processing a remittance
`coupon captured by a mobile device comprises: receiving an
`image of a remittance coupon captured by a mobile device;
`correcting at least one aspect of the image to produce a cor-
`rected image; performing a first content recognition pass on
`the corrected image to extract content from the remittance
`coupon; identifying an address of a biller on the remittance
`coupon by comparing address content in the extracted content
`with an address database; determining biller profile informa-
`tion of the biller, including an identity of the biller on the
`remittance coupon, by comparing the identified address ofthe
`biller with a database of biller profile information; and pro-
`
`Page 51 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`Page 51 of 82
`
`ROTHSCHILD EXHIBIT 1002
`
`

`
`US 8,379,914 B2
`
`3
`ducing a set of billing information, including the extracted
`content and the identity ofthe biller, for processing a payment
`of the bill.
`
`The method may also comprise using the biller profile
`information of the biller to perform a second content recog-
`nition pass on the corrected image to extract content from the
`remittance coupon, wherein the biller profile information
`includes at least one of a remittance coupon format, a remit-
`tance coupon mask, a location of at least one field on the
`remittance coupon and a format of at least one field.
`The method may also comprise reading a code line on the
`remittance coupon and correcting a scale of the remittance
`coupon based on a size of the code line.
`The correcting ofthe at least one image may include at least
`one of a perspective correction, an aspect ratio correction, a
`warping correction and a shadow correction.
`The first content recognition pass may be performed using
`optical character recognition (OCR).
`The method may also comprise comparing the address
`content
`in the extracted content with address content
`
`extracted by reading a barcode on the remittance coupon
`before comparing the address content with the address data-
`base.
`
`10
`
`15
`
`20
`
`The address database may be populated with a plurality of
`addresses from a United States Postal Service (USPS) data-
`base.
`
`25
`
`Comparing the address content in the extracted content
`with the address database may include comparing a “zip code
`plus four digit” field in the extracted content with a zip code
`plus four digit field in the address database.
`Comparing address content in the extracted content with an
`address database may involve performing a fuzzy search of
`the address database using the address content.
`The method may also comprise transmitting the set of
`billing information to the mobile device to display to a user.
`In another embodiment, a method of processing a remit-
`tance coupon on a mobile device comprises: activating an
`image