as) United States
`a2) Patent Application Publication co) Pub. No.: US 2013/0155474 Al
`(43) Pub. Date: Jun. 20, 2013
`
`Roachet al.
`
`US 20130155474A1
`
`(54)
`
`(71)
`
`(72)
`
`SYSTEMS AND METHODS FOR AUTOMATIC
`IMAGE CAPTUREON A MOBILE DEVICE
`
`Applicant: MITEK SYSTEMS,San Diego, CA
`(US)
`
`Inventors: John J. Roach, San Diego, CA (US);
`Grigori Nepomniachtchi, San Diego,
`CA (US); Robert Couch, Poway, CA
`(US); Oleg Rybakov, San Diego, CA
`(US); Michael Gillen, San Diego, CA
`(US); Kevin Andrew Bell, Seattle, WA
`(US)
`
`(73)
`
`Assignee: MITEK SYSTEMS,San Diego, CA
`(US)
`
`(21)
`
`Appl. No.: 13/769,833
`
`(22)
`
`Filed:
`
`Feb. 19, 2013
`
`Related U.S. Application Data
`
`(63)
`
`Continuation of application No. 13/461,726, filed on
`May1, 2012, Continuation-in-part of application No.
`12/906,036,filed on Oct. 15, 2010, which is a contin-
`uation-in-part of application No. 12/778,943, filed on
`May12, 2010, Continuation-in-part ofapplication No.
`12/346,026, filed on Dec. 30, 2008, now Pat. No.
`7,978,900.
`
`(60) Provisional application No. 61/561,772, filed on Nov.
`18, 2011, provisional application No. 61/022,279,
`filed on Jan. 18, 2008.
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`HOAN 1/00
`(52) U.S.CL
`CPC vieecccccesceeteceeeeeseeeene HOAN 1/00129 (2013.01)
`USPC ceeececcccccessesseceseetseeteesaeeneee 358/505; 358/473
`
`(2006.01)
`
`(57)
`
`ABSTRACT
`
`Real-time evaluation and enhancementof image quality prior
`to capturing an image of a document on a mobile device is
`provided. An image capture processis initiated on a mobile
`device during which a user of the mobile device prepares to
`capture the image of the document, utilizing hardware and
`software on the mobile device to measure and achieve optimal
`parameters for image capture. Feedback maybe provided to a
`user of the mobile device to instruct the user on how to
`
`manually optimize certain parametersrelating to image qual-
`ity, such as the angle, motion and distance of the mobile
`device from the document. When the optimal parameters for
`image capture of the document are achieved, at least one
`image of the document is automatically captured by the
`mobile device.
`
`MIP Science
`Phase 4
`
`$502
`
`Biller DB
`
`
`
`
`Phase 2
`
`$506
`
`Match >0
`
`
`
`
`Phase 4
`
`
`
`
`Phase 5
`
`PNC 1040
`
`PNC 1040
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 1 of 55
`
`US 2013/0155474 Al
`
`100
`
`Mobile
`Device
`
`102
`
`104
`
`Remote
`
`110
`
`M
`
`a seer >
`Address
`Content
`Database
`Database
`
`106
`
`Biller/Payee
`Database
`
`Banking
`Server
`
`112
`
`108
`
`FIG. 1
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 2 of 55
`
`US 2013/0155474 Al
`
`Processing of
`Image(s) or Data
`
`Initiate Financial
`Transaction
`
`S216
`
`S218
`
`'
`
`I I
`
`I
`
`I I I
`
`I
`
`Shilo
`alle
`one
`Lilo
`"Bl
`|©
`oll
`=|15
`
`I I
`
`I
`
`I I I
`
`I
`
`,E
`
`FIG. 2
`
`S202
`
`S004
`
`S206
`
`$208
`
`Initialize mobile
`application
`
`Measure IQA
`Parameters
`
`Determineif
`Thresholds Met
`
`Provide User
`Feedback
`
`sip.
`
`Capture Image(s)
`
`$212
`
`S014
`
`Processing of
`Image(s)/IQA
`
`‘
`
`Transmit
`Image(s) or Data
`
`to Server
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 3 of 55
`
`US 2013/0155474 Al
`
`300
`
`302
`
`SooeRSESECERESNESESEENESRERS
`
`we
`
`============
`
`=
`
`FIG. 3
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 4 of 55
`
`US 2013/0155474 Al
`
`402
`
`404
`
`402
`
`404
`
`2
`
`410
`
`Kosh
`
`
`FIG. 4A
`
`410
`
`FIG. 4B
`
`406
`
`408
`
`406
`
`408
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 5 of 55
`
`US 2013/0155474 Al
`
`404
`
`406
`
`408
`
` 402
`406
`
`FIG. 4C
`
`A410
`
`FIG. 4D
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 6 of 55
`
`US 2013/0155474 Al
`
`410
`
`410
`
`404
`404
`
`FIG. 4E
`
`FIG, 4F
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 7 of 55
`
`US 2013/0155474 Al
`
`406
`406
`
`FIG. 4G
`
`412
`
`404
`
`410
`
`FIG. 4H
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 8 of 55
`
`US 2013/0155474 Al
`
`S502
`
`Biller DB
`
`MIP Science
`
`Phase 1
`
`Phase 3
`
`
`
`
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 9 of 55
`
`US 2013/0155474 Al
`
`Second Content Recognition Process
`
`S602
`
`S606
`
`S608
`
`S610
`
`Verify Second
`Pass Needed
`
`Load biller
`settings
`
`Update RegEx
`for Fields
`
`
`
`Perform Second
`Pass
`
`Evaluate
`Extracted Data
`
`FIG. 6
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 10 of 55
`
`US 2013/0155474 Al
`
`702
`
`704
`
`702
`
`700
`
`700
`
`704
`706
`
`FIG. 7A
`
`704
`
`FIG. 7B
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 11 of 55
`
`US 2013/0155474 Al
`
`800
`925
`
`aa
`
`FIG. 8
`
`a
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 12 of 55
`
`US 2013/0155474 Al
`
`
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 13 of 55
`
`US 2013/0155474 Al
`
`Detect document &
`identify corners of
`remittance couponin
`mobile image
`1205
`
` Identify and read code
`
`line from binarized
`subimage
`1230
`
`Perform second
`geometrical
`transformation to
`
`generate scanner
`quality image
`1235
`
`Perform adaptive
`binarization of
`scanner quality image
`1240
`
`Build perspective
`transformation
`1210
`
`Perform geometrical
`transformation on
`coupon subimage
`1215
`
`Perform de-warping of
`remittance coupon
`subimage
`1220
`
`Perform adaptive
`binarization of
`remittance coupon
`subimage
`1225
`
`FIG. 12
`
`image
`Correction
`Module
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 14 of 55
`
`US 2013/0155474 Al
`
`Receive original color
`image from mobile
`device
`1502
`
`Convert original color
`image into smaller
`color image (color
`“icon” image)
`1504
`
`Perform color depth
`reduction
`1506
`
`1510
`
`Find document
`corners
`
`1508
`
`Output document
`cornerpositions
`
`FIG. 13
`
`Detect
`Document in
`Image
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 15 of 55
`
`US 2013/0155474 Al
`
`
`
`
`Receive gray-scale
`document image
`1402
`
`Choosepixel on gray-
`scale image
`1404
`
`
`
`
`
`
`
`
`Compute average
`value and standard
`deviation of pixel
`intensity
`4406
`
`
`
`
`
`
`
`
`More pixels?
`1418
`
`
`Std dev too
`small?
`
`1408
`
`Output bi-tonal image
`of document
`41420
`
`Convert pixel to black
`or white based on
`1412
`intensity
`
`
`
`
`Convert low-contrast
`pixels to white
`1410
`
`Choose next pixel
`1414
`
`FIG. 14
`
`

`

`Patent Application Publication
`
`Jun. 20, 2013 Sheet 16 of 55
`
`US 2013/0155474 Al
`
`tra MORTON ALONG WITH YOUR PAYMENT
`PLEAD
`PLEASE MAKE YOUA CHECK PAYABLE TO CITY OF LEVISVILLE,
`ONeens
`Total Que
`$50.95
`CURRENT AMOUNT DUE DATE:
`ox0t2nh07
`
`Faymeni
`
`
`
`
`
`W228 TO: O1R807
`
`
`
`parenENCLOSED
`
`Nreckdstodsh beatldfecablatidubalfeabaleahll
`CIY.OF LEWISVILLE
`PO BOX 850868
`DALLAS TX 75265-0568
`
`0590003050001
`
`Oonoanoessts
`
`i
`
`FIG. 15
`
`
`
`
`
`Receive original color
`Suppression of High
`Local Contrast
`image from mobile
`
`
`device
`
`
`Objects
`
`1601
`1606
`
`
`
`
`Eliminate over-
`
`
`sharpening
`41602
`
`
`
`Output color“icon”
`
`image
`
`1608
`
`
`
`Eliminate small “dark”
`
`
`objects
`
`
`4604
`
`
`
`FIG. 16
`Convertoriginal
`color image into
`smaller color image
`{color “icon” image)
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 17 of 55
`
`US 2013/0155474 Al
`
`A \
`
`‘
`FIG. 17A
`
`FIG. 17B
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 18 of 55
`
`US 2013/0155474 Al
`
`Receive Color “icon” {mage
`1301
`
`
`
`i~ComputeAverageCColor!”
`at “Central Part” of “icon”
`1304
`
`Z
`
`
`
`|ComputeAverage Color “Outside” of
`“Central Part’ of “icon”
`
`
`
`|
`
`|
`
`Find LinearTransformation
`
`(Map “Central Part” Color on White)
`(Map “Outside” Color on Black)
`1308 |
`
`|
`
`|TransformationofRGB-valuesinto
`
`|
`
`Gray-Shades
`
`:
`
`Output Gray “icon” Image
`1312
`
`FIG. 18
`
`

`

`Patent Application Publication
`
`Jun. 20, 2013 Sheet 19 of 55
`
`US 2013/0155474 Al
`
`
`FIG
`
`. 19A
`
`FIG. 19B
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 20 of 55
`
`US 2013/0155474 Al
`
`
`Receive Gray “icon” Image
`
`2001
`
`
`Find “voting” Points on Gray “icon”
`Image for Each Side of Document
`
`2002
`
`Find Side Candidates
`
`ee _ a
`
`2004
`
`Choose Best Side Candidates
`| a
`
`2006
`
`|
`
`|
`L.
`
`Define Document Corners
`
`2008
`
`ee
`Output Position of Document Corners ,
`2009
`
`
`
`|
`
`FIG. 20
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 21 of 55
`
`US 2013/0155474 Al
`
`
`wer
`
`_
`
`
`tgs 7 ~ peo a~~7Receive Position of Document Corners Receive Original Color image
`
`in Gray “icon” Image
`1504
`OLOo
`:
`'
`poceneeeceeeeenepennegennennmeneennnetecenceeeeee nenetmscenee
`cenenccceecececeeenneceneenceencncenceNccneneenecceeeeeeeeeeectmneneerneneney
`t
`Correct Landscape Orientation
`|
`Reduce Color Depth
`1506
`1508
`|
`
`Lenteene neenneeneetneteeed
`q
`|
`
`pcnnnteseeenenenntnnenen
`Ncsatmteninnenen
`postmeneneetcneenneneYO
`Build Projective Transformation Model
`Output Original Gray-Scale Image
`
`
`
`_
`
`|~ a
`
`
`
`
`1512
`1510
`nea _—
`ii
`
`|
`
`|ApplyProjectiveTransformationto
`
`Interior of ABCD Quadrangle
`
`1515
`
`
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 22 of 55
`
`US 2013/0155474 Al
`
`
`oN RS
`
`FIG. 22A
`The checkin this image is intentionally
`depicted in landscapeorientation.
`
`S
`ROAR
`SEANADNAGEARNENOTES
`ee
`.
`See
`ee
`
`SS
`
`S < Se
`S
`SS ES
`. x BN NS
`RONoS
`. Sw
`SS
`XXSe
`3
`oN
`o eee
`RRS
`SERS
`Co OC
`
` LOSSS
`
`aN
`
`FIG. 22B
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 23 of 55
`
`US 2013/0155474 Al
`
`Receive Bi-tonal Image of Check
`1702
`
`|ReadMICR-LineontheBottomand|
`Generate MICR-Confidence Value...
`1704
`
`
`
`Compare MICR-Canfidence to
`Threshold
`
`
`
` Output Bi-tonal Image of Check
`a Confidence #1=~~}¥:(esi (Original Bi-tonal Image
`
`
`
`> Threshold?
`is Right-Side Up)
`i
`1708
`1710
`|
`| No
`oooeeececneeeeeneeenenneceerenenWo,
`Rotate Image 180 Degrees,
`Read MICR-Line on the Bottom and
`| Generate MICR-Confidence Value
`
`|
`
`1712
`
`|
`
`|
`
`|
`
`|
`
`|
`
`Compare Two MICR-Confidences|
`1714
`|
`
`eneennaa
`
`
`tN
`Yes
`Confidence #2 > MICR:
`
`Confidence #1 + Delta?
`1716
`So
`No
`
`Ouiput Rotated Bitonal
`Image ofCheck
`|
`(Original Bi-tonalImage
`is Right-Side Up)
`|
`1718
`i
`eae :
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 24 of 55
`
`US 2013/0155474 Al
`
`|ReceiveRightSideUp
`Bi-tonal Image of Check
`1802
`|
`
`ennenencnenniennnenenens rectecreneepe aan
`i
`
`
`Read MICR-Line on the Bottom
`1804
`bp
`P~
`Compute Average Width of
`MICR-Characters
`1806
`
`|
`|
`
`Compute Scaling Factor
`1808
`
`
`
`1 + Delta
`
`\
`Scaling Factorng |OutputBi-tonalImageofCheck&|
`
`
`
`|
`Repeat Geometrical Correction &
`| Binarization Using Recomputed Check |---
`|
`Dimensions
`|
`L
`1814 |
`
`
`el
`
`Gray-Scale Image of Check
`
`FIG. 24
`
`

`

`Patent Application Publication
`
`Jun. 20, 2013 Sheet 25 0f55
`
`US 2013/0155474 Al
`
`MDIPE module
`2100
`
`Mobile Image
`2105
`
`Processing
`Parameters
`2107
`
`Preprocessing
`Module
`2110
`
`Document
`Snippet
`2120
`
`Test Execution
`Module
`2130
`
`Selected Tests
`and Parameters
`2112
`
`2138
`
`Results
`
`FIG. 25
`
`Test Results /
`Messages
`2140
`
`

`

`Patent Application Publication
`Jun. 20, 2013 Sheet 26 of 55==.US 2013/0155474 Al
`
`Receive mobile image
`and processing
`parameters
`
`Process mobile image
`to generate snippet
`
`Document Processing
`
`FIG. 26
`
`Select tests from data
`store
`
`Retrieve test
`parameters
`
`2205
`2210
`2215
`2220
`
`
`
`Mark transaction for
`review
`
`2265
`
`
`
`
`
`Test Affect
`Load detailed test
`
`
`Overall
`
`messages
`
`
`
`status?
`2255
`
`
`2250
`
`
`
`
`Execute test on
`Threshold
`
`
`Add test messagesto
`exceeded?
`snippet
`test results
`
`
`
`
`2225
`2235
`2260
`
`
`
`
`
`
`
`Comparetest results
`to test threshold
`
`
`
` Output test results /
`2230
`More tests?
`
`
`messages
`
`2240
`
`2270
`
`
`

`

`Patent Application Publication
`Jun. 20, 2013 Sheet 27 0f 55==.US 2013/0155474 Al
`
`Receive Check Data
`2307
`
`Receive mobile
`
`image(s) and
`processing
`parameters
`2305
`
`Process mobile image
`to generate snippet(s)
`2310
`
`Select tests from data
`store
`2315
`
`FIG. 27
`
`Check Processing
`
`Image
`quality
`acceptable?
`2375
`
`Provide results to user
`
`to re-take picture of
`check
`2365
`
`
`
`
`2330
`
`
`Retrieve test
`parameters
`2320
`
`Execute Check 21
`test on snippet(s)
`2340
`
`Add test messagesto
`test results
`2360
`
`Execute test on
`snippet(s)
`2325
`
`Comparetest results
`to test thresholds
`2345
`
`Load detailed test
`messages
`2355
`
`Comparetest results
`to test thresholds
`
`Passsnippet(s) to
`
`Image
`mabile application for
`
`quality
`further processing and
`acceptable?
`depositing
`2350
`2370
`
`
`
`
`Yes
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 28 of 55
`
`US 2013/0155474 Al
`
`FIG.28B
`
`FIG.28A
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 29 of 55
`
`US 2013/0155474 Al
`
`
`
`
`FIG. 29A
`
`FIG. 29B
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 30 of 55
`
`US 2013/0155474 Al
`
`
`
`FIG. 30
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 31 of 55
`
`US 2013/0155474 Al
`
`
`
`FIG. 31
`
`

`

`Patent Application Publication
`
`Jun. 20, 2013 Sheet 32 of 55
`
`US 2013/0155474 Al
`
`Receive mobile image
`2805
`
`Process mobile image
`to generate grayscale
`snippet
`2810
`
`Build histogram using
`snippet
`2815
`
`FIG, 32
`
`Test Document Contrast
`
`Determine median
`black value
`2820
`
`
`
`
`Caiculate difference
`between median black
`& median white
`2830
`
`Determine median
`white value
`2825
`
`Normalize difference
`2835
`
`Return test result
`2840
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 33 of 55
`
`US 2013/0155474 Al
`
`
`
`FIG. 33A
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 34 of 55
`
`US 2013/0155474 Al
`
`
`
`FIG. 33B
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 35 of 55
`
`US 2013/0155474 Al
`
`Receive mobile image
`3005
`
`FIG. 34
`
`Test Document Skew
`
`Identify corners of
`sub-image in mobile
`image
`3030
`
`3040
`
`Calculate skew score
`3015
`
`Return test result
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 36 of 55
`
`US 2013/0155474 Al
`
`
`
`FIG. 35
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 37 of 55
`
`US 2013/0155474 Al
`
`
`
`
`identify corners of
`Receive mobile image
`
`Select Corner
`sub-image in mobile
`& width / height
`
`
`
`3215
`image
`parameters
`
`
`
`
` 3210 3205
`
`
`
`
`
`
`
`
`Determine whether
`
`corneris in mobile
`
`FIG. 36
`image
`3220
`
`Cut-Off Corner Test
`
`Update maximum cut-
`off measure
`3235
`
`
`
`
`
`
`
`Determine corner cut-
`
`off measure
`
`3230
`
`
`Cornerin
`image?
`3225
`
`
`
`
`
`
`More
`
`corners?
`3225
`
`
`
`
`Computetest results
`Retumtest results
`
`
`using maximum cut-
`3245
`off measure
`
`3240
`
`
`
`
`

`

`Patent Application Publication
`
`Jun. 20, 2013 Sheet 38 of 55
`
`US 2013/0155474 Al
`
`
`
`FIG. 37
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 39 of 55
`
`US 2013/0155474 Al
`
`
`
` Receive mobile image
`
`Identify corners of
`
`Select Side
`sub-imagein mobile
`& width / height
`
`
`
`
`3415
`image
`parameters
`
`
`
`3410
`3405
`
`
`
`
`
`
`Determine whether
`side is in mobile
`
`
`FIG. 38
`image
`3420
`
`
`Cut-Off Corner Test
`
`Side in
`
`
`
`image?
`3420
`
`
`
`
`Set test results to “fail”
`3430
`
`
`
`
`
`More sides?
`3425
`
`
`
`Set test results to
`“pass”
`3440
`
`
`
`
`
`Return test results
`3445
`
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 40 of 55
`
`US 2013/0155474 Al
`
`
`
`FIG. 39
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 41 of 55
`
`US 2013/0155474 Al
`
`
`
`
`Receive mobile image
`Identify corners of
`Select Side
`
`& width / height
`sub-image in mobile
`
`3615
`parameters
`image
`
`
`
`3610
`3605
`
`
`
` Calculate piecewise-
`linear approximation
`FIG. 40
`for side
`3620
`
`Warped Image Test
`
`
`
`
`
`
`
`Calculate maximum
`
`
`Update overall
`Normalize maximum
`deviation between
`
`
`maximum deviation
`deviation value
`
`
`straight line and
`value
`3630
`approximation
`
`
`3635
`
`3625
`
`
`
`
`
`
`
`More sides?
`3640
`
`
`
`Return test results
`Compute test results
`3650
`3645
`
`
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 42 of 55
`
`US 2013/0155474 Al
`
`
`
`FIG. 41
`
`

`

`Patent Application Publication
`
`Jun. 20, 2013 Sheet 43 of 55
`
`US 2013/0155474 Al
`
`Receive mobile image
`and height and width
`parameters
`3805
`
`
`
`
`
`
`
`
`identify corners of
`sub-image
`3810
`
`FIG. 42
`
`Image Size Test
`
`
`
`
`
`
`
`Compute sub-image
`average width
`3815
`
`
`
`Compute sub-image
`average height
`3820
`
`
`
`Normalize the
` S
`average width and
`average height value
`
`3822
`
`Determine minimum
`average value
`3825
`
`
`
`Return test results
`3865
`
`
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 44 of 55
`
`US 2013/0155474 Al
`
`Receive mobile image
`3955
`
`3975
`
`Determine code line
`confidences
`3970
`
`Generatebitonal
`image
`3960
`
`Identify code line in
`bitonal image
`3965
`
`Return test results
`
`FIG. 43
`Code line Test
`
`

`

`
`
`Receive mobile rear
`image
`4010
`
`FIG. 44
`Aspect Ratio Test
`For Two-Side Documents
`
`Patent Application Publication
`Jun. 20, 2013 Sheet 45 of 55.=US 2013/0155474 Al
`
`Receive mobile front
`image
`4005
`
`Identify front sub-
`image in mobile front
`image
`4015
`
`
`
`Identify rear sub-
`image in mobile rear
`image
`4020
`
`Determinerelative
`difference between
`aspect ratios
`4035
`
`Determine aspect
`ratio of front image
`4025
`
`Calculate test results
`4040
`
`Determine aspect
`ratio of rear image
`4030
`
`Return test results
`4045
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 46 of 55
`
`US 2013/0155474 Al
`
`Receive bi-tonal
`image of remittance
`coupon
`4205
`
`Execute matching
`4210
`
`
`
`
`
`
`
`
`
`
`
`Extract data from
`
`
`coupon using
`Matching
`template information
`template
`
`
`& associate data with
`found?
`
`
`field values
`
`
`4215
`4220
`
`
`
`
`
`
`Perform dynamic data
`capture to extract
`information from
`coupon
`4225
`
`FIG. 45
`
`Form Identification
`and Data Extraction
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 47 of 55
`
`US 2013/0155474 Al
`
`Locate datafields
`4310
`
`Extract data from
`fields
`4315
`
`Receive bi-tonal
`image of remittance
`coupon
`4305
`
`4320
`
`Analyze extracted
`data
`
`FIG. 46
`Dynamic Data
`Capture
`
`

`

`Patent Application Publication
`Jun. 20, 2013 Sheet 48 of 55=US 2013/0155474 Al
`
`Launch remittance
`application
`4702
`
`Display splash
`screen
`
`4704
`
`4708
`
`Display login screen
`4706
`
`Display main menu
`
` Recurring
`
`payment option
`selected?
`A710
`
`
`
`
`
`
`
`
`
`Finished
`
`
`
`updating
`data fields?
`
`
`4722
`
`
` Display data fields
`4720
`
` Load data for
`
`
`preexisting
`recurring payment
`
`4718
`
`
`
`
`
`
`
`
`
`Preexisting
`
`
`Display list of
`
` New recurring
`recurring paymen
`
`
`recurring payments
`
`
`payment
`selected?
`
`4712
`
`
`selected?
`4714
`
`4716
`
`
`FIG. 47
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 49 of 55
`
`US 2013/0155474 Al
`
`
`
`
`Le
`Listen for user input |
`4802
`|
`
`
`
`“Immediately”
`selected?
`
`Y
`
`“Manually”
`selected?
`4820
`
`“By Schedule’
`selected?
`
`Y
`
`Y
`
`“Return”
`selected?
`
`4840
`
`
`
`4810
`
`
`
`
`4830
`
`
`
`Configure
`application
`4812
`
`
`
`
`
`
`
` Configure
`
`application
`4822
`
`Configure
`application
`4832
`
`Return to previous
`menu
`
`
`
`4842
`
`FIG. 48
`
`

`

`Patent Application Publication
`
`FIG. 49
`
`Jun. 20,2013 Sheet 50 of 55
`
`US 2013/0155474 Al
`
`Display Menu
`4902
`
`Reminder
`options
`selected?
`4904
`
`4910
`
`Display sub-menu
`4906
`
`ser-finished
`specifying
`reminder
`options?
`4908
`
`Save settings
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 51 of 55
`
`US 2013/0155474 Al
`
`Mobile Device
`4400
`
`Processor
`4410
`
`Image Capture
`4430
`
`Data Storage
`4440
`
`4460
`
`Wireless Interface
`4450
`
`1/O Interface
`
`FIG. 50
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 52 of 55
`
`US 2013/0155474 Al
`
`1900
`
`MEDIA DRIVE
`1912
`
`MEDIA
`
`1914
`
` PROCESSOR 1904 STORAGE DEVICES 1910
`
`STORAGE
`STORAGE
`
`
`
`
`
`UNIT I/F
`UNIT
`
`
`
`1922
`1920
`
`COMM I/F
`4924
`
`
`CHANNEL 14 KO28
`
`
`
`FIG. 51
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 53 of 55
`
`US 2013/0155474 Al
`
`5202 S Input Image
`
`Feature matching-
`based cropping
`
`5206
`
` 5204
`
`Edge detection-
`based cropping
`
` a
`
`FIG. 52
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 54 of 55
`
`US 2013/0155474 Al
`
`5306A
`
`5304
`
`5306B
`
`5302
`
`5300
`
`5300 J
`5308
`
`FIG. 538A
`
`5306C
`
`FIG. 53B
`
`5306D
`
`5312
`
`5310
`
`9314
`
`FIG. 53C
`
`FIG. 53D
`
`

`

`Patent Application Publication
`
`Jun. 20,2013 Sheet 55 of 55
`
`US 2013/0155474 Al
`
`5404
`
`5402
`
`
`
`5406
`
`5400
`
`FIG. 54A
`
`FIG. 54B
`
`FIG. 540
`
`

`

`US 2013/0155474 Al
`
`Jun. 20, 2013
`
`SYSTEMS AND METHODS FOR AUTOMATIC
`IMAGE CAPTUREON A MOBILE DEVICE
`
`RELATED APPLICATIONS INFORMATION
`
`[0001] This application claimspriority to U.S. patent appli-
`cation Ser. No. 13/461,726, filed May 1, 2012, which claims
`priority to U.S. Provisional Patent Application No. 61/561,
`772,filed Nov. 18, 2011, now pending; and whichis a con-
`tinuation in part of U.S. patent application Ser. No. 12/906,
`036 filed on Oct. 15, 2010, now pending, which itself is a
`continuation in part of U.S. patent application Ser. No.
`12/778,943 filed on May 12, 2010, now pending,as well as a
`continuation in part of U.S. patent application Ser. No.
`12/346,026 filed Dec. 30, 2008, now U.S. Pat. No. 7,978,900,
`which in turn claims the benefit of U.S. Provisional Applica-
`tion Ser. No. 61/022,279 filed Jan. 18, 2008, now expired,all
`ofwhich are incorporated herein by reference in their entirety
`as if set forth in full. This application is also related to U.S.
`patent application Ser. No. 12/717,080 filed Mar. 3, 2010,
`whichis now U.S. Pat. No. 7,778,457, which is incorporated
`herein by reference in its entirety as if set forth in full.
`
`BACKGROUND
`
`1. Technical Field
`[0002]
`[0003] The embodiments described herein generally relate
`to automatic capture of an imageofa financial or other docu-
`ment on a mobile device, and moreparticularly to automati-
`cally detecting and determining image quality on the mobile
`device prior to capturing the image of the document.
`[0004]
`2. Related Art
`[0005] Banks and other businesses have becomeincreas-
`ingly interested in electronic processing of check and other
`documents in order to expedite processing of these docu-
`ments. Users can scan a copy ofthe documentusing a scanner
`or copierto create an electronic copy ofthe documentthat can
`be processed instead of routing a hardcopy of the document
`from oneplace to another for processing. For example, some
`banks can process digital images of checks and extract check
`information from the image needed to process the check
`without requiring that the physical check by routed through-
`out the bank for processing.
`[0006] Mobile devices that incorporate cameras have also
`becomeubiquitous. However, the quality of images captured
`varies greatly, and many factors cause images captured using
`a mobile device to be of poor quality. Therefore, images
`captured by mobile devices are often not of sufficiently high
`quality to be used for electronic processing ofdocuments. For
`systems which utilize images of documents captured on the
`mobile device, the process of evaluating a mobile image to
`determine if it is of sufficient image quality can be time
`consuming and cumbersome for the user of the mobile
`device. Therefore, it would be advantageousto streamline and
`automate the process of capturing images of documents with
`mobile devices and verifying that the quality of the imageis
`sufficient processing.
`
`SUMMARY
`
`Systems and methods are provided for real-time
`[0007]
`evaluation and enhancementof image quality prior to captur-
`ing an image of a document on a mobile device. An image
`capture process is initiated on a mobile device during which a
`user of the mobile device prepares to capture the image of the
`document, utilizing hardware and software on the mobile
`
`device to measure and achieve optimal parameters for image
`capture. Feedback may be provided to a user of the mobile
`device to instruct the user on how to manually optimize cer-
`tain parameters relating to image quality, such as the angle,
`motion and distance of the mobile device from the document.
`Whenthe optimal parameters for image capture of the docu-
`ment are achieved, at least one image of the documentis
`automatically captured by the mobile device.
`[0008] Other features and advantagesofthe present inven-
`tion should becomeapparent from the following description
`of the preferred embodiments, taken in conjunction with the
`accompanying drawings, which illustrate, by way of
`example, the principles of the invention.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0009] The various embodiments provided herein are
`described in detail with reference to the following figures.
`The drawings are provided for purposesofillustration only
`and merely depict typical or example embodiments. These
`drawings are providedto facilitate the reader’s understanding
`of the invention and shall not be considered limiting of the
`breadth, scope, or applicability ofthe embodiments. It should
`benotedthatfor clarity and ease ofillustration these drawings
`are not necessarily made toscale.
`[0010]
`FIG. 1 is a block diagram which illustrates one
`embodimentof a system for mobile image capture and remit-
`tance processing, according to one embodimentofthe inven-
`tion.
`FIG. 2 illustrates one embodiment of a method of
`[0011]
`automatic image capture on a mobile device, according to one
`embodimentofthe invention.
`
`FIG. 3 illustrates one embodiment of a graphical
`[0012]
`user interface (GUI) which may bepresented to the user on a
`display screen of the mobile device during a mobile image
`capture process, according to one embodimentofthe inven-
`tion.
`
`FIGS. 4A-4H illustrate visual feedback displayed
`[0013]
`on the display screen of the mobile device during the mobile
`image capture process, according to one embodimentof the
`invention.
`[0014]
`FIG. 5 is a flowchart illustrating a biller lookup
`process, according to one embodimentof the invention.
`[0015]
`FIG.6 is a flow diagram illustrating a process for a
`second data recognition process on a remittance coupon,
`according to one embodimentofthe invention.
`[0016]
`FIGS. 7A and 7B are imagesof remittance coupons
`whichillustrate the results of a first data recognition process
`and a second data recognition process, according to one
`embodimentofthe invention.
`[0017]
`FIG. 8 is an imageof a remittance coupon captured
`by a mobile device.
`[0018]
`FIG. 9 is a geometrically corrected image created
`using image processing techniques disclosed herein using the
`mobile image of the remittance coupon illustrated in FIG.8.
`[0019]
`FIG. 10 andits related description above provide
`some examples of how a perspective transformation can be
`constructed for a quadrangle defined by the corners A, B, C,
`and D according to an embodiment.
`[0020]
`FIG. 11 isa diagram illustrating an example original
`image, focus rectangle and document quadrangle ABCD in
`accordance with the example of FIG. 10.
`[0021]
`FIG. 12 is a flow diagram illustrating a method for
`correcting defects to mobile image according to an embodi-
`ment.
`
`

`

`US 2013/0155474 Al
`
`Jun. 20, 2013
`
`FIG. 13 is a flow chart for a method that can be used
`[0022]
`to identify the corners of the remittance coupon in a color
`image according to an embodiment.
`[0023]
`FIG. 141s a flow diagram of a methodfor generating
`a bi-tonal image according to an embodiment.
`[0024]
`FIG. 15 illustrates a binarized image of a remittance
`coupon generated from the geometrically corrected remit-
`tance coupon imageillustrated in FIG. 9, according to one
`embodiment.
`
`FIG. 16is a flow diagram of a methodfor converting
`[0025]
`a document imageinto a smaller color icon image according
`to an embodiment.
`[0026]
`FIG. 17A is a mobile image of a check according to
`an embodiment.
`
`FIG. 17B is an example of a color icon image gen-
`[0027]
`erated using the method of FIG. 12 on the example mobile
`image of a check illustrated in FIG. 13A according to an
`embodiment.
`
`FIG. 18 is a flow diagram of a method for reducing
`[0028]
`the color depth of an image according to an embodiment.
`[0029]
`FIG. 19A depicts an example of the color “icon”
`image of FIG. 17B after operation 1302 has divided it into a
`3x3 grid in accordance with one embodimentof the inven-
`tion.
`FIG. 19B depicts an example of the color “icon”
`[0030]
`image ofFIG. 17B converted to a gray “icon” image using the
`methodillustrated in FIG. 18 according to an embodiment.
`[0031]
`FIG. 20 is a flowchart
`illustrating an example
`method for finding document corners from a gray “icon”
`image containing a documentaccording to an embodiment.
`[0032]
`FIG. 21 is a flowchart that illustrates an example
`method for geometric correction according to an embodi-
`ment.
`
`FIG. 22A is an imageillustrating a mobile image of
`[0033]
`a checkthat is oriented in landscape orientation according to
`an embodiment.
`
`FIG. 22B example gray-scale image of the docu-
`[0034]
`ment depicted in FIG. 17A once a geometrical correction
`operation has been applied to the image according to an
`embodiment.
`
`FIG. 23 is a flow chart illustrating a method for
`[0035]
`correcting landscape orientation of a document
`image
`according to an embodiment.
`[0036] FIG.24 providesa flowchart illustrating an example
`method for size correction of an image according to an
`embodiment.
`
`FIG. 29Billustrates an example of an out of focus
`[0043]
`document.
`
`[0044]
`ment.
`
`FIG. 30 illustrates an example of a shadowed docu-
`
`FIG. 31 illustrates an example ofa grayscale snippet
`[0045]
`generated from a mobile document image of a check where
`the contrast of the image is very low according to an embodi-
`ment.
`
`FIG. 32 illustrates amethodfor executing a Contrast
`[0046]
`IQA Test according to an embodiment.
`[0047]
`FIG. 33A is an example of a mobile document
`image that includes a check that exhibits significant planar
`skew according to an embodiment.
`[0048]
`FIG. 33B illustrates an example of a documentsub-
`imagethat exhibits view skew according to an embodiment.
`[0049]
`FIG. 34 is a flow chart illustrating a method for
`testing for view skew according to an embodiment.
`[0050]
`FIG. 35 illustrates an example ofa mobile document
`imagethat features an image of a document whereoneofthe
`corners of the documenthas been cut off in the picture.
`[0051]
`FIG.36 illustrates a Cut-Off Corner Test that can be
`usedfor testing whether comers of a document in a document
`subimage have been cut off when the document was imaged
`according to an embodiment.
`[0052]
`FIG. 37 illustrates an example ofa mobile document
`imagethat features a document where oneofthe ends of the
`documenthas been cut off in the image.
`[0053]
`FIG. 38 is a flow diagram of a method for determin-
`ing whether one or more sides of the documentare cut off in
`the document subimage according to an embodiment.
`[0054]
`FIG. 39 illustrates an example ofa mobile document
`image where the documentis warped according to an embodi-
`ment.
`
`FIG. 40isa flow diagram ofa methodfor identifying
`[0055]
`a warped imageandfor scoring the image based on how badly
`the document subimage is warped according to an embodi-
`ment.
`
`FIG. 41 illustrates an example of a document sub-
`[0056]
`image within a mobile document imagethat is relatively
`small in comparison to the overall size of the mobile docu-
`ment image according to an embodiment.
`[0057]
`FIG. 42 is a flow diagram of a process that for
`performing an ImageSize Test on a subimage according to an
`embodiment.
`
`FIG. 43 is a flow chart of a method for executing a
`[0058]
`code line test according to an embodiment.
`FIG. 25 illustrates a mobile document imagepro-
`[0037]
`[0059]
`FIG. 44 illustrates a methodfor executing an Aspect
`cessing engine (MDIPE) module for performing quality
`Ratio Test according to an embodiment.
`assurancetesting on mobile document images according to an
`embodiment.
`
`[0060] FIG. 451saflow chart of a methodfor processing an
`image using form identification according to an embodiment.
`[0038]
`FIG. 26 is a flow diagram of a process for perform-
`[0061]
`FIG. 46isa flow chart of a methodfor processing an
`ing mobile image quality assurance on an image captured by
`image using dynamic data capture according to an embodi-
`a mobile device according to an embodiment.
`ment.
`[0039]
`FIG. 27 is a flow diagram of a process for perform-
`ing mobile image quality assurance on an image of a check
`captured by a mobile device according to an embodiment.
`[0040]
`FIG. 28Aillustrates a mobile image where the docu-
`ment captured in the mobile document image exhibits view
`distortion.
`
`FIG. 28Billustrates an example of a grayscale geo-
`[0041]
`metrically corrected subimage generated from the distorted
`image in FIG. 28A according to an embodiment.
`[0042]
`FIG. 29A illustrates an example of an in-focus
`mobile documentimage.
`
`FIG. 47 is a flow diagram illustrating an exemplary
`[0062]
`method of configuring a recurring payment schedule accord-
`ing to an embodiment.
`[0063]
`FIG. 48 is a flow diagram illustrating an exemplary
`method ofselecting a specific scheduling preference accord-
`ing to an embodiment.
`[0064]
`FIG. 49 is a flow diagram illustrating an exemplary
`method of enabling a user to set one or more reminders
`associated with a recurring bill payment according to an
`embodiment.
`
`

`

`US 2013/0155474 Al
`
`Jun. 20, 2013
`
`FIG. 50 is a block diagram of various functional
`[0065]
`elements of a mobile device that can be used with the various
`
`systems and methods described herein according to an
`embodiment.
`
`FIG. 51 isablock diagram of functional elements of
`[0066]
`a computer system that can be used to implementthe mobile
`device and/or the servers described in the systems and meth-
`ods disclosed herein.
`[0067]
`FIG. 52 is a flow diagram of a process for edge and
`feature detection, according to an embodiment.
`[0068]
`FIGS. 53A-53Dillustrate images being processed
`using the edge and feature detection methodologies, accord-
`ing to an embodiment.
`template-matching
`[0069]
`FIG. 54A-54C illustrate a
`method used during feature detection, according to an
`em