I lllll llllllll Ill lllll lllll lllll lllll lllll lllll lllll lllll llllll llll llll llll
`
`US007234937B2
`
`c12) United States Patent
`Sachdeva et al.
`
`( JO) Patent No.:
`(45) Date of Patent:
`
`US 7,234,937 B2
`Jun. 26,2007
`
`(75)
`
`(54) UNIFIED WORKSTATION FOR VIRTUAL
`CRANIOFACIAL DIAGNOSIS, TREATMENT
`PLANNING AND THERAPEUTICS
`Inventors: Rohit Sachdeva, Plano, TX (US);
`Sanjeev Taneja, Plano, TX (US); Peer
`Sporbert, Berlin (DE); Phillip Getto,
`Plano, TX (US); Stephan Maetzel,
`Berlin (DE); Hans Imgrund, Berlin
`(DE); Charles L. Abraham, Dallas, TX
`(US)
`(73) Assignee: OraMetrix, Inc., Richardson, TX (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 291 days.
`
`(21) Appl. No.: 10/429,123
`
`(22) Filed:
`
`May 2, 2003
`
`(65)
`
`Prior Publication Data
`
`US 2004/0015327 Al
`
`Jan. 22, 2004
`
`Related U.S. Application Data
`
`(63) Continuation-in-part of application No. 10/340,404,
`filed on Jnn. 9, 2003, now abandoned, which is a
`continuation-in-part of application No. 09/835,039,
`filed on Apr. 13. 2001 , now Pat. No. 6,648,640, which
`is a continuation of application No. 091560 641 , filed
`on Apr. 28, 2000, now Pat. No. 6,512,994, which is a
`continuation-in-part of application No. 09/452,034,
`filed on Nov. 30, 1999, now abandoned.
`
`(51 )
`
`Int. Cl.
`(2006.01 )
`A61C 3100
`(52) U.S. Cl. .................................................... .... 433/24
`(58) Field of Classification Search .................... 433/6,
`433/18, 20, 24
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4 ,837,732 A *
`5,060,171 A •
`5,278,756 A
`5,338,198 A
`5,372,502 A "
`5,879,158 A
`5,975,893 A
`
`6/ 1989 Brandeslini el al. . .. ....... 433/29
`10/1991 Steir et al. ............. ..... 345/630
`111994 Lemchen et al.
`... ... .. ... 364/413
`8/1994 Wu el al.
`................... 433/213
`12/1994 Massen el al . ...... ........ 433/215
`3/ 1999 Doyle el al. .. ... ........... .. 433/24
`1111999 Chishti et al.
`... .............. 433/6
`
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`WO
`
`WO 0180761
`
`11/2001
`
`OTHER PUBLICATIONS
`
`Yamany el al., A System for Human Jaw Mod,-,/i11g Using I11tra-Oral
`Images, Proc. IEEE Eng. Med. Biol. Soc. (EMBS) Conf. vol. 20,
`Hong Kong, pp. 563-566, Oct. 1998.
`
`(Continued)
`
`Primary Examiner- Ralph A. Lewis
`(74) Attorney, Agent, or Finn-McDonnell, Boehnen,
`Hulbert & Bergholf LLP
`
`(57)
`
`ABSTRACT
`
`An integrated system is described in which digital image
`data of a patient, obtained from a variety of image sources,
`including CT scanner, X-Ray, 2D or 3D scanners and color
`photographs, are combined into a common coordinate sys~
`tern to create a virtual three-dimensional patient model.
`Software tools are provided for manipulating the virtual
`patient model to simulation changes in position or orienta(cid:173)
`tion of craniofacial structures (e.g., jaw or teeth) and sinm(cid:173)
`late their affect on the appearance of the patient. The
`simulation (which may be pure simulations or may be
`so-called "morphing" type simulations) enables a compre(cid:173)
`hensive approach to planning trealmenl for the patient. In
`one embodiment, the treatment may encompass orthodontic
`treatment. Similarly, surgical treatment plans can be created.
`Data is extracted from the virtual patient model or simula(cid:173)
`tions thereof for purposes of manufacture of customized
`ther.ipeutic devices for any component of the craniofacial
`structures, e.g., orthodontic appliances.
`
`4,575,805 A •
`
`3/ 1986 Moermann el al .
`
`...... ... 700/163
`
`62 Claims, 23 Drawing Sheets
`
`exocad GmbH, et al.
`Exhibit 1013
`
`0001
`
`

`

`US 7,234,937 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`6,015,289 A
`6,068,482 A
`6,081,739 A
`6,099,314 A
`6,217,325 Bl
`6,227,850 Bl
`6,283,858 Bl *
`6,431,870 Bl
`6,464,496 Bl
`6,471,512 Bl
`6,512,994 Bl
`6,540,512 Bl
`6,621,491 Bl*
`6,648,640 B2
`6,688,886 B2 *
`6,739,869 Bl*
`6,739,870 B2 *
`6,767,208 B2 ..
`
`112000 Andreiko et al. ..... ......... 433/3
`512000 Snow ............... .......... 433/223
`6/2000 Lemchen .... ................ 600/407
`8/2000 Kopelman et al. . . .. ... . .. 433/213
`4/2001 Chishti et al.
`.. .............. 433/24
`5/2001 Chishti et al.
`................ 433/24
`912001 Hayes et al. .............. .... 463/31
`8/2002 Sachdeva .................... 433/2 13
`I 0/2002 Sachdeva et al ......... ..... 433/24
`10/2002 Sachdeva et al .............. 433/24
`1/2003 Sachdeva
`4/2003 Sachdcva ct al .............. 433/24
`9/2003 Baumrind et al.
`. ......... 345/419
`11/2003 Sachdeva
`2/2004 Hughes et al. .............. .. 433/24
`5/2004 Taub et al. . .................. 433/24
`5/2004 Lai et al ... .................. .. 433/24
`7/2004 Kaza .... ....................... 433/24
`
`1/2005 Kopelman et al.
`6,845,175 B2
`6,851,949 Bl• 212005 Sachdeva et al ............ 433/213
`6,947,038 Bl
`9/2005 Anh et al.
`2002/0025503 Al
`2/2002 Chapoulaud et al .......... 433/24
`2002/0028417 Al
`3/2002 Chapoulaud et al. . ........ 433/24
`
`OTHER PUBLICATIONS
`
`Yamany et al, A 3-D Reconsrruction System for the Human Jaw
`Using a Sequence of Optical Images, IEEE Transactions on Medical
`Imaging, vol. 19, No. 5, pp. 538-547, May 2000.
`Pighin ct al., Synthesizing Realistic Facial faprossfo11s from Pho(cid:173)
`tographs, Computer Graphics Proceedings SIGGRAPH '98, pp.
`78-94 (1998).
`Pighin et al., Realistic Facial Animation Using Image-based 3D
`Morphing, Technical Report, No. UW-CSE-97001-03, University of
`Washington (May 9, 1997).
`Blantz et al., A Morphable Model for the Synthesis of 3D Faces,
`Computer Graphics Proceedings SIGGRAPH '99, (Aug. 1999).
`"' cited by examiner
`
`0002
`
`

`

`U.S. Patent
`
`Jun.26,2007
`
`Sheet 1of23
`
`US 7,234,937 B2
`
`14
`
`32
`
`12
`
`FIG. 1
`
`OTHER CLINICS I SPECIALISTS
`
`DEVICE MANUFACTURES
`
`DEVICE MANUFACTURES
`
`.1QQ
`
`j
`
`22
`
`IMAGE
`DATA
`
`28
`
`SCANNING
`NODE
`
`18
`
`OlHER IMAGING
`DEVICES:
`X-AAY
`CT SCANNER
`MRI
`
`36
`
`0003
`
`

`

`U.S. Patent
`
`Jun.26,2007
`
`Sheet 2 of 23
`
`US 7,234,937 B2
`
`46
`
`;
`
`3DFACE
`SCANNING
`LASER, OPTICAL
`
`NO
`
`NO
`
`50
`
`54
`
`FIG. 2
`
`48
`
`CREATE MORPHABLE
`MOOR WITH OPTIC
`FLOW ALGORITHM
`
`42
`
`CREATE 30
`FACE USING
`ACTIVE MODEL
`MATCHING
`STRATEGY
`
`3DFACE
`MODEL
`
`~
`44
`
`0004
`
`40
`
`2DCOLOR
`FACE DIGITAL
`IMAGE SET
`24
`
`CREATE 30
`FACE TEXTURE
`USING CYLINDRICAL
`PROJECTION
`TECHNIQUE
`
`~
`56
`
`

`

`U.S. Patent
`
`Jun.26,2007
`
`Sheet 3 of 23
`
`US 7,234,937 Bl
`
`FIG. 3
`
`56
`
`62
`
`64/
`20 FACE !-::I
`'----M-OD_E_L __, '--P-IC..,.TU_R_E__. L::_J
`
`CT SKULL
`
`52
`
`54
`
`3DCOLOR
`FACE MODEL
`
`3DCOLOR
`TEETH
`MODEL
`
`ALIGNING
`TRANSFORMATION
`
`SB
`
`30 FACE MODEL
`COMPLETE
`
`60
`
`OVERLAY
`TRANSFORMATION
`
`66
`
`c DATABASE
`
`(
`
`'--.,.
`68
`
`0005
`
`

`

`U.S. Patent
`
`Jun. 26, 2007
`
`Sheet 4 of 23
`
`US 7,234,937 B2
`
`FIG. 4
`
`70
`
`3D DENTAL
`MODEL
`
`(72
`
`2DCOLOR
`DENTAL
`PHOTOGRAPHS
`
`74
`
`76
`
`3DTEXTURE
`MODEL OF TEETH
`USING CYLINDRICAL
`PROJECTION
`TECHNIQUE
`
`3DCOLOR
`TEETH MODEL
`
`STAT\C OR OYNAM\C
`SiMULATIONS
`
`0006
`
`

`

`U.S. Patent
`
`Jun.26,2007
`
`Sheet 5 of 23
`
`US 7,234,937 B2
`
`FIG. 4A
`
`-- ..... - ... -- --
`
`·-~ ~ 75
`
`x=14.512372
`
`I y=23.351378 I Unit IN= mm
`
`I Unit SYS= mm!
`
`FIG. 48
`
`~ ~·;:;::-. .;-_- -
`
`-
`
`~
`
`-
`
`___ - _- -
`
`_-- _-__ -
`
`-
`
`-
`
`-
`
`75
`
`0007
`
`

`

`U.S. Patent
`
`Jun.26,2007
`
`Sheet 6 of 23
`
`US 7,234,937 B2
`
`FIG. 4C
`
`71
`
`75
`
`I
`-
`'-,'~: l ... -
`
`...
`
`.>.,, -..
`
`._ - -
`
`..... _ -"
`
`FIG. 40
`
`-- - - -
`'. ii?"G~'.!·;:··_:~"~'
`
`~-- - - - __=-:..:.:_ ~l!!l~
`
`..... -
`
`-
`
`-
`
`..,I , , ...
`
`-
`
`"
`
`-
`
`-
`
`-
`
`-
`
`,,_
`
`I,., I ~. - '
`
`,.. ~ -
`
`"""·-- ...... ,_.,..,,,,"'--- ... _
`
`I x=s.111014
`
`11 y= 2.a155s2
`
`II Unit IN = mm I
`
`6.32
`
`UnitSYS=mm
`
`0008
`
`

`

`U.S. Patent
`
`Jun.26,2007
`
`Sheet 7 of 23
`
`US 7,234,937 B2
`
`FIG. 4E
`
`\ ------
`~,,.. .......... ___ ,,,
`
`I
`...... r, -
`
`,
`
`_
`
`.. -_ , . . . , __ .,,., ..
`
`-
`
`......... _.._,_(
`
`-
`
`x=B.929511
`
`I y= 0.579335
`
`11 Unit IN = mm
`
`6.32
`I Unit SYS = mm I
`
`0009
`
`

`

`U.S. Patent
`
`Jun.26,2007
`
`Sheet 8 of 23
`
`US 7,234,937 B2
`
`x
`
`. I
`
`In
`C"l
`
`x
`0
`
`0
`if
`D
`sr.:i
`.9-
`~ #
`(lJ
`L:r
`1 ~
`(!]
`([
`c:
`~ ~
`is lSl
`.. gZJ
`g;i
`e
`CJ. ! = x
`aB
`IC
`~ ~
`!Bl
`~ §
`H3
`~ >C
`& a
`~
`.c g
`ID DI
`i
`::i s
`
`'C
`
`'Cl
`
`~
`~
`lOO
`~
`~
`
`>-
`
`~ N~
`-....... _
`.. -
`-- - - --
`
`\
`
`x
`
`I 0
`~o
`~D 5
`
`E
`E
`II
`~
`·2
`=:i.
`
`N !§
`.-
`0 co
`..-
`
`~
`
`II -
`
`IO
`..-
`M
`0 -.r
`-.r
`~ cc
`11
`>< >.
`"O
`ctl
`Ill
`0::
`
`"
`
`~ = !bb
`
`0010
`
`

`

`FIG. 6
`
`Cloud
`
`.Mesh
`
`~urve &urface
`
`Inspect Wjndow
`
`Help
`
`35
`
`- Cl
`
`)C·
`
`~
`~
`&>
`
`- Selecl
`
`IRi
`=
`
`~ -
`
`"""
`
`Align Meshes
`
`Pick Po!nl!
`
`II (15)
`II (8)
`
`Poinl Grid II
`
`11
`
`..... _................
`102
`".)
`/
`/·
`\.
`!
`}
`.
`
`•
`
`Y5
`.S'l9
`4lJ,
`.I-
`
`,..
`
`-1-
`.at
`
`106
`
`Ready
`
`L
`
`j__
`
`0011
`
`

`

`
`
`:.
`"" .. ..c :g
`_3:32.55
`
`:ii:
`c:
`
`°' = <
`
`I
`
`g.
`'li!
`i5
`
`C'-· +1'
`J'
`~
`!:!l f§J
`~ ~ ~
`~
`m
`i.tl ~ c i::
`~
`
`U.S. Patent
`U.S. Patent
`
`Jun.26,2007
`Jun. 26, 2007
`
`Sheet 10 of 23
`Sheet 10 of 23
`
`US 7,234,937 B2
`US 7,234,937 B2
`
`x
`Cl
`
`)(
`Cl
`
`I D
`0
`D
`
`N
`0 ..-
`
`§:
`
`0..
`
`Ja
`·8
`£
`
`00 [i]
`
`10
`fl
`
`0012
`0012
`
`>.
`"O
`ra
`Q) a:
`
`~~~~ij@S~II
`
`~
`= c:=
`
`...... .
`(!) -LL
`
`

`

`U.S. Patent
`U. S. Patent
`
`Jun.26,2007
`Jun.26,2007
`
`Sheet 11 of 23
`Sheetll of23
`
`US 7,234,937 B2
`US 7,234,937 B2
`
`mmmnmemfii
`
`\
`
`'"""
`
`(")
`
`0013
`0013
`
`E
`E
`II
`~
`·r::
`=>
`
`EEu2_=23
`W383m
`
`C'\I
`~
`~
`C> co
`,.._
`a)
`11
`>.
`
`h _m
`
`C")
`10
`LO
`lO r.n
`~
`...,
`~
`' II x
`
`;::.-.
`"O
`Ctl
`a.>
`0:::
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jun.26,2007
`Jun. 26,2007
`
`Sheet 12 of 23
`Sheet 12 of 23
`
`US 7,234,937 B2
`US 7,234,937 B2
`
`IEEEW
`UEEEQ§Eufl=XmeGQmm:
`EEE.932.—Bmzfi95a:32256Efiningymgmlhmmfanmw
`
`)(
`IJ
`
`0
`cf
`D
`
`"a
`::::i _g
`0
`
`......
`""'"
`.....
`
`.
`C) -u...
`m.0.“—
`
`0014
`0014
`
`E
`E
`II
`
`;?;; -"E
`
`::::>
`
`N
`Ln
`r-
`0
`N
`N
`t<'i
`N
`1i ;::....
`
`~
`CD
`a>
`,.._
`CTJ
`cr:i co
`
`r1
`>C
`
`>-
`~
`CL> c:::
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jun.26,2007
`Jun.26,2007
`
`Sheet 13 of 23
`Sheet13 0f23
`
`US 7,234,937 B2
`US 7,234,937 B2
`
`E
`E
`
`II z -"E
`
`=:i
`
`~
`~
`
`co
`r..n
`tC
`N
`0
`co
`1l')
`,......
`II
`"""
`
`i
`
`"O
`::i
`R
`u
`
`
`sanm>w=5gwmwmsml882.3%.
`
`
`_J 0
`
`C")
`
`N
`0
`LO
`c<)
`..-
`Lri
`II x >.
`-c IB .
`cc I
`
`2.91
`
`0015
`0015
`
`

`

`U.S. Patent
`
`Jun. 26, 2007
`
`Sheet 14 of 23
`
`US 7,234,937 B2
`
`FIG.11A
`
`202
`
`FIG. 11 B
`
`FIG. 11 C
`
`FIG. 11 D
`
`FIG. 11E
`
`0016
`
`

`

`U.S. Patent
`
`Jun.26,2007
`
`Sheet 15 of 23
`
`US 7,234,937 B2
`
`FIG.12
`
`,,,.. __
`
`I
`I
`I
`202~
`I
`I
`I
`
`I ~ \
`
`208
`
`'
`..
`
`'
`
`'
`
`fJ.
`,
`'
`, ~ '
`,
`'
`,
`206
`,
`.I
`\
`- - - -
`,' /
`I
`,
`I
`I ~------
`1
`208 --...,.., -1---
`1
`I
`I
`
`' \. --------
`-.. .... _____ __
`
`I
`I
`I
`I
`
`206
`
`208
`
`210 __ _
`
`FIG. 13
`..... _,. __ .....
`
`r
`~---
`I
`
`'
`... .--..
`I
`202
`:
`r--_)
`I
`I
`I
`l
`I
`
`______ ._ ___ __
`,_
`
`_.,,
`
`212
`
`0017
`
`

`

`U.S. Patent
`
`Jun.26,2007
`
`Sheet 16 of 23
`
`US 7,234,937 B2
`
`222
`
`~220
`
`206
`
`FIG.14
`
`FIG. 15
`
`0018
`
`230
`
`(_____
`
`226
`
`

`

`U.S. Patent
`
`Jun. 26, 2007
`
`Sheet 17 of 23
`
`US 7,234,937 B2
`
`FIG.16A
`
`262
`
`264 ___.)
`
`FIG.16C
`
`262
`
`26B
`
`0019
`
`

`

`U.S. Patent
`
`Jun.26,2007
`
`Sheet 18 of 23
`
`US 7,234,937 B2
`
`FIG.17
`
`I
`I
`I
`I
`
`I 0----__3s2
`
`280
`
`276
`
`~
`
`266
`
`274
`
`x
`
`282
`
`z
`
`0020
`
`

`

`U.S. Patent
`
`Jun.26,2007
`
`Sheet 19 of 23
`
`US 7 ,234,93 7 B2
`
`FIG.18
`
`y
`
`262
`
`~o------
`
`2B4
`
`266
`
`~ _ ..
`
`--
`
`274
`
`.. -
`
`282
`
`270
`
`0021
`
`

`

`U.S. Patent
`
`Jun.26,2007
`
`Sheet 20 of 23
`
`US 7,234,937 B2
`
`FIG 19
`•
`r 28/35
`
`20 SCANNING DEVICES
`
`20 PICTURES: XRAYS,
`PHOTOGRAPHS
`
`( 302
`
`PATIENT HISTORY
`
`MEDICAUDENTAL
`DEMOGRAPHIC
`
`CUSTOM
`304~ DENTAUORTHODONTIC
`APPLIANCES
`B!;MQ~ABl.E
`ALIGNERS
`FUNCTIONAL-
`APPLIANCES
`RETAINERS
`DENTURE
`
`Elm
`BRACKETS
`BANDS
`ARCHWIRES
`RETAINERS
`CROWNS
`BRIDGES
`
`.____
`
`300
`
`PLANNING
`
`SOFTWARE
`
`'
`
`( 30/36
`
`3DSCANNING
`DEVICES
`
`3DMODELS:
`LASER
`SCANNER,
`WHITE LIGHT
`SCANNER,
`CT,MRI
`
`VARIABLE
`
`PATIENT:
`AGE, WEIGHT,
`FACIAL
`ATIRIBUTES
`'(SMILE, FROWN,
`ETC.)
`TEETH:
`TEXTURE,
`POSITION.
`LIGHT:
`REFLECTIVITY,
`AMBIENT
`INTENSITY,
`DIRECTION.
`
`\
`308
`
`1
`
`.;
`
`TAGGING
`I
`
`322.)
`320 \
`
`,,
`PARAMETERS H 'MORPHABLE' MODEL
`
`\.
`326
`KNOWLEDGE
`DATABASE
`. .,
`\
`~ ~ CRANIOFACIAL ANALYSIS I 316
`
`do
`
`I TREATMENT PLANNING/
`
`APPLIANCE SELECTION
`
`I
`
`328
`
`330
`
`PRACTICE MANAGEMENT
`TASK ALLOCATION
`APPOINTMENT SCHEDULING
`
`I\_
`312
`
`0022
`
`'\
`TREATMENT
`,.
`r:.i 30 MODEL I
`CREATION
`314) 1
`•
`l 30 GEOMETRY DATA I
`318) ..--
`
`CUSTOM
`APPLIANCE ~
`......._ MANAGEMENT I'- 315
`MANUFACTURING
`CATALOG
`
`I
`
`( 324
`
`*
`I APPLIANCE DATA ~
`I
`-;.-1 META-ANALYSIS I
`
`STANDARD
`DENTAU
`ORTHODONTIC
`APPLIANCES
`
`I
`
`f
`306
`
`-
`
`DIAGNOSIS
`AND
`SIMULATION
`
`TEETH
`POSITION,
`30 FACE,
`FACIAL
`ATTRIBUTES
`f
`
`310
`
`

`

`~
`r.n
`~
`
`~ '"'" ~ =
`'"'"
`.... = := ....
`!" ....
`g
`
`-i
`
`t:ll
`
`=" "' "' ... .... ...
`Q ... ....
`
`'M
`
`FIG.20
`
`14
`
`UNIFIED WORKSTATION
`
`402
`404
`
`408
`
`IMAGE DATA
`20
`30
`STATIC
`DYNAMIC
`COLOR MODELS
`OTHER
`
`MFG.#1
`MFG. #2
`MFG. #3
`FG.#4
`OTHER
`
`~ 422
`
`424
`
`426
`42B
`
`APPLIANCE
`MANUFACTURER
`#1
`
`APPLIANCE
`MANUFACTURER
`#2
`
`0023
`
`

`

`8 ·Nol yet Submitted
`' ~} ·Tiea.lmenl Plans
`~ !Setup I
`~ ·Selup
`• • Selup
`I±} ·Verification Plans
`: ffi .Progress/Update lvlodels
`ffi ·In Production al Oramelrix
`~ - To be Reviewed
`!±} ·Rnished
`'.· Canceled
`I B@le.J I By fVDi! ii By Date U
`Order Information
`Treatment Strategy
`Midfine and Aesthetic Occlusal Plane
`Occlusal Plane end AP Poslllons
`Reference Tooth
`Alignment 20 • 30
`Mandible Space Management
`Maxilla Space Management
`Space Management fMandlbte and
`Maxilla)
`Additional Instructions
`
`For Help, press F1
`
`FIG. 21
`
`505
`
`Dlgllal Treatment P annln
`
`~
`
`[Z]
`~
`!ill
`[gj
`
`~
`~
`
`-= a ~ = .....
`
`'"' c
`:=
`N
`F"
`N =
`= -...)
`
`.,,
`t't> .. N
`='
`t't>
`~ ...
`
`N
`
`N
`!M
`
`MldlinoSl!ll\
`Uppei
`
`lOM!r
`
`ill] nan• m
`10..~IJ
`[fil .... ill
`
`min
`
`501
`
`Length di.sCi18pancy
`
`fu\ ~r L ·l.7
`
`Righi
`0.2mm
`
`Lott
`1.4mm
`
`\......___/
`
`Orametrix AcUve Real &Order in Work
`
`0024
`
`

`

`FIG. 22
`
`505
`
`Space Management (Mandible and Maxllla)
`
`Ceph Image
`
`~
`
`!ZJ
`~
`181
`181
`
`~
`f'.l
`~ = ......
`~ = ......
`.... = :::i
`
`N
`~~
`
`N e e
`
`-..I
`
`C'-l =-~ !.
`= ....
`
`N
`(M
`
`N
`(M
`
`Camera NavtgaUon
`
`Mldlin&Shllt
`Upp•
`
`Lower
`
`,0-]
`!ID nOM ITJ
`[fil nooe m
`l°::m]
`
`m
`
`503
`
`Length dlscmpancy
`Right
`.0.9nvn
`
`(.\
`
`Righi
`0.2mm
`
`Left
`1.4mm
`
`Left
`·1.0mm
`
`\J
`
`B ·Not yet Submitted
`' G Treatment Plans
`~ I Setup J
`~ ·Serup
`' - Setup
`ffi .Yerificalion Plans
`,
`!±} -Progress/Update Models
`1±1 ·In Production at Orametrix
`• - To be Reviewed
`IB ·Finished
`' - - Canceled
`IBl'..Slam.!I B.Y.hpJ,JI BJ..Ra~I
`Order Information
`Treatment Strategy
`Mldllne and Aesthetic Occlusal Plane
`Occfusel Plane and AP Positions
`Reference Tooth
`Alignment 2D - 30
`Mandible Space Management
`MaxBla Space Management
`Space Management (Mandible and
`Maxllla)
`Addlllonal lnslruclions
`
`For f'lelp, press F1
`
`Orametrix AcUve Real & Order \n Work
`
`0025
`
`

`

`US 7,234,937 B2
`
`1
`UNIFIED WORKSTATION FOR VIRTUAL
`CRANIOFACIAL DIAGNOSIS, TREATMENT
`PLANNING AND THERAPEUTICS
`
`RELATED APPLICATIONS
`
`BACKGROUND OF THE INVENTION
`
`This application claims priority benefits pursuant to 35
`U .S.C. § 120 as a continuation-in-part of application Ser.
`No. I 0/340,404 filed Jan. 9, 2003, now abandoned which is
`a continuation of application Ser. No. 09/560,641, filed Apr.
`28, 2000, now U.S. Pat. No. 6,512,994, which is a continu(cid:173)
`ation-in-part of application Ser. No. 09/452,034 filed Nov.
`30, 1999, abandoned. This application also claims priority
`benefits pursuant to 35 U.S.C. § 120 as a continuation-in(cid:173)
`part of application Ser. No . 09/835,039 filed Apr. 13, 2001,
`now U.S. Pat. No. 6,648,640. The entire contents of the
`related applications are fully incorporated by reference
`herein.
`This application is also related to a patent application filed
`on the same date as this application, inventors Rohit Sach(cid:173)
`deva et al., entitled INTERACTlVE UNIFIED WORKSTA(cid:173)
`TION FOR BENCHMARKING AND CARE PLANNING,
`Ser. No. 10/429,074, pending the entire contents of which
`are incorporated by reference herein.
`This application is also related to a patent application filed
`on the same date as this application, inventors Rohit Sach(cid:173)
`deva et al., entitled METIIOD AND SYSTEM FOR INTE(cid:173)
`GR..'\.TED ORTHODONTIC TREATMENT PLANNING
`USING UNIFIED WORKSTATION, Ser. No. 10,429,074,
`pending the entire contents of which are incorporated by
`reference herein.
`
`2
`including the straightening of teeth, have been proposed. See
`Andreiko, U.S. Pat. No. 6,015,289; Snow, U.S. Pat. No.
`6,068,482; Kopelmann et al., U.S. Pat. No. 6,099,314;
`Doyle, et al., U.S. Pat. No. 5,879,158; Wu el al., U.S. Pat.
`No. 5,338,198, and Chisti et al., U.S. Pat. Nos. 5,975,893
`and 6,227,850, the contents of each of which is incorporated
`by reference herein. Also, imaging and medical diagnostic
`software and related products are marketed by Dolphin
`Imaging, 661 Independence Avenue, Canoga Park, Calif.
`10 91309-2944. A method for generation of a 3D model of the
`dentition from an in-vivo scan of the patient, and interactive
`computer-based treatment planning for orthodontic patients,
`is described in published PCT patent application of OraMe(cid:173)
`trix, Inc., lhe assignee of this invention, publication no. WO
`15 01/807 6 L the contents of which are incorporated by refer(cid:173)
`ence herein. Other background references related to captur(cid:173)
`ing three dimensional models of dentition and associated
`craniofacial structures include S. M. Yamany and A. A.
`Farag, "A System for Human Jaw Modeling Using Intra-
`20 Oral Images" in Proc. IEEE Eng. Med. Biol. Soc. (EMBS)
`Conj, Vol. 20, Hong Kong, October 1998, pp. 563-566; and
`M. Yamany, A. A. Farag, David Tasman, A.G. Farman, "A
`3-D Reconstruction System for the Human Jaw Using a
`Sequence cifOplical Images," IEEE Transactions on Medi-
`25 cal Imaging, Vol. 19, No. 5, May 2000, pp. 538-547. The
`contents of these references are incorporated by reference
`herein.
`The technical literature further includes a body of litera(cid:173)
`ture describing the creation of 3D models of faces from
`30 photographs, and computerized facial animation and mor(cid:173)
`pha ble modeling of faces. See, e.g., Pighin et al., Synthe(cid:173)
`sizing Realistic Facial Expression from Photographs, Com(cid:173)
`puter Graphics Proceedings SIGGRAPH '98, pp. 78-94
`(1998); Pighin et al., Realistic Facial Animation Using
`A. Field of the Invention
`35 Image-based 3D Morphing, Technical Report no. UW-CSE-
`lbis invention relates to the field of computerized tech(cid:173)
`97-01-03, University of Washinton (May 9, 1997); and
`niques for diagnosis and planning medical and dental treat(cid:173)
`Blantz et al., A Morphable Model for The Synthesis of 3D
`ment of human patients. More particularly, the invention is
`Faces, Computer Graphics Proceedings SIGGRAPH '99
`directed to a unified workstation and associated computer(cid:173)
`(August, 1999). The contents of these references are incor(cid:173)
`ized techniques for creating a virtual three-dimensional 40
`porated by reference herein.
`model of the patient, including bone, soft tissue, and teeth
`The art has lacked a truly integrated and unified system in
`from data from a variety of diverse imaging sources. The
`which soft tissue (skin, lips, etc.) and the underlying bone
`invention is also related to computer software tools enabling
`and other craniofacial features, including leelh, are super(cid:173)
`a user to use such a virtual model for diagnosis and planning
`imposed and registered together in a common coordinate
`treatment of craniofacial structures of the patient, including 45
`system to create a complete virtual patient model that also
`teeth, and for export of data to diverse manufacturers of
`includes the exterior appearance of the patient, and in which
`therapeutic devices for the patient. such as orthodontic
`the user is provided with tools to study the interaction of
`appliances.
`such craniofacial features to each other and to simulate with
`B. Description of Related Art
`a computer changes in craniofacial features (such as by
`The diagnosis and treatment of patients with craniofacial 50
`means of proposed tooth extraction, orthodontic manipula(cid:173)
`problems or disease typically begins with the obtaining of
`tion, or surgery) and their effects on the external, visual
`clinical history, medical history, dental history, ultrasonic
`appearance of the patient, and design optimal therapeutics
`scanned images, 2D or 3D scanned images, photographs,
`based upon lhe unified virtual patient.
`and 2D or 3D X-rays. Such X-rays are taken from the front
`A principal benefit of the invention is that it provides a
`and the side view. X-rays are also taken to show the 55
`powerful tool to the physician, dentist or orthodontist for
`condition of the teeth and the jaws. At this stage, diagnosis
`diagnosis and treatment planning. The unified workstation
`and treatment planning is often done by the practitioner on
`provides comprehensive, multiple functionalities in the
`a sheet of acetate over the X-rays. Generally, this process is
`same unit, thus eliminating the need for more expensive and
`nol very scientific, and ii is time consuming and requires
`less efficient multiple workstations wherein each worksta-
`experience. There is no guarantee how good the results of 60
`tion is dedicated to performing one specific function or a
`the treatment will be. Similarly, orthodontists typically men(cid:173)
`limited sub-set of functions necessary for the practitioner's
`tally visualize a target or desired occlusion for an orthodon-
`practice. Moreover, the three-dimensional virtual patient
`tic patient and attempt to bend archwires by hand to move
`model described herein is useful datum for use in a diverse
`teeth lo the desired position. This approach also lacks
`set of possible treatment regimes for treatment of the patient.
`reliability, reproducibility and precision.
`65 As such, the virtual patient model (or perhaps some subset
`of data from the model) can be provided or exported to
`More sophisticated, computer-based approaches to diag(cid:173)
`nosis and treatment planning of craniofacial structures,
`manufacturers of appliance systems for their use in design-
`0026
`
`

`

`US 7,234,937 B2
`
`3
`ing and/or fabricating customized appliances for treatment
`of the patient, e.g., customized orthodontic appliances.
`
`SUMMARY OF THE INVENTION
`
`In a first aspect, a system for use in diagnosis and planning
`treatment of a human patient is provided. TI1e system
`includes a general-purpose computer system having a pro(cid:173)
`cessor (e.g., central processing unit) and a user interface.
`The details of the computer system are not important. A 10
`memory is provided which is accessible to the general(cid:173)
`purpose computer system, such as a hard disk or a file server
`on a network to which the general-purpose computer is
`connected. The memory stores a first set of digital data
`representing patient craniofacial image information obtained 15
`from a first imaging device. For example, the first set of
`digital data may be 3-D scan data obtained from a scan of the
`patient's face using a scanner, 3D scan data from a scan of
`the dentition of the patient, X-ray data, CT scan, MRI, video,
`a set of two-dimensional digital color photographs of the 20
`patient, etc. The memory further includes a second set of
`digital data representing patient craniofacial image informa(cid:173)
`tion obtained from a second image device different from the
`first image device. For example, if the first set of data
`represents CT scan data the second set of data may represent 25
`3D sC<ln dllta of the teeth of the ratient. 'l11e first and second
`sets of data represent, at least in part, common craniofacial
`anatomical structures of the patient. In other words, there are
`some anatomical features that are conunon to the two sets of
`data; they overlap 10 some extent. One of the fil'Sl and second 30
`sets of data will typically include data representing the
`surface configuration or external appearance of the patient's
`face, for example a two dimensional digital photograph of
`the face (black and white or color), a 3D scan of the face, or
`other face data.
`The system further includes a set of computer instructions
`stored on a machine-readable storage medium accessible to
`said general-purpose computer system. The computer
`instructions need not necessarily be stored on the same
`memory as the first and second sets of data. In the i.llustratcd
`embodiment, rhe instmctioas are stored in the bard disk
`memory of the general-purpose computer system and arc
`executed by the computer's host processor, but that need not
`always be the case. The set of instructions cause the general
`purpose computer system to perform several tasks:
`1) Firstly, automatically, and/or with the aid of operator
`interaction, the set of instructions includes instruction
`that operate to superimpose the first set of digital data
`and the second set of digital data so as to provide a
`composite, combined digital representation oftbe crnn(cid:173)
`iofacial anatomical stnicturcs in a common. Preferably,
`but not necessarily, this representation will be a three(cid:173)
`dimensional representation in a common 3D coordinate
`system. This representation is referred to herein occa(cid:173)
`sionally as a "virtual patient model." In this aspect, the 55
`techniques of creation of a 3-D model disclosed in the
`patent application of Rohit Sachdeva et al., Ser. No.
`09/560,641 filed Apr. 28, 2000 may be employed.
`Scaling techniques may be used to scale the data from
`one set of images to the other so as to created correctly 60
`scaled composite model that accurately reflects the
`patient's anatomy.
`2) Secondly, the instructions include instructions for dis(cid:173)
`playing the composite, combined digital representation
`of the craniofacial anatomical structures to a user of the 65
`system, for example on the user interface of the general
`purpose computer system.
`
`45
`
`50
`
`4
`Preferably, the instructions include instructions providing
`the user with tools on the user interface for visually studying
`the interaction of the craniofacial anatomical structures and
`their relationship to the external, visual appearance of the
`patient. The set of tools including tools for simulating
`changes in the anatomical position or shape of the cranio(cid:173)
`facial anatomical structures and measuring their effect on the
`external, visual appearance of the patient.
`In a representative embodiment, 3D data of the face, skull
`and jaw is obtained from various scanning or imaging
`devices (CT scan, X-Ray, color photographs) and stored in
`the memory. Then, the general-purpose computer superim-
`poses the data to place all the data in one common coordi(cid:173)
`nate system to create a virtual patient model. Scaling of the
`data may be performed in this step. The virtual patient model
`is displayed to the user of the system. The software instruc-
`tions in the system provide modeling or "morphing" tools
`which allow the user to manipulate various parameters and
`simnlate the effect of such changes on the appearance of the
`patient, such as the position of one or more teeth or jaw, the
`shape of the arches, the age of the patient; the color and
`texture of the teeth; and the reflectivity and an1bient condi(cid:173)
`tions of the light shining on the patient.
`In another aspect of this invention an onbodontic treat-
`ment planning system is provided comprising a 30 scanner
`for scanning the dentition of the patient, a general-purpose
`computer receiving scan data from the scanner and respon(cid:173)
`sively generating a three-dimensional virtual model of the
`dentition of the patient, and software stored on a machine(cid:173)
`readable memory accessible 10 the general-purpose com(cid:173)
`puter. The software contains instructions for combining,
`either automatically or with the aid of an operator, scan data
`from the scanner with digital data of the facial appearance of
`the patient. The digital data oft.be Incial appearance of the
`35 patient can be obtained from a variety of sources, such as
`color digital camera or from a scanning of the face with the
`3D scanner. 111e software combines (e.g., superimposes) the
`two sets of data to thereby create a combined digital three(cid:173)
`dimellsional representatioll of the deolition and ihe facial
`40 appearance in a common three-dimensional coordinaie sys(cid:173)
`tem.
`The software further includes instructions providing the
`user with tools to manipulate the position of the virtual teeth
`in the three-dimensional virtual model of the dentition
`relative to other anatomical structures of the patient and to
`visualize the effect of proposed changes in tooth position on
`the facial appearance of the patient. Thus, the tools thereby
`provide the user with the ability to design with the computer
`a·dcsired three-dimensional configuration of the virtual teeth
`while viewing the effect of chauging tooth position on t.be
`visual appearance of the face of the patient.
`In a preferred embodiment, the scanner comprises a
`hand-held, three-dimensional optical scanner. The digital
`data of the facial appearance of the patient can be obtained
`from the hand-held, three-dimensional optical scanner,
`thereby obviating the need for any other data acquisition
`devices. On the other hand the digital dala could be obtained
`from a color camera, a video camera, or other type of
`imaging or scanning device. Other types of imaging devices
`could be used, such as radiographic images, CAT scan
`images, or MRI images.
`In one possible embodiment, the system can include
`software combining the digital three-dimensional represen(cid:173)
`tation of the dentition and facial appearance with X-ray data
`superimposed on the scan data and the digital data of the
`facial appearance of the patient.
`0027
`
`

`

`US 7,234,937 B2
`
`6
`which a practitioner can integrate the acquisition of data, the
`treatment plan, and the appliance design and manufacture
`into one seamless system.
`
`5
`
`BRJEF DESCRJPTION OF THE DRAWINGS
`
`Presently preferred embodiments of the invention are
`described below in reference to the appended drawings,
`wherein like reference numerals refer to like elements in the
`various views, and in which:
`FIG. 1 is block diagram of a system for creating a
`tlrree-dimensional virtual patient model and for diagnosis
`and planning treatment of the patient.
`FIG. 2 is a flow chart showing a method of tlrree(cid:173)
`dimensional face creation from scanning systems, which
`may be executed in software in the computer system of FIG.
`L
`
`5
`With the system of this invention, the elements of the
`craniofacial dental complex can be analyzed quickly in
`either a static or dynamic fonnat, using the unified work(cid:173)
`station and simulation tools provided in software in the
`workstation. The virtual patient model enables the simula-
`tion of facial expressions such as smiling, grimacing, the
`aging of the patient, and functional movements such as
`chewing and other complex motions of the jaw, in both a
`static manner and in a dynamic manner. For example, the
`virtual patient model is displayed and current smile of the 1 o
`patient is viewed, and changes to the smile are simulated, as
`for example by the simulation of tooth movement and its
`effect on soft tissue, lips etc. and its effect on the smile. The
`simulations could be performed as a dynamic simulation, in
`which the series of changes in tooth position (intermediate 15
`positions), and their effect on soft tissue during the smile, is
`demonstrated in a manner showing the motion of the teeth
`FIG. 3 is a flow chart showing an alternative method of
`and tissues. Alternatively, the simulations could be static, for
`tlrree-dimensional face model face creation using a plurality
`example, movement of one or more teeth from one position
`of possible input image or data formats, which may be
`to another is performed, and the virtual patient model is 20
`executed in software in the computer system of FIG. L
`shown with the effect on that movement on the change in
`FIG. 4 is a flow chart showing a method of creating a
`soft tissue configuration (e.g., lip) or on the overall smile.
`complete textured tlrree-dimensional model of teeth; simu(cid:173)
`There is also a possibility of simulations in between purely
`lations, either static or dynainic, can be performed with the
`static simulations and dynamic simulalions, such as stepping
`textur