`
`Application Information
`
`Application Number:
`
`Filing Date:
`
`Application Type:
`
`Provisional
`
`Subject Matter:
`
`Utility
`
`Suggested Classification:
`
`Suggested Group Art Unit:
`
`CD-ROM or CD-R?::
`
`None
`
`Number of CD Disks:
`
`Number of Copies of CDs:
`
`Sequence Submission?:
`
`Computer Readable Form (CRF)?: No
`
`Number of Copies of CRF:
`
`Title:
`
`2D IMAGE ARRANGEMENT
`
`Attorney Docket Number:
`
`0079124—000004
`
`Request for Early Publication?:
`
`Request for Non—Publication?:
`
`No
`
`No
`
`Suggested Drawing Figure:
`
`Fig. 1
`
`Total Drawing Sheets:
`
`Small Entity?::
`
`Latin Name:
`
`16
`
`No
`
`Variety Denomination Name:
`
`Petition lncluded?:
`
`No
`
`exocad GmbH, et. al.
`Exhibit 1004 - Part 2
`
`Page # 1
`
`0401
`
`Initial 03/18/11
`
`0401
`
`
`
`Petition Type:
`
`Licensed US Govt. Agency:
`
`Contract or Grant Numbers:
`
`Secrecy Order in Parent Appl.?::
`
`No
`
`Applicant Information
`
`Applicant Authority Type:
`
`Primary Citizenship Country:
`
`Status:
`
`Inventor
`
`Sweden
`
`Full Capacity
`
`Given Name:
`
`Tais
`
`Middle Name:
`
`Family Name:
`
`Name Suffix:
`
`CLAUSEN
`
`City of Residence:
`
`Klagshamn
`
`State or Province of Residence:
`
`Country of Residence:
`
`Sweden
`
`Street of Mailing Address:
`
`Signe Lofdahls vag 12
`
`City of Mailing Address:
`
`Klagshamn
`
`State or Province of Mailing
`Address:
`
`Country of Mailing Address:
`
`Postal or Zip Code of Mailing
`Address:
`
`Applicant Authority Type:
`
`Primary Citizenship Country:
`
`Sweden
`
`21851
`
`Inventor
`
`Denmark
`
`Status:
`
`Full Capacity
`
`Page # 2
`
`0402
`
`initial 03/18/11
`
`0402
`
`
`
`Given Name:
`
`Rune
`
`Middle Name:
`
`Family Name:
`
`Name Suffix:
`
`FiSKER
`
`City of Residence:
`
`Virum
`
`State or Province of Residence:
`
`Country of Residence:
`
`Denmark
`
`Street of Mailing Address:
`
`Kaplevej 87
`
`City of Mailing Address:
`
`Virum
`
`State or Province of Mailing
`Address:
`
`Country of Mailing Address:
`
`Postal or Zip Code of Mailing
`Address:
`
`Applicant Authority Type:
`
`Primary Citizenship Country:
`
`Status:
`
`Given Name:
`
`Middle Name:
`
`Family Name:
`
`Name Suffix:
`
`Denmark
`
`2830
`
`Inventor
`
`Sweden
`
`Fuli Capacity
`
`Nikolaj
`
`DEICHMANN
`
`City of Residence:
`
`Klagshamn
`
`State or Province of Residence:
`
`Country of Residence:
`
`Sweden
`
`Street of Mailing Address:
`
`Jarnolofs Vég 10
`
`Page # 3
`
`0403
`
`Initial 03/18/11
`
`0403
`
`
`
`City of Mailing Address:
`
`Klagshamn
`
`State or Province of Mailing
`Address:
`
`Country of Mailing Address:
`
`Sweden
`
`Postal or Zip Code of Mailing
`Address:
`
`21851
`
`Correspondence Information
`
`Correspondence Customer Number: 21839
`
`Phone Number:
`
`Fax Number:
`
`(703) 836-6620
`
`(703) 836—2021
`
`Representative Information
`
`Representative Customer Number:
`
`21839
`
`Assignee Information
`
`Assignee Name:
`
`3Shape A/S
`
`Street of Mailing Address:
`
`Holmens Kanal 7, 4. saE
`
`City of Mailing Address:
`
`Copenhagen K
`
`State or Province of Mailing
`Address:
`
`Country of Mailing Address:
`
`Denmark
`
`Postal or Zip Code of Mailing
`Address:
`
`1060
`
`
`
`Signature
`'% M Date
`March 18, 2011
`
`
`
`l Name
`
`l William C. Rowland
`
`Registration No.
`
`30888
`
`1
`
`Page # 4
`
`0404
`
`Initial 03/18/11
`
`0404
`
`
`
`1/16
`
`101
`
`102
`
`104
`
`103
`
`Fig. 1
`
`0405
`
`0405
`
`
`
`2/16
`
`202
`
`208
`
`207
`
`EMmM
`
`
`
`203
`
`Ffig.2a)
`
`0406
`
`0406
`
`
`
`202
`
`3/16
`
`207
`
`\
`
`201////1
`
`204
`
`Fig. 2b)
`
`0407
`
`0407
`
`
`
`4/16
`
`
`. " ‘R! - - Efiufiwrm: S‘F‘ }'§(€q‘.¢'§¥‘~tfi&&3&‘$§&d&&§“¥“LV- M:
`
`302
`
`307
`
`
`
`
`
`~:\\w.~.~.
`_
`.
`“Nu“ *
`
`
`
`W‘\<\\\\\\§\
`
`
`
`. 3.06:
`
`Ffig.3a)
`
`0408
`
`0408
`
`
`
`5/16
`
`
`
`Fig. 3b)
`
`0409
`
`0409
`
`
`
`6/16
`
`301
`
`302
`
`Ffi; 3Q)
`
`0410
`
`0410
`
`
`
`Ffi; 4a)
`
`Fig. 4b)
`
`k
`
`0411
`
`0411
`
`
`
`
`8/16
`
`
`
`Ffl; 4c)
`
`0412
`
`0412
`
`
`
`9/16
`
`509
`
`510
`
`511
`
`512
`
`
`
`506
`
`505
`
`FR; 58)
`
`0413
`
`0413
`
`
`
`Fig. 5b)
`
`509
`
`10/16
`
`
`
`509
`
`
`\514
`
`Fig. 5c)
`
`0414
`
`0414
`
`
`
`Fig 5d)
`
`11/16
`
`
`
`515
`
`501
`
`
` \&
`
`M,
`
`Fig_ 5e)
`
`0415
`
`0415
`
`
`
`
`
`0416
`
`0416
`
`
`
`13/16
`
`
`
`Ffi; 6a)
`
`0417
`
`0417
`
`
`
`x
`
`FWg.6b)
`
`0418
`
`0418
`
`
`
`15/16
`
`
`
`701
`
`
`
`
`
`0419
`
`0419
`
`
`
`_
`Ffig.9
`
`901
`
`16/16
`
`w
`
`
`
`\\\\ \x
`
`Fig. 10
`
`1001
`
`
`
`0420
`
`0420
`
`
`
`2D image arrangement
`
`Field of the invention
`
`This
`
`invention generally relates to a computer—implemented method of
`
`visualizing and modeling a set of teeth for a patient. More particularly, the
`
`invention relates to providing a 3D virtual model of the patient’s set of teeth.
`
`10
`
`Background of the invention
`
`Visualization and modeling or design of teeth are known in the field of dental
`
`restorations.
`
`When a patient requires a dental
`
`restoration, such as crowns, bridges,
`
`abutments, or implants,
`
`the dentist will prepare the teeth e.g. a damaged
`
`tooth is grinded down to make a preparation where a crown is glued onto. An
`
`alternative treatment is to insert implants, such as titanium screws,
`
`into the
`
`jaw of
`
`the patient and mount crowns or bridges on the implants. After
`
`preparing the teeth or
`
`inserting an implant
`
`the dentist can make an
`
`impression of the upper jaw, the lower jaw and a bite registration or a single
`
`impression in a double-sided tray,
`
`also known as triple trays. The
`
`impressions are sent
`
`to the dental
`
`technicians who manufacture the
`
`restorations 9.9. the bridge. The first step to manufacture the restoration is
`
`traditionally to cast the upper and lower dental models from impressions of
`
`the upper and the lower jaw, respectively. The models are usually made of
`
`gypsum and often aligned in a dental articulator using the bite registration to
`
`simulate the real bite and chewing motion. The dental technician builds up
`
`the dental
`
`restoration inside the articulator
`
`to ensure a nice visual
`
`appearance and bite functionality.
`
`15
`
`2O
`
`25
`
`30
`
`0421
`
`0421
`
`
`
`CAD technology for manufacturing dental restoration is rapidly expanding
`
`improving quality, reducing cost and facilitating the possibility to manufacture
`
`in attractive materials otherwise not available. The first step in the CAD
`
`manufacturing process is to create a 3—dimensional model of the patient's
`
`teeth. This is traditionally done by 3D scanning one or both of the dental
`
`gypsum models. The 3—dimensional replicas of the teeth are imported into a
`
`CAD program, where the entire dental
`
`restoration,
`
`such as a bridge
`
`substructure,
`
`is designed. The
`
`final
`
`restoration 3D design is
`
`then
`
`manufactured e.g. using a milling machine, 3D printer,
`
`rapid prototyping
`
`manufacturing or other manufacturing equipment. Accuracy requirements for
`
`the dental restorations are very high otherwise the dental restoration will not
`
`be visual appealing, fit onto the teeth, could cause pain or cause infections.
`
`WO1OOS1404A relates to tools in a system for the design of customized
`
`three—dimensional models
`
`of
`
`dental
`
`restorations
`
`for
`
`subsequent
`
`manufacturing, where the dental restorations are such as implant abutments,
`
`copings, crowns, wax—ups, and bridge frameworks. Moreover, the invention
`
`relates to a computer-readable medium for implementing such a system on a
`
`computer.
`
`Visualizing and modeling teeth for a patient based are also known from the
`
`field of orthodontics.
`
`U82006127836A discloses orthodontic systems and methods for determining
`
`movement of a tooth model from a first position to a second position by
`
`identifying one or more common features on the tooth model; detecting the
`
`position of the common features on the tooth model at the first position;
`
`detecting the position of the common features on the tooth model at the
`
`second position; and determining a difference between the position of each
`
`common feature at the first and second positions.
`
`10
`
`15
`
`2O
`
`25
`
`30
`
`0422
`
`0422
`
`
`
`Thus orthodontics relates to movement of teeth, so the desired position of a
`
`tooth or teeth is determined, and based on the present position of that tooth
`
`or teeth, the movement from the present position to the desired position is
`
`determined. Thus within orthodontics the desired or resulting position of a
`
`tooth or teeth is/are is known before planning the steps of the movement.
`
`It remains a problem to provide an improved method and system for
`
`providing esthetically beautiful and/or physiologically suitable results of
`
`modeling teeth, both within the field of restorations,
`
`implants, orthodontics
`
`10
`
`etc.
`
`My
`
`15
`
`2O
`
`Disclosed is a computer-implemented method of visualizing, designing and
`
`modeling a set of teeth for a patient, wherein the method comprises the steps
`
`of:
`
`— providing one or more 2D digital images;
`
`- providing a 3D virtual model of at least part of the patient’s oral cavity;
`
`- arranging at least one of the one or more 2D digital images relative to the
`
`3D virtual model in a 3D space such that the at least one 2D digital image
`
`and the 3D virtual model are aligned when viewed from a viewpoint, whereby
`
`the 3D virtual model and the at least one 2D digital image are both visualized
`
`in the 3D space; and
`
`- modeling the 3D virtual model based on at least one of the one or more 2D
`
`25
`
`digital images.
`
`Consequently, it is an advantage that the 3D CAD modeling of the 3D virtual
`
`model
`
`is based on a 2D digital
`
`image, since the 2D image determines or
`
`indicates what kind of modeling is suitable, where the expression suitable
`
`30
`
`may comprise physiologically suitable or esthetically suitable or appealing.
`
`Thus the 2D image is used to perform a correct modeling of the 3D model,
`
`0423
`
`0423
`
`
`
`since the 2D image functions as a benchmark or rule for what kind of
`
`modeling is possible or how the modeling can be with the limits provided by
`
`the 2D image. Thus the modeling of the 3D virtual model
`
`is decided and
`
`performed based on the one or more 2D image,
`
`i.e. such as that
`
`the
`
`modeling of the 3D virtual model
`
`is based on the visualization of the 2D
`
`image.
`
`10
`
`15
`
`2O
`
`25
`
`30
`
`The patient’s oral cavity may comprise at least the patient’s present set of
`
`teeth, such as prepared teeth or unprepared teeth,
`
`if
`
`the patient
`
`is not
`
`toothless, and maybe part of the gums.
`
`If the patient is toothless, then the
`
`oral cavity may comprise the gums of the patient.
`
`It
`
`is an advantage that the 2D digital image and the 3D virtual model are
`
`aligned when viewed from one viewpoint, since hereby the user or operator
`
`of the system performing the method, can view the 2D image and the 3D
`
`model from a viewpoint where they are aligned, since this enables and
`
`facilitates that modeling of the 3D model is based on the 2D image. That the
`
`2D image and 3D model are aligned when seen from a viewpoint means that
`
`at least some structures of the 2D image and the 3D model are coinciding
`
`when seen from a viewpoint. Thus the 2D image and 3D model may not be
`
`aligned when seen from any viewpoint, thus there may be only one viewpoint
`
`from which the 2D image and the 3D model are aligned.
`
`Furthermore,
`
`it
`
`is an advantage that the 2D image and the 3D model are
`
`arranged and remain as separate data representations which are not merged
`
`or
`
`fused
`
`together
`
`into
`
`one
`
`representation. By
`
`keeping
`
`the data
`
`representations as separate representations,
`
`time is saved and data
`
`processing time and capacity is
`
`reduced. Thus the 2D image is not
`
`superimposed or overlaid onto the 3D virtual model
`
`for creating one
`
`representation with all data included. Prior art documents describe that the
`
`data from e.g. a color image is added to the 3D model, such that the color
`
`0424
`
`0424
`
`
`
`content from the image is transferred to the 3D model, whereby the result is
`
`one representation, i.e. the 3D model including color. Creating such models
`
`requires more time and exhaustive data processing.
`
`Thus,
`
`it is an advantage that the present method may be performed faster
`
`than prior art methods.
`
`The method is for use when modeling teeth, but can of course also with
`
`advantage be used by students within the dental field when learning how to
`
`model teeth and what to take into consideration when modeling teeth.
`
`10
`
`Modeling of teeth is defined as comprising modeling of one or more dental
`
`restorations, modeling of one or more implants, modeling orthodontic
`
`movement of one or more teeth, modeling one or more teeth in a denture,
`
`e.g. a fixed or removable denture, to provide a visually pleasing appearance
`
`15
`
`of the set of teeth etc.
`
`Thus the modeling may comprise modeling of
`
`restorations, orthodontic
`
`planning and/or treatment, modeling of implants, modeling of dentures etc.
`
`When the CAD modeling comprises for example restorations,
`
`the virtually
`
`modeled restorations, such as crowns and bridges, can be manufactured by
`
`means of CAM, and the manufactured restorations can then eventually be
`
`inserted onto the patient’s teeth by a dentist.
`
`Arranging, placing, or positioning the 2D digital image on the 3D virtual model
`
`is performed digitally on a computer and shown on a user interface such as a
`
`screen, such that the user or operator obtains a visual representation of the
`
`2D image and the 3D model together in the same field of view, whereby the
`
`operator can perform the modeling based on the simultaneous view of the 2D
`
`image and the 3D model
`
`instead of based on either one combined
`
`representation or separate views of the 2D image and/or the 3D model.
`
`2O
`
`25
`
`30
`
`0425
`
`0425
`
`
`
`For facilitating the arrangement of the 2D image and the 3D model relative to
`
`each other, edge detection may be performed, whereby the contour of the
`
`teeth on the 2D image and/or on the 3D model is automatically derived. Edge
`
`detection can be performed by means of a software algorithm. Edges are
`
`points where there is a boundary or edge between to image regions, and
`
`edges can thus be defined as sets of points in the image which have a strong
`
`gradient magnitude. The contour of the teeth may thus be detected by
`
`detecting the edge between image portions showing the teeth and the
`
`gingival.
`
`One or more 2D images may be provided in the method, and the 2D images
`
`may e.g. show the patient’s face from different directions, show different
`
`parts of the patient’s face, such as the lips and the eyes or nose for example
`
`for determining facial lines, show different examples of new teeth which the
`
`teeth of the 3D model can be modeled to look like, show the patient’s teeth
`
`before preparing the teeth for restorations and after preparing the teeth, etc.
`
`When aligning the 2D image and the 3D model, the 2D image may be of the
`
`patient’s unprepared teeth, since it may be easier to align the 2D image and
`
`the 3D model, when the teeth on the 2D image are unprepared. When
`
`modeling the teeth of the 3D model,
`
`the 2D image may then be of the
`
`patient’s prepared teeth, since e.g. restorations normally are modeled after
`
`having prepared the teeth by cutting part of the teeth such that crowns etc.
`
`can be attached to the prepared part of the teeth.
`
`In some embodiments the 3D virtual model
`
`is generated by scanning a
`
`physical model of the patient’s teeth, by scanning an impression of the
`
`patient’s teeth, and/or by performing a direct scanning of the patient’s teeth. If
`
`the patient is toothless, then the gums, a model or an impression of the gums
`
`may be scanned for creating a 3D model of the oral cavity.
`
`10
`
`15
`
`2O
`
`25
`
`30
`
`0426
`
`0426
`
`
`
`In 3D scanning the object
`
`is analyzed to collect data on its shape. The
`
`collected data can then be used to construct digital,
`
`three dimensional
`
`models.
`
`In 3D scanning usually a point cloud of geometric samples on the
`
`surface of
`
`the subject
`
`is created. These points can then be used to
`
`extrapolate the shape of the subject.
`
`In some embodiments the one or more 2D digital image comprises a patient—
`
`specific image of at least part of the patient’s face.
`
`An advantage of this embodiment is that the modeling can be based on an
`
`image of the patient, such that the modeling is performed with respect to the
`
`look or appearance of the patient, or with respect to some, a few or a single,
`
`specific visual features of the patient, such as the lips.
`
`In some embodiments the one or more 2D digital image comprises a generic
`
`image of at least part of a human face.
`
`An advantage of this embodiment is that the modeling can be based on a
`
`generic image, whereby it is not patient-specific features which determine the
`
`modeling, but instead it
`
`is a general image, e.g. of some visually pleasing
`
`teeth from another person, or a drawing of some ideal teeth.
`
`In some embodiments the one or more 2D digital image is retrieved from a
`
`library comprising a number of images of teeth.
`
`An advantage of this embodiment is that the 2D image, such as a generic
`
`image, can be selected from a library which contains for example several
`
`images of teeth, so that the patient e.g. can choose his/her desired new set
`
`of teeth from the library. The library may be a so called smile guide library
`
`comprising images of teeth and/or mouths which are shown while smiling,
`
`since visually pleasing teeth may be most important when smiling, since this
`
`may be when most teeth are shown to the surroundings.
`
`The images of teeth in the library may be photos of teeth, may be drawings of
`
`teeth, etc.
`
`10
`
`15
`
`2O
`
`25
`
`30
`
`0427
`
`0427
`
`
`
`In some embodiments the one or more 2D digital
`
`image is a template for
`
`supporting designing and/or modeling the patient’s teeth.
`
`An advantage of this embodiment is that when the 2D image is a template,
`
`then the operator can arrange and model
`
`teeth using this template for
`
`obtaining a visually pleasing result of the modeling.
`
`In some embodiments the template comprises the midline of a face.
`
`In some embodiments the template comprises a horizontal line passing along
`
`the anterior teeth.
`
`In some embodiments the template comprises the occlusal plane of a face.
`
`An advantage of
`
`the embodiments where the template comprises some
`
`feature, such as the midline of the face, a horizontal line, an occlusal plane
`
`etc, is that these features may assist in arranging the 2D image and the 3D
`
`model relative to each other and in modeling of the 3D model.
`
`In some embodiments the template comprises boxes adapted to fit
`
`the
`
`centrals, the laterals and the cuspids.
`
`An advantage of this embodiment is that it enables the operator to easily
`
`model the different anterior teeth to be visually pleasing. For example the
`
`laterals can with advantage be 2/3 of the width of the centrals, and the
`
`cuspids or canines can with advantage be slightly narrower than the centrals.
`
`In some embodiments the template comprises one or more long axes of
`
`anterior teeth.
`
`An advantage of this embodiment is that the long axes can be used for
`
`indicating the long axis alignment of teeth and/or the vertical direction of teeth
`
`for support in modeling.
`
`10
`
`15
`
`2O
`
`25
`
`30
`
`0428
`
`0428
`
`
`
`In some embodiments the long axes of at least the upper anterior teeth
`
`converge toward the incisal edge or biting edge.
`
`An advantage of this embodiment is that it is visually pleasing when the long
`
`axes of at least the upper anterior teeth converge toward the incisal.
`
`In some embodiments the template comprises a contour of teeth.
`
`In some embodiments the contour comprises a shape of one or more teeth
`
`10
`
`seen from the front.
`
`An advantage of the embodiments relating to the contour of teeth is that
`
`using the visually pleasing contour of some suitable teeth may be a simple
`
`and easy way to model the teeth of the 3D model.
`
`In some embodiments the template comprises a curve.
`
`An advantage of this embodiment is that by means of a curve, distances and
`
`angles can be measured or viewed. For example a distance can be
`
`measured from the centre of the curve, and in one example the operator may
`
`measure x mm from a certain point on the curve, and at
`
`this distance
`
`something specific may be arranged, such as a distal point on a lateral.
`
`Furthermore the curve may a symmetry curve for ensuring that the modeled
`
`teeth will be symmetric.
`
`In some embodiments the curve comprises an arch following the upper
`
`and/or lower anterior teeth seen from the front or from above.
`
`In some embodiments the curve comprises a smile line adapted to follow the
`
`lower lip in a natural smile and the incisal edges of the upper teeth.
`
`15
`
`2O
`
`25
`
`30
`
`0429
`
`0429
`
`
`
`10
`
`In some embodiments the template comprises one or more curves for
`
`indicating the position of the gingival tissue.
`
`An advantage of these embodiments relating to curves of the teeth and/or of
`
`the mouth and lips is that using some kind of curve(s) may be a simple and
`
`easy way to model the teeth of the 3D model.
`
`In some embodiments the one or more 2D digital
`
`image shows at least a
`
`number of front teeth.
`
`In some embodiments the one or more 2D digital
`
`image is a photograph
`
`showing at least the patient’s lips and teeth seen from the front.
`
`An advantage of this embodiment is that when the 2D image shows the
`
`patient’s lips and existing teeth,
`
`then the modeling of
`
`the teeth can be
`
`performed such that
`
`they suits the patient’s lips and unchanged teeth
`
`providing a visually pleasing result of the modeling.
`
`In some embodiments the method further comprises virtually cutting at least
`
`a part of the teeth out of the one or more 2D digital image, if the 2D image
`
`comprises teeth, such that at least the lips remains to be visible in the 2D
`
`digital image.
`
`An advantage of this embodiment is that when the lips and no or only some
`
`teeth are visible in the 2D image then it is easy to visualize the modeled teeth
`
`with the patient’s lips and determined whether it is a good result of modeling.
`
`The cutting of teeth out of the 2D image may be performed virtually or
`
`digitally such that the information in the 2D image relating to the teeth is
`
`removed, deleted, made invisible etc..
`
`In some embodiments the 3D virtual model is visible behind the lips.
`
`10
`
`15
`
`2O
`
`25
`
`0430
`
`0430
`
`
`
`11
`
`An advantage of this embodiment is that when the 3D model can be seen
`
`behind the lips,
`
`then the modeling of
`
`the teeth can be performed while
`
`viewing the lips for determining if the modeling is satisfactory.
`
`In some embodiments the one or more 2D digital image shows the face of
`
`the patient such that facial lines, such as the midline and the bi—pupillar line,
`
`are detectable.
`
`An advantage of this embodiment is that facial lines determines the geometry
`
`of the patient’s face, and for obtaining a visually pleasing result of modeling,
`
`the teeth should fit with this overall geometry.
`
`In some embodiments the one or more 2D digital image is an X—ray image of
`
`the patient’s teeth.
`
`An advantage of this embodiment is that when using or applying an X-ray
`
`image of the patient’s teeth, the entire teeth with roots under the gingival can
`
`be seen, and thus broken or weak teeth or roots can be detected. Hereby for
`
`example implants exerting force on the teeth and roots can be planned to be
`
`arranged to exert force on non-broken or strong teeth and teeth roots instead
`
`of on the broken and weak teeth and roots.
`
`In
`
`some embodiments the method further comprises providing a 3D
`
`computed tomography scan of the patient’s face for facilitating aligning the
`
`one or more 2D image and the 3D model and/or for modeling the 3D virtual
`
`model.
`
`In some embodiments the one or more 2D digital image is a still image from
`
`a video recording.
`
`10
`
`15
`
`2O
`
`25
`
`In some embodiments the one or more 2D digital image is derived from a 3D
`
`30
`
`face scan.
`
`0431
`
`0431
`
`
`
`12
`
`In some embodiments the method further comprises providing a 3D face
`
`scan of the patient for facilitating aligning the one or more 2D image and the
`
`3D model and/or for modeling the 3D virtual model.
`
`In some embodiments a face scan of the patient provides a measure of the
`
`distance that the upper and/or lower lip moves when the patient smiles, and
`
`the distance is adapted to be used for measuring the ideal length of at least
`
`some of the teeth.
`
`An advantage of this embodiment is that at least the length of the front teeth
`
`is important for the visual appearance of the teeth.
`
`In some embodiments the method further comprises providing at least part of
`
`the one or more 2D digital image to be at least partly transparent, such that
`
`the 3D virtual model is visual through the 2D digital image.
`
`In some embodiments the one or more 2D digital image is adapted to be
`
`smoothly faded in and out of the view.
`
`An advantage of this embodiment is that when smoothly fading the 2D image
`
`in and out of view this provides that the visualization of the 2D digital image
`
`changes from being entirely visible to be partly visible and then maybe
`
`invisible and vice versa. Hereby the 2D image can be viewed as the user
`
`wishes.
`
`10
`
`15
`
`2O
`
`In some embodiments the 3D virtual model comprises the patient’s set of
`
`25
`
`teeth.
`
`In some embodiments the method further comprises scaling the one or more
`
`2D digital image and the 3D virtual model to show at least part of the teeth in
`
`the same size.
`
`30
`
`An advantage of this embodiment is that the 2D image and the 3D model
`
`should be shown in the same scale in order for optimally performing the
`
`0432
`
`0432
`
`
`
`13
`
`modeling. The scaling may be an automatic modification of the size of e.g.
`
`the 3D virtual model
`
`to the size of
`
`the 2D digital
`
`image or vice versa.
`
`Alternatively, the scaling may be of both the 2D image and the 3D model to
`
`resize them to a predetermined scale.
`
`In some embodiments the method further comprises aligning the one or more
`
`2D digital image and the 3D virtual model.
`
`An advantage of this embodiment is that when the 2D image and the 3D
`
`model are aligned then modeling may be performed easier and with a better
`
`result. Alignment may be defined as the adjustment of an object in relation
`
`with another object, such that structures of the objects are coinciding. Thus
`
`common or alike structures of the 2D image and the 3D model are aligned.
`
`In some embodiments the silhouette of the biting edge of at least the upper
`
`anterior teeth on the one or more 2D image and on the 3D virtual model is
`
`used to perform the alignment of the 2D image and the 3D virtual model.
`
`An advantage of this embodiment is that in many cases the biting edge of the
`
`upper anterior teeth are seen on both the 2D image and on the 3D model,
`
`and therefore this biting edge may be an advantageous physical point of
`
`10
`
`15
`
`2O
`
`alignment.
`
`In some embodiments the method further comprises projecting the plane of
`
`the one or more 2D digital image to the 3D virtual model.
`
`An advantage of this embodiment is that when projecting the plane of 2D
`
`25
`
`image to the 3D model or to a plane of the 3D model, the 3D model and the
`
`2D image can be viewed in the same plane which may be an advantage
`
`when modeling the teeth. The viewing of the 3D model and the 2D image in
`
`the same plane may otherwise be complex.
`
`0433
`
`0433
`
`
`
`14
`
`In
`
`some embodiments
`
`the method further comprises
`
`changing the
`
`perspective view of the one or more 2D digital image and/or of the 3D virtual
`
`model to obtain the same perspective view.
`
`An advantage of this embodiment is that modeling may be facilitated when
`
`the 2D image and the 3D model can be seen in the same perspective view.
`
`In
`
`some embodiments
`
`the method further comprises de—warping the
`
`perspective view of the one or more 2D image for visually aligning the 2D
`
`image and the 3D virtual model.
`
`An advantage of this embodiment is that when de-warping or correcting the
`
`perspective view of the 2D image, then the view is digitally manipulated, and
`
`hereby points on the perspective view of the 2D image can be mapped to
`
`points on the 3D model or its plane. After de-warping or correcting the
`
`perspective of the 2D image, the 3D model can be re-aligned, such that the
`
`2D image and the 3D model are aligned again.
`
`In some embodiments scaling, aligning, projecting to a plane, de—warping
`
`perspective and changing perspective are defined as virtual actions for
`
`arrangement.
`
`In some embodiments one or more of the virtual actions for arrangement
`
`comprises rotations and translations left/right and back/forth of the one or
`
`more 2D digital image and/or of the 3D virtual model.
`
`An advantage of this embodiment is that by providing rotations, translations
`
`etc. then different movements of the 2D image and/or of the 3D model may
`
`be performed for facilitating the scaling, aligning, perspective changing and
`
`ultimately for facilitating the modeling of the teeth.
`
`In some embodiments the method further comprises the steps of:
`
`10
`
`15
`
`2O
`
`25
`
`0434
`
`0434
`
`
`
`15
`
`- detecting anatomical points on the teeth, where the anatomical points are
`
`present and detectable both on the one or more 2D digital image and the 3D
`
`virtual model, and
`
`—
`
`performing
`
`the
`
`virtual
`
`actions
`
`for
`
`arrangement based on
`
`these
`
`corresponding anatomical points.
`
`An advantage of this embodiment is that using common or mutual anatomical
`
`points on the 2D image and the 3D model may be an easy way to perform
`
`alignment of the 2D image and the 3D model, where after modeling of the
`
`teeth can be performed.
`
`10
`
`In some embodiments at
`
`least one corresponding anatomical point
`
`is
`
`selected to perform the virtual actions for arrangement.
`
`An advantage of this embodiment is that one common or mutual point on the
`
`2D image and the 3D model may be sufficient for arranging the 2D image
`
`15
`
`and the 3D model relative to each other. However in other cases the 2D
`
`image and the 3D model should be aligned using more points, such as two,
`
`three or four points. In general three points may be suitable. Four points can
`
`be used for performing an even better arrangement or for use in more difficult
`cases.
`
`2O
`
`In some embodiments the method further comprises the steps of:
`
`- providing a virtual measurement bar, and
`
`— performing the virtual actions for arrangement of the one or more 2D digital
`
`image and/or of the 3D virtual model by means of adjustment to the virtual
`
`25
`
`measurement bar.
`
`An advantage of this embodiment is that it may be easy and fast to use a
`
`virtual measurement bar to perform the virtual actions for arrangement such
`
`as scaling, where the sizes of the 2D image and the 3D model are adjusted
`
`to correspond to each other.
`
`30
`
`0435
`
`0435
`
`
`
`16
`
`In some embodiments the method further comprises that a user performs the
`
`virtual actions for arrangement of the one or more 2D digital image and/or of
`
`the 3D virtual model by means of eye measure.
`
`An advantage of this embodiment is that just by using simple eye measure,
`
`the operator can very quickly and reliably perform the arrangement of the 2D
`
`image and the 3D model relative to each other or perform a rough starting
`
`point for a more detailed adjustment.
`
`In some embodiments the anatomical points are upper and/or lower distal
`
`and/or mesial points on a number of specific anterior teeth.
`
`An advantage of this embodiment
`
`is that anatomical point on the upper
`
`and/or lower distal and/or mesial parts of the anterior teeth are normally easy
`
`to detect both on the 2D image and on the 3D model.
`
`In
`
`some embodiments
`
`the modeling of
`
`the 3D model
`
`is performed
`
`automatically based on the one or more 2D digital image.
`
`An advantage of this embodiment is that the user does not need to perform
`
`any manual modeling of the 3D model on the screen, when the modeling can
`
`be performed fully automatic. However,
`
`typically if an automatic modeling
`
`takes place, then the user may check that the modeling is satisfying, and
`
`maybe perform small corrections to the modeling.
`
`10
`
`15
`
`2O
`
`In some embodiments the method further comprises automatically selecting
`
`one or more 2D digital image which provides an optimal fit to the 3D virtual
`
`25
`
`model.
`
`An advantage of this embodiment is that a 2D image with an optimal, good or
`
`the best fit to the 3D model can automatically be selected, and hereby a good
`
`result of modeling can be obtained, and furthermore the time used for
`
`performing the modeling can be reduced, since no person needs to spend
`
`30
`
`time on looking through a larger number of 2D images. The 2D image may
`
`be selected from a library of 2D digital
`
`images, or
`
`from any source
`
`0436
`
`0436
`
`
`
`17
`
`comprising a number of
`
`images of
`
`teeth and smiles. The library may
`
`comprises templates, photos, drawings etc.
`
`In some embodiments the optimal
`
`fit
`
`is determined based on specific
`
`parameters for providing an esthetically, visually pleasing appearance.
`
`An advantage of this embodiment is that the optimal, best or just a good fit
`
`can be determined based on different parameters, such as the present size
`
`of the patient’s teeth, on the curves of the patient’s present teeth set, etc.
`
`New teeth which are very big may not suit a person who used to have very
`
`10
`
`small teeth or a person who has thin lips. Likewise a new teeth set with a
`
`strong composition may not suit a person who used to have a teeth set with a
`
`soft composition or a person who has full lips etc. 80 based on the present
`
`structures, features, shapes etc. of the patient’s teeth, new teeth which will
`
`look natural and suit the patient can be determined from e.g. a template
`
`15
`
`library of photos, drawings etc.
`
`In some embodiments the align