(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT)
`
`(19) World Intellectual Property
`Organization
`International Bureau
`
`(43) International Publication Date
`13 December 2012 (13.12.2012)
`
`it
`_/
`WIPOI PCT
`
`(10) International Publication N umber
`
`WO 2012/170354 A2
`
`(51)
`
`International Patent Classification:
`G06T13/40 (2011.01)
`A63F 13/00 (2006.01)
`
`(21)
`
`International Application Number:
`
`PCT/1182012 ’040768
`
`(22)
`
`International Filing Date:
`
`Filing Language:
`
`Publication Language:
`
`4 June 2012 (04.06.2012)
`
`English
`
`English
`
`Priority Data:
`61/493,839
`13/297,890
`
`6 June 2011 (06.06.2011)
`16 November 2011 (16.11.2011)
`
`US
`US
`
`(81)
`
`(25)
`
`(26)
`
`(30)
`
`(71)
`
`(72)
`
`LICH, Mark, Thomas; c/o Microsoft Corporation, LCA -
`International Patents, One Microsoft Way, Redmond, WA
`98052-6399 (US). OSBORN, Dan; c/o Microsoft Corpor-
`ation, LCA - International Patents, One Microsoft Way,
`Redmond, WA 98052-6399 (US). CAMPBELL, Andrew,
`R.; 0/0 Microsoft Corporation, LCA - International Patents,
`One Microsoft Way, Redmond, WA 98052-6399 (US).
`MARTIN, Charles, Everett; c/o Microsoft Corporation,
`LCA - International Patents, One Microsoft Way, Red-
`mond, WA 98052—6399 (US). HILL, David, M.; c/o Mi—
`crosoft Corporation, LCA - International Patents, One Mi-
`crosoft Way, Redmond, WA 98052-6399 (US).
`
`Designated States (unless otherwise indicated, for every
`kind of national protection available): AE, AG, AL, AM,
`AO, AT, AU, AZ, BA, BB, BG, BH, BR, BW, BY, BZ,
`CA, CH, CL, CN, CO, CR, CU, CZ, DE, DK, DM, DO,
`DZ, EC, EE, EG, ES, FI, GB, GD, GE, GH, GM, GT, HN,
`HR, HU, 11), IL, IN, IS, JP, KE, KG, KM, KN, KP, KR,
`KZ, LA, LC, LK, LR, LS, LT, LU, LY, MA, MD, ME,
`MG, MK, MN, MW, MX, MY, MZ, NA, NG, NI, NO, NZ,
`OM, PE, PG, PH, PL, PT, QA, RO, RS, RU, RW, SC, SD,
`SE, SG, SK, SL, SM, ST, SV, SY, TH, TJ, TM, TN, TR,
`TT, TZ, UA, UG, US, UZ, VC, VN, ZA, ZM, ZW.
`
`(84)
`
`Designated States (unless otherwise indicated, for every
`kind of regional protection available): ARTPO (BW, GH,
`GM, KE, LR, LS, MW, MZ, NA, RW, SD, SL, SZ, TZ,
`UG, ZM, ZW), Eurasian (AM, AZ, BY, KG, KZ, RU, TJ,
`
`[Continued on next page]
`
`(for all designated States except US): MI-
`Applicant
`CROSOFT CORPORATION [US/US]; One Microsoft
`Way, Redmond, WA 98052-6399 (US).
`
`Inventors: EVERTT, Jeffrey, Jesus; c/o Microsoft Cor-
`poration, LCA - International Patents, One Microsoft Way,
`Redmond, WA 98052—6399 (US). CLARK, Justin, A'-
`ram; c/o Microsoft Corporation, LCA - International Pat-
`ents, One Microsoft Way, Redmond, WA 98052-6399
`(US). MIDDLETON, Zachary, Tyler; c/o Microsoft Cor-
`poration, LCA - International Patents, One Microsoft Way,
`Redmond, WA 98052—6399 (US). PULS, Matthew, J.; 0/0
`Microsoft Corporation, LCA - International Patents, One
`Microsoft Way, Redmond, WA 98052-6399 (US). MIHE-
`
`(54) Title: GENERATION OF AVATAR REFLECTING PLAYER APPEARANCE
`
`600
`
`604
`
`SENSORS
`
`
`DATA ACQUISITION COMPONENT
`
`
`
`
`CLOTHING
`SLEEVE
`SKIN COLOR
`
`CAPTURE
`CAPTURE
`CAPTURE
`DETECTION
`CAPTURE
`COMPONENT
`
`
`
`
`COMPONENT
`COMPONENT
`COMPONENT
`COMPONENT
`
`
`
`
`
`
`
`
`AVATAR GENERATION COMPONENT
`
`|
`
`616
`
`FIG. 6
`
`(57) Abstract: Systems, methods, and computer media for generating an avatar reflecting a player's current appearance. Data dc-
`scribing the player's current appearance is received. The data includes a visible spectrum image of the player, a depth image includ -
`ing both the player and a current background, and skeletal data for the player. The skeletal data indicates an outline of the player's
`skeleton. Based at least in part on the received data, one or more of the following are captured: a facial appearance of the player; a
`hair appearance of the player; a clothing appearance of the player; and a skin color of the player. A 3D avatar resembling the player
`is generated by combimng the captured facial appearance, hair appearance, clothing appearance, and/or skin color with predeter—
`mined avatar features.
`
`
`
`W2012/170354A2|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
`
`

`

`WO 2012/170354 A2 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
`
`TM), European (AL, AT, BE, BG, CII, CY, CZ, DE, DK,
`EE, ES, FI, FR, GB, GR, HR, HU, IE, IS, IT, LT, LU,
`LV, MC, MK, MT, NL, NO, PL, PT, R0, Rs, SE, SI, SK,
`SM, TR), OAPI (BF, BJ, CF, CG, CI, CM, GA, GN, GQ, Published:
`GW’ ML’ MR’ NE’ SN’ TD’ TG)‘
`— without international search report and to be republished
`Declarations under Rule 4.17:
`upon receipt of that report (Rule 48.2(’g))
`
`— as to the applicant’s entitlement to claim the priority of
`the earlier application (Rule 4.1 7(iiz))
`
`* as to applicant’s entitlement to apply for and be granted
`apatent (Rule 4.1 7(ii))
`
`

`

`WO 2012/170354
`
`PCT/U52012/040768
`
`GENERATION OF AVATAR REFLECTING PLAYER APPEARANCE
`
`BACKGROUND
`
`[0001]
`
`Video games have advanced substantially in recent years. Improvements in
`
`processing power,
`
`image processing, and graphics, among other areas, have enabled
`
`games of increasing conceptual and graphical complexity. Conventionally, players
`
`selected a default character or selected one of a group of simple, predefined characters. As
`
`video games have evolved, the use of player avatars rather than simple characters has
`
`become increasingly common. Some games allow a player to select among a group of
`
`predefined avatars or customize various features of an avatar. Players can attempt to
`
`crcatc an avatar rcscmbling thcmsclvcs by manually customizing fcaturcs such as hair
`
`color, gender, height, glasses, etc. Such manual customization is time consuming and still
`
`does not result in an avatar that actually resembles the appearance of the player.
`
`SUMMARY
`
`[0002]
`
`Embodiments of the present
`
`invention relate to systems, methods, and
`
`computer media for generating an avatar reflecting a player’s current appearance. Data
`
`describing the player’s current appearance is received.
`
`The data includes a visible
`
`spectrum image of the player, a depth image including both the player and a current
`
`background, and skeletal data for the player. The skeletal data indicates an outline of the
`
`player’s skeleton. Based at least in part on the received data, one or more of the following
`
`are captured: a facial appearance of the player; a hair appearance of the player; a clothing
`
`appearance of the player; and a skin color of the player. A shirt sleeve, pants or shorts leg,
`
`or skirt or dress hem may also be detected. A 3D avatar resembling the player is
`
`generated by combining the captured facial appearance, hair appearance, clothing
`
`appearance, skin color, and/or detected shirt sleeve, pants or shorts leg, or skirt or dress
`
`10
`
`15
`
`20
`
`25
`
`hcm with prcdctcrmincd avatar fcaturcs.
`
`[0003]
`
`This Summary is provided to introduce a selection of concepts in a
`
`simplified form that are further described below in the Detailed Description.
`
`This
`
`Summary is not intended to identify key features or essential features of the claimed
`
`subject matter, nor is it intended to be used to limit the scope of the claimed subject
`
`30
`
`matter.
`
`

`

`WO 2012/170354
`
`PCT/U32012/040768
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0004]
`
`The present invention is described in detail below with reference to the
`
`attached drawing figures, wherein:
`
`[0005]
`
`FIG. 1 is a block diagram of an exemplary computing environment suitable
`
`5
`
`for use in implementing embodiments of the present invention;
`
`[0006]
`
`FIG. 2 is a perspective view of a visible spectrum image of a player’s head
`
`in accordance with embodiments of the present invention;
`
`[0007]
`
`FIG. 3 is a perspective view of a visible spectrum image of a player in
`
`accordance with embodiments of the present invention;
`
`10
`
`[0008]
`
`FIG. 4 is a perspective view of a 3D avatar resembling the player in FlGS.
`
`2 and 3 in accordance with embodiments of the present invention;
`
`[0009]
`
`FIG. 5 is a flow chart of an exemplary method for generating an avatar
`
`reflecting a player’s current appearance in accordance with an embodiment of the present
`
`invention;
`
`15
`
`[0010]
`
`FIG. 6 is an exemplary avatar generation system in accordance with an
`
`embodiment of the present invention;
`
`[0011]
`
`FIG. 7 is a flow chart of an exemplary method for generating an avatar
`
`reflecting a player’s currcnt appcarancc in accordancc with an embodiment of the present
`
`invention in which sub—steps are shown for facial appearance capture, hair appearance
`
`20
`
`capture, clothing capture, and skin color capture;
`
`[0012]
`
`FIG. 8 is a perspective view of a visible spectrum image of a player in
`
`accordance with embodiments of the present invention;
`
`[0013]
`
`FIG. 9 is a perspective view of a visible spectrum image of a player’s head
`
`in accordance with embodiments of the present invention;
`
`25
`
`[0014]
`
`FIG. 10 is a partial perspective view of a visible spectrum image of a
`
`player’s head in accordance with embodiments of the present
`
`invention,
`
`the image
`
`showing the face and other portions of the head;
`
`[0015]
`
`FIG. 11 the image of FIG. 10 shown with alignment points in accordance
`
`with embodiments of the present invention;
`
`30
`
`[0016]
`
`FIG. 12 is a template texture map with destination points in accordance
`
`with embodiments of the present invention;
`
`[0017]
`
`FIG. 13 is a face texture map resulting from warping the image in FIG.
`
`11
`
`by matching the alignment points to thc destination points in FlG. 12;
`
`

`

`WO 2012/170354
`
`PCT/U52012/040768
`
`[0018]
`
`FIG. 14 is a 3D avatar reflecting the current appearance of the player in the
`
`image in FIGS. 8-11 and 13;
`
`[0019]
`
`FIG. 15 is a perspective view of a visible spectrum image of a player in
`
`accordance with embodiments of the present invention; and
`
`[0020]
`
`FIG. 16 is a depth image of the player shown in FIG. 15 in which a
`
`simplified skeleton has been identified.
`
`DETAILED DESCRIPTION
`
`[0021]
`
`Embodiments of the present invention are described with specificity herein
`
`to meet statutory requirements. However, the description itself is not intended to limit the
`
`scope of this patent. Rather, the inventors have contemplated that the claimed subject
`
`matter might also be embodied in other ways, to include different steps or combinations of
`
`steps similar to the ones described in this document, in conjunction with other present or
`
`future technologies. Moreover, although the terms “step” and/or “block” or “module” etc.
`
`might be used herein to connote different components of methods or systems employed,
`
`the terms should not be interpreted as implying any particular order among or between
`
`various steps herein disclosed unless and except when the order of individual steps is
`
`explicitly described.
`
`[0022]
`
`Embodiments of the present
`
`invention relate to systems, methods, and
`
`computer media for generating an avatar reflecting a player’s current appearance.
`
`In
`
`accordance with embodiments of the present invention, an avatar is generated from data
`
`describing the player’s current appearance. Various aspects of the player’s appearance are
`
`captured based on the data and combined with predetermined avatar features to generate a
`
`3D avatar resembling the player.
`
`[0023]
`
`As discussed above, players can attempt to create an avatar resembling
`
`themselves by manually customizing certain features, but the resulting avatar still does not
`
`actually reflect the appearance of the player.
`
`[0024]
`
`Various sensors can be used, however, to gather data describing a player’s
`
`current appearance. These sensors provide visible spectrum images, depth images, and
`
`skeletal data, among other data, that contain detailed information describing the player’s
`
`appearance and can be used to generate an avatar that accurately reflects the player’s
`
`appearance. The combination of visible spectrum data, depth data, and skeletal data
`
`allows capture of detailed aspects of a player’s appearance not previously possible.
`
`Additionally, an avatar resembling the player can be automatically generated without
`
`player selection of various features after sensor data is acquired.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`

`

`WO 2012/170354
`
`PCT/U52012/040768
`
`[0025]
`
`In one embodiment of the present invention, data describing the player’s
`
`current appearance is received. The data includes: a visible spectrum image of the player,
`
`a depth image including both the player and a current background, and skeletal data for the
`
`player. The skeletal data indicates an outline of the player’s skeleton. A facial appearance
`
`of the player is captured based at least in part on the received data. The facial appearance
`
`is captured by identifying the player’s face; identifying facial features of the player’s face;
`
`and warping a portion of the received visible spectrum image that includes the player’s
`
`face into a face texture map based on the identified facial features. A hair appearance of
`
`the player is captured based at least in part on the received data. The hair appearance is
`
`captured by identifying an extent of the player’s hair; matching the identified extent of the
`
`player’s hair to a predetermined hairstyle template; and identifying a color of the player’s
`
`hair. A 3D avatar reflecting the captured facial appearance and hair appearance of the
`
`player is generated such that the 3D avatar resembles the player. The 3D avatar is
`
`generated by combining the face texture map, hairstyle template that matches the
`
`identified extent of the player’s hair, and the identified player hair color with
`
`predetermined avatar features.
`
`[0026]
`
`In another embodiment,
`
`a data acquisition component
`
`receives data
`
`describing the player’s current appearance. The data includes: a visible spectrum image
`
`of the player, a depth image including both the player and a current background, and
`
`skeletal data for the player. The skeletal data indicates an outline of the player’s skeleton.
`
`A face capture component captures a facial appearance of the player based at least in part
`
`on the received data. A hair capture component captures a hair appearance of the player
`
`based at least in pa1t 011 the received data. A clothing capture component captures a
`
`clothing appearance of the player based at least in part on the received data. An avatar
`
`generation component generates a 3D avatar resembling the player by combining the
`
`captured facial appearance, hair appearance, and clothing appearance with predetermined
`
`avatar features.
`
`[0027]
`
`In still
`
`another
`
`embodiment,
`
`data describing the player’s
`
`current
`
`appearance is received. The data includes: a visible spectrum image of the player, a depth
`
`image including both the player and a current background, and skeletal data for the player.
`
`The skeletal data indicates an outline of the player’s skeleton. A facial appearance of the
`
`player is captured based at least in part on the received data. The facial appearance is
`
`captured by identifying the player’s face; identifying facial features of the player’s face;
`
`and warping a portion of the received visible spectrum image that includes the player’s
`
`10
`
`15
`
`20
`
`25
`
`30
`
`

`

`WO 2012/170354
`
`PCT/U52012/040768
`
`face into a face texture map based on the identified facial features. A hair appearance of
`
`the player is captured based at least in part on the received data. The hair appearance is
`
`captured by identifying an extent of the player’s hair; matching the identified extent of the
`
`player’s hair to a predetermined hairstyle template; and identifying a color of the player’s
`
`hair.
`
`[0028]
`
`A clothing appearance of the player is captured based at least in part on the
`
`received data. The clothing appearance is captured, using the received depth image
`
`including both the player and the current background and the received visible spectrum
`
`image of the player, by determining an outer clothing boundary using edge detection. The
`
`clothing appearance is also captured by, using the received visible spectrum image of the
`
`player, determining at least one inner clothing boundary by distinguishing clothing from
`
`skin based on a determined color gradient. The clothing appearance is further captured by
`
`identifying player clothing as the area bounded at least in part by the outer clothing
`
`boundary and the at least one inner clothing boundary.
`
`[0029]
`
`A skin color of the player is captured based at least in part on the received
`
`data. The skin color is captured by sampling the color of one or more areas of the received
`
`visible spectrum image in areas corresponding to the player’s face and blending the
`
`sampled colors of the one or more areas to determine a player skin color. A processor of a
`
`computing device is used to generate a 3D avatar reflecting the captured facial appearance,
`
`hair appearance, clothing appearance, and skin color of the player such that the 3D avatar
`
`resembles the player. The 3D avatar is generated by combining the face texture map,
`
`hairstyle template that matches the identified extent of the player’s hair, and identified
`
`player hair color with predetermined avatar features.
`
`[0030]
`
`Having briefly described an overview of some embodiments of the present
`
`invention, an exemplary operating environment in which embodiments of the present
`
`invention may be implemented is described below in order to provide a general context for
`
`various aspects of the present invention. Referring initially to FIG.
`
`1
`
`in particular, an
`
`exemplary operating environment for implementing embodiments of the present invention
`
`is shown and designated generally as computing device 100. Computing device 100 is but
`
`one example of a suitable computing environment and is not intended to suggest any
`
`limitation as to the scope of use or functionality of embodiments of the present invention.
`
`Neither should the computing device 100 be interpreted as having any dependency or
`
`requirement relating to any one or combination of components illustrated.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`

`

`WO 2012/170354
`
`PCT/U52012/040768
`
`[0031]
`
`Embodiments of the present invention may be described in the general
`
`context of computer code or maehine—useable instructions, including computer-executable
`
`instructions such as program modules, being executed by a computer or other machine,
`
`such as a personal data assistant or other handheld device. Generally, program modules
`
`including routines, programs, objects, components, data structures, etc., refer to code that
`
`perform particular tasks or implement particular abstract data types. Embodiments of the
`
`present invention may be practiced in a variety of system configurations, including hand-
`
`held devices, consumer electronics, general-purpose computers, more specialty computing
`
`devices, etc. Embodiments of the present invention may also be practiced in distributed
`
`computing environments where tasks are performed by remote-processing devices that are
`
`linked through a communications network.
`
`[0032]
`
`With reference to FIG. 1, computing device 100 includes a bus 110 that
`
`directly or indirectly couples the following devices: memory 112, one or more processors
`
`114, one or more presentation components 116,
`
`input/output ports 118,
`
`input/output
`
`components 120, and an illustrative power supply 122. Bus 110 represents what may be
`
`one or more busses (such as an address bus, data bus, or combination thereof). Although
`
`the various blocks of FIG.
`
`1 are shown with lines for the sake of clarity,
`
`in reality,
`
`delineating various components is not so clear, and metaphorically, the lines would more
`
`accurately be grey and fuzzy. For example, one may consider a presentation component
`
`such as a display device to be an I/O component. Also, processors have memory. We
`
`recognize that such is the nature of the art, and reiterate that the diagram of FIG.
`
`1 is
`
`merely illustrative of an exemplary computing device that can be used in connection with
`
`one or more embodiments of the present invention. Distinction is not made between such
`
`categories as “workstation,” “server,” “laptop,” “hand-held device,” etc., as all are
`
`contemplated within the scope of FIG. 1 and reference to “computing device.”
`
`[0033]
`
`Computing device 100 typically includes a variety of computer—readable
`
`media. Computer-readable media can be any available media that can be accessed by
`
`computing device 100 and includes both volatile and nonvolatile media, removable and
`
`non-removable media. By way of example, and not limitation, computer—readable media
`
`may comprise computer storage media and communication media. Computer storage
`
`media includes both volatile and nonvolatile,
`
`removable and non-removable media
`
`implemented in any method or technology for storage of information such as computer-
`
`readable instructions, data structures, program modules, or other data. Computer storage
`
`media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other
`
`10
`
`15
`
`20
`
`25
`
`30
`
`

`

`WO 2012/170354
`
`PCT/U52012/040768
`
`memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage,
`
`magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage
`
`devices, or any other medium which can be used to store the desired information and
`
`which can be accessed by computing device 100.
`
`[0034]
`
`Communication media typically embodies computer-readable instructions,
`
`data structures, program modules, or other data in a modulated data signal such as a carrier
`
`wave. The term “modulated data signal” refers to a propagated signal that has one or more
`
`of its characteristics set or changed to encode information in the signal. By way of
`
`example, and not limitation, communication media includes wired media, such as a wired
`
`network or direct-wired connection, and wireless media such as acoustic, RF, infrared,
`
`radio, microwave, spread—spectrum, and other wireless media. Combinations of the above
`
`are included within the scope of computer-readable media.
`
`[0035]
`
`Memory 112 includes computer storage media in the form of volatile
`
`and/or nonvolatile memory.
`
`The memory may be removable, nonremovable, or a
`
`combination thereof. Exemplary hardware devices include solid—state memory, hard
`
`drives, optical-disc drives, etc. Computing device 100 includes one or more processors
`
`that read data from various entities such as memory 112 or I/O components 120.
`
`Presentation component(s) 116 present data indications to a user or other device.
`
`Exemplary presentation components
`
`include
`
`a display device,
`
`speaker, printing
`
`10
`
`15
`
`20
`
`component, vibrating component, etc.
`
`[0036]
`
`I/O ports 118 allow computing device 100 to be logically coupled to other
`
`devices including I/O components 120, some of which may be built in.
`
`Illustrative
`
`components include a microphone, joystick, game pad, satellite dish, scanner, printer,
`
`wireless device, etc.
`
`25
`
`30
`
`[0037]
`
`As discussed previously, embodiments of the present invention relate to
`
`systems, methods, and computer media for generating an avatar resembling a player’s
`
`current appearance.
`
`Embodiments of the present
`
`invention will be discussed with
`
`reference to FIGS. 2-16.
`
`[0038]
`
`FIGS. 2-4 illustrate an implementation of an embodiment of the present
`
`invention to acquire data describing a player and generate an avatar resembling the
`
`player’s current appearance. FIG. 2 illustrates a visible spectrum image 200 of player 201.
`
`Image 200 includes face 202, hair 204, and clothes 206 of player 201.
`
`In one
`
`embodiment, player 200 positions herself in front of a camera according to guide markers
`
`208 to facilitate capture of image 201. FIG. 3 illustrates another visible spectrum image
`
`

`

`WO 2012/170354
`
`PCT/USZ012/040768
`
`300 of player 201. Substantially all of the body of player 201 is included in image 300.
`
`FIG. 3 also includes guide markers 308 to facilitate capture of image 308.
`
`[0039]
`
`In addition to visible spectrum images 200 and 300, sensors gather one or
`
`more depth images of player 201. A depth image indicates the depth of each pixel in the
`
`image, which allows the background of an image to be distinguished from a person or
`
`object in the image and also allows identification of various body features of a person,
`
`such as facial features, based on the depth. The person can be recognized as including the
`
`pixels that are of a similar depth in the foreground of the image, whereas the background
`
`is determined to be the pixels having a larger depth.
`
`[0040]
`
`Depth data also allows creation of skeletal data as known in the art.
`
`In one
`
`embodiment, the portion of an image representing a person can be distinguished from the
`
`background of the image using depth information. The portion of the image representing
`
`the person can then be compared to a database of poses to identify a match to the pose of
`
`the identified person. Once a match is
`
`identified, a simplified skeletal structure
`
`corresponding to the match can be identified. The simplified skeletal structure can
`
`indicate the outline of a player’s skeleton and can include points that represent joints and
`
`lines connecting the points to represent appendages or other body features. Other methods
`
`for identifying a simplified player skeleton from depth data are also envisioned.
`
`[0041]
`
`Visible spectrum images 200 and 300, along with skeletal data and a depth
`
`image, are combined with predetermined avatar features to generate an avatar 400 shown
`
`in FIG. 4 resembling player 201.
`
`In some embodiments, the generated avatar has certain
`
`predetermined proportions, such as height and width, and certain predetermined features
`
`such as hands, feet, eyes, arm and leg width, or other features. Avatar 400 reflects face
`
`202, hair 204, and clothes 206 of player 201, achieving an avatar that much more
`
`accurately represents player 201 that manual selection of various features can accomplish.
`
`[0042]
`
`FIG. 5 illustrates a method of generating an avatar reflecting a player’s
`
`current appearance.
`
`In step 502, data describing the player’s current appearance is
`
`received. The data includes:
`
`a visible spectrum image of the player, a depth image
`
`including both the player and a current background, and skeletal data for the player. The
`
`skeletal data indicates an outline of the player’s skeleton.
`
`In step 504, a facial appearance
`
`of the player is captured based at least in part on the received data. Also based at least in
`
`part on the received data: a hair appearance and clothing appearance are captured in steps
`
`506 and 508; sleeves, pant legs, or hems are detected in step 510; and skin color is
`
`captured in step 512.
`
`In step 514, a 3D avatar resembling the player is generated by
`
`10
`
`15
`
`20
`
`25
`
`30
`
`

`

`WO 2012/170354
`
`PCT/U52012/040768
`
`combining the captured facial appearance, hair appearance,
`
`clothing appearance,
`
`sleeves/legs/hems, and skin color with predetermined avatar features.
`
`[0043]
`
`FIG. 6 illustrates a system 600 in accordance with an embodiment of the
`
`present invention. Data acquisition component 602 receives data describing the player’s
`
`current appearance from sensors 604. The data includes: a visible spectrum image of the
`
`player, a depth image including both the player and a current background, and skeletal
`
`data for the player. The skeletal data indicates an outline of the player’s skeleton. Sensors
`
`604 may include an RGB camera and an infrared or other depth sensor.
`
`In one
`
`embodiment, a player stands in the path of sensors 604 in a variety of preferred positions
`
`to facilitate acquisition of sensor data. Data acquisition component 602 provides received
`
`data to face capture component 606, hair capture component 608, clothing capture
`
`component 610, sleeve detection component 612, and skin color capture component 614.
`
`Components 606, 608, 610, 612, and 614 capture or detect information and provide this
`
`information to avatar generation component 616, which generates a 3D avatar resembling
`
`the player using the information.
`
`[0044]
`
`In various embodiments, any combination of components 606, 608, 610,
`
`612, and 614 are included. Face capture component 606 captures a facial appearance of
`
`the player based at least in part on the received data. Hair capture component 608 captures
`
`a hair appearance of the player based at least in part on the received data. Clothing
`
`capture component 610 captures a clothing appearance of the player based at least in part
`
`on the received data. Sleeve detection component 612 identifies, based at least in part on
`
`the received data, at least one of:
`
`(1) a shirt sleeve, (2) a pants or shorts leg, and (3) a skirt
`
`or dress hem. Skin color capture component 614 identifies a skin color of the player based
`
`at least in part on the received data. Avatar generation component 616 generates a 3D
`
`avatar
`
`resembling the player by combining the captured facial appearance, hair
`
`appearance, and clothing appearance with predetermined avatar features.
`
`It is envisioned
`
`that the functionality of components 602, 606, 608, 610, 612, 614, and 616 may reside on
`
`separate physical components or devices or may be implemented together.
`
`[0045]
`
`FIG. 7 illustrates a method 700 of generating an avatar reflecting a player’s
`
`current appearance. The steps of method 700 may be implemented by the components of
`
`system 600 of FIG. 6.
`
`In step 702, data describing a player’s current appearance is
`
`received. This data includes at least one visible spectrum image 702A of the player, at
`
`least one depth image 702B including both the player and a current background, and
`
`10
`
`15
`
`20
`
`25
`
`30
`
`

`

`WO 2012/170354
`
`PCT/U52012/040768
`
`skeletal data 702C for the player. Skeletal data 702C indicates an outline of the player’s
`
`skeleton and may be derived from the at least one depth image 7023.
`
`[0046]
`
`In step 704, a facial appearance of the player is captured based at least in
`
`part on the received data. Step 704 can be implemented by sub—steps 704A—704C. In sub—
`
`step 704A, the player’s face is identified.
`
`In sub—step 704B, facial features of the player’s
`
`face are identified. In sub—step 704C, a portion of the received visible spectrum image that
`
`includes the player’s face is warped into a face texture map based on the identified facial
`
`features. The face texture map is a UV space map that identifies how to map 2D image
`
`points to a 3D model.
`
`[0047]
`
`In step 706, a hair appearance of the player is captured based at least in part
`
`on the received data. Step 706 can be implemented by sub—steps 706A—706C.
`
`In sub—step
`
`706A, the extent of the player’s hair is identified.
`
`In sub-step 706B, the identified extent
`
`of the player’s hair is matched to a predetermined hairstyle template.
`
`In sub—step 706C, a
`
`color of the player’s hair is identified.
`
`[0048]
`
`In step 708, a clothing appearance of the player is captured based at least in
`
`part on the received data. Step 708 can be implemented by sub-steps 708A—708C. In sub-
`
`step 708A, using the received depth image including both the player and the current
`
`background and the received visible spectrum image of the player, an outer clothing
`
`boundary is determined using edge detection.
`
`In sub—step 708B, using the received visible
`
`spectrum image of the player, at
`
`least one inner clothing boundary is determined by
`
`distinguishing clothing from skin based on a determined color gradient.
`
`In sub-step 708C,
`
`player clothing is identified as the area bounded at least in part by the outer clothing
`
`boundary and the at least one inner clothing boundaiy.
`
`[0049]
`
`A skin color of the player is captured in step 710, the capture based at least
`
`in part on the received data. Step 710 can be implemented by sub-steps 710A-710B.
`
`In
`
`sub—step 710A, the color of one or more areas of the received visible spectrum image are
`
`sampled in areas corresponding to the player’s face.
`
`In sub-step 710B, the sampled colors
`
`of the one or more areas are blended to determine a player skin color.
`
`[0050]
`
`In step 712, a 3D avatar
`
`is generated reflecting the captured facial
`
`appearance, hair appearance, clothing appearance, and skin color of the player such that
`
`the 3D avatar resembles the player by combining the face texture map, hairstyle template
`
`that matches the identified extent of the player’s hair, and identified player hair color with
`
`predetermined avatar features.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`10
`
`

`

`WO 2012/170354
`
`PCT/U52012/040768
`
`[0051]
`
`Facial appearance capture, hair appearance capture, clothing appearance
`
`capture, and sleeve/leg/hem detection, discussed with reference to FIGS. 5-7 will now be
`
`discussed in greater detail.
`
`[0052]
`
`[0053]
`
`Facial Appearance Capture
`
`As discussed above, a visible spectrum image, depth image, and skeletal
`
`data can be used to capture a player’s facial appearance. The captured facial appearance
`
`can then be incorporated into a 3D avatar.
`
`In one embodiment, a players head can be
`
`identified by analyzing the received skeletal data for the player. A head portion