(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2003/0220796 A1
`Aoyama et al.
`(43) Pub. Date:
`Nov. 27, 2003
`
`US 2003O220796A1
`
`(54) DIALOGUE CONTROL SYSTEM, DIALOGUE
`CONTROL METHOD AND ROBOTC
`DEVICE
`(76) Inventors: Kazumi Aoyama, Saitama (JP); Hideki
`Shimomura, Kanagawa (JP); Keiichi
`Yamada, Tokyo (JP)
`Correspondence Address:
`William S. Frommer, Esq.
`FORMMER LAWRENCE & HAUG LLP
`745 Fifth Avenue
`New York, NY 10151 (US)
`(21) Appl. No.:
`10/379,440
`(22) Filed:
`Mar. 4, 2003
`(30)
`Foreign Application Priority Data
`
`Mar. 6, 2002 (JP)...................................... 2002-060428
`
`Publication Classification
`
`(51) Int. Cl." ..................................................... G10L 21/00
`(52) U.S. Cl. .............................................................. 7041275
`
`(57)
`
`ABSTRACT
`
`A dialogue control System, a dialogue control method and a
`robotic device are capable of remarkably improving the
`entertainment factor. In the dialogue control System in which
`a robot and the information processing device are connected
`via the network, in the case of conducting the conversation
`by word games between the robot and the user, the history
`data regarding the word game in Said user's Speech content
`is formed and transmitted to the information processing
`device. Then, Said information processing device Selectively
`reads out the contents best Suited to the user based on Said
`history data from the memory means and provides to the
`original robot.
`
`
`
`3 HEAD UNIT
`
`4B---
`ARM UNT
`
`
`
`5B LEG UNIT
`
`1 ROBOT
`
`IPR2023-00037
`Apple EX1019 Page 1
`
`

`

`Patent Application Publication
`
`Nov. 27, 2003 Sheet 1 of 25
`
`US 2003/0220796 A1
`
`
`
`3 HEAD UN
`
`1 ROBOT
`
`FIG. 1
`
`IPR2023-00037
`Apple EX1019 Page 2
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 2 of 25
`
`US 2003/0220796 A1
`
`
`
`3 HEAD UNT
`
`4A
`ARM UNT
`
`IPR2023-00037
`Apple EX1019 Page 3
`
`

`

`Patent Application Publication
`
`Nov. 27, 2003 Sheet 3 of 25
`
`US 2003/0220796 A1
`
`
`
`3 HEAD UNT
`
`4A ARM UN
`
`IPR2023-00037
`Apple EX1019 Page 4
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 4 of 25
`
`US 2003/0220796 A1
`
`4. ARM UNIT
`
`2 BODY UNIT
`
`A3-GO !---3 HEAD UNIT
`A. - CD438
`i losio.
`4A ARM UNIT
`UT
`- - - - - 49- ---41. 42
`Erstei: Y
`Erstein
`ciplicity. PEER-cio:
`UNIT
`UNIT
`;
`L UNIT
`;430
`GO
`As
`^ As
`
`CRC
`
`::
`
`SUB-
`CONTROL
`UNIT
`
`; :
`
`As
`
`SE: SB
`;
`: CONTROL:
`CONTROL
`(; ; L UNIT J
`L UNIT
`D
`(f
`
`A9
`
`Ag:
`
`A8
`; :----------
`
`- - - - - - - - - -
`
`- - - - - - - - - - -
`
`A13
`
`1 ROBOT
`
`IPR2023-00037
`Apple EX1019 Page 5
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 5 of 25
`
`US 2003/0220796A1
`
`887
`
`
`
`
`
`Gy
`
`| | | | | | –
`
`| | |
`
`IPR2023-00037
`Apple EX1019 Page 6
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 6 of 25
`
`US 2003/0220796 A1
`
`
`
`g
`
`IPR2023-00037
`Apple EX1019 Page 7
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 7 of 25
`
`US 2003/0220796 A1
`
`NETWORK
`NTERFACE
`UNIT
`
`S I D
`
`NAME
`
`
`
`IPR2023-00037
`Apple EX1019 Page 8
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 8 of 25
`
`US 2003/0220796 A1
`
`£S
`
`98
`
`1?N?N ONIZISEHINXS
`
`H03BdS ,·
`
`| || NÍ TOH1N00
`
`#8MOMEN
`
`
`
`
`
`
`
`
`
`|0
`
`08
`
`11Nm NO11|N|90058
`
`HOBBdS
`
`IPR2023-00037
`Apple EX1019 Page 9
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 9 of 25
`
`US 2003/0220796A1
`
`OF FROM SD?
`
`SP2
`
`:
`
`CONFIRM NAME
`
`
`
`SP3
`
`RECEIVE SID, RECEIVE
`SPEECH RECOGNITION RESULT
`
`SP4
`
`AFFIRMATIVE
`ANSWERT
`
`Y
`
`SP5
`
`ADDITIONAL STUDY START COMMAND
`
`SP6
`
`l
`
`GOSSIP PROCESSING
`
`SP7
`
`ADDITIONAL STUDY STOP COMMAND
`
`STOP
`
`SP20
`
`FIG 10
`
`IPR2023-00037
`Apple EX1019 Page 10
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 10 of 25
`
`US 2003/0220796 A1
`
`NEW PERSON spi
`NEW STUDY START COMMAND
`
`
`
`
`
`
`
`SP8
`
`QUESTION
`"Oh! What's your name?"
`SP
`9
`
`RECEIVE SID
`RECEIVE SPEECH
`RECOGNITION RESULT
`
`
`
`
`
`
`
`
`
`NEW PERSON?
`JUDGEMEN
`KNOWN PERSO
`SPS ASSEE
`GOSS P PROCESSING
`
`
`
`
`
`
`
`
`
`GOSSP PROCESSING .
`
`SP13
`
`AMOUNT O
`SUDY is ENOUGH
`
`NEW STUDY STOP COMMAND
`
`MEMORY REGISTRATION
`
`
`
`ADDITIONAL STUDY
`START COMMAND
`
`GOSS P PROCESSING
`
`ADDITIONAL STUDY
`STOP COMMAND
`
`F.G. 11
`
`IPR2023-00037
`Apple EX1019 Page 11
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 11 of 25
`
`US 2003/0220796A1
`
`R: Are you Mr. Fujita?
`H: Yes, I am.
`R. It's a fine day.
`H. Yes, it is.
`
`F.G. 12
`
`
`
`R. Ahl Are you. Mr. Fujita?
`H: No, I am not.
`R. Tell me your name, please.
`My name is Yamamoto.
`H:
`am Robot. Please give your favors toward me.
`R.
`ask your favors too.
`H.
`Mr. Yamamoto, it's a nice day, isn't it?
`R.
`Yes, it is.
`H:
`
`FIG. 13
`
`IPR2023-00037
`Apple EX1019 Page 12
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 12 of 25
`
`US 2003/0220796A1
`
`
`
`
`
`S I D
`
`NAME
`
`6
`
`
`
`YAMAMOTO
`FIG. 14
`
`
`
`(SA. you are Mr. Fujita, aren't you)
`HNo, I am not.
`R. Ah, so, then tell me your name, please.
`H: I am Yoshida.
`R. Ah, you are Mr. Yoshida.
`remember you.
`H. Yes, it is.
`R. When did we meet last time?
`H. Oh, I think we met the day before yesterday.
`
`It's a fine day, isn't it?
`
`FIG 15
`
`
`
`(SR: are you Mr. Fujita.)
`H: No, I am not.
`R. Ah, then, tell me your name.
`H. My name is Yamamoto.
`R. Ah, is that so. Are you fine?
`H. Yes, I am fine.
`
`FG 16
`
`IPR2023-00037
`Apple EX1019 Page 13
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 13 of 25
`
`US 2003/0220796 A1
`
`S1B
`
`90
`
`AD CONVERTER UNIT
`91
`
`FEATURE EXTRACTION
`UNIT
`
`MATCHING UNIT
`
`92
`
`
`
`
`
`96
`
`
`
`
`
`
`
`level wa?
`UNREGISTERED WORD
`SECTIONERCESSING
`--
`C C
`FEATURE VECTOR
`BUFFER
`
`- -
`-
`ACCOUSTIC MODEL
`MEMORY UNIT
`
`93
`
`94
`
`DCTIONARY
`MEMORY UNIT
`7
`st
`c
`
`GRAMMER
`MEMORY UNIT
`
`
`
`
`
`
`
`CLUSTERING UNIT
`
`SCORE SHEET
`MEMORY UNIT
`
`
`
`MAINTENANCE UNIT
`
`HEADLINE (TITLE)
`
`m
`
`F.G. 17
`
`IPR2023-00037
`Apple EX1019 Page 14
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 14 of 25
`
`US 2003/0220796 A1
`
`TITLE
`
`PHONEME
`SERIES
`
`chigau
`doko
`genki
`iro
`janai
`
`
`
`
`
`
`
`NO)
`WHERE)
`WELL
`COLOR
`NOT
`HATE
`PLEASE
`
`
`
`F.G. 18
`
`
`
`Sco
`Sthis
`$neg
`Snull
`$des
`Snot
`Scolor
`Scoor2
`Spati
`
`sono) iro wa:
`Ikono
`kore (ga
`wa
`mo));
`Sigal iie) ISs it):
`S ,
`(desu
`da) (yo)
`yo;
`janai yo),
`Snegl $this,
`$null
`$neg
`Sneg) Scol
`(iro) (desu
`janai
`da) (yo):
`Scolor $garbage Scolor 2,
`
`FIG. 19
`
`IPR2023-00037
`Apple EX1019 Page 15
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 15 of 25
`
`US 2003/0220796A1
`
`ID | PHONEME SERIES
`-
`
`FEATURE VECTURE SERIES
`F
`-
`
`
`
`
`
`
`
`N
`
`
`
`-
`
`PHONEME SERIES
`doroa:
`kuro
`Nfuro
`NhoNn
`hoNN
`NhoNda
`NhoNde. Su
`ohoN
`hoNgdawasorea:
`amodori.
`Omidor i:
`e: imidor i :
`Ninidor i :
`amidor iiroiresu
`Nro. ka
`NrokaNa
`
`-
`
`UTTERANCE WORD
`FUR0 x 1.
`FUR0 x 3.
`FUR0 x 20,
`HON 18,
`HONX 6;
`HON x 10;
`.
`.
`HONX 4,
`ORANGEx HON X 19:
`HON x 2,
`MDOR x 11,
`MDOR x 10;
`MDOR x 3.
`MDOR x 5;
`MDOR x 4,
`ROKA x 10;
`ROKA x 10:
`
`
`
`FIG. 25
`
`IPR2023-00037
`Apple EX1019 Page 16
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 16 of 25
`
`US 2003/0220796 A1
`
`
`
`s
`
`IPR2023-00037
`Apple EX1019 Page 17
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 17 of 25
`
`US 2003/022079 A1
`
`RT2.--O START D-u-SP30
`
`--a
`
`N
`
`
`
`AREIHERE ANY
`UNREGISTERED WORDS?
`
`Y-u-SP34
`
`IPR2023-00037
`Apple EX1019 Page 18
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 18 of 25
`
`US 2003/0220796A1
`
`RT3
`
`START.
`
`SP40
`
`READ OUT FEATURE WECTOR
`OF NEW REGISTERED WORD AND D
`
`SP41
`
`SP42
`
`SP44
`
`CLUSTER EXISTS?
`
`CALCULATE SCORE
`
`
`
`
`
`SP45
`
`
`
`
`
`DELECT CLUSTER HAVING
`REPRESENTATIVE REMBER TO MAKE
`THE SCORE OF NEW UNREGISTERED
`WORD THE HIGHEST
`
`SP43
`FORM CLUSER HAVING
`NEW UNREGISTERED WORD
`AS REPRESENTAT VE
`MEMBER AND REGISTER
`
`SP53
`
`
`
`
`
`OBTAIN REPRESENTAT VE
`MEMBER OF DETECTED
`CLUSTER
`
`DSTANCE
`BETWEEN
`CLUSERS
`
`
`
`X e 2
`
`DCTIONARY MANTENANCE
`
`SP52
`
`SP54
`
`FIG. 23
`
`IPR2023-00037
`Apple EX1019 Page 19
`
`

`

`Patent Application Publication Nov. 27, 2003. Sheet 19 of 25
`
`US 2003/0220796 A1
`
`SP60
`
`SELECT 2 OPTIONAL MEMBERS FROM DETECTED
`CLUSTER AND MAKE THESE TO TENTATIVE
`REPRESENTATIVE MEMBERS
`
`CAN DETECTED CLUSTER BE DIVIDED SO THAT 2
`TENTATIVE REPRESENTATIVE MEMBERS CAN BECOME
`REPRESENTATIVE MEMBERS
`
`DDWE DETECTED CLUSTER AND MAKE
`THESE TO CAND DATE CLUSTERS
`
`ARE THERE ANY COMBINATION OF
`MEMBERS NOT SELECTED?
`N
`
`--SP64
`
`A GROUP OF CAND DATE CLUSTER EXSS
`Y
`DETECT A GROUP OF CAND DATE CLUSTERS
`HAV ING THE SMALLES DISTANCE BETWEEN
`CLUSTERS, AND MAKE THIS DVISION RESULT
`OF DETECTED CLUSTER
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`RETURN
`
`SP67
`
`FIG. 24
`
`IPR2023-00037
`Apple EX1019 Page 20
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 20 of 25
`
`US 2003/0220796A1
`
`Request from
`user for Word
`game
`
`
`
`
`
`
`
`
`
`
`
`
`
`COMPARE USERS PROFILE
`AND OPTION DATA OF
`CONTENT DB
`
`LOAD CONTENT DATA
`MATCHED TO USER'S
`PROFILE
`
`SP75
`SEND CUT-OFF REQUEST
`TO CONTENT DB
`
`ESTABLSH CONNECTION
`WITH ROBOT
`
`SP82
`
`SP83
`TRANSMIT DATA REQUESTED
`BY ROBOT
`
`SP84
`CUT OFF CONNECTION
`WITH ROBOT
`
`SP85
`
`FG. 26
`
`IPR2023-00037
`Apple EX1019 Page 21
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 21 of 25
`
`US 2003/0220796A1
`
`NAME OF USER
`
`Sankakuko
`Shikakuyama
`Batsuo
`
`nazonazo
`yamanote line
`
`2
`
`2
`
`LEVELD ALREADY PLAYED THE NUMBER PLAYED
`3.
`2,
`2, 5
`
`3.
`
`
`
`
`
`2
`
`FIG. 27
`
`D
`
`1
`
`3
`
`
`
`
`
`REASON
`
`g
`
`OPTION DATA
`LEVE POPULARITY
`
`
`
`
`
`5
`
`7
`
`3
`
`2
`
`3
`
`5
`
`2
`-
`
`CONTENT DATA
`QUESTION
`ANSWER
`Where is the foreign city
`in which only 4 and 5 (shikago)
`years old children live?
`What kind of car in which
`2 only few people ride but Ambulance, (Kyukyusha
`ful of people?
`(kyukyu de ful)
`What part of the house
`(genkan)
`having poor heating
`(very cold
`system?
`de genkan)
`If you eat twice, you will
`get excited even when you
`4 are in sad mood, what's
`the name of that food?
`
`Entrance
`
`(nori)
`
`(twice makes
`nor inori)
`
`FIG. 28
`
`IPR2023-00037
`Apple EX1019 Page 22
`
`

`

`Patent Application Publication
`
`Nov. 27, 2003 Sheet 22 of 25
`
`US 2003/0220796 A1
`
`‘Uosead9U}49117
`
`von
`
`
`
`jdnAdin£27e8AON
`
`‘aj{ppi4e@Aejds49a]
`
`Aas]
`
`LON
`
`‘uoseadBY}19337]
`
`i000S!4emsueaul
`
`NOK0g‘“BUOUM$I‘ON
`
`{4emsueel}Jeey0}URN
`
`
`
`YAMSNYLOSYYOON|
`
`g4emsue94}Jee0}JUN
`NOA0g‘SUOIMS3!‘ON
`
`
`‘eqepaiinboyjuajuod
`
`i000S!4emsuesuli0QS!4amsueaul
`
`
`
`QW!}paxilay}4azj4eyeedsjousecpJasf
`
`Pauleiqo-ys9saquessoe}WY)jys!4aseno,}oeqn[&xxx]Y"GdWot}
`
`i00S]Jomsueau]gourdj3nsdasn-.[|
`
`YIMSNV1934409a|pp!44ndqngq+
`
`
`‘dnones|PNZNLON—)
`cqo=9OURI}INSjOdOL...[_]
`
`
`
`6¢Sls
`
`IPR2023-00037
`Apple EX1019 Page 23
`
`IPR2023-00037
`Apple EX1019 Page 23
`
`
`
`
`
`
`
`
`
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 23 of 25
`
`US 2003/0220796 A1
`
`RT8
`
`SP 100
`
`SP10
`START CONNECTION
`WTH ROBOT
`
`
`
`
`
`-a- - - -
`
`SEND QUESTION, e.g.,
`"Is this question
`interesting?" TO USER
`
`- as
`
`a
`
`RECEIVE ANSWER FROM USER
`
`It was not fun
`
`t was interesting.
`
`
`
`SP96
`SEND INCREMENT
`REQUES OF POPULARITY
`VALUE TO SERVER
`
`SEND DECREMENT
`REQUEST OF POPULARTY
`VALUE TO SERVER.
`
`
`
`-
`
`- -
`
`-
`
`CHANGE OPTION DATA
`VALUE BASED ON
`REQUEST FROM ROBOT
`
`FG. 30
`
`IPR2023-00037
`Apple EX1019 Page 24
`
`

`

`Patent Application Publication Nov. 27, 2003 Sheet 24 of 25
`
`US 2003/0220796 A1
`
`
`
`RTO
`
`SP110
`SP111
`
`VSP20
`SP21
`
`w
`
`WITH ROBOT
`
`SP122
`ADD NEW CONTENT D TO
`DATA TRANSMITTED FROM
`ROBOT
`
`SP123
`
`STORE NEW DATA IN
`DATABASE
`
`INFORM ROBOT OF
`TERMINATION OF DATA
`ADDITION
`
`
`
`SP 125
`CUT OFF CONNECTION
`WITH ROBOT
`
`Acquire question
`contents from
`USe
`
`
`
`
`
`a gigs -
`
`TO CONTENT DB
`SP 12
`
`
`
`
`
`
`
`
`
`SEND CUT-OFF REQUEST
`TO CONTENT DB
`
`FIG. 31
`
`IPR2023-00037
`Apple EX1019 Page 25
`
`

`

`Patent Application Publication
`
`Nov. 27, 2003 Sheet 25 of 25
`
`US 2003/0220796 A1
`
`
`
`
`[B4-1U)|JOUT1OU|paieadasaouanbes©soyewaolmy
`
`‘gJ0WG0UGJeMSUeBl|{a
`(a|dwex3)
`{40U110UapuoseedalyseGLON
`
`
`Jejoeseyo103s|Foy|PesULa4eNOKUAH)—opoyguiyjouseyjogos94}PUY
`
`vVVdajpoeseyo49481994
`
`pajeadasaouenbesauSJURA‘poow
`
`uo|jsenb9yyse.(poo,yeyJoeweu
`VIONeLON
`yous!3
`
`“a40W3OUOJOMSURsli|[alN
`
`uoljsend3u!}saiajqul
`
`(agmsue94}}eeday)
`Oa]
`quBla8yey"S84
`(uoljsenbey}jJeedey)6XXXTIS|
`
`
`_.40msueSe
`
`
`
`49joe1eYD493S/394
`
`pajeedeisouenbes
`
`“joust41
`
`“ON
`
`SON
`
`j4ouseyons
`
`‘00
`
`400§}|
`
`"xxxxX
`
`uarepajloxejad||IMNOK
`
`
`‘a91M}4eaNOK4|
`‘gasequed90k)$43S
`
`
`
`ueall||3}asee|q
`
`(uosea49}1Beday)4]
`
`49481394||,|
`
`
`
`VVY"qUud14S$7eYR‘Sa,
`
`
`
`“g40WBOUOUOSES494}OW|(2)
`
`‘yOuSs!4!
`
`“ON
`
`YALS19Y
`
`
`
`NOILS3N0
`
`ceOld
`
`‘Oo
`
`"ALJOSWl
`|40
`
`gourszynsjogoi.-.[|
`
`
`
`gours}ynsjsesn---[|
`
`IPR2023-00037
`Apple EX1019 Page 26
`
`IPR2023-00037
`Apple EX1019 Page 26
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`US 2003/0220796 A1
`
`Nov. 27, 2003
`
`DIALOGUE CONTROL SYSTEM, DIALOGUE
`CONTROL METHOD AND ROBOTC DEVICE
`
`BACKGROUND OF THE INVENTION
`0001) 1. Field of the Invention
`0002 The present invention relates to a dialogue control
`System, a dialogue control method and a robotic device and
`is Suitably applicable to Such as an entertainment robot.
`0003 2. Description of the Related Art
`0004 Entertainment robots for general households have
`been developed and commercialized in many companies in
`recent years. Some of these entertainment robots are
`equipped with various external Sensors Such as a charge
`coupled device (CCD) camera and a microphone, and these
`can recognize the external conditions based on these exter
`nal Sensors and can function automatically based on Such
`recognition.
`0005. In the case of constructing the audio interactive
`System in which a robot and the user conduct the audio
`conversation, an audio interactive System aimed at accom
`plishing Some task, Such as receiving the telephone shop
`ping, and informing the telephone number, can be consid
`ered.
`0006 Assuming the scene in which the daily conversa
`tion is conducted between a robot and a man, the robot
`should come up in conversation Such as gossip talk and
`playing on words, i.e., the conversation that would not be
`tiring even if it is conducted every day, in addition to the
`dialogue just to accomplish his task. However, in the inter
`active System aimed at accomplishing Such task, Since the
`data Such as the telephone number list and the Shopping item
`list in the System were fixed to the Specific contents, the
`conversation of the robot could not have fun. And further
`more, the data in Said System could not be changed accord
`ing to the taste of a person who was using Said System.
`0007 Especially, in the case where the robot and the man
`conduct the conversation by playing on words, Such as
`giving a riddle and Yamanote-line game (the game to
`eXchange words having contents related to the Specific item
`not repeating the same word each other) as the daily con
`Versation, it is necessary for the robot to hold a large Volume
`of data showing the conversation contents (hereinafter
`referred to as content data).
`0008. In recent years, Web (i.e., World Wide Web:
`WWW), an information net that made various kinds of
`documents among Servers distributed on the Internet Search
`able connecting each form of document, has been widely
`used as an information Service. And using Such Web, the
`content Server having a large Volume of contents exchanges
`the content data to be held by the robot eXchanging the
`content data among robots, and thus, it is considered that the
`user facing to Said robot can conduct the daily conversation.
`0009 Said content server stores database to which all
`robots capable of using a large Volume of content data can
`access, and reading out content data corresponding to Said
`database as occasion demands, can make the robot utter via
`the network.
`0.010 However, in the case of conducting the word game
`between the robot and the user, the method that the robot
`
`acquiring the content data randomly from enormous Volume
`of content data Stored in the database cannot Satisfy needs of
`all users Since each user has his own taste and the Skill to
`cope with the difficulty is diversified each other.
`0011. As a method to solve this problem, the profile
`information showing the user's taste and his level and
`classification information having Supplemental contents
`would be Stored in the database in advance, and the method
`that the content Server Selects the content data associated
`with the profile information and the classification informa
`tion when the content Server acquires the content data that
`the user desires from the database in response to the request
`of the robot can be considered.
`0012 However, in the dialogue aiming at the word game
`Such as playing riddles and Yamanote-line game, rhythm and
`amusingness of the conversation will be required between
`the robot and the user. However, according to the present
`Speech recognition processing technique, the recognition
`error to the user's Speech cannot be prevented, and if the
`robot confirms contents of the user's Speech in each time, the
`conversation between the user becomes unnatural.
`0013 More specifically, in the case where the user
`answers “nori (seaweed)” when the robot proposes playing
`a riddle, "If you eat twice, you will get eXcited, what's the
`name of that food?”, if the robot utters as “it’s nori' directly
`confirming, it stops the flow of conversation and at the same
`time loses amusingneSS.
`0014. On the other hand, if the robot continues the
`conversation ignoring the contents of user's speech, the user
`could not confirm how the robot recognized the contents of
`conversation and the user had the Sense of anxiety during the
`conversation.
`
`SUMMARY OF THE INVENTION
`0015. In view of the foregoing, an object of this invention
`is to provide a dialogue control System, a dialogue control
`method and a robotic device capable of remarkably improv
`ing the entertainment factor.
`0016. According to the present invention described
`above, in the dialogue control System in which the robot and
`the information processing device are connected via the
`network, Since in the case of interacting by playing word
`games between the robot and the user, the history data
`concerning the word game in the user's Speech contents is
`formed and transmitted to the information processing device
`and Said information processing device Selectively reads out
`the content data best Suited to the user from the memory
`means based on Said history data and provides to the original
`robot, the conversation between the user and the robot can
`have amusingness and rhythm, and can be brought closer to
`natural daily conversation as if the fellow men are talking.
`Thereby, the dialogue control System capable of remarkably
`improving the entertainment factor can be realized.
`0017 According to the present invention, in the dialogue
`control method in which the robot and the information
`processing device are connected via the network, Since in the
`case of interacting by playing on words between the robot
`and the user, the history data concerning the word game in
`the user's Speech contents is formed and transmitted to the
`information processing device, and Said information pro
`cessing device Selectively reads out the content data best
`
`IPR2023-00037
`Apple EX1019 Page 27
`
`

`

`US 2003/0220796 A1
`
`Nov. 27, 2003
`
`Suited to the user from multiple content data based on the
`history data and provides to the original robot, the conver
`sation between the user and the robot can have amusingneSS
`and rhythm and can be brought closer to natural daily
`conversation as if the fellow men are talking. Thereby, the
`dialogue control method capable of remarkably improving
`the entertainment factor can be realized.
`0.018 Moreover, according to the present invention, in
`the robotic device to which the information processing
`device is connected via the network, Since the interactive
`means having the function to interact with the man and for
`recognizing the user's Speech through the conversation, the
`forming means for forming the history data on the word
`game from the user's Speech contents by the interactive
`means, the updating means for updating the history data
`formed by the forming means based on user's Speech
`contents obtained through the word game and the commu
`nication means for transmitting the history data to the
`information processing device via the network when Starting
`the word game are provided; and when content data Selected
`based on the history data transmitted from the communica
`tion means is transmitted via the network out of content data
`showing the contents of multiple word games memorized in
`advance in the information processing device, the interactive
`means outputs contents of the word game based on Said
`content data, the conversation between the user and the robot
`can have amusingness and rhythm and can be brought closer
`to natural daily conversation as if the fellow men are talking.
`Thereby the robotic device capable of remarkably improving
`the entertainment factor can be realized.
`0019. The nature, principle and utility of the invention
`will become more apparent from the following detailed
`description when read in conjunction with the accompany
`ing drawings in which like parts are designated by like
`reference numerals or characters.
`
`BRIEF DESCRIPTION OF DRAWINGS
`In the accompanying drawings:
`0020
`FIG. 1 is a perspective view showing the external
`0021
`construction of a robot according to the present invention;
`0022 FIG. 2 is a perspective view showing the external
`construction of a robot according to the present invention;
`0023 FIG. 3 is a perspective view showing the external
`construction of a robot according to the present invention;
`0024 FIG. 4 is a block diagram showing the internal
`construction of a robot;
`0.025
`FIG. 5 is a block diagram showing the internal
`construction of a robot;
`0.026
`FIG. 6 is a brief linear diagram showing the
`construction of the dialogue control System according to the
`present invention;
`0.027
`FIG. 7 is a block diagram showing the construction
`of a content server shown in FIG. 6;
`0028 FIG. 8 is a block diagram showing the processing
`of main control unit 40;
`0029 FIG. 9 is a conceptual diagram showing the rela
`tionship between SID and name in the memory;
`
`0030 FIG. 10 is a flow chart showing the name study
`processing procedure;
`0031
`FIG. 11 is a flow chart showing the name study
`processing procedure;
`0032 FIG. 12 is a diagram showing dialogue examples
`at the time of name Study processing;
`0033 FIG. 13 is a diagram showing dialogue examples
`at the time of name Study processing;
`0034 FIG. 14 is a conceptual diagram showing the new
`registration of SID and name;
`0035 FIG. 15 is a diagram showing dialogue examples
`at the time of name Study,
`0036 FIG. 16 is a diagram showing dialogue examples
`at the time of name Study,
`0037 FIG. 17 is a block diagram showing the construc
`tion of audio recognition unit;
`0038 FIG. 18 is a conceptual diagram illustrating the
`word dictionary;
`0039 FIG. 19 is a conceptual diagram illustrating the
`grammatical rule;
`0040 FIG. 20 is a conceptual diagram illustrating the
`memory contents of feature vector buffer;
`0041
`FIG. 21 is a conceptual diagram illustrating the
`Score sheet,
`0042 FIG. 22 is a flow chart showing the audio recog
`nition processing procedure;
`0043 FIG. 23 is a flow chart showing the unregistered
`word processing procedure;
`0044 FIG.24 is a flow chart showing the cluster division
`processing procedure;
`004.5
`FIG. 25 is a conceptual diagram showing the
`Simulation result;
`0046 FIG. 26 is a flow chart showing the content data
`acquisition processing procedure and the content data offer
`ing processing procedure;
`0047 FIG. 27 is a conceptual diagram illustrating the
`profile data;
`0048 FIG. 28 is a conceptual diagram illustrating the
`content data;
`0049 FIG. 29 is a conceptual diagram illustrating the
`dialogue Sequence according to the Word game;
`0050 FIG. 30 is a flow chart showing the popularity
`indeX Summing processing procedure and the option data
`updating processing procedure,
`0051 FIG. 31 is a flow chart showing the content col
`lection processing procedure and the content data add-up
`registration processing procedure; and
`0052 FIG. 32 is a conceptual diagram illustrating the
`dialogue Sequence according to the word game.
`DETAILED DESCRIPTION OF THE
`EMBODIMENT
`0053 Preferred embodiments of this invention will be
`described in detail with reference to the accompanying
`drawings:
`
`IPR2023-00037
`Apple EX1019 Page 28
`
`

`

`US 2003/0220796 A1
`
`Nov. 27, 2003
`
`0054 (1) Construction of Robot According to the Present
`Invention
`0055. In FIGS. 1 and 2, Reference numeral 1 generally
`shows a two-foot walking type robot according to the
`present invention. This robot comprises a head unit 3 which
`is provided on the upper part of a body unit 2, and arm units
`4A, 4B having the same construction which are placed on
`the left and right of the upper part of said body unit 2
`respectively, and leg units 5A, 5B having the same con
`Struction which are attached respectively to the predeter
`mined positions on the right and left of the lower part of the
`body unit 2.
`0056. The body unit 2 is comprised of a frame 10 forming
`the upper part of the main body and a waste base 11
`connected via the waste joint System 12, by driving each
`actuator 1A, A2 of the waste joint system 12 fixed to the
`waste base 11 of the lower part of the body, the upper part
`of the body can be rotated about the roll axis 13 and the pitch
`axis 14 independently shown in FIG. 3, which are orthogo
`nal to each other.
`0057. Furthermore, the head unit 3 is attached to the
`upper surface central part of the shoulder base 15 fixed to the
`upper edge of the frame 10 via the head joint system 16, and
`by driving each actuator A3, A4 of the neck joint system 16
`respectively, the head unit 3 can be rotated about the pitch
`axis 17 and the yawing axis 18, which are orthogonal to each
`other, shown in FIG. 3.
`0.058. Furthermore, arm units 4A, 4B are attached to the
`right and left of the shoulder base 15 via the shoulder joint
`system 19 respectively, and by driving the actuators A5, A6
`of the corresponding shoulder joint System 19 respectively,
`the arm units 4A, 4B can be rotated about the pitch axis 20
`and the roll axis 21, which are orthogonal to each other,
`shown in FIG. 3, respectively.
`0059. In this case, each of arm units 4A and 4B is
`comprised of an actuator A8 forming the front arm part
`connected to the output axis of the actuator A7 forming its
`upper arm part via the elbow joint System 22 and a hand unit
`23 is attached to the edge of Said front arm part.
`0060. Then, in the arm units 4A and 4B, the front arm part
`can be turned about the yawing axis 24 shown in FIG. 3 by
`driving the actuator A7, and the front arm part can be turned
`about the pitch axis 25 shown in FIG. 3 by driving the
`actuator A8.
`0061. On the other hand, leg units 5A and 5B are attached
`to the waste base 11 of the lower body part via the coxa
`System 26 respectively, and by driving the corresponding
`actuator A9-A11 of the coxa system 26, these can be rotated
`about the yawing axis 27, roll axis 28 and the pitch axis 29,
`which are orthogonal each other, shown in FIG. 3 indepen
`dently.
`0.062. In this case, in leg units 5A, 5B, frame 32 forming
`the lower thigh part is connected to the lower edge of the
`frame 30 forming the thigh part via the knee joint system 31,
`and the leg part 34 is connected to the lower edge of the
`frame 32 via the ankle joint system 33.
`0063 Thus, in the leg units 5A and 5B, by driving the
`actuator A12 forming the knee joint System 31, its lower
`thigh part can be rotated about the pitch axis 35, and by
`driving actuators A13, A14 of the ankle joint system 33
`
`respectively, the leg part 34 can be rotated about the pitch
`axis 36 and the roll axis 37 orthogonal to each other, shown
`in FIG. 3 independently.
`0064. On the other hand, on the back side of the waste
`base 11 forming the body stem lower part of the body unit
`2, a main control unit 40 for controlling the whole operation
`of the robot 1, as shown in FIG.4, a control unit 42 in which
`the peripheral circuit 41 Such as the power Source circuit and
`the communication circuit, and a battery 45 (FIG. 5) are
`stored in the box is provided.
`0065. Then, this control unit 42 is connected respectively
`to each of Sub-control units 43 A-43D provided in each of
`construction units (body unit 2, head unit 3, arm units 4A,
`4B and leg units 5A, 5B), and it supplies the required power
`Source voltage to these sub-control units 43A-43D and can
`communicate with these Sub-control units 43A-43D.
`0066 Furthermore, these sub-control units 43A-43D are
`connected respectively to corresponding actuators A1-A14
`in construction units, and can drive actuators A1-A14 in Said
`construction unit in the State Specified based on various
`control commands to be given from the main control unit 40.
`0067 Furthermore, as shown in FIG. 5, in the head unit
`3 a (charge coupled device) CCD camera 50 to function as
`“eyes” of the robot 1, a microphone 51 to function as “ears'
`of the robot 1, and an external sensor 53 formed of Such as
`touch sensor 52, and a speaker 54 to function as “mouth” are
`placed respectively on the predetermined positions. And the
`internal sensor 57 formed of such as the buttery sensor 55
`and the acceleration Sensor 56 are provided in the control
`unit 42.
`0068. Then, the CCD camera 50 of the external sensor 53
`takes pictures of the Surrounding conditions, and outputs the
`resultant image Signal S1A to the main control unit. While,
`the microphone 51 collects various command Sounds Such as
`“walk”, “lie down” or “chase after a ball” to be given from
`the user as the Speech input, and transmits the resultant audio
`signal S1B to the main control unit 40.
`0069. Moreover, as is clear from FIGS. 1 and 2, the
`touch Sensor 52 is provided on the upper part of the head unit
`3 and detects the preSSure received by the physical influence
`such as “hit' and “pat” from the user and outputs the
`detection result to the main control unit 40 as the pressure
`detection signal S1C.
`0070) Furthermore, the battery sensor 55 of the internal
`Sensor unit 57 detects the remaining quantity of energy in the
`battery 45 at the predetermined cycle and transmits the
`detection result to the main control unit 40 as the battery
`remaining quantity detection Signal S2A. On the other hand,
`the acceleration Sensor 56 detects the acceleration of 3-axis
`direction (X-axis, y-axis and Z-axis) at the predetermined
`cycle and transmits the detection result to the main control
`unit 40 as the acceleration detection signal S2B.
`0071. The main control unit 40 judges the surrounding
`condition and the internal condition of the robot 1, and the
`existence or non-existence of the command from the user
`and the influence of the user based the image Signal S1A,
`audio signal S1B and the pressure detection signal S1C to be
`supplied respectively from the CCD camera 50, microphone
`51 and touch sensor 52 of the external sensor unit 53
`(hereinafter referred to as external sensor Signal S1) and the
`
`IPR2023-00037
`Apple EX1019 Page 29
`
`

`

`US 2003/0220796 A1
`
`Nov. 27, 2003
`
`battery remaining quantity detection signal S2A and the
`acceleration detection Signal S2B to be Supplied from the
`battery sensor 55 and the acceleration sensor of the internal
`sensor unit 57 (hereinafter referred to as an internal sensor
`Signal S2).
`0.072 Then, the main control unit 40 determines the
`action to be followed based on Said judgment result and the
`control program Stored in advance in the internal memory
`40A and various control parameters Stored in the external
`memory 58 equipped at that time, and outputs the control
`command based on the determination result to the corre
`sponding sub-control units 43 A-43D. As a result, based on
`this control command, the corresponding actuators A1-A14
`will be driven under the control of the Sub-control units 43A
`-43D and thus, actions Such as making the head unit 3 Swing
`up and down, right and left and the arm units 4A, 4B put up,
`and to walk, can be realized by the robot 1.
`0073. Furthermore, in this case the main control unit 40,
`giving the predetermined audio signal S3 to the Speaker 54
`as necessary, outputs Speeches based on Said audio Signal S3,
`and by outputting the driving Signal to the LED provided on
`the predetermined part of the head unit 3 functioning as
`“eyes” by appearances, flushes this hea