I 1111111111111111 1111111111 11111 111111111111111 1111111111111111 IIII IIII IIII
`
`
`
`
`
`USO 100623 77B2
`
`c12) United States Patent
`Larri et al.
`
`(IO) Patent No.: US 10,062,377 B2
`* Aug. 28, 2018
`(45) Date of Patent:
`
`(54) DISTRIBUTED PIPELINED PARALLEL
`SPEECH RECOGNITION SYSTEM
`
`(71) Applicant: Zentian Limited, Cambridge (GB)
`
`(72)
`
`Inventors: Guy Larri, Cambridge (GB); Mark
`Catchpole, Prickwillow (GB); Damian
`Kelly Harris-Dowsett, Conventry
`(GB); Timothy Brian Reynolds,
`Cambridge (GB)
`
`(73) Assignee: Zentian Limited, Cambridge (GB)
`
`(58) Field of Classification Search
`CPC ....................................................... Gl0L 15/10
`(Continued)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,881,312 A
`6,374,220 Bl
`
`3/1999 Dulong
`4/2002 Kao
`(Continued)
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`EP
`GB
`
`This patent is subject to a terminal dis(cid:173)
`claimer.
`
`FOREIGN PATENT DOCUMENTS
`
`2/2002
`1 178 466 A2
`7 /1999
`2 333 172 A
`(Continued)
`
`OTHER PUBLICATIONS
`
`(21) Appl. No.: 14/788,164
`
`(22) Filed:
`
`Jun. 30, 2015
`
`(65)
`
`Prior Publication Data
`
`US 2016/0005397 Al
`
`Jan. 7, 2016
`
`Related U.S. Application Data
`
`(60) Continuation of application No. 13/735,091, filed on
`Jan. 7, 2013, now Pat. No. 9,076,441, which is a
`(Continued)
`
`(30)
`
`Foreign Application Priority Data
`
`Sep. 14, 2004
`
`(GB) ................................... 0420464.0
`
`(51)
`
`Int. Cl.
`Gl0L 15110
`Gl0L 15108
`
`(2006.01)
`(2006.01)
`(Continued)
`
`(52) U.S. Cl.
`CPC .............. Gl0L 15108 (2013.01); Gl0L 15102
`(2013.01); Gl0L 15106 (2013.01); Gl0L 15110
`(2013.01); Gl0L 151285 (2013.01)
`
`Glinski et al., "Spoken Language Recognition on DSP Array
`Processor" IEEE Transactions on Parallel and Distributed Systems,
`vol. 5, No. 7, pp. 697-703, Jul. 1, 1994.
`(Continued)
`
`Primary Examiner - Susan McFadden
`(74) Attorney, Agent, or Firm - Blank Rome LLP
`
`ABSTRACT
`(57)
`A speech recognition circuit comprising a circuit for pro(cid:173)
`viding state identifiers which identify states corresponding
`to nodes or groups of adjacent nodes in a lexical tree, and for
`providing scores corresponding to said state identifiers, the
`lexical tree comprising a model of words. The circuit
`includes: a memory structure for receiving and storing state
`identifiers identified by a node identifier identifying a node
`or group of adjacent nodes, the memory structure being
`adapted to allow lookup to identify particular state identi(cid:173)
`fiers, reading of the scores corresponding to the state iden(cid:173)
`tifiers, and writing back of the scores to the memory
`structure after modification of the scores; an accumulator for
`receiving score updates corresponding to particular state
`identifiers from a score update generating circuit which
`generates the score updates using audio input, for receiving
`(Continued)
`
`107
`
`105
`
`103
`
`104
`
`Front End
`
`Digital
`Digital
`Audio Data Audio Data
`
`Featllre
`Vectors
`
`Reoognlsed
`Text
`
`IIJ Functions performed by the accelerator are shown in grey
`O Functions performed by other hardware or software systems are shown in white
`
`IPR2023-00035
`Apple EX1001 Page 1
`
`

`

`US 10,062,377 B2
`Page 2
`
`scores from the memory structure, and for modifying said
`scores by adding said score updates to said scores; and a
`selector circuit for selecting at least one node or group of
`adjacent nodes of the lexical tree according to said scores.
`
`6 Claims, 27 Drawing Sheets
`
`Related U.S. Application Data
`
`continuation of application No. 13/162,128, filed on
`Jun. 16, 2011, now Pat. No. 8,352,262, which is a
`division of application No. 11/662,704, filed as ap(cid:173)
`plication No. PCT/GB2005/003554 on Sep. 14, 2005,
`now Pat. No. 7,979,277.
`
`(51)
`
`Int. Cl.
`(2013.01)
`Gl0L 15128
`(2013.01)
`Gl0L 15106
`(2006.01)
`Gl0L 15102
`( 58) Field of Classification Search
`USPC
`.................................................. 704/238, 251
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`7,328,153 B2
`7,979,277 B2
`8,036,890 B2
`8,352,262 B2
`8,612,227 B2
`9,076,441 B2
`2003/0061046 Al
`2004/0111264 Al
`
`2/2008 Wells et al.
`7/2011 Larri et al.
`10/2011 Catchpole
`1/2013 Larri et al.
`12/2013 Kato
`7/2015 Larri et al.
`3/2003 Zhao et al.
`6/2004 Wang et al.
`
`FOREIGN PATENT DOCUMENTS
`
`GB
`WO
`WO
`WO
`
`2 391 679 A
`WO 01/48737 A2
`WO 01/75862 A2
`WO 03/067572 A2
`
`2/2004
`7/2001
`10/2001
`8/2003
`
`OTHER PUBLICATIONS
`
`S. Chatterjee et al., "Connected Speech Recognition on a Multiple
`Processor Pipeline." ICASSP 89, pp. 774-777, May 23, 1989.
`
`IPR2023-00035
`Apple EX1001 Page 2
`
`

`

`-....l = N
`N w
`"""' = = 0--,
`d r.,;_
`
`-....l
`
`0 ....
`('D a ....
`rJ'1 =(cid:173)
`
`-....J
`N
`
`0 ....
`
`QO
`
`~
`
`N
`QO
`N
`
`> = ~
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`figure 1b
`to
`
`..__...,Joins
`
`, '
`
`-
`
`' "'
`
`r
`
`word hist 1/)'
`
`~
`
`~ -
`
`recognltlon
`
`Interface
`output
`
`-...
`
`outpu
`recoq r1itio1 -
`
`...... .,. ...
`,,
`
`"'"-~ ~--·
`wordlat__node_id
`word history
`
`generator
`
`Word
`
`-graph
`
`-
`
`word lattlce
`
`start.Jrame ' l
`
`score
`word_id
`
`mph
`
`prev_wordlat_no<:re_id
`word level
`
`GPIO, timers
`clocks, PLL,
`controller,
`
`watchdog
`
`controller.
`
`OMA
`
`system
`
`processor+
`floatill) point
`
`ARM9 +
`
`cache
`
`-
`
`-
`
`~ -
`~ -
`
`I
`
`memoiy interfaces
`
`AHB
`
`--
`
`~ -
`I
`
`-
`
`'
`
`i~
`
`one best
`n best list
`word graph
`
`Word Engine
`
`next grammer_no •e_id(s)
`next wordj:J(s}
`nextLT_id
`
`arammar Id
`grammar_node_id
`
`serial, host,
`
`display i/f
`debug,
`
`.,
`
`-~
`
`-l:ngine
`-iararnmar {select, dby) Pl)
`
`gran mar_f,
`
`...
`
`-
`
`,. bridge
`AHB
`APB to
`
`-
`
`chip/IP core
`
`model(s) --
`
`grammar
`
`-
`
`APB
`end
`front
`
`t
`i/f
`front end
`audio 1--tio--to
`
`models
`speech--t,,.
`
`level
`grammar
`
`i/f
`
`codec
`audio
`
`,.
`
`~
`
`-
`
`level
`system
`
`flash memory
`storage e.g.
`nonvolatile
`
`debug
`display, main memory
`host,
`
`Figure lA
`
`CODEC
`audio
`AC97
`
`..
`
`-
`
`input
`audio
`
`IPR2023-00035
`Apple EX1001 Page 3
`
`

`

`---l = N
`N w
`'"= = 0--,
`"'""'
`d r.,;_
`
`---l
`
`0 ....
`N
`('D .....
`rJJ =(cid:173)
`
`('D
`
`-....J
`N
`
`0 ....
`
`CIO
`
`N
`'"CIO
`N
`~
`~
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`Joins to
`
`-flgure1a
`
`control
`
`2eno system 1
`
`(optimode)
`
`engine
`
`mojeJ factor
`
`language
`
`(optimode)
`
`engine
`
`adaptation
`
`model
`
`state_ids
`
`lookup
`
`lookup
`state_id
`
`Triphone to
`
`triphone_id
`
`Phone Book
`
`phone generator
`
`phone
`
`wavefront
`
`new
`
`PINE array
`merge unit in
`
`to path
`
`Unit
`Merge
`Path
`
`phone CAM
`
`r wave front
`
`RAM
`
`phone record
`
`wave front
`
`monophone_id/s.
`
`Lexical tree(s)
`
`prev _phone_ no
`grammar_id
`grammar_node_id
`next_leftmph]d
`word/at node id
`current
`
`manager
`Front
`Wave
`
`LM interface
`
`Figure lB
`
`B
`
`beam pruning
`
`word-end pruning
`max model pruning
`
`pruning control
`
`l~P::;.
`
`unit
`
`manager
`wave front
`
`merge unit in
`
`from path
`
`next phone_ no
`last phone_no
`
`token
`
`passing
`
`inter word token
`
`Phone Instance Network Engine Array (PINE array)
`
`-~~~
`
`transition
`
`state:] PINE 1
`
`1
`
`probabiliti~s
`
`1
`
`output
`state
`
`engine
`MHD
`
`mean & covar data
`
`~---~
`
`• vectors
`ifeature
`
`end ilf
`front
`from
`
`frame count,
`
`Level
`State
`
`~-------~--
`
`acoustic models
`
`compressed
`
`buffer for model
`feature vector
`
`ilence
`ull transition vector models (e.g.
`
`models
`sf)_eech
`
`next phone request
`
`nextf)_hone
`
`score
`word_e-;,d_flag
`word/at node id
`current phone-no
`
`models
`speech
`
`engine
`
`adaptation
`
`model
`
`next phone request
`
`state_id(x3), trans_vec_id, word_end_flag, ful/_trans_flag,
`
`recor'l,rev_phone_no, phone_no, wordlat_node_id, LM factor
`
`switches
`buttons/
`
`models
`speech
`
`LMfa/;tor
`
`next phone response
`
`IPR2023-00035
`Apple EX1001 Page 4
`
`

`

`-....l = N
`N w
`'"= = 0--,
`"'""'
`d r.,;_
`
`-....l
`
`Joins to Figure 2b
`
`C
`
`Figure 2A
`
`wordlat_node_id = 5
`neither(worr.f_id = 3556)
`
`wordlat_node_id =6
`
`(word_id = OJ
`
`word level
`
`0 ....
`('D .....
`rJJ =(cid:173)
`
`-....J
`N
`
`~
`
`('D
`
`~
`
`which (word _ _id = 238)
`attrce
`
`silence < wordlat_node_id "'4
`
`0 ....
`
`CIO
`
`N
`'"CIO
`N
`~
`~
`
`LT_id=4
`grammar_node_Jd = 9
`word id=O
`silence
`9
`
`9
`
`grammar__id = 2
`
`LT_id=3
`grammar_node_id = 2
`word_id = 3556
`neither
`25
`
`23
`
`7
`
`LT_id=4
`grammar_node_id = 7
`word_id= O
`silence
`7
`
`grammar_id = 2
`
`grammar level
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`LT_id=3
`grammar node Id = 17
`word_id=45
`house
`17
`
`-
`
`LT_id=3
`grammar_node_id = 23
`word_id = 238
`which
`23
`
`grammar models
`
`IPR2023-00035
`Apple EX1001 Page 5
`
`

`

`.. ···············-····
`
`-
`
`-@
`
`•either"
`
`"either"
`
`D -@-
`
`-
`
`\al
`
`•neither'
`
`-
`
`_@
`
`"nelthlH"
`
`D -@-
`
`n < t -D
`
`el -
`
`•house•
`
`'\.2 _
`
`------
`
`S-"hOi.1$6"
`
`h-au<
`
`! .
`
`...._ 'which•
`
`-/-+S....._
`
`w
`
`'
`I
`
`'~"---!:=--:--=~=---···-=:;;;~-=::
`
`__,,,.,-,
`
`•••••••• I
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`Joins to Figure 2a
`
`lexical trees
`
`•
`
`Figure 2B
`
`"1 = N
`N w
`'"= = 0--,
`"'""'
`d r.,;_
`
`"1
`
`0 ....
`('D .....
`rJJ =(cid:173)
`
`.i;...
`
`('D
`
`-...J
`N
`
`0 ....
`
`CIO
`
`N
`'"CIO
`N
`~
`~
`
`trans lmJactor
`
`4
`
`0
`
`0
`
`0 1
`
`345 3456 456 7112
`
`320
`
`llldlctates phone Is
`
`on ,..,,vafront
`
`237
`
`22
`
`prev_phone_no wora/al_Mll11_ld tnwt lmJa-.tor
`
`phone instance RAM
`
`/JJ1on.i_nc
`
`phone instance CAM
`
`·····-·+·····-···········-···
`
`I -D
`
`I :
`I
`J
`i
`
`I
`
`h-1!1U+Z
`
`n9xt triphones: h •a/J+s
`
`phone level
`
`phone record
`
`wave front
`
`RAM
`
`IPR2023-00035
`Apple EX1001 Page 6
`
`

`

`U.S. Patent
`
`Aug. 28, 2018
`
`Sheet 5 of 27
`
`US 10,062,377 B2
`
`Figure 3A
`
`next
`phcf1IJ
`l'fJqt.l68t
`
`neid
`phones
`
`phooe manager
`
`Kword8deep
`
`f18Xtphone's
`phone_no
`
`phone_no
`
`nexlphone':i
`phont:JIQ
`
`mod N counter
`Is synchronous
`to MHO bus
`
`repeatcount
`fifo
`'----..s--,
`
`o utlll,Jlhaod
`
`tokens 18!'.foosf.s
`
`1----------------------1~
`
`H
`rtomotherPINE.
`tokens to other
`.,.◄.,.e.,../h..,_IE.IJ ___
`__,IJEJ141◄11-----------------(i)
`__ am .... wldlh=--------------------------►..-0
`_,..g;i.lobal=_be
`..,◄•Jocsl;..;,;;;.;be~st;.;.fok,=''";;;..SCCW=;,;._------------------------®
`..,..► (9
`
`
`__,sta=ts ... tta=ns!U=on-"mxf:='els_... _____________________
`
`IPR2023-00035
`Apple EX1001 Page 7
`
`

`

`U.S. Patent
`
`Aug. 28, 2018
`
`Sheet 6 of 27
`
`US 10,062,377 B2
`
`Figure 3B
`
`pointer lo previoua phone
`
`st_dum st_1 st_2 st_3
`
`tmp
`
`~ordlat_nodejd
`
`trans LM_factor
`
`end
`
`CAM addfeSS maps to RAM
`
`st_dum st_1 st_2
`
`st_3 tmp prev_phone_no wordlat_node_id trans
`
`Wrfte
`claf9
`
`phone Instance RAM
`model !!\$lances
`Kwords deep
`
`116W
`model
`address
`
`encoded
`address
`OU/pl.It
`
`match
`flfo
`
`updai0d
`tokttn
`
`$X)fflS
`
`output
`llklihood
`
`Ht-------_...
`0
`... ---------1..._
`
`17b 17b 17b 16b
`
`22b
`
`16b
`
`6_b
`
`16b
`
`1b
`
`ReadPott
`
`trans
`
`state
`1rans1t1ont...11-----
`RAM
`
`~
`tmnsltior,
`Hk/Jhood
`
`_____________________
`
`_.
`
`IPR2023-00035
`Apple EX1001 Page 8
`
`

`

`-....l = N
`N w
`'"= = 0--,
`"'""'
`
`-....l
`
`r.,;_
`
`d
`
`0 ....
`-....J
`('D .....
`rJJ =(cid:173)
`
`('D
`
`-....J
`N
`
`0 ....
`
`CIO
`
`N
`'"CIO
`N
`~
`~
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`to path merge
`
`wd_his/2
`wd_hist 1
`word id
`
`-
`
`st_fra'me
`word id
`word/at node id unit
`to word engine
`
`Figure 4
`
`grammar_id
`grammar_node_id
`to grammar engine
`
`ne;t_leff;ph_1d
`LT node id
`to phone_book
`
`10b
`
`20b
`
`20b
`
`20b
`
`4b
`
`10b
`
`20b
`
`13b
`
`Wntedata
`
`• possibly reducing memory requirements
`kinds of phones -word end phones and intra word phones
`NB Tne CAM-RAM structure can be split to handle to
`
`wave front phone record RAM
`
`Kwords deep
`
`address
`
`wordiat_node_id(not strictly needed)
`word td
`grammar_id
`grammar_node_id
`LT_node_id
`
`1b
`
`19b
`
`data
`
`binary
`single match
`K words deep
`CAM
`phone
`
`I write address
`I
`I
`!
`I
`i
`
`I
`I
`i
`
`phone __ no
`"vord_1d
`grammar_rd
`grammar_node_1d
`LT_node_1d
`next_Jeftmph _id
`from phone book
`
`t
`
`fifo
`
`phone_no
`
`r----+--twnte
`
`£ddress
`next_em~
`
`iempty_count I
`
`manager
`occupancy
`CAM RAM
`
`RAM
`maps to
`address
`CAM
`
`wave front
`
`control
`
`match dDtQ
`
`word_end_flag
`
`request
`phone
`nex I
`
`word_end_fiaQ
`word/at node 1d
`phone_no
`
`I word id '4d_hist1 Y.,d_hist2 ,t_frame
`
`next,,Jeftrnph)dJL_T_11ode,..i~J~~!_!!mar_node_id lgrammar_id
`
`phone ... no __ .1. _ occupy
`
`IPR2023-00035
`Apple EX1001 Page 9
`
`

`

`U.S. Patent
`
`Aug. 28, 2018
`
`Sheet 8 of 27
`
`US 10,062,377 B2
`
`Figure 5
`
`situation at frame m
`
`co O 0) II
`
`first
`trlphone
`In word
`
`II
`
`word
`It 0 0 QU---411l...---r'---
`last
`link
`trlphone
`in word
`
`II
`
`• II
`
`word
`end
`
`situation at frame n
`(n>m)
`
`first trl hone
`
`co o o) • • • _o_o_o
`__
`
`word k
`
`ti-lphooo
`
`word k (with identical word history}
`(0 00)
`II O O 000
`
`trlphone
`
`•
`• •
`
`word
`ena
`
`word
`
`IPR2023-00035
`Apple EX1001 Page 10
`
`

`

`U.S. Patent
`
`Aug. 28, 2018
`
`Sheet 9 of 27
`
`US 10,062,377 B2
`
`word_id
`
`d_hist1 wd_hist2 occupy
`
`i
`~li_n_k_-n_o_d_e-_i-d-~-- - --\
`
`word_id
`wd_his/1
`__ w_d._-... m ... ·• ... 12--------,-i
`
`B
`
`CAM RAM occupancy
`mana er
`empty_count
`
`match data
`
`Path Merge
`CAM
`
`K words deep
`single match
`binary
`
`CAM
`address
`maps to
`RAM
`
`encoded
`address
`
`Path Merge RAM
`K words deep
`address
`
`Write data
`
`20b
`
`20b
`
`20b
`
`1b
`
`20b
`
`link_node_id
`
`counter
`
`L _____ ~
`
`prev _phone_no
`
`prev_phone _ no
`
`from wave front manager
`word_id
`wd hist1
`wd-hist2
`f}ftW_phone_no
`
`to Path Merge Unit in
`PINEs
`Unk node id
`prevyhor:e_no
`
`Figure 6
`
`IPR2023-00035
`Apple EX1001 Page 11
`
`

`

`--..l = N
`N w
`'"= = 0--,
`"""'
`d r.,;_
`
`--..l
`
`0 ....
`0
`....
`('D .....
`rJJ =(cid:173)
`
`('D
`
`--.J
`N
`
`0 ....
`
`CIO
`
`N
`'"CIO
`N
`~
`~
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`i,. state_ids(x3), transvec_id, word_end_flag, full_trans_flag
`
`prev_phone_no, phone_no, wordlat_node_id
`
`Figure 7
`
`I
`
`,
`
`___________
`
`word/at node_id I to L....,....,_,_..,....,.,....,.=.:..,_,
`
`transvec_id
`
`I
`
`phone no
`
`(counter)
`generator
`
`phone number
`
`word_ end_f/ag
`prev _phone_ no
`
`3 x 1-3 bits = 39 bits
`
`deep
`64k
`
`data
`
`lookup RAM
`
`state id
`
`memory
`
`transvec_ld
`
`monophone id
`central
`
`6,6,6 = 18 bits
`triphone_id
`1
`
`phone generator
`
`L....,
`~,~--~-~~~
`l state_fds (x3)
`,,, ____
`..
`address to I
`
`deep
`256k
`
`16 bits
`address
`
`data
`RAM
`lookup
`address
`
`I
`
`i......
`
`to
`
`triphone
`
`word id
`
`-,.
`
`manager
`wavefront
`
`1
`phone no
`
`word id
`grammar_,
`d.d
`
`l/raW?M~jode _i
`
`B
`
`next_leftmph_id
`prev _phone no -
`current grammar Id
`current grammar noae id
`current word/at node id
`current LT node id
`
`-
`
`instruction: address offset
`data: monophone_idlword_id
`
`(lexical tree address)
`LT_node_id
`
`grammar_1d
`grammac._ryode _id
`LT node id
`RAM abbreviations
`phone_no -unique-phone instance fndentifier
`CAM abbreviations
`
`Mi:ire_~s __ d§:J1&
`
`clata fields
`
`instruction and
`
`FtOMwith
`
`micro-coded
`lexical tree
`
`current grammar_node_id
`current grammar_id
`
`grammar node id
`grammar_id
`
`jl.
`
`IPR2023-00035
`Apple EX1001 Page 12
`
`

`

`--..l = N
`--..l
`~ w
`o--,
`0
`~
`rJJ_
`d
`
`0 ....
`....
`....
`('D ....
`rJ'1 =(cid:173)
`
`('D
`
`-...J
`N
`
`0 ....
`
`CIO
`
`N
`~CIO
`N
`~
`~
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`11
`17
`
`mm
`4
`kk W
`00
`16
`
`Wl
`
`11
`15
`
`kk
`00
`14
`
`13
`
`kk
`jj
`11
`jj
`00 00
`11 12
`
`l,.!:..~~1 00 ~
`
`aa I ~~ . 6.
`
`bb
`
`6
`
`d
`b
`
`_O ~
`
`bb
`
`aa -bb !'.'. dd
`cc
`
`7
`
`2
`
`1
`
`,_______
`
`0
`"01" -two br.:10_9J)(;)§_,_
`
`___
`
`•
`
`cc
`
`bb
`
`"
`
`2
`
`1
`
`0
`
`? ~~o ??
`
`sequence Implied.
`Next element in
`
`aa
`
`aa -bb-cc
`
`"00" -Single branch, le. mono-phone to mono-phone.
`
`8
`
`5
`
`ii
`ff
`hh
`bb
`00 00 11 00 00
`10
`6
`
`. hh
`bb
`
`Wz
`
`gg
`ff
`
`9
`
`8
`
`7
`
`W4
`w2
`
`w2
`
`w1
`
`mm-sil
`'
`
`II -sil
`
`'
`
`Figure SA
`
`w1
`
`11
`5
`
`ee
`dd
`cc
`dd
`00 00
`4
`3
`
`2
`
`(monopltone tree)
`
`JJ -lick-
`sll
`
`11 -
`ff -gg -
`
`'hh-
`
`cc-dd-ee-sil
`
`aa -bb -
`/
`
`IPR2023-00035
`Apple EX1001 Page 13
`
`

`

`-....l = N
`
`-....l
`t
`~
`'"=
`"'""'
`~
`
`-....J
`:
`0
`N
`
`"'"'" ....
`g
`
`('D
`
`~
`~
`~
`~
`~
`e •
`
`=
`►
`a
`
`0 ....
`
`CIO
`
`N
`'"CIO
`N
`•
`(JQ
`
`-.i""=:====:::, _ _,,,,,,
`
`27
`
`ff
`bb
`00
`
`21
`
`ea
`bb
`00
`
`15
`
`7
`
`4
`
`3
`
`2
`
`1
`
`O
`
`"'-....&~~:t;~t=:J....
`bb
`aa
`00
`
`,_-,llliil-.lt-,~::.:.:.-_-_-:_
`
`_ _,
`
`aa
`
`5 (odd)
`
`gg
`\ff
`~
`-ee
`?dd
`
`cc
`
`aa -tib
`
`"10" -Mbranch: 5 (odd)
`
`••
`
`;~~fMb;;;~~h_-·-- -.....;..::~
`
`4 (even)
`
`~
`
`A--1-'
`
`I
`
`:o ~ ~ L
`
`e
`b
`
`d
`b
`
`bb
`
`2
`
`1
`
`"10" -Mbranch: 4 (even) ---
`
`bb
`aa
`~o
`
`aa
`
`ff
`\ ee
`~
`aa -bb :! ::
`
`., -··E: .. :: .:, :: ~~ ~ l 1
`
`M
`
`ee
`
`cc L dd
`
`•
`6
`
`1
`
`bb
`
`ea
`
`--.--..--
`
`"01" -three branches.
`
`Figure 8B
`
`IPR2023-00035
`Apple EX1001 Page 14
`
`

`

`U.S. Patent
`
`Aug. 28, 2018
`
`Sheet 13 of 27
`
`US 10,062,377 B2
`
`Figure 9
`
`memory address
`word Id
`pointer mph start_frame end frame score
`
`( pointer mph slart_frame end_frame score I
`
`. • •
`
`0
`sll
`nullsQ 1 3 0 0
`
`2
`one
`0 t 410 -20
`
`seven
`0 !4 12 -20
`0 t 4 13 -20
`0 t 4 14 ~20_
`
`4
`two
`2 f 11 1955
`2 f 11 20 60
`
`s
`two
`1 f 13 17 -84
`
`5
`ten
`1 f 13 17 -60
`1th 1318~5
`.111519-56,
`
`6
`five
`462027 -75
`3s1827
`-80
`
`7
`five
`5s 18 27-79
`5s 2027-82
`
`8
`three
`5 s 19 27 -79
`
`IPR2023-00035
`Apple EX1001 Page 15
`
`

`

`U.S. Patent
`
`Aug. 28, 2018
`
`Sheet 14 of 27
`
`US 10,062,377 B2
`
`Figure 10
`
`won dlalnod&_ld
`
`logloal lo
`physical
`address
`translation
`memory
`
`next spare
`location
`
`t
`
`control
`
`empty count
`
`word_kl I head wlr_ptr I
`I
`20b
`13b
`I
`
`spare
`
`pointer prev_worldat node_ld I mph I start,Jrame I ern;U'rame soore
`I 6b I
`I 16b
`13b I
`10b
`10b
`10b
`I
`
`phys/cal
`address
`ofward
`
`record . - er:Jdress
`
`word link record memory
`
`Bk deep ------ ~
`D
`
`_,,_
`two
`\!
`pointer
`~
`~ pointer
`1 polnter
`
`I head_wlr __ptr I
`2
`4
`2
`
`t
`t
`t
`
`spare
`
`11
`12
`12
`
`17
`17
`18
`
`-55
`·66
`-55
`
`Vlf)fdfal..J1ode_ld
`....
`
`: counter I
`
`~
`
`J~
`
`add new record
`to llxlstlng record:
`new record:
`prov_wordlat_node_ld word_{d
`mph
`head_record.,pfr
`start.frame
`endJt'Bmlil
`$CO/l1
`
`,
`
`read record during traceback
`or read whim adding WLR to
`existing word record
`
`IPR2023-00035
`Apple EX1001 Page 16
`
`

`

`--..l = N
`f
`=
`~
`~
`C
`
`--..l
`
`"'
`
`-....:i
`S. N
`ui
`.,..
`; ('D .....
`
`rJJ
`
`~
`=
`>
`
`N
`
`S
`
`QO
`
`=
`
`~
`
`;--
`~
`~
`•
`00
`
`e •
`
`I
`
`I
`
`--Write data
`
`r----:isb
`I
`I
`I
`
`1 K words deep
`address
`
`RAM
`
`factor
`
`-
`
`LM
`
`im••·•••-~ LM_factor
`
`RAM
`m~fu
`address
`CAM
`
`:..J., wd_hlst1 wd_hist2
`
`I LMJactocl
`
`to Phone Book
`
`...a
`
`Figure 11
`
`,.,_,.,.,
`
`....., _________________________
`
`1:::1
`
`I 1b
`
`I 20b
`
`I 20b
`
`20b
`
`I
`
`write data
`
`I 18b
`
`1
`
`tdphooe_id I
`
`1
`
`LM factor
`w,(hlst2 "
`wd hist1
`LT node id
`fromARM9
`
`-
`
`11 I
`
`I
`
`~ I I
`I I
`l-"".....-1
`
`•
`
`t--lllllO-I
`
`encoded
`
`encoder
`address
`
`next_empty address II 1 ► I write address
`
`sing/~ match
`
`binary
`
`1 k words deep
`
`CAM
`-·
`LM factor
`
`I
`
`I
`
`II
`
`empty_count
`
`CAM RAM occupancy I
`
`manager
`
`B
`
`match data
`
`-
`
`wd-hist2
`~~~
`LT node id
`triphone.Jd
`
`IPR2023-00035
`Apple EX1001 Page 17
`
`

`

`U.S. Patent
`
`Aug. 28, 2018
`
`Sheet 16 of 27
`
`US 10,062,377 B2
`
`Figure 12
`
`one
`
`two
`
`IPR2023-00035
`Apple EX1001 Page 18
`
`

`

`U.S. Patent
`
`Aug. 28, 2018
`
`Sheet 17 of 27
`
`US 10,062,377 B2
`
`Figure 13
`
`grammar
`memory
`
`address
`
`' .
`
`grammar_nodejd
`
`data
`grammar_id
`1 • grammar_node_ld
`
`current grammar_ld
`current grammar node_Jd.
`
`control
`
`next grammar_node _fd{ s)
`next word_id(s)
`1 , next L T_id
`next L T_node_ld(s)
`
`IPR2023-00035
`Apple EX1001 Page 19
`
`

`

`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`20.8 us ( 48 kHz), 12 x 20 bit slots+ 16 bit tag slot= 256 bits
`
`one codec· frame
`
`I t I add I dat J adc I :_adc I I I I I
`I t I add I <lat I ,: I I I I I
`n
`
`0
`◄
`
`(12.288 MHz, 81.4ns)
`
`BIT_CLK
`
`SDATA_IN
`
`( codec output)
`SDATA IN
`
`( codec input)
`SDATA OUT
`
`SYNC
`
`Slot#
`
`-....l = N
`N w
`'"= = 0--,
`"'""'
`d r.,;_
`
`-....l
`
`Figure 14
`
`rtJ7..fUL
`
`0
`
`1
`
`2
`
`3
`
`16
`
`17
`
`19 18
`
`n..n..nrL
`
`0 ....
`CIO
`....
`('D .....
`rJJ =(cid:173)
`
`('D
`
`-....J
`N
`
`0 ....
`
`CIO
`
`N
`'"CIO
`N
`~
`~
`
`I
`I
`
`I I I
`I I I
`
`12
`
`11
`
`10
`
`9
`
`8
`
`7
`
`6
`
`5
`
`4
`
`3
`
`2
`
`1
`
`IPR2023-00035
`Apple EX1001 Page 20
`
`

`

`-....l = N
`N w
`'"= = 0--,
`'"""'
`d r.,;,
`
`-....l
`
`0 ....
`....
`('D .....
`rJJ =(cid:173)
`
`1,0
`
`('D
`
`-....J
`N
`
`0 ....
`
`CIO
`
`N
`'"CIO
`N
`~
`~
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`I
`
`l
`:
`
`I
`l
`
`l
`
`I
`
`1
`I
`:
`:
`
`i
`
`I
`
`Processor
`
`RAM depth
`
`~Branch
`idle
`
`idle
`
`token passing
`
`I
`I
`:
`pruning thresholds calculated fot
`ax model, word end and bear,1
`
`next frame
`
`' I
`
`I
`I
`
`1
`
`1024 0
`
`1
`
`!81921
`
`:
`
`•
`l
`!8192j
`I
`
`feature vectors
`
`:
`
`instance RAM location
`Branch Processor model
`
`!3538l
`
`·-i -----------4--""-r---~-----------f
`
`I
`
`word link records added
`
`return next phone responses
`requests
`
`be::i111 prune, t!)ken score update
`
`•j'"" ----r-e'""'ce'""'iv_e_n_e_xt_p,hone
`: •
`::
`I
`l
`,
`
`MHD processing latency
`
`dlgits management
`
`resource
`!3538]
`
`connected
`
`r--
`I
`l
`
`memory to MHD transfer latency
`
`:
`front end to MHD transfer latency
`
`I
`,_
`
`I
`
`I
`
`I I
`
`-
`
`! I O j 1 j 2 I J121j
`! f
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`l
`:
`:
`! '
`I
`I
`l
`l
`l I
`! ! I O j 1 I 2 I j121j
`-:--J
`I
`I
`l
`l
`I ,
`
`scale
`not to
`
`frame 1
`
`IFigure 15
`
`LM 1/F
`
`word engine
`
`phone book
`
`PINE array
`
`MHD engine
`
`-
`
`--:
`
`mean & covar reg
`
`feature vector reg
`
`frame_count
`
`push_to_dictale_button
`
`IPR2023-00035
`Apple EX1001 Page 21
`
`

`

`-....l = N
`N w
`'"= = 0--,
`"'""'
`d r.,;_
`
`-....l
`
`0 ....
`
`0
`N
`('D .....
`rJJ =(cid:173)
`
`('D
`
`-....J
`N
`
`0 ....
`
`CIO
`
`N
`'"CIO
`N
`~
`~
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`Search Stage
`
`Front End
`
`106
`
`models
`Various
`
`107
`
`108
`
`Acoustic model
`
`104
`
`103
`
`105
`
`102
`
`♦
`
`.+-I Buffer I
`
`Data
`Audio
`
`ADC
`
`101
`
`100
`
`Figure 16
`
`D Functions performed by other hardware or software systems are shown in white
`!~) Functions performed by the accelerator are shown in grey
`
`Text
`
`Recognised
`
`Likelihoods
`
`or Probabilities or
`
`Distances
`
`I
`I
`I
`
`Vectors
`Feature
`
`Audio Data
`
`Audio Data
`
`Digital
`
`Digital
`
`IPR2023-00035
`Apple EX1001 Page 22
`
`

`

`U.S. Patent
`
`Aug. 28, 2018
`
`Sheet 21 of 27
`
`US 10,062,377 B2
`
`FIGURE 17
`
`212
`
`---r--------.--l:.....-----i-------...-....---------:.--
`
`202
`
`203
`
`Memory
`A
`
`Memory
`B
`
`Contro\
`Sequencing
`and
`Address
`Generation
`
`'210
`
`206
`
`204
`
`Distance
`Calculation ------i
`Engine
`
`CRC
`Check
`
`Decompressor
`
`207
`
`Acoustic
`Model Memory
`
`211
`
`FV
`Memory
`A
`
`208
`
`FV
`Registers
`and
`optional
`FIFO
`
`209
`
`I
`I
`1
`I
`I
`
`I ' l
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`i
`I
`I
`I
`I
`I
`I
`I
`I
`
`I '
`
`I
`1
`
`_________________________
`L~~
`~---------------
`'
`
`I
`I
`I
`f
`I
`I
`I
`I
`I
`1
`I
`I
`I
`I
`I
`I
`I
`I
`
`I ' i ' I
`
`1
`I
`I
`I
`I
`I
`
`' I
`
`I
`I
`I
`I
`I
`1
`
`205
`Distance Accelerator Subsystem
`L----------------------------------------------1
`
`IPR2023-00035
`Apple EX1001 Page 23
`
`

`

`-....l = N
`N w
`'"= = 0--,
`"'""'
`d r,r;_
`
`-....l
`
`0 ....
`
`-....J
`N
`
`N
`N
`('D .....
`rJJ =(cid:173)
`
`('D
`
`0 ....
`
`CIO
`
`N
`'"CIO
`N
`~
`~
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`using dists(FV5)
`for data frame 5
`Search Stage
`
`using dists(FV4)
`for data frame 4
`Search Stage
`
`using dists(FV3)
`for data frame 3
`Search Stage
`
`using dists(FV2)
`for data frame 2
`Search Stage
`
`using dists(FV1)
`for data frame 1
`Search Stage
`
`Dists(FV5) -> mem A
`Cale all dist. for FV5
`
`Dists(FV4)-> mem B
`Cale all dist, for FV4
`
`Dists(FV3) -> mem A
`Cale all dist. for FV3
`
`Dists(FV2) -> mem B
`Cale all dist. for FV2
`
`dists(FV1)
`
`Dists(FV1) -> mem A
`Cale all dist. for FV1
`
`Figure 18
`
`I Data Frame 5 -> FV5
`
`I Data Frame 4 -> FV4
`
`I Data Frame 3 -> FV3
`
`I Data Frame 2 -> FV2
`
`FV1
`
`I Data Frame 1 -> FV1 I
`
`...J._
`
`Time
`Step
`
`Pipeline
`
`Time
`Step
`
`Pipeline
`
`I
`
`I
`
`I
`
`I
`
`Time
`Step
`
`Pipeline
`
`Time
`Step
`
`Pipeline
`
`Time
`Step
`
`Pipeline
`
`Time
`Step
`
`Pipeline
`
`nme
`Step
`Pipeline I
`
`Search stage on CPU
`
`Distance Cale Accelerator
`
`Front end on DSP
`
`Time
`
`IPR2023-00035
`Apple EX1001 Page 24
`
`

`

`-....l = N
`w
`=
`'"=
`"'""'
`r.,;_
`d
`
`N
`0--,
`
`-....l
`
`0 ....
`
`-....J
`N
`
`~
`N
`('D .....
`=-
`
`('D
`
`rJJ
`
`....
`
`CIO
`0
`N
`'"CIO
`N
`
`.
`=
`>
`
`(JQ
`
`~ =
`
`~
`
`~
`~
`~
`•
`00
`
`e •
`
`f
`
`awaiting distances
`
`Olsts(FV4) -> mem B ~
`
`Release memory A
`Read dists from mem A
`I Search Stage 3
`Release memory B
`Read dists from mem B
`Search Stage 2
`
`awaitin distances
`
`0 Search stalled
`
`Release memo A
`Read dists from mem A
`Search Stage 1
`
`-
`
`I -~:~s~:!
`. -
`
`l 10J -... ~,.,~~ ~-~• c:\/
`
`FV4
`
`Dists(FV3)-> mem A
`Cale all dist. for FV3
`
`Dists(FV2) -> rnem B
`Cale all dist. for FV2
`
`rv;; Imest
`
`Data Frame 4 -> FV4 I
`
`3 ~ Dists(FV1) -> mem A
`Cale all dist. for FV1
`
`FV
`
`2
`
`Stall delivering FV3
`Data Frame 3 -> FV3
`Data Frame 2 -> FV2
`Data Frame 1 -> FV1
`
`Search stage on CPU
`
`Distance Cale Accelerator
`
`Front end on OSP
`
`Time
`
`Dists(FV5) -> mem A
`I Cale all dist. for FV5
`
`~A
`
`Data Frame 6 -> FV6 I _
`
`C'\/1;
`
`Data Frame 5 -> FV5 I _
`
`Release memory B
`Read dists from mem 8
`Search Stage 6
`
`Release memory A
`Read dists from mem A
`Search Stage 5
`
`Figure 19
`
`Oists(FV6) -> mem 8
`Cale all dist. for FV6
`
`18 Release memory B
`
`release@
`
`... ----
`
`Accelerator stalled
`
`® awaiting result
`
`memory B release
`
`Read dists from mem ~
`I Search Stage 4
`I Cale all dist. for FV4 I dists(FV4} ® Search stalled
`
`readv
`
`I Otsts\t"Vb)
`
`.-are
`
`IPR2023-00035
`Apple EX1001 Page 25
`
`

`

`-....l = N
`N w
`'"= = 0--,
`""""
`d r.,;_
`
`-....l
`
`0 ....
`
`-....J
`N
`
`.i;...
`N
`('D .....
`rJJ =(cid:173)
`
`('D
`
`0 ....
`
`CIO
`
`N
`'"CIO
`N
`~
`~
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`Figure 20
`
`-
`
`Cale all dist. for FV6
`
`Cale all dist. for FV5
`
`..
`
`Cale all dist. for FV4
`
`Cale all dist. for FV3
`
`Cale all dist. for FV2
`
`Cale all dist. for FV1
`
`.
`
`.
`
`.
`
`.
`
`FV6
`
`dists(FV4)
`
`FVS
`
`dists(FV3)
`
`FV4
`
`dists(FV2)
`
`FV3
`
`dists(FV1)
`
`FV2
`
`-
`
`2
`FE Data Frame 3 -> FV3
`
`using dists(FV2)
`Search Stage, data frame
`~-
`
`using dists(FV1)
`Search Stage, data frame 1
`
`~· FE Data Frame 2 -> FV2
`
`ata Frame 1 -> FV1 f_
`
`I
`
`using dists(FV6)
`
`dists(FV5)
`
`using dists(FV5)
`Search stage, data frame 5
`
`I Search Stage, data frame 6 I
`---
`-· FE Data Frame 6 -> FV6
`-· FE Data Frame 5 -> FV5
`
`-
`
`using dlsts(FV4)
`Search Stage, data frame 4
`
`3
`FE Data Frame 4 -> FV4
`
`using dlsts(FV3)
`search Stage, data frame
`~-
`
`Pipeline
`
`Time
`Step
`
`Time
`Step
`
`Pipeline
`
`Time
`Step
`
`Pipeline
`
`_Time
`Step
`
`Pipeline
`
`Time
`Step
`
`Pipeline
`
`Pipeline
`
`Time
`Step
`
`Time
`Step
`
`Pipeline
`
`Time
`Step
`
`Pipeline
`
`Distance Cale Accelerator
`
`Searc:h stage on CPU
`
`Front end and
`
`Time
`
`IPR2023-00035
`Apple EX1001 Page 26
`
`

`

`U.S. Patent
`
`Aug. 28, 2018
`
`Sheet 25 of 27
`
`US 10,062,377 B2
`
`FIGURE 21
`
`Time
`
`Software on CPU
`
`Accelerator
`
`FrontEnd
`Data Frame I ➔ FVI
`FrontEnd
`Data Frame 2 ➔ FV2
`Prep search stage 1
`Search stage 1
`Read distances from
`memA
`
`Front End
`Data Frame 3 ➔ FV3
`Search stage 2
`Read distances from
`memB
`
`FrontEnd
`Data Frame 4 ➔ FV4
`Searoh~gt3
`Read dlsmru:es from mem A
`P-eleai,'e memwv A
`FrontEnd
`Data Frame 5 -+ FVS
`'Pr,-.n i:1".:m·h sfiiaP. A.
`
`Search stage 4
`Read distances from
`memB
`
`0 I
`
`0 I
`
`Cale all dist. for FVl
`Dists(FVl) ➔ memA
`
`Cale all dist. for FV2
`Dists(FV2) ➔ mem B
`
`Cale all dist. for FV3
`Dists(FV3) ➔ mem A
`
`Cale all dist. for FV 4
`. Dists(FV4) ➔ mem B
`
`Calo all dist. for FVS
`Dists(FV5) ➔ mem A
`
`IPR2023-00035
`Apple EX1001 Page 27
`
`

`

`-....l = N
`N w
`'"= = 0--,
`"'""'
`rJl.
`d
`
`-....l
`
`0 ....
`
`-....J
`N
`
`O'I
`N
`('D .....
`rJJ =(cid:173)
`
`('D
`
`0 ....
`
`CIO
`
`N
`'"CIO
`N
`~
`~
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`dist~1)
`
`from
`
`~
`dists(FV2)
`
`I,'""
`
`I#"
`·~
`dists(FV1)
`
`Dists(FV1) -> rnern A
`Cale all dist. for FV1
`
`Search stage on CPU
`
`or conflicted
`
`results across slow
`Transfer distance
`
`Accelerator
`
`Distance Calculation
`
`using dlsts(FV1)
`for data frame 1
`Search Stage
`
`~
`distfil!:V2)
`
`Accelerator ,
`
`from
`
`dist result
`Transfer
`Accelerator If'"' ~
`dist result
`Transfer
`
`dist result
`Transfer
`
`from
`
`Lr
`dists(FV4}
`
`~
`
`V
`
`dlst~3)
`
`Dlsts(FV3)-> mem A
`Cale all dist. for FV3
`
`Dists(FV5) -> mem A
`Cale all dist. for FV5
`
`Dists(FV4)-> mem B
`Cale all dist. for FV4
`
`Dists(FV2)-> mem B
`Cale all dist. for FV2
`
`using dists(FV5)
`for data frame 5
`Search Stage
`
`using dists(FV4)
`for data frame 4
`Search Stage
`
`~
`
`Accelerator r
`dist result dists(.E\/5)
`Transfer
`
`from
`
`dist result
`Transfer
`
`from
`
`distsj!:_V5)
`
`~ .. Accelerator r
`
`using dists(FV3)
`for data frame 3
`Search Stage
`
`~
`dlsts(.E\/4)
`
`using dists(FV2)
`for data frame 2
`Searcti Stage
`
`~
`
`distsi.(fV3}
`
`~ Accelerator II"'
`
`Fiaure 22
`
`Pipeline
`
`Time
`Step
`
`.q Accelerator ~
`
`~
`
`FV5
`
`Accelerator - "
`
`FV to
`Transfer
`
`'4li1
`
`FV4
`
`<QI Accelerator V
`
`~ FV to
`Transfer
`
`~ Accelerator ~
`
`FV3
`
`FV to
`Transfer
`
`FV4
`
`i,
`
`V
`
`V
`
`<Ill
`
`"
`
`FV1
`
`Accelerator
`
`~ FV to
`
`Transfer
`
`'Ill
`~
`
`tr
`
`FV2
`
`FV to
`Transfer
`
`FV3
`
`Pipeline
`Time
`Step
`
`Pipeline
`
`llrne
`Step
`
`Data Frame 5 -> FV5
`
`Data Frame 4 -> FV4
`
`Data Frame 3 -> FV3
`
`Data Frame 2 -> FV2
`
`Data Frame 1 -> FV1
`
`Pipeline
`
`Time
`Step
`
`Pipeline
`
`llme
`Step
`
`nme
`Step
`Pipeline
`
`Pipeline
`
`Time
`Step
`
`Pipeline
`
`Time
`Step
`
`Pipeline
`
`Time
`Step
`
`Interface
`
`slow or confllc:ted
`Transfer FV across
`
`Front end on OSP
`
`Time
`
`IPR2023-00035
`Apple EX1001 Page 28
`
`

`

`-....l = N
`N w
`-..= = O',
`'"'"'
`d r.,;_
`
`-....l
`
`0 ....
`
`-....J
`N
`
`-....J
`N
`('D .....
`rJJ =(cid:173)
`
`('D
`
`0 ....
`
`CIO
`
`N
`'"CIO
`N
`~
`~
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`Figure 23
`
`dists(FV6)
`
`<-Xfer
`
`dists(FV5)
`
`using dists(FV5)
`
`I Search Stage, data frame 61""'"'
`I Search Stage, data frame 51
`
`using dists(FV6)
`
`Cale all dist. for FV6
`
`dists(FV5)
`
`<-Xfer
`
`._dists(FV4)
`
`using dists(FV4)
`Search Stage, data frame 4
`
`Cale all dist. for FV1
`
`FV2.
`
`FV1 .._
`
`Distance Cale Accelerator
`
`interface
`
`results across slow
`Transfer distance
`
`or conflicted
`
`Cale all dist. for FVS
`
`dists(FV4)
`
`<-Xfer
`
`Xfer ->
`
`FV6 .._
`
`Cale all dist. for FV4
`
`Cate all dist. for FV3
`
`FVS.,
`
`FV4., -
`
`'d1sts(FV3)
`
`dists(FV2)
`
`FV3.
`
`<-Xfer
`
`Xfer->
`
`<-Xfer
`
`Xfer->
`
`Cale all dist. for FV2
`
`• dists(FV1)
`
`<-Xfer
`
`Xfer ->
`
`Xfer->
`
`Xfer ->
`
`..
`
`...
`
`.
`
`.
`
`...
`
`.
`
`slow or conflicted
`Transfer FV across
`
`interface
`
`stage on CPU
`
`Front encl and search
`
`' dists(FV3)
`
`FV6
`
`-·
`
`~ dists(FV2)
`
`using dists(FV3)
`Search Stage, data frame 3
`
`FE Data Frame 6 -> FV6
`
`using dists(FV2)
`Search Stage, data frame 2
`
`FVS
`
`FE Data Frame 5 -> FVS
`
`using dists(FV1)
`search Stage, data frame 1 .._dlsts(FV1)
`FE Data Frame 4 -> FV4 -· FV4
`
`FV1
`
`I FE Data Frame 3 -> FV3 l--FV3
`I FE Data Frame 2 -> FV2 L--FV2
`FE Data Frame 1 -> FV1 L
`
`Time
`Step
`
`Pipeline
`
`Time
`Step
`
`Pipeline
`
`Time
`Step
`
`Plpefinei
`
`Time
`Step
`
`Pipeline
`
`Time
`Step
`
`Plpa!lne
`
`Time
`Step
`
`Pipel.ine
`
`Time
`Step
`Pipeline
`
`Pipeline
`
`Time
`Step
`
`Time
`Step
`
`Pipeline
`
`Time .
`
`IPR2023-00035
`Apple EX1001 Page 29
`
`

`

`US 10,062,377 B2
`
`1
`DISTRIBUTED PIPELINED PARALLEL
`SPEECH RECOGNITION SYSTEM
`
`This is a continuation of application Ser. No. 13/735,091,
`filed Jan. 7, 2013, now U.S. Pat. No. 9,076,441, which is a 5
`continuation of application Ser. No. 13/162,128, filed Jun.
`16, 2011, now U.S. Pat. No. 8,352,262, which is a divisional
`of application Ser. No. 11/662,704, filed Mar. 14, 2007, now
`U.S. Pat. No. 7,979,277, which is a 371 of PCT/GB2005/
`003554, filed Sep. 14, 2005, which claims priority to UK 10
`Application No. 0420464.0, filed Sep. 14, 2004, the disclo(cid:173)
`sures of which are hereby incorporated by reference in their
`entireties.
`
`BACKGROUND OF THE INVENTION
`
`2
`each phone in the network. An alternative suggested by
`Sukun Kim et al is to provide a processor for each state in
`the network.
`In an article entitled "Dynamic Programming Search for
`Continuous Speech Recognition" in IEEE Signal Processing
`Magazine, September 1999, Ney et al discuss language
`model lookahead. Language model
`lookahead
`involves
`computation of a language model factor for each node (i.e.
`phone) in the lexical tree. This technique is also known as
`smearing. Each phone instance in the search network can be
`given a language model factor when it is used in the lexical
`tree search. Ney et al show that for an example bigram
`language model, the average number of states per 10 ms
`15 frame can be reduced from around 168,000 states with no
`language model lookahead to around 8,000 states when
`language model lookahead is used. They also show that
`bigram language model lookahead requires about a quarter
`of the states compared with