UNITED STATES PATENT AND TRADEMARK OFFICE
`
`____________________
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`____________________
`
`GOOGLE INC.
`
`Petitioner
`
`v.
`
`IXI MOBILE (R&D) LTD.
`
`Patent Owner
`
`____________________
`
`Patent No. 7,552,124
`____________________
`
`DECLARATION OF
`JASON FLINN, PH.D.
`
`
`
`
`
`
`
`
`
`
`
`
`
`Page 1 of 155
`
`GOOGLE EXHIBIT 1002
`
`

`
`
`
`TABLE OF CONTENTS
`
`Introduction ..................................................................................................... 1
`I.
`Qualifications .................................................................................................. 1
`II.
`Summary of Opinions ..................................................................................... 5
`III.
`IV. Level of Ordinary Skill ................................................................................... 6
`V.
`The ’124 Patent ............................................................................................... 6
`VI. Claim Construction ....................................................................................... 13
`VII. Technical Background & Prior Art Considered ........................................... 15
`A.
`Technical Background ........................................................................ 15
`B. Maes ................................................................................................... 17
`Preston ................................................................................................ 24
`C.
`Pazandak ............................................................................................ 27
`D.
`E. White ................................................................................................... 32
`F. Manson ............................................................................................... 35
`VIII. The Prior Art Discloses All of the Features of Claims 1-10 the ’124
`Patent ............................................................................................................ 37
`A. Maes and Preston Disclose or Suggest the Features of Claims
`1-10 ..................................................................................................... 38
`Pazandak, White, and Manson Disclose or Suggest the Features
`of Claims 1-10 .................................................................................... 91
`IX. Conclusion .................................................................................................. 142
`
`B.
`
`-i-
`
`
`
`
`
`
`
`
`
`Page 2 of 155
`
`

`
`
`
`
`
`I, Jason Flinn, declare as follows:
`
`I.
`
`INTRODUCTION
`
`1.
`
`I have been retained as an independent expert consultant in this
`
`proceeding before the United States Patent and Trademark Office (“PTO”)
`
`regarding U.S. Patent No. 7,552,124 (“the ’124 patent,” which I understand is Ex.
`
`1001 in this proceeding) based on my experience, education, and knowledge in the
`
`field of computer science as applied to mobile computing, distributed computing,
`
`and natural language processing. I have been asked to consider whether certain
`
`references disclose the features recited in claims 1-10 of the ’124 patent. My
`
`opinions are set forth below.
`
`2.
`
`I am being compensated at my rate of $425 per hour for the time I
`
`spend on this matter. My compensation is in no way contingent on the nature of
`
`my findings, the presentation of my findings in testimony, or the outcome of this or
`
`any other proceeding. I have no other interest in this proceeding.
`
`II. QUALIFICATIONS
`I earned my Ph.D. in Computer Science from Carnegie Mellon
`3.
`
`University in 2001. I earned a M.S. in Computer Engineering from Syracuse
`
`University in 1996. I earned a B.S.E. in Computer Science and Engineering from
`
`the University of Pennsylvania in 1991. I also received a B.S. in Economics from
`
`
`
`
`
`1
`
`
`
`Page 3 of 155
`
`

`
`the University of Pennsylvania, with a specialization in strategic management, in
`
`
`
`
`
`1991.
`
`4.
`
`I am currently a tenured Professor in the Division of Computer
`
`Science and Engineering (CSE) of the Department of Electrical Engineering and
`
`Computer Science (EECS) at the University of Michigan in Ann Arbor, MI. Since
`
`2014, I have also been director of the Software Systems Laboratory at the
`
`University of Michigan in Ann Arbor. I have held these positions since 2014. I
`
`have been a full-time, tenure-track professor in the CSE Division of the EECS
`
`Department at the University of Michigan in Ann Arbor, MI for 14 years.
`
`Previously, I was a tenured Associate Professor of CSE and EECS from 2008 to
`
`2014, and an Assistant Professor of CSE and EECS from 2002 to 2008. From
`
`2001 to 2002, I was employed as a post-doctoral intern by Intel Corp. in
`
`Pittsburgh, PA. From 1991 to 1996, I was employed as a programmer by IBM in
`
`East Fishkill, NY.
`
`5. My areas of research specialization include mobile computing,
`
`distributed systems, and operating systems. I have published over 60 peer-
`
`reviewed scholarly articles on these topics in journals, conferences, and
`
`workshops. I have received 8 best paper awards and 1 best demo award in these
`
`research areas. I also received the University of Michigan Faculty Recognition
`
`award in 2015, the University of Michigan College of Engineering Education
`
`
`
`
`
`2
`
`
`
`Page 4 of 155
`
`

`
`
`
`
`
`Excellence Award in 2012, the University of Michigan EECS Department
`
`Outstanding Achievement Award in 2009, the Morris Wellman Faculty
`
`Development Professorship from 2006 to 2008, and the CAREER award from the
`
`National Science Foundation in 2004.
`
`6.
`
`I have conducted extensive research on the topic of distributing
`
`functionality between mobile devices and fixed servers in the cloud and elsewhere.
`
`My Ph.D. dissertation in 2001 was one of the first scholarly works on this topic. I
`
`have published at least 16 scholarly papers addressing the topic of distributed
`
`systems that offload application functionality from mobile devices such as
`
`smartphones to servers in the cloud and other fixed locations; the majority of these
`
`publications address the offload of natural language application functionality such
`
`as speech recognition and language translation. I have also written a book on these
`
`topics, entitled “Cyber Foraging: Bridging Mobile and Cloud Computing,” for the
`
`Morgan & Claypool Synthesis Lectures on Mobile Computing series.
`
`7.
`
`I was co-chair of the technical program committee for the ACM
`
`International Conference on Mobile Systems, Applications, and Services
`
`(MobiSys) in 2009, the USENIX Conference on File and Storage Technologies
`
`(FAST) in 2012, and the USENIX Symposium on Operating Systems Design and
`
`Implementation (OSDI) in 2014. I have also been a member of the technical
`
`program committee for over 30 academic conferences and workshops. From 2011
`
`
`
`
`
`3
`
`
`
`Page 5 of 155
`
`

`
`
`
`
`
`to 2015, I was an Associate Editor of ACM Transactions on Storage. I have been a
`
`Guest Editor of IEEE Transactions on Mobile Computing (in 2010) and a member
`
`of the Editorial Board for ACM Mobile Computing and Communications Review
`
`(in 2004).
`
`8.
`
`I have advised 7 students who have graduated with a Ph.D. in
`
`Computer Science and Engineering and 3 students who have graduated with a M.S.
`
`in Computer Science and Engineering from the University of Michigan in Ann
`
`Arbor, MI. I am currently advising 4 graduate students. At the University of
`
`Michigan, I have developed a graduate-level course on mobile computing, and the
`
`classes I have taught include a graduate-level class on distributed systems, as well
`
`as graduate-level and undergraduate-level classes on operating systems.
`
`9.
`
`Based on my experience and education, I believe that I am qualified to
`
`opine as to knowledge and level of skill of a person of ordinary skill in the art at
`
`the time of the alleged invention of the ’124 patent (which I further describe
`
`below) and what such a person would have understood at that time, and the state of
`
`the art during that time.
`
`10. My curriculum vitae, which includes a more detailed summary of my
`
`background, experience, and publications, is attached as Appendix A.
`
`
`
`
`
`4
`
`
`
`Page 6 of 155
`
`

`
`
`
`
`
`III. SUMMARY OF OPINIONS
`11. All of the opinions contained in this Declaration are based on the
`
`documents I reviewed, my knowledge, and professional judgment. In forming the
`
`opinions expressed in this Declaration, I reviewed the ’124 patent (which I
`
`understand is Ex. 1001 in this proceeding), the prosecution file history for the ’124
`
`patent (which I understand is Ex. 1003 in this proceeding), U.S. Patent No.
`
`7,003,463 to Maes (“Maes”) (which I understand is Ex. 1005 in this proceeding),
`
`U.S. Patent Application Publication No. 2003/0046061 to Preston (“Preston”)
`
`(which I understand is Ex. 1006 in this proceeding), U.S. Patent No. 7,027,975 to
`
`Pazandak (“Pazandak”) (which I understand is Ex. 1007 in this proceeding), PCT
`
`Publication No. WO 02/12982 A2 to Applicant Object Services and Consulting
`
`(“Object”) (which is the PCT application corresponding to Pazandak, and which I
`
`understand is Ex. 1011 in this proceeding), U.S. Patent Application Publication
`
`No. 2002/0072918 to White (“White”) (which I understand is Ex. 1008 in this
`
`proceeding), U.S. Patent No. 7,085,708 to Manson (“Manson”) (which I
`
`understand is Ex. 1009 in this proceeding), U.S. Patent Application Publication
`
`No. 2003/0182132 to Niemoeller (“Niemoeller”) (which I understand is Ex. 1012
`
`in this proceeding), U.S. Patent No. 7,693,720 to Kennewick (“Kennewick”)
`
`(which I understand is Ex. 1013 in this proceeding), and any other materials I refer
`
`
`
`
`
`5
`
`
`
`Page 7 of 155
`
`

`
`
`
`
`
`to in this Declaration in support of my opinions, while drawing on my experience
`
`and knowledge of natural language programming and distributed systems.
`
`12. Based on my experience and expertise, it is my opinion that certain
`
`references disclose or suggest all the features recited in claims 1-10 of the ’124
`
`patent, as I discuss in detail below.
`
`IV. LEVEL OF ORDINARY SKILL
`13. Based on my knowledge and experience, I understand what a person
`
`of ordinary skill in the art would have known at the time of the alleged invention.
`
`My opinions herein are, where appropriate, based on my understandings as to one
`
`of ordinary skill in the art at that time.
`
`14.
`
`In my opinion, based on the materials and information I have
`
`reviewed, and on my extensive experience in the technical areas relevant to the
`
`’124 patent in the early 2000s, a person of ordinary skill in the art would have had
`
`a Bachelor’s degree or equivalent in computer science, electrical engineering, or a
`
`similar related field, as well as 1-2 years of experience working with natural
`
`language programming. I apply this understanding in my analysis herein.
`
`V. THE ’124 PATENT
`15. The ’124 patent, entitled “Natural language for programming a
`
`specialized computing system,” describes a “method for programming a mobile
`
`communication device based on a high-level code comprising operative language.”
`
`
`
`
`
`6
`
`
`
`Page 8 of 155
`
`

`
`
`
`
`
`Ex. 1001 at Title, Abstract. The ’124 patent states in its background section that
`
`“mobile communication devices (e.g., cellular phones) and data organizers (e.g.,
`
`personal digital assistants (PDAs)) are particularly popular these days.” Id. at
`
`1:28-30. The ’124 patent acknowledges that cellular phones had voice-activated
`
`features and states that “most consumers find it tedious to program the device to
`
`perform the special features, and therefore forgo using said features altogether.”
`
`Id. at 38-41 (“Other programming features may include voice-activated dialing,
`
`voice mail management, or other functions that may be configured in accordance
`
`with occurrence of particular conditions and events.”), 1:46-48.
`
`16. Claims 1 and 6 are the only independent claims in the ’124 patent,
`
`with claim 6 being a system claim that is similar in many respects to method claim
`
`1. Claims 7-10, which are dependent from claim 6, are similar in many respects to
`
`claims 2-5, which are dependent from claim 1.
`
`17. The alleged invention of the ’124 patent is directed to a method (claim
`
`1) and corresponding system (claim 6) “for programming a mobile communication
`
`device based on a high-level code comprising operative language.” Id. at 8:59-61
`
`(claim 1), 9:47-49 (claim 6).
`
`18. The ’124 patent discloses that “high-level code 150” is received from
`
`a user. Id. at 8:62 (“receiving a high-level code”). The ’124 patent describes that
`
`“[h]igh-level code 150, in a preferred embodiment, comprises text formatted in the
`
`
`
`
`
`7
`
`
`
`Page 9 of 155
`
`

`
`
`
`
`
`context of a natural language (e.g., English, French, Spanish, Japanese, etc.)” and
`
`may include a sentence such as ‘Transfer call to voice mail if call is from Bob.’”
`
`Id. at 4:15-17, 4:25-26. “[A]pplication software 1122 can act as a natural language
`
`compiler to processes high-level code 150 to control the operation of mobile
`
`device 120 based a defined set of conditions.” Id. at 4:42-45. The application
`
`software and mobile device are shown in FIG. 1 of the ’124 patent:
`
`
`
`Id. at FIG. 1; see also id. at 2:62-63 (“FIG. 1 illustrates an exemplary
`
`communications environment”).
`
`
`
`
`
`
`
`8
`
`
`
`Page 10 of 155
`
`

`
`
`
`
`
`19. As shown in FIG. 1, the application software may be located at the
`
`mobile device 120 or at a network server 100. See id. at FIG. 1. The ’124 patent
`
`discloses a distributed computing environment and that processing of high-level
`
`code 150 occurs at either the network server 100 or mobile device 120 depending
`
`on whether the high-level code comprises a “complex set of instructions” or a “less
`
`complex structure.” Id. at 4:49-55 (“if high-level code 150 comprises a complex
`
`set of instructions, then high-level code 150 is transmitted to network server 100,
`
`so that a more powerful system is utilized to process and compile high-level code
`
`150. Therefore, in one embodiment, application software 1122 or a portion thereof
`
`is installed and executed on network server 100 to process high-level code 150”),
`
`4:58-61 (“Alternatively, if high-level code 150 comprises a less complex structure,
`
`then application software 1122 or a portion thereof is installed and executed on
`
`mobile device 120 to process high-level code 150”), FIG. 1.
`
`20. The ’124 patent discloses various processing steps shown in FIG. 2:
`
`
`
`
`
`9
`
`
`
`Page 11 of 155
`
`

`
`
`
`
`
`S120
`
`
`3251!}
`
` Dctermi ne
`relationships
`and
`Conditions
`
`326 I
`
`FIG. 2
`
`
`
`Id. at FIG. 2; see also 2:65-67 (“FIG. 2 is a flow diagram of a method for
`Id. at FIG. 2; see also 2:65-67 (“FIG. 2 is a flow diagram of a method for
`
`providing a natural programming language for a specialized computing device, in
`providing a natural programming language for a specialized computing device, in
`
`
`
`
`Page 12 of 155
`
`10
`
`10
`
`
`
`Page 12 of 155
`
`

`
`accordance with one or more embodiments”), 5:31-6:7 (describing FIG. 2), 8:59-
`
`
`
`
`
`9:38 (claim 1).
`
`21. The ’124 patent discloses that application software 1122 performs
`
`each of the steps shown in FIG. 2, namely: “processes the high-level code 150
`
`(S210),” “[parses] high-level code 150 for keywords in an attempt to recognize any
`
`operative language included in high-level code 150 (S220),” “parse[s] high-level
`
`code 150 for keywords in an attempt to recognize any data sources (S230),”
`
`“determines the requested operation that is to be performed in accordance with the
`
`recognized keywords (S240),” “determines the relationships and conditions that
`
`are to be taken into account for the operation to be performed (S250),” and
`
`“produces executable code 160 (S260).” Id. at 5:35-66. The application software
`
`1122 that performs these steps is shown in FIG. 3B:
`
`
`
`
`
`11
`
`
`
`
`
`Page 13 of 155
`
`

`
`
`
`
`
`Id. at FIG. 3B; see also id. at 3:1-4.
`
`22. The application software disclosed in the ’124 patent runs on a
`
`general purpose computer. See id. at 6:51-61 (“Referring to FIG. 3A, an
`
`embodiment of application software 1122 can be implemented as computer
`
`software in the form of computer readable code executed on a general purpose
`
`hardware environment 1110”), FIG 3A:
`
`
`
`23. As discussed above, the ’124 patent describes converting high-level
`
`code into executable instructions (see, e.g., id. at FIG. 2) using application
`
`software (see, e.g., id. at FIG. 3B) running in a distributed computing environment
`
`(see, e.g., id. at FIG. 1). But, all of those features were already in the prior art, as
`
`explained below at Sections VII and VIII.
`
`
`
`
`
`12
`
`
`
`Page 14 of 155
`
`

`
`
`
`
`
`VI. CLAIM CONSTRUCTION
`I understand that a claim subject to inter partes review receives the
`24.
`
`broadest reasonable interpretation in light of the specification and file history of
`
`the patent in which it appears. I also understand that any term that is not construed
`
`should be given its plain and ordinary meaning under the broadest reasonable
`
`interpretation. I have followed these principles in my analysis. I discuss certain
`
`claim terms below and what I understand to be Petitioner’s construction of these
`
`terms, which I apply in my analysis. For the remaining claim terms in the ’124
`
`patent, I apply the plain and ordinary meaning under the broadest reasonable
`
`interpretation.
`
`25.
`
`I understand that Petitioner has proposed that the broadest reasonable
`
`interpretation of the claimed term “operative language” is “language associated
`
`with one or more operations to be performed.” I agree with this broadest
`
`reasonable construction based on the claims and specification of the ’124 patent.
`
`In my opinion, “operative language” is not a term of art that would have carried
`
`any special meaning. “Operative language” appears in claims 1 and 6 in the
`
`context of being “associated with controlling one or more operations of the mobile
`
`communication device.” Ex. 1001 at 9:2-4, 10:6-9. Similarly, the specification
`
`states that the “operative language” may “defin[e] an instruction for a function or
`
`an operation to be performed.” Id. at 4:19-21; see also id. at 4:25-28 (explaining
`
`
`
`
`
`13
`
`
`
`Page 15 of 155
`
`

`
`
`
`
`
`that the “operative language” in the sentence “Transfer call to voice mail if call is
`
`from Bob” is “transfer”). The construction proposed by Petitioner for “operative
`
`language” is also consistent with my review of the file history and how a person of
`
`ordinary skill in the art would have understood the term in context of the ’124
`
`patent. I have applied this understanding in my analysis.
`
`26.
`
`I have been asked to assume that the broadest reasonable
`
`interpretation of the claimed “means for receiving a high-level code comprising
`
`one or more keywords” recited in claim 6 includes a “keypad, pointing device,
`
`touchscreen, keyboard, microphone, or equivalents thereof” performing the
`
`“receiving a high-level code comprising one or more keywords.” I have applied
`
`this understanding in my analysis.
`
`27.
`
`I have been asked to assume that the broadest reasonable
`
`interpretation of the other “means for” limitations recited in claim 6 includes
`
`software running on a processor configured to perform the functions recited for
`
`each of those “means for” limitations or equivalents thereof. In particular, the
`
`other “means for” limitations includes the “means for parsing . . . ,” “means for
`
`determining at least one operation . . . ,” “means for determining whether high-
`
`level code . . . ,” “means for producing . . . ,” “means for determining level of
`
`complexity and implementation . . . ,” and the “means for designation . . . .” I have
`
`applied this understanding in my analysis.
`
`
`
`
`
`14
`
`
`
`Page 16 of 155
`
`

`
`
`
`
`
`VII. TECHNICAL BACKGROUND & PRIOR ART CONSIDERED
`A. Technical Background
`28. The use of natural language input to program devices was well
`
`established by June 17, 2004, which is the filing date of the ’124 patent. Ex. 1001
`
`at Cover. For example, U.S. Patent No. 7,693,720 to Kennewick (“Kennewick”)
`
`describes natural language techniques for performing commands using mobile
`
`device in connection with remote computing devices. See, e.g., Ex. 1013 at
`
`Abstract, 2:50-3:36, 5:51-6:16. Kennewick discloses parsing and interpreting user
`
`commands inputted to a mobile device, determining the domain and context, and
`
`based on that determination, invoking one or more agents for processing the
`
`commands. See id. at 3:61-4:64, 9:27-56. The device described in Kennewick may
`
`operate in a distributed environment where more complicated actions are
`
`performed at remote computing devices. Id. at 3:14-60, 4:5-64, 5:27-50. The
`
`control functions performed by the disclosed system include the control of
`
`electronics (e.g., entertainment systems), cellular phones, SMS systems, e-mail,
`
`vehicle control systems, and the like. Id. at 9:57-10:12; see also id. at 10:13-15:24.
`
`29.
`
`I have reviewed the file history for the ’124 patent. I understand from
`
`my review that during prosecution, the Applicant acknowledged that many aspects
`
`of natural language processing were known in the prior art. For example, during
`
`prosecution the Applicant stated as follows:
`
`
`
`
`
`15
`
`
`
`Page 17 of 155
`
`

`
`
`
`
`
`Pazandak discloses a system and method for a light-
`
`weight guided natural language interface (NLI) client.
`
`The disclosed system and method support parser farms
`
`on servers, available currently and in real time to a
`
`plurality of users, over disperse and geographically
`
`disparate networks. Pazandak teaches a system and
`
`method directed to inputting to a thin client a query;
`
`communicating to an interface intermediary;
`
`communicating to an interface descriptor data source;
`
`generating an interface descriptor; communicating the
`
`interface descriptor to the interface intermediary;
`
`communicating the interface descriptor to a parser farm;
`
`and requesting the appropriate parser corresponding to
`
`the interface descriptor.
`
`Ex. 1003 at 53 (Amendment dated November 29, 2007). It is my understanding
`
`that the Applicant amended claims and the Examiner ultimately allowed the claims
`
`over Pazandak, but that the Examiner did not cite any other prior art for making
`
`claim rejections and did not combine Pazandak with any other prior art references.
`
`As discussed below, even with the amendments that the Applicant made, claims 1-
`
`
`
`
`
`16
`
`
`
`Page 18 of 155
`
`

`
`
`
`
`
`10 of the ’124 patent do not disclose anything that was not already in the prior art.
`
`See Sections VII.B-F, VIII.
`
`B. Maes
`30. Maes is entitled “System and method for providing network
`
`coordinated conversational services.” Maes describes a “system and method for
`
`providing automatic and coordinated sharing of conversational resources, e.g.,
`
`functions and arguments, between network-connected servers and devices and their
`
`corresponding applications.” Ex. 1005 at Abstract. Maes explains in its
`
`background section that a problem with conversational systems was that
`
`“[a]lthough such conversational systems are becoming increasingly popular,
`
`typically all the conversational processing is performed either on the client side or
`
`on the server side (i.e., all the configurations are either fully local or fully
`
`client/server).” Id. at 1:31-35. Maes further discloses “speech embedded
`
`conversational applications in portable client devices,” id. at 1:47-48, and describes
`
`a drawback associated with such systems:
`
`Unfortunately, because of limited resources, it is to be expected that
`
`such client devices may not be able to perform complex
`
`conversational services such as, for example, speech recognition
`
`(especially when the vocabulary size is large or specialized or when
`
`domain specific/application specific language models or grammars are
`
`
`
`
`
`17
`
`
`
`Page 19 of 155
`
`

`
`
`
`
`
`needed), NLU (natural language understanding), NLG (natural
`
`language generation), TTS (text-to-speech synthesis), audio capture
`
`and compression/decompression, playback, dialog generation, dialog
`
`management, speaker recognition, topic recognition, and
`
`audio/multimedia indexing and searching, etc. For instance, the
`
`memory and CPU (and other resource) limitations of a device can
`
`limit the conversational capabilities that such device can offer.
`
`Id. at 1:49-62.
`
`31. Therefore, Maes solved that problem by disclosing “a system and
`
`method that allows a network device with limited resources to perform complex
`
`specific conversational tasks automatically using networked resources in a manner
`
`which is automatic and transparent to a user.” Id. at 2:27-31.
`
`32. For example, Maes discloses that a cooperative, distributed computing
`
`environment enables a user to control a smartphone by uttering a command as
`
`follows:
`
`The user can then utter a command such as “dial first
`
`name last name at . . . possible qualifier (home, office,
`
`cell phone), and upon recognition/understanding of the
`
`command (via the local conversational engines 102), the
`
`smartphone will automatically dial the phone number
`
`
`
`
`
`18
`
`
`
`Page 20 of 155
`
`

`
`
`
`
`
`associated with the person in the address book (via the
`
`local applications 104). On the other hand, when a name
`
`is uttered that is not within the address book (and
`
`therefore not recognized/understood), but which is in a
`
`larger corporate (or public) directory (as contained in
`
`remote server 106), the request can be saved (in features
`
`or in waveform) and transmitted to a remote server 106
`
`for recognition.
`
`Id. at 15:49-61.
`
`33. The distributed computing environment that Maes uses is shown in
`
`FIG. 1:
`
`
`
`
`
`19
`
`
`
`Page 21 of 155
`
`

`
`
`
`
`
`Id. at FIG. 1 (showing client device 100 and server 106).
`
`34. Maes discloses that “[t]he client device 100 may be, for example, a
`
`smartphone or any speech-enabled PDA (personal digital assistant).” Id. at 4:18-
`
`
`
`
`
`
`
`20
`
`
`
`Page 22 of 155
`
`

`
`
`
`
`
`20. Maes discloses various end-to-end, speech-to-speech processing associated
`
`with the client device, as follows:
`
`The client device 100 further comprises one or more
`
`local conversational engines 102 for processing the
`
`acoustic features and/or waveforms generated and/or
`
`captured by the acoustic front-end 101 and generating
`
`dialog for output to the user. The local conversational
`
`engines 102 can include, for instance, an embedded
`
`speech recognition, a speaker recognition engine, a TTS
`
`engine, a NLU and NLG engine and an audio capture and
`
`compression/decompression engine as well as any other
`
`type of conversational engine.
`
`Id. at 4:20-29.
`
`35.
`
`In other words, the client device 100 includes a local conversational
`
`engine, which in turn can include various components such as a speech recognition
`
`engine and a natural language understanding (NLU) engine. Id.
`
`36. Maes also discloses functionality associated with the server:
`
`As with the local engines 102, the server engines 107 can
`
`include, for instance, an embedded speech recognition, a
`
`TTS engine, a NLU and NLG engine, an audio capture
`
`
`
`
`
`21
`
`
`
`Page 23 of 155
`
`

`
`
`
`
`
`and compression/decompression engine, as well as any
`
`other type of conversational engine. The server 106
`
`comprises a server dialog manager 108 which operates in
`
`a manner similar to the local dialog manager 103 as
`
`described above. For example, the server dialog manager
`
`108 determines whether a request for a conversational
`
`service from the local dialog manager 103 is to be
`
`processed and executed by the server 106 or on another
`
`remote network-connected server or device.
`
`Id. at 4:57-5:1.
`
`37. Maes’s processing technique “allows a low resource client device to
`
`transparently perform simple tasks locally, as well as complex tasks in binary or
`
`analog connection with a server (or other device) having more complex
`
`conversational capabilities.” Id. at 3:1-5. For example, “a NLU/FSG [natural
`
`language understanding / finite state grammar] system can be designed in
`
`accordance with the present invention so that if the user's request requires FSG
`
`(finite state grammar), the request can be processed locally unless the request is
`
`more complex and natural, thereby requiring forwarding to a remote server for
`
`recognition.” Id. at 16:42-47.
`
`
`
`
`
`22
`
`
`
`Page 24 of 155
`
`

`
`
`
`
`
`38. The following flow chart in Maes shows local and remote processing
`
`can be harnessed in a cooperative way:
`
`39.
`
`Id. at FIG. 2. FIG. 2 is further described as follows in Maes:
`
`
`
`
`
`
`
`23
`
`
`
`Page 25 of 155
`
`

`
`
`
`
`
`If it is determined that local processing is available
`
`(affirmative determination in step 201), then processing
`
`will be performed locally (step 202) via local engines
`
`102. On the other hand, if it is determined that local
`
`processing is not available (negative determination in
`
`step 201), then the relevant
`
`features/waveforms/information is automatically
`
`transmitted to a remote network-connected server (step
`
`204) (via IP, LAN, Bluetooth, IR, RF or via phone or IP
`
`phone), wherein remote processing (e.g., speech
`
`recognition/synthesis) is performed (step 205) (possibly
`
`with some user/server interaction).
`
`Id. at 11:23-33.
`
`40.
`
`In short, Maes discloses a natural language processing system for,
`
`e.g., controlling a device such as a phone, and even discloses the use of a
`
`distributed computing environment that the ’124 patent purports to have invented.
`
`Preston
`C.
`41. Preston is entitled “Apparatus for automatically generating source
`
`code.” Preston discloses a system in which “a user can input a natural language
`
`instruction . . . and it will be analysed and used to put together source code
`
`
`
`
`
`24
`
`
`
`Page 26 of 155
`
`

`
`
`
`
`
`customised for carrying out the user’s wishes in that environment.” Ex. 1006 at ¶
`
`[0049]. Preston discloses natural language input, e.g., sentences such as “Please
`
`divert my phone to Frank’s,” in the form of either speech or text. Id. at ¶¶ [0049],
`
`[0156]-[0157] (“In earlier embodiments, the data entries are typed into the terminal
`
`101 as text via the keyboard 701 shown in FIG. 7. In the present embodiment the
`
`terminal 101 is provided with a microphone 703, and the input text is dictated and
`
`transliterated by a speech-to-text conversion program”).
`
`42. Preston discloses a technique that can be used to control various types
`
`of devices:
`
`In a preferred embodiment, the invention is used for
`
`control of terminal devices used in a communications
`
`system, of which the telephone has been discussed above
`
`as an example. A more complete (though non limiting)
`
`list would include: telephones, video cameras, 3D
`
`displays, personal digital assistants, cellular telephones,
`
`satellite telephones, pagers, video phones, facsimiles,
`
`payphones, quertyphones, personal computers, lap top
`
`portable computers, engineering workstations, audio
`
`microphones, video conference suites, telemetry
`
`equipment.
`
`25
`
`
`
`
`
`
`
`Page 27 of 155
`
`

`
`
`
`
`
`Id. at ¶ [0183]; see also id. at ¶ [0184].
`
`43. The technique disclosed in Preston involves parsing natural language
`
`inputs. See id. at ¶ [0179] (“The embodiments of the present invention concern
`
`natural language inputs, where input statements are syntactically and semantically
`
`analysed using a parser.”). The parser is used to parse an input statement into
`
`constituent parts, e.g., to identify various conditions, relations, and other
`
`components relevant to understanding the semantic content of the user’s input
`
`statement. See, e.g., id. at ¶¶ [0122]-[0123]. For example, an input statement such
`
`as “If I am in an important meeting and the call is urgent, you should forward the
`
`telephone call to my mobile,” id. at ¶ [0118], may be parsed as follows to extract
`
`semantic content:
`
`Id. at [0122]-[0123].
`
`
`
`
`
`26
`
`
`
`
`
`Page 28 of 155
`
`

`
`
`
`
`
`44. Preston explains how to produce the code that is executable on a
`
`particular machine. For example, a code generator 103 “is used for generating
`
`source code from whichever predefined functions have been identified by the data
`
`analyser 102,” and the code generated by generator 100 “will be compiled into
`
`object code when run on a particular platform . . . .” Id. at ¶¶ [0059], [0185].
`
`Pazandak
`D.
`45. Pazandak is entitled “Guided natural language interface system and
`
`method.” Pazandak “generally relates to computers and computing and, more
`
`particularly, to natural language interfaces (NLI) for user interfacing with
`
`computing devices to interact with computer applications, for exam