`
`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`______________________________________________
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`______________________________________________
`
`
`
`
`GOOGLE LLC, SAMSUNG ELECTRONICS CO. LTD.,
`AND SAMSUNG ELECTRONICS AMERICA, INC.,
`Petitioner
`
`
`v.
`
`MULTIMODAL MEDIA LLC,
`Patent Owner.
`
`_______________________
`
`Case No. IPR2024-00063
`U.S. Patent No. 9,185,227
`_______________________
`
`
`DECLARATION OF DR. BENJAMIN B. BEDERSON IN SUPPORT
`OF PETITION FOR INTER PARTES REVIEW OF
` U.S. PATENT NO. 9,185,227
`
`
`
`
`
`
`
`
`Petitioner Exhibit 1002, Page 1 of 127
`
`

`

`TABLE OF CONTENTS
`
`
`Page
`
`
`
`I. 
`INTRODUCTION .......................................................................................... 1 
`BACKGROUND AND QUALIFICATIONS ................................................ 2 
`II. 
`INFORMATION CONSIDERED .................................................................. 9 
`III. 
`IV.  RELEVANT LEGAL STANDARDS ............................................................ 9 
`A. 
`Claim Interpretation ............................................................................. 9 
`B. 
`Perspective of One of Ordinary Skill in the Art ................................. 10 
`C. 
`Obviousness ........................................................................................ 10 
`LEVEL OF ORDINARY SKILL IN THE ART .......................................... 13 
`V. 
`VI.  SUMMARY OF MY OPINIONS ................................................................ 14 
`VII.  TECHNOLOGICAL BACKGROUND ....................................................... 15 
`A. 
`Call completion applications were well known. ................................ 15 
`B. 
`Call completion actions, other than voicemail, were well known.
` ............................................................................................................ 23 
`VIII.  THE ’227 PATENT AND ITS FILE HISTORY ......................................... 29 
`IX.  CLAIM CONSTRUCTION ......................................................................... 32 
`A. 
`“call completion application” (Claims 1, 5, 6, 9, 10, 11, and 15) ...... 32 
`1. 
`The specification supports the plain and ordinary meaning
` .................................................................................................. 33 
`The dependent claims contradict the Oppo construction ......... 33 
`2. 
`The agreed-upon construction makes no sense ....................... 34 
`3. 
`“one of text data, audio data, video data, audiovisual data, image
`data, multimedia data, message data, and any combination
`thereof” (Claims 8, 17) ....................................................................... 35 
`X.  OVERVIEW OF THE APPLIED PRIOR ART REFERENCES ................ 36 
`A. 
`Prikowitsch (Ex-1011) ....................................................................... 36 
`B. 
`Roujinsky (Ex-1012) .......................................................................... 38 
`C. 
`Sar-Shalom (Ex-1013) ........................................................................ 39 
`
`B. 
`
`i
`
`Petitioner Exhibit 1002, Page 2 of 127
`
`

`

`TABLE OF CONTENTS
`(continued)
`
`Page
`
`b. 
`
`
`XI.  DETAILED EXPLANATION OF THE UNPATENTABILITY
`GROUNDS ................................................................................................... 41 
`A.  Ground 1: Claims 1, 2, 5, 8-12, 15, and 17-19 are Obvious Over
`Prikowitsch ......................................................................................... 41 
`1. 
`Independent Claims 1 and 11 ................................................... 41 
`a. 
`Elements 1[pre]/11[pre]: A computer implemented
`method/[A computer system] for completing an
`incomplete call made by a calling party to a called
`party, comprising ........................................................... 41 
`Element 1[a]: providing a call completion
`application executable by at least one processor on
`a calling party device; .................................................... 42 
`Element 11[a]: a non-transitory computer readable
`storage medium configured to store a call
`completion application on a calling party device; ......... 42 
`Element 11[b]: at least one processor
`communicatively coupled to said non-transitory
`computer readable storage medium, said at least
`one processor configured to execute said call
`completion application on said calling party
`device; and ..................................................................... 42 
`Element 1[b]: detecting said incomplete call made
`by said calling party to said called party, by said
`call completion application on said calling party
`device, ............................................................................ 48 
`Element 1[c]: wherein said incomplete call is a call
`that is not connected to a called party device due to
`occurrence of one or more of a plurality of events; ...... 48 
`Element 11[c]: said call completion application on
`said calling party device, comprising: ........................... 48 
`Element 11[d]: a call detection module configured
`to detect said incomplete call made by said calling
`party to said called party, ............................................... 48 
`
`c. 
`
`ii
`
`Petitioner Exhibit 1002, Page 3 of 127
`
`

`

`
`
`d. 
`
`e. 
`
`f. 
`
`g. 
`
`TABLE OF CONTENTS
`(continued)
`
`Page
`
`Element 1[d(i)]: receiving one or more of a
`plurality of call completion actions to be
`performed by said call completion application on
`said calling party device to complete said detected
`incomplete call, .............................................................. 50 
`Element 11[e(i)]: an action determination module
`configured to receive one or more of a plurality of
`call completion actions to be performed to
`complete said detected incomplete call, ........................ 50 
`Elements 1[d(ii)]/11[e(ii)]: wherein said plurality
`of call completion actions are selected according to
`one or more responses provided by said calling
`party through a graphical user interface of said
`calling party device, ....................................................... 52 
`Elements 1[d(iii)]/11[e(iii)]: wherein said call
`completion actions comprise [1] setting a reminder
`to call back said called party at a configurable
`time, [2] recording media data on said calling party
`device, [3] transmitting said media data to a called
`party device, [4] transmitting a missed call alert to
`said called party device, [5] transmitting a
`notification of said detected incomplete call to said
`called party device, [6] transmitting a notification
`on availability of said called party, [7] transmitting
`said media data to a social networking platform,
`[8] transmitting an automated message requesting
`said called party to call back said calling party
`when available, and [9] any combination thereof;
`and .................................................................................. 53 
`Element 1[e]: triggering execution of said received
`one or more of said call completion actions by said
`call completion application on said calling party
`device based on one or more of action execution
`criteria for said completion of detected incomplete
`call. ................................................................................. 56 
`
`iii
`
`Petitioner Exhibit 1002, Page 4 of 127
`
`

`

`
`
`2. 
`
`TABLE OF CONTENTS
`(continued)
`
`Page
`
`b. 
`
`Element 11[f]: an action execution module
`configured to trigger execution of said received
`one or more of said call completion actions based
`on one or more of action execution criteria one or
`more responses selected by said calling party for
`said completion of said detected incomplete call. ......... 56 
`Dependent Claims 2, 5, 8-10, 12, 15, and 17-19 ..................... 58 
`a. 
`Element 12[a]: The computer system of claim 11,
`wherein said incomplete call is a call that is not
`connected to a called party device due to
`occurrence of one or more of a plurality of events, ....... 58 
`Claim 2/Element 12[b]: [The computer
`implemented method of claim 1, / The computer
`system of claim 11,] wherein said events comprise
`said called party being busy, said called party
`device being in an out of coverage area, said called
`party device being unreachable, said called party
`device being switched off, network congestion,
`and said call not being answered by said called
`party. .............................................................................. 58 
`Claims 5 and 15 ............................................................. 60 
`Claims 8/17: [The computer implemented method
`of claim 1 / The computer system of claim 11],
`wherein said media data comprises one of text
`data, audio data, video data, audiovisual data,
`image data, multimedia data, message data, and
`any combination thereof. ............................................... 66 
`
`c. 
`d. 
`
`iv
`
`Petitioner Exhibit 1002, Page 5 of 127
`
`

`

`TABLE OF CONTENTS
`(continued)
`
`Page
`
`
`
`B. 
`
`2. 
`
`e. 
`
`f. 
`
`Claims 9/18: [The computer implemented method
`of claim 1, further comprising determining / The
`computer system of claim 11, wherein said action
`execution module is further configured to
`determine] a mode of transmission of media data
`from said calling party device to a called party
`device via a network, by said call completion
`application on said calling party device based on
`one or more of said action execution criteria. ............... 67 
`Claims 10/19: The computer implemented method
`of claim 1, further comprising monitoring/[The
`computer system of claim 11, wherein said call
`detection module is further configured to monitor]
`said incomplete call by said call completion
`application on said calling party device as said
`incomplete call progresses from said incomplete
`call being initiated, said incomplete call being one
`of answered and rejected, and said incomplete call
`being terminated. ........................................................... 69 
`Ground 2: Claims 3, 4, 9, 13, 14, and 18 are Rendered Obvious
`by Prikowitsch in View of Roujinsky ................................................ 71 
`1. 
`A POSITA would have been motivated to combine
`Prikowitsch and Roujinsky and had a reasonable
`expectation of success .............................................................. 71 
`Dependent Claims 3, 4, 9, 13, 14, and 18 ................................ 74 
`a. 
`Claims 3/13: [The computer implemented method
`of claim 1/The computer system of claim 11],
`wherein said incomplete call is a call of a short
`duration that does not enable an intended
`communication to occur between said calling party
`and said called party. ..................................................... 74 
`
`v
`
`Petitioner Exhibit 1002, Page 6 of 127
`
`

`

`
`
`C. 
`
`TABLE OF CONTENTS
`(continued)
`
`Page
`
`b. 
`
`c. 
`
`Claim 4: The computer implemented method of
`claim 1, wherein said one or more of said call
`completion actions are received by said call
`completion application based on a duration of said
`detected incomplete call ................................................ 75 
`Claim 14: The computer system of claim 11,
`wherein said action determination module is
`further configured to receive said one or more of
`said call completion actions based on a duration of
`said detected incomplete call. ........................................ 75 
`Claims 9/18: [The computer implemented method
`of claim 1, further comprising determining / The
`computer system of claim 11, wherein said action
`execution module is further configured to
`determine] a mode of transmission of media data
`from said calling party device to a called party
`device via a network, by said call completion
`application on said calling party device based on
`one or more of said action execution criteria. ............... 77 
`Ground 3: Claims 1-2, 6, 8-12, and 17-19 are Rendered Obvious
`by Sar-Shalom .................................................................................... 78 
`1. 
`Independent Claims 1 and 11 ................................................... 78 
`a. 
`Elements 1[pre]/11[pre]: A computer implemented
`method/[A computer system] for completing an
`incomplete call made by a calling party to a called
`party, comprising ........................................................... 78 
`Element 1[a]: providing a call completion
`application executable by at least one processor on
`a calling party device; .................................................... 79 
`Element 11[a]: a non-transitory computer readable
`storage medium configured to store a call
`completion application on a calling party device; ......... 79 
`
`b. 
`
`vi
`
`Petitioner Exhibit 1002, Page 7 of 127
`
`

`

`
`
`TABLE OF CONTENTS
`(continued)
`
`Page
`
`Element 11[b]: at least one processor
`communicatively coupled to said non-transitory
`computer readable storage medium, said at least
`one processor configured to execute said call
`completion application on said calling party
`device; and ..................................................................... 79 
`Element 1[b]: detecting said incomplete call made
`by said calling party to said called party, by said
`call completion application on said calling party
`device, ............................................................................ 82 
`Element 1[c]: wherein said incomplete call is a call
`that is not connected to a called party device due to
`occurrence of one or more of a plurality of events; ...... 82 
`Element 11[c]: said call completion application on
`said calling party device, comprising: ........................... 82 
`Element 11[d]: a call detection module configured
`to detect said incomplete call made by said calling
`party to said called party ................................................ 82 
`Element 1[d(i)]: receiving one or more of a
`plurality of call completion actions to be
`performed by said call completion application on
`said calling party device to complete said detected
`incomplete call, .............................................................. 84 
`Element 11[e(i)]: an action determination module
`configured to receive one or more of a plurality of
`call completion actions to be performed to
`complete said detected incomplete call, ........................ 84 
`Elements 1[d(ii)]/11[e(ii)]: wherein said plurality
`of call completion actions are selected according to
`one or more responses provided by said calling
`party through a graphical user interface of said
`calling party device, ....................................................... 86 
`
`c. 
`
`d. 
`
`e. 
`
`vii
`
`Petitioner Exhibit 1002, Page 8 of 127
`
`

`

`
`
`2. 
`
`TABLE OF CONTENTS
`(continued)
`
`Page
`
`f. 
`
`g. 
`
`Elements 1[d(iii)]/11[e(iii)]: wherein said call
`completion actions comprise [1] setting a reminder
`to call back said called party at a configurable
`time, [2] recording media data on said calling party
`device, [3] transmitting said media data to a called
`party device, [4] transmitting a missed call alert to
`said called party device, [5] transmitting a
`notification of said detected incomplete call to said
`called party device, [6] transmitting a notification
`on availability of said called party, [7] transmitting
`said media data to a social networking platform,
`[8] transmitting an automated message requesting
`said called party to call back said calling party
`when available, and [9] any combination thereof;
`and .................................................................................. 89 
`Element 1[e]: triggering execution of said received
`one or more of said call completion actions by said
`call completion application on said calling party
`device based on one or more of action execution
`criteria for said completion of detected incomplete
`call. ................................................................................. 91 
`Element 11[f]: an action execution module
`configured to trigger execution of said received
`one or more of said call completion actions based
`on one or more of action execution criteria one or
`more responses selected by said calling party for
`said completion of said detected incomplete call. ......... 91 
`Dependent Claims 2, 6, 8-10, 12, and 17-19 ........................... 93 
`a. 
`Element 12[a]: The computer system of claim 11,
`wherein said incomplete call is a call that is not
`connected to a called party device due to
`occurrence of one or more of a plurality of events, ....... 93 
`
`viii
`
`Petitioner Exhibit 1002, Page 9 of 127
`
`

`

`
`
`b. 
`
`c. 
`
`d. 
`
`e. 
`
`TABLE OF CONTENTS
`(continued)
`
`Page
`
`Claim 2/Element 12[b]: [The computer
`implemented method of claim 1,] wherein said
`events comprise said called party being busy, said
`called party device being in an out of coverage
`area, said called party device being unreachable,
`said called party device being switched off,
`network congestion, and said call not being
`answered by said called party. ....................................... 94 
`Claim 6: The computer implemented method of
`claim 1, wherein said one or more of said call
`completion actions to be performed by said call
`completion application on said calling party device
`to complete said detected incomplete call is based
`on one or more of configurable criteria configured
`by said calling party. ...................................................... 95 
`Claims 8/17: [The computer implemented method
`of claim 1 / The computer system of claim 11],
`wherein said media data comprises one of text
`data, audio data, video data, audiovisual data,
`image data, multimedia data, message data, and
`any combination thereof. ............................................... 95 
`Claims 9/18: [The computer implemented method
`of claim 1, further comprising determining / The
`computer system of claim 11, wherein said action
`execution module is further configured to
`determine] a mode of transmission of media data
`from said calling party device to a called party
`device via a network, by said call completion
`application on said calling party device based on
`one or more of said action execution criteria. ............... 96 
`
`ix
`
`Petitioner Exhibit 1002, Page 10 of 127
`
`

`

`TABLE OF CONTENTS
`(continued)
`
`Page
`
`
`
`f. 
`
`Claims 10/19: [The computer implemented method
`of claim 1, further comprising monitoring / The
`computer system of claim 11, wherein said call
`detection module is further configured to monitor]
`said incomplete call by said call completion
`application on said calling party device as said
`incomplete call progresses from said incomplete
`call being initiated, said incomplete call being one
`of answered and rejected, and said incomplete call
`being terminated. ........................................................... 97 
`D.  Ground 4: Claims 3-5 and 13-15 are Obvious Over Sar-Shalom
`in View of Roujinsky ......................................................................... 98 
`1. 
`A POSITA would have been motivated to combine Sar-
`Shalom and Roujinsky and had a reasonable expectation
`of success ................................................................................. 98 
`Dependent Claims 3-5 and 13-15 .......................................... 101 
`a. 
`Claims 3/13: [The computer implemented method
`of claim 1 / The computer system of claim 11],
`wherein said incomplete call is a call of a short
`duration that does not enable an intended
`communication to occur between said calling party
`and said called party. ................................................... 101 
`Claim 4: The computer implemented method of
`claim 1, wherein said one or more of said call
`completion actions are received by said call
`completion application based on a duration of said
`detected incomplete call .............................................. 103 
`Claim 14: The computer system of claim 11,
`wherein said action determination module is
`further configured to receive said one or more of
`said call completion actions based on a duration of
`said detected incomplete call. ...................................... 103 
`Claims 5 and 15 ........................................................... 104 
`
`2. 
`
`b. 
`
`c. 
`
`x
`
`Petitioner Exhibit 1002, Page 11 of 127
`
`

`

`TABLE OF CONTENTS
`(continued)
`
`Page
`
`
`XII.  CONCLUSION ........................................................................................... 112 
`
`
`
`xi
`
`Petitioner Exhibit 1002, Page 12 of 127
`
`

`

`Declaration of Dr. Benjamin B Bederson
`U.S. Patent No. 9,185,227
`
`LIST OF EXHIBITS
`
`Ex-1001 U.S. Patent No. 9,185,227
`Ex-1002 Declaration of Dr. Benjamin B. Bederson
`Ex-1003 Curriculum Vitae of Dr. Benjamin B. Bederson
`Ex-1004 File History of U.S. Patent No. 9,185,227
`Ex-1005
`INTENTIONALLY LEFT BLANK
`Ex-1006
`INTENTIONALLY LEFT BLANK
`Ex-1007
`INTENTIONALLY LEFT BLANK
`Ex-1008
`INTENTIONALLY LEFT BLANK
`Ex-1009
`INTENTIONALLY LEFT BLANK
`Ex-1010
`INTENTIONALLY LEFT BLANK
`Ex-1011 PCT Publication WO 2010139478 (“Prikowitsch”)
`Ex-1012 PCT Publication WO2008050325 (“Roujinsky”)
`Ex-1013 U.S. Patent Publication No. 20040267884 (“Sar-Shalom”)
`Ex-1014
`INTENTIONALLY LEFT BLANK
`Ex-1015
`INTENTIONALLY LEFT BLANK
`Ex-1016
`INTENTIONALLY LEFT BLANK
`Ex-1017
`INTENTIONALLY LEFT BLANK
`Ex-1018
`INTENTIONALLY LEFT BLANK
`Ex-1019
`INTENTIONALLY LEFT BLANK
`Ex-1020 Claim Mapping Table
`Ex-1021 Multimodal Media LLC v. Guangdong Oppo Mobile
`Telecommunications Corp., Ltd., 2:21-cv-00436, D.I. 65 (E.D. Tex.
`Apr. 24, 2023) (Claim Construction Order) (“Oppo Litigation”)
`
`xii
`
`Petitioner Exhibit 1002, Page 13 of 127
`
`

`

`Declaration of Dr. Benjamin B Bederson
`U.S. Patent No. 9,185,227
`Ex-1022 Multimodal Media LLC v. TCL Technology Group Corporation et
`al., 2:22-cv-00463, D.I. 23 (E.D. Tex. Feb. 22, 2023) (Stay Order)
`(“TCL litigation”)
`Ex-1023 Multimodal Media LLC v. Samsung Electronics Co., Ltd. et al., 2:22-
`cv-00462, Infringement Contentions (E.D. Tex. May 30, 2023)
`(“Samsung Infringement Contentions”)
`INTENTIONALLY LEFT BLANK
`Ex-1024
`INTENTIONALLY LEFT BLANK
`Ex-1025
`Ex-1026 Benjamin B. Bederson & James D. Hollan, Pad++: A Zooming
`Graphical Interface for Exploring Alternate Interface Physics, UIST
`‘94 Proceedings of the 7th Annual ACM Symposium on User
`Interface Software and Technology 17 (1994), DOI:
`http://dx.doi.org/10.1145/192426.192435
`INTENTIONALLY LEFT BLANK
`Ex-1027
`INTENTIONALLY LEFT BLANK
`Ex-1028
`INTENTIONALLY LEFT BLANK
`Ex-1029
`INTENTIONALLY LEFT BLANK
`Ex-1030
`Ex-1031 David Dearman, Amy Karlson, Brian Meyers, and Ben Bederson,
`Multi-modal text entry and selection on a mobile device (2010), In
`Proceedings of Graphics Interface 2010 (GI ‘10), Canadian
`Information Processing Society, CAN, 19–26
`INTENTIONALLY LEFT BLANK
`INTENTIONALLY LEFT BLANK
`INTENTIONALLY LEFT BLANK
`INTENTIONALLY LEFT BLANK
`INTENTIONALLY LEFT BLANK
`INTENTIONALLY LEFT BLANK
`INTENTIONALLY LEFT BLANK
`
`Ex-1032
`Ex-1033
`Ex-1034
`Ex-1035
`Ex-1036
`Ex-1037
`Ex-1038
`
`xiii
`
`Petitioner Exhibit 1002, Page 14 of 127
`
`

`

`Declaration of Dr. Benjamin B Bederson
`U.S. Patent No. 9,185,227
`
`INTENTIONALLY LEFT BLANK
`Ex-1039
`INTENTIONALLY LEFT BLANK
`Ex-1040
`INTENTIONALLY LEFT BLANK
`Ex-1041
`INTENTIONALLY LEFT BLANK
`Ex-1042
`INTENTIONALLY LEFT BLANK
`Ex-1043
`INTENTIONALLY LEFT BLANK
`Ex-1044
`INTENTIONALLY LEFT BLANK
`Ex-1045
`Ex-1046 Multimodal Media LLC v. Guangdong Oppo Mobile
`Telecommunications Corp., Ltd., 2:21-cv-00436, D.I. 74 (E.D. Tex.
`Sept. 6, 2023) (Order of Dismissal)
`Ex-1047 Rebecca L.J. Chen, et al., Business Models and Drivers for Nex-
`Generation IMS Services (2007)
`Ex-1048 U.S. Patent No. 6,639,973 (“Wheeler”)
`Ex-1049 U.S. Patent Publication No. 20030169865 (“Oren”)
`Ex-1050 U.S. Patent Publication No. 20090142085 (“Anderl”)
`Ex-1051 U.S. Patent No. 8,340,702 (“Bychkov”)
`Ex-1052 U.S. Patent No. 8,229,409 (“Bluvand”)
`Ex-1053 Harvey G. Cragon, Computer Architecture and Implementation
`(2000)
`Ex-1054 Nokia 7650 User’s Guide
`
`
`
`xiv
`
`Petitioner Exhibit 1002, Page 15 of 127
`
`

`

`Declaration of Dr. Benjamin B. Bederson
`U.S. Patent No. 9,185,227
`I, Dr. Benjamin B. Bederson, declare as follows.
`
`1.
`
`I.
`
`INTRODUCTION
`2.
`I have been retained by Google LLC, Samsung Electronics Co. Ltd.,
`
`and Samsung Electronics America, Inc. (collectively, “Petitioner”) as an
`
`independent expert consultant in this inter partes review (“IPR”) proceeding before
`
`the United States Patent and Trademark Office (“PTO”).
`
`3.
`
`I have been asked by Petitioner Counsel (“Counsel”) to consider
`
`whether certain references disclose, teach and/or suggest the features recited in
`
`Claims 1-6, 8-15, and 17-19 of U.S. Patent No. 9,185,227 (“the ’227 Patent”)
`
`(Ex-1001), which I understand is currently assigned to Patent Owner Multimodal
`
`Media LLC (“PO”).1 My opinions and the bases for my opinions are set forth below.
`
`4.
`
`I am being compensated at my ordinary and customary consulting rate
`
`for my work, which is $600 per hour. My compensation is in no way contingent on
`
`the nature of my findings, the presentation of my findings in testimony, or the
`
`outcome of this or any other proceeding. I have no other direct financial interest in
`
`this proceeding.
`
`
`1 Where appropriate, I refer to exhibits that I understand are attached to the petition
`for IPR of the ’227 Patent.
`
`1
`
`Petitioner Exhibit 1002, Page 16 of 127
`
`

`

`Declaration of Dr. Benjamin B. Bederson
`U.S. Patent No. 9,185,227
`
`II. BACKGROUND AND QUALIFICATIONS
`5.
`All of my opinions stated in my Declaration are based on my own
`
`personal knowledge and professional judgment. In forming my opinions, I have
`
`relied on my knowledge and experience in designing, developing, researching, and
`
`teaching the technology referenced in my Declaration.
`
`6.
`
`I am over 18 years of age and, if I am called upon to do so, I would be
`
`competent to testify as to the matters set forth in my Declaration. I understand that a
`
`copy of my current curriculum vitae, which details my education and professional
`
`and academic experience, is being submitted as Ex-1003. The following provides a
`
`brief overview of some of my experience that is relevant to the matters set forth in
`
`my Declaration.
`
`7.
`
`I am currently Professor Emeritus of Computer Science at the
`
`University of Maryland (“UMD”). From 2014 to 2018, I was the Associate Provost
`
`of Learning Initiatives and Executive Director of the Teaching and Learning
`
`Transformation Center at the UMD. I am a member and previous director of the
`
`Human-Computer Interaction Lab (“HCIL”), the oldest and one of the best known
`
`Human-Computer Interaction (“HCI”) research groups in the country. I am co-
`
`founder of the J.S. Bryant School, a therapeutic high school planning to open in
`
`2025. I was also co-founder and Chief Scientist of Zumobi, Inc. from 2006 to 2014,
`
`a Seattle-based startup that is a publisher of content applications and advertising
`
`2
`
`Petitioner Exhibit 1002, Page 17 of 127
`
`

`

`Declaration of Dr. Benjamin B. Bederson
`U.S. Patent No. 9,185,227
`platforms for smartphones. I am also co-founder and co-director of the International
`
`Children’s Digital Library (“ICDL”), a web site launched in 2002 that provides the
`
`world’s largest collection of freely available online children’s books from around
`
`the world at www.childrenslibrary.org with an interface aimed to make it easy for
`
`children and adults to search and read children’s books online. I was also co-founder
`
`and prior Chief Technology Officer of Hazel Analytics, a data analytics company to
`
`improve food safety and better public health whose product sends alerts in warranted
`
`circumstances. In addition, I have for more than 15 years consulted for numerous
`
`companies in the area of user interfaces, including Microsoft, Logitech, the Palo Alto
`
`Research Center, Sony, Lockheed Martin, Hillcrest Labs, and NASA Goddard Space
`
`Flight Center.
`
`8.
`
`The claims of the ’227 Patent generally relate to voice short message
`
`service messaging system. For more than 30 years, I have studied, designed, and
`
`worked in the field of computer science and HCI. My experience includes 30 years
`
`of teaching and research, with research interests in HCI and the software and
`
`technology underlying today’s interactive computing systems. This includes the
`
`design and implementation of hardware and software systems including the use of
`
`interactive voice interfaces as well as many other kinds of sensors and interfaces for
`
`electronic devices on a range of devices, including embedded systems, controllers,
`
`smart phones and PDAs.
`
`3
`
`Petitioner Exhibit 1002, Page 18 of 127
`
`

`

`Declaration of Dr. Benjamin B. Bederson
`U.S. Patent No. 9,185,227
`At UMD, I have been focused primarily on the area of HCI, a field that
`
`9.
`
`relates to the development and understanding of computing systems to serve users’
`
`needs. Researchers and practitioners in this field are focused on making universally
`
`usable, useful, efficient, and appealing systems to support people in their wide range
`
`of activities. My approach is to balance the development of innovative technology
`
`that serves people’s practical needs. Example systems following this approach that
`
`I have built include Cortex-I (1992 embedded computer vision system that sensed
`
`licensed plates with custom motor, camera and controller), Audio Augmented
`
`Reality (1995 embedded system for sensing a user’s location and playing audio
`
`suited to that location), Fisheye Menus (2000 software for sensing movement within
`
`and selection of linear list of items in a menu), PhotoMesa (2001 software for end
`
`users to browse personal photos), DateLens (2002 software for end users to use their
`
`mobile devices to efficiently access their calendar information), SlideBar (2005
`
`linear sensor to control scrolling), LaunchTile (2005 “home screen” software for
`
`mobile devices to allow users to navigate apps in a zoomable environment),
`
`SpaceTree (2001 software for end users to efficiently browse very large hierarchies),
`
`ICDL (as I describe above), and StoryKit (a 2009 iPhone app for children to create
`
`stories).
`
`4
`
`Petitioner Exhibit 1002, Page 19 of 127
`
`

`

`Declaration of Dr. Benjamin B. Bederson
`U.S. Patent No. 9,185,227
`I built a voice interface system in 2009, described in a 2010 paper2, to
`
`10.
`
`support user’s rich text entry with multi-modal input on mobile devices to support
`
`communications including text messages. As illustrated in this figure from the paper,
`
`we enabled users to enter formatted text (bold, colored, etc.) using a touch screen,
`
`voice recognition, tilt, and a foot pedal and we measured which approach was most
`
`effective.
`
`11. Throughout the 1990s and 2000s, I worked on a range of “zoomable
`
`user interfaces,” which are systems that support the multi-scale and spatial
`
`organization of and magnification-based navigation among multiple documents or
`
`visual objects. I built several different “ZUI” systems over the years, including
`
`
`
`
`2 David Dearman, Amy Karlson, Brian Meyers, and Ben Bederson, Multi-modal text
`entry and selection on a mobile device (2010), In Proceedings of Graphics Interface
`2010 (GI ‘10), Canadian Information Processing Society, CAN, 19–26 (Ex-1031).
`
`5
`
`Petitioner Exhibit 1002, Page 20 of 127
`
`

`

`Declaration of Dr. Benjamin B. Bederson
`U.S. Patent No. 9,185,227
`Pad++, Jazz and Piccolo. In those systems, I used a range of solutions to allow users
`
`to control zooming through the information space.3
`
`12. My work with ZUIs led to my creation of Zumobi in 2006, where I was
`
`responsible for investigating new software platforms and developing new user
`
`interface designs that provide efficient and engaging interfaces to permit end users
`
`to access a wide range of content on mobile platforms (including the iPhone and
`
`Android-based devices). For example, I designed and implemented software called
`
`“Ziibii,” a “river” of news for iPhone that used a capacitive sensor for controlling
`
`linear movement through news, software called “ZoomCanvas,” a zoomable user
`
`interface for several iPhone apps, and iPhone apps including “Inside Xbox” for
`
`Microsoft and Snow Report for REI.
`
`13. At the ICDL, I have since 2002 been the technical director responsible
`
`for the design and implementation of the web site, www.childrenslibrary.org
`
`(originally at www.icdlbooks.org). In particular, I have been closely involved in
`
`designing the user interface as well as the software architecture for the web site since
`
`its inception in 2002. Beginning in the mid-1990s, I have been responsible for the
`
`design and implementation of numerous other web sites in addition to the ICDL. For
`
`
`3 Benjamin B. Bederson & James D. Hollan, Pad++: A Zooming Graphical Interface
`for Exploring Alternate Interface