Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`
`
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`
`
`
`FACEBOOK, INC., INSTAGRAM, LLC, and WHATSAPP INC.,
`Petitioners
`
`v.
`
`BLACKBERRY LIMITED
`Patent Owner
`
`
`
`
`IPR2019-00516
`IPR2019-00528
`U.S. Patent No. 8,279,173 B2
`
`
`DECLARATION OF SANDEEP CHATTERJEE, PH.D.
`IN RESPONSE TO PATENT OWNER’S CONDITIONAL
`MOTION TO AMEND UNDER 37 C.F.R. § 42.121
`
`
`
`
`
`
`
`
`
`
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`001
`
`

`

`Table of Contents
`
`
`Page
`
`
`
`
`
`I. 
`
`THE NEW LIMITATIONS IN THE SUBSTITUTE CLAIMS ARE
`DISCLOSED BY THE PRIOR ART ............................................................. 1 
`A. 
`Zuckerberg discloses “displaying a photograph comprising at
`least one subject or object; receiving a user selection of a
`location in the photograph corresponding to the at least one
`subject or object” and “associating at least one of the tags in the
`tag list with the at least one subject or object.” .................................... 2 
`1. 
`Zuckerberg discloses “displaying a photograph
`comprising at least one subject or object; receiving a user
`selection of a location in the photograph corresponding to
`the at least one subject or object” .............................................. 2 
`Zuckerberg discloses “associating at least one of the tags
`in the tag list with the at least one subject or object.” ............... 5 
`Rothmuller and Plotkin disclose and render obvious “wherein
`the tags in the tag list are displayed in a vertical list, and
`wherein: a first tag from a first tag source is displayed above a
`second tag from a second tag source; and the second tag from
`the second tag source is displayed above a third tag from the
`first tag source” .................................................................................... 5 
`1. 
`Rothmuller (Exs. 1004 and 1005) .............................................. 5 
`2. 
`Plotkin (Ex. 1008) .................................................................... 12 
`THE TEACHINGS OF ZUCKERBERG, ROTHMULLER AND
`PLOTKIN RENDER THE SUBSTITUTE CLAIMS OBVIOUS ............... 15 
`A.  How Instituted Grounds in IPR2019-00516 Render the
`Substitute Claims Unpatentable ......................................................... 15 
`How Instituted Grounds in IPR2019-00528 Render the
`Substitute Claims Unpatentable ......................................................... 20 
`III.  CONCLUSION ............................................................................................. 22 
`
` 
`
`2. 
`
`B. 
`
`II. 
`
`B. 
`
`
`
`
`
`-i-
`
`
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`002
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`I, Sandeep Chatterjee, Ph.D., declare as follows:
`
`1.
`
`I have been asked to review and respond to certain points raised in the
`
`“Patent Owner’s Conditional Motion to Amend,” which I understand was filed in
`
`both IPR2019-00516 and IPR2019-00528 (“Motion”). I understand that Patent
`
`Owner did not submit a declaration from its expert, Dr. Rajeev Surati, in support of
`
`this motion. I have nevertheless been asked to determine whether the proposed
`
`substitute claims identified in the Motion would have been non-obvious over the
`
`prior art. As I will explain below, in my opinion, the proposed substitute claims add
`
`limitations that are already disclosed by and rendered obvious by the prior art
`
`identified in IPR2019-00516 and IPR2019-00528.
`
`I.
`
`THE NEW LIMITATIONS IN THE SUBSTITUTE CLAIMS ARE
`DISCLOSED BY THE PRIOR ART
`2.
`Substitute claims 21-32 would replace claims 1, 2, 4, 6-7, 10, 12-16,
`
`16, and 18, the challenged claims. The substitute claims introduce new limitations
`
`only with respect to independent claims 21, 25, and 29, which would replace original
`
`independent claims 1, 7, and 13, respectively. The remaining substitute dependent
`
`claims are unchanged from their corresponding original claim other than modifying
`
`the claim dependency. (Motion at 2.)
`
`3.
`
`To summarize, the new limitations purport to introduce three features
`
`into the independent claims: (1) display of a photograph and user selection of a
`
`subject or object in the photograph; (2) display of a “vertical” tag list showing at
`
`
`
`1
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`003
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`least three tags from two tag sources; and (3) associating at least one of the tags in
`
`the list with the selected subject or object. As I will explain below, each of these
`
`features is disclosed by the prior art.
`
`A. Zuckerberg discloses “displaying a photograph comprising at least
`one subject or object; receiving a user selection of a location in the
`photograph corresponding to the at least one subject or object” and
`“associating at least one of the tags in the tag list with the at least
`one subject or object.”
`These claim limitations correspond to features (1) and (3) listed above.
`
`4.
`
`As shown below, Zuckerberg discloses them.
`
`1.
`
`Zuckerberg discloses “displaying a photograph comprising
`at least one subject or object; receiving a user selection of a
`location in the photograph corresponding to the at least one
`subject or object”
`This feature is plainly disclosed by Zuckerberg. Figure 5 shows:
`
`
`
`
`2
`
`5.
`
`
`
`
`
`
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`004
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`
`
`
`(Zuckerberg, Fig. 5 (partial figure; annotations added).) As shown highlighted in
`
`the screenshot above, the system instructs the user to “[c]lick on people in the photo
`
`to tag them.” (Id.) As shown in the red box, the user has selected a region 520
`
`corresponding to an individual’s face. (Id.) Zuckerberg provides the following
`
`explanation of these features in the context of Figure 5:
`
`FIG. 5 is an exemplary screen shot of a tag web page 500 illustrating
`various functions of the components of the exemplary tag component
`340. The region selection component 410 is configured to receive input
`from a user 101 (e.g., the media owner 101a, the tagged user 101b, and
`
`
`
`3
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`005
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`
`etc.) and/or a non-member, (e.g., the tagged non-user 102). The region
`selection component 410 is further configured to select a region (e.g., a
`selected region 520) within a digital image 362 according to the input.
`In some embodiments, the user 101 moves a cursor 530 on the user
`device 110 to a point in the digital image 362 using a mouse, trackball,
`track pad, or the like. The user 101 clicks on the point and the region
`selection component 410 places a border 525 around the selected region
`520. In various embodiments, the shape of the selected region 520 may
`be a rectangle, circle, ellipse, or polygon. The size of the selected region
`520 may be fixed, may be determined by the user 101, or may be
`automatically determined.
`
`(Zuckerberg, 8:13-31 (underlining added); see also id., 8:27-37.)
`
`6.
`
`Zuckerberg discloses “displaying a photograph comprising at least
`
`one subject or object,” as shown by the display of digital image 362 in Figure 5.
`
`The photograph has “at least one subject or object,” in this case a human being.
`
`7.
`
`Zuckerberg also discloses “receiving a user selection of a location in
`
`the photograph corresponding to the at least one subject or object.” Zuckerberg
`
`discloses that “[t]he region selection component 410 is configured to receive input
`
`from a user 101,” and “select a region (e.g., a selected region 520) within a digital
`
`image 362 according to the input.” (Zuckerberg, 8:15-22.) The “location in the
`
`photograph” in Zuckerberg corresponds to a point in the digital image 362 selected
`
`by the user (such as the cross-hatch shown in Figure 5), which may correspond to a
`
`subject or object in the photo. Zuckerberg explains that in some embodiments, “the
`4
`
`
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`006
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`user 101 moves a cursor 530 on the user device 110 to a point in the digital image
`
`362,” and then “clicks on the point and the region selection component 410 places a
`
`border 525 around the selected region 520.” (Zuckerberg, 8:22-27.) The particular
`
`photo shown in Figure 5 shows that the point corresponds to a human being, which
`
`is consistent with the instruction in Figure 5: “Click on people in the photo to tag
`
`them.” (Zuckerberg, Fig. 5.) It would have been obvious that the user in Zuckerberg
`
`could have selected a point in any photo corresponding to a subject or object.
`
`2.
`
`Zuckerberg discloses “associating at least one of the tags in
`the tag list with the at least one subject or object.”
`Zuckerberg also discloses the tag component 340 discussed above can,
`
`8.
`
`after receiving a selection of a tag in the tag list, associate that tag with the selected
`
`region in the photo. (Zuckerberg, e.g., 7:54-56 (“The tag component 340 is
`
`configured to select a region in the image and associate text with the region.”).)
`
`“Clicking on any of the previously used tags may associate the tag with the selected
`
`region 520.” (Zuckerberg, 8:66-9:1.) As explained previously, the selection region
`
`520 in Zuckerberg can correspond to a subject or object in the photo.
`
`B. Rothmuller and Plotkin disclose and render obvious “wherein the
`tags in the tag list are displayed in a vertical list, and wherein: a
`first tag from a first tag source is displayed above a second tag from
`a second tag source; and the second tag from the second tag source
`is displayed above a third tag from the first tag source”
`1.
`Rothmuller (Exs. 1004 and 1005)
`As explained in my opening declarations, Figure 9A of the Rothmuller
`
`9.
`
`
`
`5
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`007
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`Provisional discloses a “tag list” in the form of a list of recently used tags, with each
`
`tag shown with a respective tag type indicator icon. (Ex. 1002 in IPR2019-00516,
`
`¶¶88, 89, 171-172, 197-198; Ex. 1002 in IPR2019-00528, ¶¶87-89.) The recently
`
`used tag list of Figure 9A of the Rothmuller Provisional, which I have annotated
`
`below, shows this list and how it applies to the substitute claims:
`
`
`
`(Rothmuller Provisional, Fig. 9A (partial figure; highlighting and annotations
`
`added).) Figure 9A above discloses or suggests displaying tags in a tag list “wherein
`
`the tags in the tag list are displayed in a vertical list,” as recited. The three tags
`
`shown in highlighting above, i.e. Landscapes, Lori, and Animals, are shown listed
`
`one after another in a vertical arrangement. As shown, the “first tag” (“Landscapes”)
`
`and the “third tag” (“Animals”) both come from “a first tag source,” as indicated by
`
`the common tag type indicator icon next to them. The “second tag” (“Lori”) comes
`
`from a second and different tag source as reflected by the different indicator icon.
`
`10. More specifically, an example of the claimed “a first tag from a first
`6
`
`
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`008
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`tag source” is shown in “Landscapes” tag shown in yellow, which is shown
`
`alongside a tag type indicator icon (
`
` ). For all of the grounds that I identified in
`
`IPR2019-00516 and IPR2019-00528, the “first tag source” corresponds to the
`
`collection of tags associated with the displayed tag type indicator icon (e.g.,
`
` ).
`
`11. For those grounds that rely on Rothmuller as the primary reference (i.e.
`
`Grounds 6-7 of IPR2019-00516), the “first tag source” corresponds to the tag
`
`category or tag type associated with the displayed icon. (Ex. 1002 in IPR2019-
`
`00516, ¶¶199-201.) For those grounds that rely on Zuckerberg as the primary
`
`reference in combination with Rothmuller (i.e. Grounds 1-5 of IPR2019-00516), the
`
`“first tag source” corresponds to either the collection of tags used to populate the
`
`friends list 546, or the collection of tags used to populate the text list 544. (Id., ¶¶75-
`
`77.) It does not matter whether the “first tag source” is applied to the “friends list”
`
`tag collection or the “text list” tag collection in Zuckerberg. If the “friends list”
`
`collection is chosen as the “first tag source,” for example, then the “text list”
`
`collection would become the “second tag source,” and vice versa.
`
`12. For those grounds that rely on MacLaurin as the primary reference in
`
`combination with Rothmuller (i.e. Grounds 3 and 4 in IPR2019-00528), the “first
`
`tag source” corresponds to (1) either the collection of “automatic” tags or the
`
`collection of “explicit” tags, or (2) in the alternative application of MacLaurin, one
`
`of at least two “external” tag sources. (Ex. 1002 in IPR2019-00528, ¶97.) As with
`
`
`
`7
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`009
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`Zuckerberg, it is not important to identify precisely which of the two tag sources
`
`under (1) or (2) corresponds to the “first tag source,” since one can choose either tag
`
`source as the “first” and choose the other as the “second.”
`
`13. With respect to the “vertical list” limitation, I note that Figure 9A of
`
`Rothmuller shows the tags in the recently used tag list arranged in two adjacent
`
`vertical columns, rather than a single vertical list. But this does not change my
`
`opinion that Rothmuller discloses the claimed “vertical” tag list. This is because the
`
`claim only requires “a list including tags… wherein the tags in the tag list are
`
`displayed in a vertical list,” which on its face merely requires a vertical list with
`
`more than one tag. The claim does not require that the claimed “list” include every
`
`tag that might be presented on the display. The listing on the right side of the
`
`recently used tags in Figure 9A, therefore, satisfies the claim requirement and can
`
`independently be applied to the “tag list” limitation. The presence of other tags on
`
`the left side of the recently used tag area does not change this result.
`
`14.
`
`It would also have been obvious to a person of ordinary skill in the art
`
`that the recently used tag area in Figure 9A could have included fewer tags such that
`
`all of the tags would be listed in a single vertical list. For example, if the user only
`
`had three recently used tags (instead of the six shown in Figure 9A), all of the
`
`recently used tags would have appeared in a single vertical list on the left side of the
`
`recently used tag area of Figure 9A. It would have been obvious to a person of
`
`
`
`8
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`010
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`ordinary skill in the art that, at some earlier point during the use of the Rothmuller
`
`system, the recently used tag area in Figure 9A would have included fewer tags such
`
`that all of them would have been displayed in a single vertical list.
`
`15. But even if one interpreted the claim and Rothmuller to require that the
`
`claimed “vertical list” include every one of the tags within the two columns in the
`
`recently used tag area of Figure 9A, arranging those tags in the form of a single
`
`vertical list (as opposed to two lists) would have been obvious to a person of ordinary
`
`skill in the art. The ability to display icons or other objects in a single column, or
`
`multiple columns, was basic knowledge of persons of ordinary skill in the art and
`
`even casual users. For example, the popular Microsoft Windows operating system
`
`since at least the 1990s has had the ability (through the ubiquitous Windows “View”
`
`menu) to list items in a window as a single vertical list or in a multi-column
`
`arrangement similar to Figure 9A. This is illustrated in Fred Davis, The Windows
`
`95 Bible (1996) [Ex. 1028]; for example:
`
`
`
`9
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`011
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`
`
`
`
`
`(Ex. 1028, at 034-036.)
`
`16. A person of ordinary skill in the art would have understood that
`
`displaying the tags in Rothmuller in the form of a single vertical list was one of a
`
`finite number of techniques for arranging or listing items on the screen. The use of
`
`a single vertical list, or two adjacent columns as shown in Figure 9A, present easily-
`
`
`
`10
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`012
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`understood and predictable tradeoffs; a single vertical list for example may show
`
`fewer items at one time (without scrolling) but it can show more information about
`
`each item and accommodate longer item names. A person of ordinary skill in the art
`
`would have been motivated to arrange the tag list in Rothmuller as a vertical list, for
`
`example, to accommodate longer tag names that might be truncated or cut off under
`
`the two-column arrangement shown in Figure 9A. In any case, the ability to arrange
`
`items (such as the recently used tags of Figure 9A) in a single vertical list was so
`
`basic and well-understood that it does not even remotely recite a patentable feature.
`
`17. Figure 9A of the Rothmuller Provisional also discloses or renders
`
`obvious the claimed “first tag… above a second tag from a second tag source.”
`
`An example of the claimed “second tag” is the tag “Lori” shown in blue, which sits
`
`below the “Landscapes” tag and has a different tag type indicator icon (
`
` ). The
`
`“second tag source” corresponds to the group or collection of tags associated with
`
`the displayed tag type indicator icon (
`
` ). As explained previously, the identity of
`
`the “second tag source” depends on the primary reference cited; for Rothmuller, it
`
`is a collection of tags associated with a second tag category or type. For Zuckerberg,
`
`it is one of either the “friends list” or “text list” tag collections, depending on which
`
`was chosen as the first tag source. For MacLaurin, it is either (a) one of the
`
`“automatic” or “explicit” tag collection depend on which was chosen as the first tag
`
`source, or (b) a second “external” tag source.
`
`
`
`11
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`013
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`
`18. Finally, Figure 9A of the Rothmuller Provisional discloses or renders
`
`obvious that “the second tag from the second tag source is displayed above a
`
`third tag from the first tag source.” An example is the tag “Animals” above
`
`shown in yellow, which is shown alongside the same tag type indicator icon as the
`
`“first tag” (
`
` ), indicating that it also comes from “the first tag source.”
`
`2.
`Plotkin (Ex. 1008)
`19. Plotkin discloses these limitations for many of the same reasons as
`
`Rothmuller discussed extensively above. As explained in my opening declarations,
`
`Plotkin discloses a “tag list” in the form of a list of tags used to import tags, with
`
`each tag shown with a respective tag type indicator icon. (Ex. 1002 in IPR2019-
`
`00516, ¶¶54, 110; Ex. 1002 in IPR2019-00528, ¶¶55, 107.) The tag list of Plotkin,
`
`which I have annotated below, shows the list and how it applies to the claims:
`
`(Plotkin, p.328 (red annotations added; partial figure).)
`
`20. The figure above from Plotkin discloses or suggests displaying tags in
`
`
`
`12
`
`
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`014
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`a tag list “wherein the tags in the tag list are displayed in a vertical list,” as
`
`recited. The three tags shown alongside the red annotations above, i.e. “Birds,”
`
`“David P,” and “Seals/Sealions,” are shown listed in a vertical arrangement. As
`
`shown, the “first tag” (“Birds”) and the “third tag” (“Seals/Sealions”) both come
`
`from the “first tag source,” as indicated by the common tag type indicator icon (in
`
`this case a bird). The “second tag” (“David P”), as shown, comes from a second and
`
`different tag source, as indicated by a different tag type indicator icon (in this case
`
`an icon showing two people).
`
`21. Plotkin was not cited as a primary reference for any of the grounds, so
`
`the first, second, and third tags and tag sources come from the primary references
`
`that were combined with Plotkin, i.e. Zuckerberg and MacLaurin. In the case of
`
`Zuckerberg, as mentioned, the tags and tag sources come from and correspond to
`
`either the “friends list” or “text list” tag collections, as mentioned. For MacLaurin,
`
`the tags and tag sources come from and correspond to (a) the “automatic” or
`
`“explicit” tag collections, or (b) “external” tag sources.
`
`22. The arrangement shown above discloses and renders obvious that the
`
`first tag from the first tag source “is displayed above a second tag from a second
`
`tag source,” which in turn “is displayed above a third tag from the first tag
`
`source.” I note that in the exemplary tag list in Plotkin, the “second tag” (e.g. “David
`
`P”) is above the “third tag” (e.g. “Seals/Sealions”) in the list, but not immediately
`
`
`
`13
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`015
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`above it as there is an intermediate tag (e.g. “Russ H”) between them. But the claim
`
`does not require that that the three recited tags be displayed one-after-another with
`
`nothing in between them; it merely requires “a first tag” displayed “above a second
`
`tag,” which “is displayed above a third tag,” which Plotkin provides.
`
`23. But even if the claim were interpreted to require that the first, second,
`
`and third tags be vertically displayed with no intermediate tags between them, this
`
`would have been obvious to a person of ordinary skill in the art. The list shown in
`
`Plotkin is alphabetically organized, and it would have been obvious to a person of
`
`ordinary skill in the art that the sequence of tag type indicator icons in the list (and
`
`thus tag sources) was simply the happenstance of the particular tags that were being
`
`imported as captured in that screenshot. It would have been obvious to a person of
`
`ordinary skill in the art that the Plotkin user interface could have generated different
`
`arrangements that would meet this narrower construction; if for example the “Russ
`
`H” tag was not present in the tag import data, there would be no intermediary tag
`
`displayed between the “second” and “third” tags.
`
`24. At a broader level, in my opinion, the claimed visual and vertical
`
`arrangement of the first, second, and third tags is not a significant or patentable
`
`feature. It would have been obvious to a person of ordinary skill in the art that
`
`different collections of tags will generate different lists, and it only takes the right
`
`combination of input data to generate a tag list with the display arrangement required
`
`
`
`14
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`016
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`by the claim. Rothmuller and Plotkin both disclose the ability to display a tag type
`
`indicator adjacent to each tag in a tag list, so even without the figures that I cited
`
`above, it would have been obvious that both references would display three tags in
`
`the claimed vertical arrangement if it reflected the underlying tag data.
`
`25. My view is consistent with the fact that this claimed feature is not
`
`discussed anywhere in the textual description of the ’173 patent. Patent Owner’s
`
`sole support for this limitation is the fortuity that Figure 4B shows an alphabetically
`
`sorted vertical tag list that happens to show three items in this particular
`
`arrangement. (Motion at 4 (quoting ’173, Fig. 4B).) Nothing in the ’173 patent
`
`suggests that this particular display sequence was in any way a point of novelty over
`
`the prior art or significant in any way to the invention.
`
`II. THE TEACHINGS OF ZUCKERBERG, ROTHMULLER AND
`PLOTKIN RENDER THE SUBSTITUTE CLAIMS OBVIOUS
`26. As demonstrated above, all of the new limitations introduced in the
`
`substitute claims are disclosed or rendered obvious by Zuckerberg, Rothmuller and
`
`Plotkin. Accordingly, all challenged claims are obvious based on the same prior art
`
`that I cited in my opening declarations.
`
`A. How Instituted Grounds in IPR2019-00516 Render the Substitute
`Claims Unpatentable
`27. For IPR2019-00516, Grounds 2-5 were based on Zuckerberg in
`
`combination with either Rothmuller (Grounds 2-3) or Plotkin (Grounds 4-5). These
`
`
`
`15
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`017
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`grounds render the substitute claims obvious without the need to add any additional
`
`references. As I explained above, the additional features recited in the substitute
`
`claims are disclosed or rendered obvious by Zuckerberg and Rothmuller and Plotkin.
`
`28. As explained above, Zuckerberg discloses “displaying a photograph
`
`comprising at least one subject or object; receiving a user selection of a location in
`
`the photograph corresponding to the at least one subject or object” and “associating
`
`at least one of the tags in the tag list with the at least one subject or object.” With
`
`respect to the remaining additional limitations, in addition to the discussion of
`
`Rothmuller and Plotkin above, I already provided an analysis in my opening
`
`declaration in IPR2019-00516 that fully addresses how the additional limitations
`
`would have been obvious over Zuckerberg in view of Rothmuller or Plotkin. In
`
`particular, I explained how Zuckerberg discloses a “tag list,” and how Zuckerberg
`
`in view of Rothmuller or Plotkin discloses a “tag type indicator for each tag
`
`appearing in the tag list, said tag type indicator being indicative of a tag source
`
`associated with the tag.” (Ex. 1002 in IPR2019-00516, ¶¶74-79, 80-83, 98-104, 111-
`
`114.) The explanation that I provided in connection with combining Zuckerberg
`
`with Rothmuller or Plotkin fully addresses the new claim limitations. In particular,
`
`Zuckerberg plainly discloses a “tag list” where “the tags in the tag list are
`
`displayed in a vertical list.” (Zuckerberg, e.g., Fig. 5.) I also explained that a
`
`person of ordinary skill in the art would have been motivated to combine Rothmuller
`
`
`
`16
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`018
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`and Plotkin with Zuckerberg because doing so would have provided the benefit of a
`
`more flexibly-organized tag list, where tags could be displayed “in the tag list in any
`
`order, without the constraints of using a separate or distinct list for each type.” (Ex.
`
`1002 in IPR2019-00516, ¶102; see also id., ¶112.) “For example,” I explained, “tags
`
`of both types could be interspersed within the overall list, allowing the system to
`
`present a single list of alphabetically-sorted tags, a list sorted based on their predicted
`
`relevance to the user, and many other possibilities.” (Id. (underlining added); see
`
`also id., ¶112.) Therefore, combining Zuckerberg with Rothmuller or Plotkin in the
`
`manner explained in my opening declaration fully discloses and renders obvious
`
`“wherein the tags in the tag list are displayed in a vertical list, and wherein: a first
`
`tag from a first tag source is displayed above a second tag from a second tag source;
`
`and the second tag from the second tag source is displayed above a third tag from
`
`the first tag source.”
`
`29. For IPR2019-00516, Grounds 6-7 cited Rothmuller but did not cite
`
`Zuckerberg, which I have cited with respect to the new limitations relating to display
`
`of a photograph, user selection of a location, and tag association based on the user
`
`selection. Accordingly, in order to account for the new limitations of the substitute
`
`claims, Zuckerberg would be added to Grounds 6-7. The motivation to combine
`
`Rothmuller with Zuckerberg would have been straightforward, as shown below. As
`
`I explained extensively in my opening declaration, Rothmuller and Zuckerberg are
`
`
`
`17
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`019
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`analogous references in the field of computer-based systems for tagging content.
`
`(Ex. 1002 in IPR2019-00516, ¶99.)
`
`30.
`
`It would have been obvious to a person of ordinary skill in the art to
`
`adapt Zuckerberg’s techniques to Rothmuller, predictably resulting in the system of
`
`Rothmuller in which the system displays a photograph comprising at least one
`
`subject or object, receives a user selection of a location in the photograph
`
`corresponding to the at least one subject or object, and in response, associates a tag
`
`with the at least one subject or object. A person of ordinary skill in the art would
`
`have appreciated at least two clear benefits from this combination.
`
`31. First, the combination would have enhanced the system of Rothmuller
`
`by allowing the selection and association of tags with a particular object or subject
`
`in the photo rather than the photo as a whole. A person of ordinary skill in the art
`
`would have appreciated that photographs commonly contain many different objects.
`
`For example, a photo could include multiple people, natural or architectural
`
`landmarks (such as trees or mountains), animals, and many other things. Allowing
`
`a tag in Rothmuller to be associated with a particular subject or object in a photo
`
`makes the tag much more specific and useful than assignment to the photo as a
`
`whole, because it records where in the photo tagged item appears.
`
`32. A second but related motivation to combine flows from the ability in
`
`both Zuckerberg and Rothmuller to assign multiple tags to a single photo. As I
`
`
`
`18
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`020
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`explained in my opening declaration in IPR2019-00516, both Zuckerberg and
`
`Rothmuller allow users to assign multiple tags to a single photo. (Ex. 1002 in
`
`IPR2019-00516, ¶¶72-73 (Zuckerberg), 168-169 (Rothmuller).) Zuckerberg, for
`
`example, shows an example in which two people (i.e. “erin” and “betty jo”) were
`
`tagged in a single photo. (Id., ¶¶72-73.) Rothmuller similarly discloses an example
`
`in which a photo (“Lori on the road at Legoland”) was tagged with “Lori R” (a
`
`people tag) and “San Diego” (a places tag). (Id., ¶168.) Without the ability to
`
`granularly associate a tag with a particular subject or object in the photograph, the
`
`tag is much less useful because it cannot be used to identify where the tagged subject
`
`or object appears in the photo, or identify more significant objects in the photo. To
`
`take a concrete example, if the photograph “Lori on the road at Legoland” in
`
`Rothmuller included several other people in addition to Lori, the “Lori” people tag
`
`could not be used to determine which of the people in the photo was Lori.
`
`33. A person of ordinary skill in the art would have perceived no technical
`
`obstacle in making this combination. The addition to Rothmuller of the Zuckerberg
`
`user interface for displaying a photograph and selecting a region would have
`
`involved nothing more than routine and conventional programming techniques. The
`
`ability to display photographs and receive location input from the user (e.g. through
`
`a mouse, trackball, etc.) were basic techniques well-known to persons of ordinary
`
`skill in the art. A person of ordinary skill in the art would have found the benefits
`
`
`
`19
`
`Facebook's Ex. 1027
`IPR2019-00528
`
`021
`
`

`

`Reply Declaration of Sandeep Chatterjee, Ph.D.
`
`of Zuckerberg’s image display and region selection techniques sufficiently
`
`compelling to adapt the user interface of Rothmuller to incorporate them.
`
`B. How Instituted Grounds in IPR2019-00528 Render the Substitute
`Claims Unpatentable
`34. For IPR2019-00528, Grounds 3-6 cited MacLaurin as the primary
`
`reference and combined it with either Rothmuller (Grounds 3-4) or Plotkin (Grounds
`
`5-6), but these grounds did not cite Zuckerberg. Accordingly, in order to account
`
`for the new limitations of the substitute claims, Zuckerberg would be added to these
`
`grounds. The motivation to combine MacLaurin with Zuckerberg would have been
`
`straightforward, as shown below. As I explained extensively in my opening
`
`declaration in IPR2019-00516, MacLaurin and Zuckerberg are analogous references
`
`in the field of computer-based systems for tagging content. (Ex. 1002 in IPR2019-
`
`00516, ¶99.) The “Field of the Invention” of Zuckerberg defines the field as relating
`
`“generally to internet digital content, and more particularly to systems and methods
`
`for tagging digital media.” (Zuckerberg, 1:30-32.) MacLaurin similarly explains
`
`that “[its] subject matter relates generally to information retrieval, and more
`
`particularly to systems and methods for tagging items based on use