`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`
`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`
`
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`
`
`
`FACEBOOK, INC., INSTAGRAM, LLC, and WHATSAPP INC.,
`Petitioners
`
`v.
`
`BLACKBERRY LIMITED
`Patent Owner
`
`
`
`Case IPR2019-00516
`U.S. Patent No. 8,279,173 B2
`
`
`PETITIONER RESPONSE TO PATENT OWNER’S
`CONDITIONAL MOTION TO AMEND UNDER 37 C.F.R. § 42.1211
`
`
`
`
`
`
`
`
`
`1 An identical response is filed in IPR2019-00528.
`
`
`
`
`
`
`
`
`Table of Contents
`
`
`Page
`
`
`INTRODUCTION .......................................................................................... 1
`I.
`II. ARGUMENT .................................................................................................. 1
`A.
`Zuckerberg discloses “displaying a photograph comprising at
`least one subject or object; receiving a user selection of a
`location in the photograph corresponding to the at least one
`subject or object” and “associating at least one of the tags in the
`tag list with the at least one subject or object.” .................................... 2
`Rothmuller and Plotkin disclose and render obvious “wherein
`the tags in the tag list are displayed in a vertical list, and
`wherein: a first tag from a first tag source is displayed above a
`second tag from a second tag source; and the second tag from
`the second tag source is displayed above a third tag from the
`first tag source” .................................................................................... 5
`The Additional Teachings of Zuckerberg, Rothmuller and
`Plotkin Render the Substitute Claims Obvious When Applied to
`the Existing Instituted Grounds .......................................................... 14
`III. CONCLUSION ............................................................................................. 22
`
`B.
`
`C.
`
`
`
`
`
`-i-
`
`
`
`
`
`
`
`I.
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`INTRODUCTION
`Patent Owner filed identical contingent motions to amend in IPR2019-00516
`
`and IPR2019-00528, urging the Board to allow substitute claims 21-32 in the event
`
`the challenged claims are found unpatentable based on the instituted grounds. But
`
`the new features recited in substitute claims 21-32 do not provide any meaningful
`
`distinction over the prior art, and are obvious for the same reasons as the original
`
`challenged claims. The new limitations added by substitute claims 21-32 are so
`
`insubstantial, in fact, that they are disclosed and rendered obvious by the prior art
`
`already of record in these IPRs. Patent Owner’s motion should be denied.
`
`II. ARGUMENT
`Substitute claims 21-32 would replace claims 1, 2, 4, 6-7, 10, 12-14, 16, and
`
`18, the challenged claims. These substitute claims introduce new limitations only
`
`with respect to independent claims 21, 25, and 29, which would replace original
`
`independent claims 1, 7, and 13, respectively. The remaining substitute dependent
`
`claims are unchanged from their corresponding original claims other than modifying
`
`the claim dependency. (Motion at 2.)
`
`Generally speaking, the new limitations would add three additional features
`
`into the independent claims: (1) display of a photograph and user selection of a
`
`subject or object in the photograph; (2) display of a “vertical” tag list showing at
`
`least three tags from two tag sources in a particular arrangement; and (3) associating
`
`
`
`
`
`1
`
`
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`at least one of the tags in the list with the selected subject or object. But as explained
`
`below, each of these features is disclosed by the prior art.
`
`A. Zuckerberg discloses “displaying a photograph comprising at least
`one subject or object; receiving a user selection of a location in the
`photograph corresponding to the at least one subject or object” and
`“associating at least one of the tags in the tag list with the at least
`one subject or object.”
`These claim limitations, which correspond to features (1) and (3) in the brief
`
`summary above, are readily disclosed by Zuckerberg as shown in Figure 5:
`
`
`
`
`
`
`
`2
`
`
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`(Zuckerberg, Fig. 5 (partial figure; annotations added); Ex. 1027, ¶¶6-8.) As shown
`
`highlighted in yellow in Figure 5 above, the system instructs the user to “[c]lick on
`
`people in the photo to tag them.” (Zuckerberg, Fig. 5.) As shown in the red box,
`
`the user has selected a region 520 corresponding to an individual’s face. (Id.)2
`
`Zuckerberg discloses “displaying a photograph comprising at least one
`
`subject or object,” as shown by the display of digital image 362 in Figure 5 above.
`
`The photograph has “at least one subject or object,” in this case a human being.
`
`Zuckerberg also discloses “receiving a user selection of a location in the
`
`photograph corresponding to the at least one subject or object.” As disclosed in
`
`Figure 5 above and further discussed in Zuckerberg’s textual description, “[t]he
`
`region selection component 410 is configured to receive input from a user 101,” and
`
`“select a region (e.g., a selected region 520) within a digital image 362 according to
`
`the input.” (Zuckerberg, 8:15-22.) The “location in the photograph” in
`
`Zuckerberg thus corresponds to a point in the digital image 362 selected by the user
`
`(such as the cross-hatch shown in Figure 5), which can correspond to a subject or
`
`object in the photo. Zuckerberg explains that in some embodiments, “the user 101
`
`
`2 Zuckerberg is Exhibit 1003 in IPR2019-00516 but was not included in IPR2019-
`
`00528. Accordingly, Zuckerberg will be added as previously-unused Exhibit 1003
`
`in IPR2019-00528 upon the filing of this response.
`
`
`
`
`
`3
`
`
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`moves a cursor 530 on the user device 110 to a point in the digital image 362,” and
`
`then “clicks on the point and the region selection component 410 places a border
`
`525 around the selected region 520.” (Zuckerberg, 8:22-27.) The particular photo
`
`shown in Figure 5 shows that the point corresponds to a human being (“the at least
`
`one subject or object”), which is consistent with the instruction in Figure 5: “Click
`
`on people in the photo to tag them.” (Zuckerberg, Fig. 5.) It would have been
`
`obvious to a person of ordinary skill in the art the user in Zuckerberg could have
`
`selected a point in any photo corresponding to a subject or object. (Ex. 1027, ¶7.)
`
`Finally, Zuckerberg discloses “associating at least one of the tags in the tag
`
`list with the at least one subject or object,” recited in the substitute claims.
`
`Zuckerberg discloses the tag component 340 discussed above can, after receiving a
`
`selection of a tag in the tag list, associate that tag with the selected region in the
`
`photo. (Zuckerberg, e.g., 7:54-56 (“The tag component 340 is configured to select
`
`a region in the image and associate text with the region.”) (underlining added).)
`
`“Clicking on any of the previously used tags may associate the tag with the selected
`
`region 520.” (Zuckerberg, 8:66-9:1 (underlining added).) As explained previously,
`
`the selection region 520 in Zuckerberg can correspond to a subject or object in the
`
`photo. Zuckerberg therefore discloses this limitation. (Ex. 1027, ¶8.)
`
`
`
`
`
`4
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`
`U.S. Patent No. 8,279,173
`B. Rothmuller and Plotkin disclose and render obvious “wherein the
`tags in the tag list are displayed in a vertical list, and wherein: a
`first tag from a first tag source is displayed above a second tag from
`a second tag source; and the second tag from the second tag source
`is displayed above a third tag from the first tag source”
`Rothmuller and Plotkin disclose and render obvious these claim limitations as
`
`discussed in detail below. But as a more threshold issue, it is questionable whether
`
`the claimed visual and vertical arrangement of the first, second, and third tags is even
`
`a patentable feature. This feature is discussed nowhere in the textual description of
`
`the ’173 patent. Patent Owner’s sole support for these claim limitations under 35
`
`U.S.C. § 112 is the fortuity that Figure 4B of the ’173 patent shows a vertical tag list
`
`that happens to include three tags in this particular arrangement, i.e. a first tag from
`
`a first source, a second tag from a second source, and a third tag from the first source.
`
`(Motion at 4 (quoting and annotating ’173, Fig. 4B).) The particular sequence of
`
`tags in Figure 4B appears to be little more than the happenstance of the particular
`
`screenshot used to create that figure. Nothing in the ’173 patent suggests that this
`
`particular sequence or ordering of tags was in any way a point of novelty over the
`
`prior art or significant in any way to the invention. (Ex. 1027, ¶25.)
`
`Rothmuller and Plotkin specifically disclose this same sequence as discussed
`
`below, but even without those disclosures, the claimed arrangement would have
`
`been plainly obvious to a person of ordinary skill in the art. A skilled artisan would
`
`have appreciated that different tagging datasets will generate different lists, and it
`
`
`
`
`
`5
`
`
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`only takes the right combination and sequence of tag data to generate a list with the
`
`display arrangement recited in these new limitations. (Ex. 1027, ¶¶24-25.) It would
`
`have been obvious that Rothmuller and Plotkin could display tags in this sequence
`
`if it was consistent with the underlying tag data. (Id., ¶24.)
`
`1.
`Rothmuller (Exs. 1004 and 1005)
`Figure 9A of the Rothmuller Provisional discloses a “tag list” in the form of
`
`a list of recently used tags, with each tag shown with a respective tag type indicator
`
`icon. (Ex. 1002 in IPR2019-00516, ¶¶88, 89, 171-172, 197-198; Ex. 1002 in
`
`IPR2019-00528, ¶¶87-89.) The annotated used tag list from Rothmuller Figure 9A,
`
`below, shows this list and how it applies to the substitute claims:
`
`(Rothmuller Provisional, Fig. 9A (partial figure; highlighting and annotations
`
`added); Ex. 1027, ¶9.) Figure 9A above discloses or suggests displaying tags in a
`
`tag list “wherein the tags in the tag list are displayed in a vertical list,” as recited
`
`
`
`in the substitute claims.
`
`
`
`
`6
`
`
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`The three tags shown in highlighting above, i.e. Landscapes, Lori, and
`
`Animals, are shown listed one after another in a vertical arrangement. As shown,
`
`the “first tag” (“Landscapes”) and the “third tag” (“Animals”) both come from “a
`
`first tag source,” as indicated by the common tag type indicator icon (
`
` ) next to
`
`them. The “second tag” (“Lori”) comes from a second and different tag source as
`
`reflected by the different indicator icon (
`
` ).
`
`More specifically, an example of the claimed “a first tag from a first tag
`
`source” is shown in “Landscapes” tag shown in yellow, which is shown alongside a
`
`tag type indicator icon (
`
` ). For all of the grounds identified in IPR2019-00516
`
`and IPR2019-00528, the “first tag source” corresponds to the collection of tags
`
`associated with the displayed tag type indicator icon (e.g.,
`
` ). More information
`
`about how the teachings of Rothmuller apply in combination with the Zuckerberg
`
`and MacLaurin “tag sources” is provided in Part II.C below.
`
`With respect to the “vertical list” limitation, Figure 9A of Rothmuller shows
`
`tags in the recently used tag list arranged in two adjacent vertical columns, rather
`
`than a single vertical list. But this does not impact whether Rothmuller discloses the
`
`claimed “vertical” tag list. This is because the claim only requires “a list including
`
`tags… wherein the tags in the tag list are displayed in a vertical list,” which on
`
`its face merely requires a single vertical list with more than one tag. The claim does
`
`not require that the claimed “list” include every tag that might be presented on the
`
`
`
`
`
`7
`
`
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`display. The listing on the right side of the recently used tags in Figure 9A, therefore,
`
`satisfies the claim requirement and can independently be applied to the “tag list”
`
`limitation. Because the substitute claims are “comprising” claims, moreover, they
`
`do not preclude the existence of additional elements not recited in the claims.
`
`Accordingly, the presence of other tags on the left side of the recently used tag area
`
`in Figure 9A does not change this result. (Ex. 1027, ¶13.)
`
`But even assuming the claim required a single vertical list containing all
`
`displayed tags (which it does not), this would have been obvious to a person of
`
`ordinary skill in the art based on the disclosures of Rothmuller, for two reasons.
`
`First, it would also have been obvious to a person of ordinary skill in the art
`
`that the recently used tag area in Figure 9A could have included fewer tags such that
`
`all of the tags would be listed in a single vertical list. (Ex. 1027, ¶14.) For example,
`
`suppose the user had only three recently used tags instead of the six shown in Figure
`
`9A; in this scenario, all of the recently used tags would have appeared in a single
`
`vertical list on the left side of the recently used tag area of Figure 9A. It would have
`
`been obvious to a person of ordinary skill in the art that, at some earlier point during
`
`the use of the Rothmuller system – when fewer tags had been used – the recently
`
`used tag area in Figure 9A could have included fewer tags such that all of them
`
`would fit in a single vertical list. (Id.) In other words, it would have been obvious
`
`to a person of ordinary skill in the art that Rothmuller could display a single column
`
`
`
`
`
`8
`
`
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`list of tags, even if the particular scenario captured by Figure 9A happens to show
`
`two columns of tags.
`
`More fundamentally, arranging those tags in the form of a single vertical list
`
`(as opposed to two lists) would have been obvious to a person of ordinary skill in
`
`the art. The ability to display icons or other objects in a single column, or multiple
`
`columns, was basic knowledge of persons of ordinary skill in the art and even casual
`
`users. (Ex. 1027, ¶15.) For example, the popular Microsoft Windows operating
`
`system since at least the 1990s had the ability (through the ubiquitous Windows
`
`“View” menu) to list items in a window as a single vertical list or in a multi-column
`
`arrangement similar to Figure 9A. (Id.) This is illustrated in Fred Davis, The
`
`Windows 95 Bible (1996) [Ex. 1028]; for example:
`
`
`
`
`
`9
`
`
`
`
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`
`
`(Ex. 1028, at 034-036 (also quoted in Ex. 1027, ¶15).)
`
`As these screenshots from the ubiquitous 1990s Microsoft Windows operating
`
`system demonstrate, a person of ordinary skill in the art would have understood that
`
`displaying the tags in Rothmuller in the form of a single vertical list (similar to the
`
`“Details” view of Figure 3.17 in Davis shown above) was one of a finite number of
`
`techniques for arranging or listing items on the screen. (Ex. 1027, ¶16.) The use of
`
`a single vertical list, or adjacent vertical columns as shown in Figure 9A (and in the
`
`“List” view in Figure 3.18 of Davis shown above), present obvious alternatives with
`
`predictable tradeoffs; a single vertical list, for example, may show fewer items at
`
`one time (without scrolling) but it can show more information about each item and
`
`accommodate longer item names. (Id.) A person of ordinary skill in the art would
`
`have been motivated to arrange the tag list in Rothmuller as a vertical list, for
`
`example, to accommodate longer tag names that might be truncated or cut off under
`
`the multiple column arrangement shown in Figure 9A. (Id.) In any case, the ability
`
`to arrange items (such as the recently used tags of Figure 9A) in a single vertical list
`
`
`
`
`
`10
`
`
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`would have been regarded by a skilled artisan as so basic and well-understood that
`
`would not even remotely have been perceived as a patentable feature. (Id.)
`
`Continuing to the next part of this claim limitation, Figure 9A of the
`
`Rothmuller Provisional discloses or renders obvious the claimed “first tag… above
`
`a second tag from a second tag source.” An example of the claimed “second tag”
`
`is the tag “Lori” shown in blue, which sits below the “Landscapes” tag and has a
`
`different tag type indicator icon (
`
` ). The “second tag source” corresponds to the
`
`group or collection of tags associated with the displayed tag type indicator icon (
`
` ).
`
`Finally, Figure 9A of the Rothmuller Provisional discloses or renders obvious
`
`that “the second tag from the second tag source is displayed above a third tag
`
`from the first tag source.” An example is the tag “Animals” above shown in
`
`yellow, which is shown alongside the same tag type indicator icon as the “first tag”
`
`(
`
` ), indicating that it also comes from “the first tag source.”
`
`2.
`Plotkin (Ex. 1008)
`Plotkin discloses these limitations for many of the same reasons as Rothmuller
`
`discussed extensively above. Plotkin discloses a “tag list” in the form of a list of
`
`tags used to import tags, with each tag shown with a respective tag type indicator
`
`icon. (Ex. 1002 in IPR2019-00516, ¶¶54, 110; Ex. 1002 in IPR2019-00528, ¶¶55,
`
`107.) The annotated tag list of Plotkin, below, shows how it applies to the claims:
`
`
`
`
`
`11
`
`
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`
`
`(Plotkin, p.328 (red annotations added; partial figure).)
`
`The figure above from Plotkin discloses or suggests displaying tags in a tag
`
`list “wherein the tags in the tag list are displayed in a vertical list,” as recited.
`
`The three tags shown alongside the red annotations above, i.e. “Birds,” “David P,”
`
`and “Seals/Sealions,” are shown listed in a vertical arrangement. As shown, the
`
`“first tag” (“Birds”) and the “third tag” (“Seals/Sealions”) both come from the “first
`
`tag source,” as indicated by the common tag type indicator icon (in this case a bird).
`
`The “second tag” (“David P”), as shown, comes from a second and different tag
`
`source, as indicated by a different tag type indicator icon (in this case an icon
`
`showing two people).
`
`The arrangement shown above discloses and renders obvious that the first tag
`
`from the first tag source “is displayed above a second tag from a second tag
`
`source,” which in turn “is displayed above a third tag from the first tag source.”
`
`Petitioner notes that in the exemplary tag list in Plotkin, the “second tag” (e.g.
`
`12
`
`
`
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`“David P”) is above the “third tag” (e.g. “Seals/Sealions”) in the list, but not
`
`immediately above it as there is an intermediate tag (e.g. “Russ H”) between them.
`
`But the claim does not require that the three recited tags be displayed one-after-
`
`another with nothing in between them; it merely requires “a first tag” displayed
`
`“above a second tag,” which “is displayed above a third tag,” which the tag list from
`
`Plotkin above discloses. Because the substitute claims are “comprising” claims, they
`
`do not preclude the existence of other displayed tags.
`
`But even if the claim were interpreted to require that the first, second, and
`
`third tags be displayed vertically in sequence, with no intermediate tags between
`
`them, this would have been obvious to a person of ordinary skill in the art. (Ex.
`
`1027, ¶16.) The tag list in Plotkin is alphabetically organized, and it would have
`
`been obvious to a person of ordinary skill in the art that the sequence of tag type
`
`indicator icons in the list (and thus tag sources) was simply the happenstance of the
`
`particular tags that were being imported as captured in that screenshot. (Id., ¶23.) It
`
`would have been obvious to a person of ordinary skill in the art that the Plotkin user
`
`interface could have generated different arrangements meeting even this narrower
`
`view; if for example the “Russ H” tag was not present in the tag import data, the
`
`software would not display any intermediary tag between the “second” and “third”
`
`tags. (Id.) As noted, the claimed visual and vertical arrangement of the first, second,
`
`
`
`
`
`13
`
`
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`and third tags is not a significant or patentable feature; different sets of input data
`
`would have predictably generated different sequences of displayed tags. (Id., ¶24.)
`
`C. The Additional Teachings of Zuckerberg, Rothmuller and Plotkin
`Render the Substitute Claims Obvious When Applied to the
`Existing Instituted Grounds
`As demonstrated above, all of the new limitations introduced in the substitute
`
`claims are disclosed or rendered obvious by Zuckerberg, Rothmuller and Plotkin.
`
`The substitute claims are thus unpatentable based on the prior art already of record
`
`in these IPRs. But how the above described teachings of Zuckerberg, Rothmuller,
`
`and Plotkin apply to the instituted grounds differs from one set of grounds to another.
`
`For the convenience of the Board, therefore, Petitioner will walk through how the
`
`existing instituted grounds render the substitute claims obvious when the additional
`
`disclosures of Zuckerberg, Rothmuller, and Plotkin are taken into account.
`
`1.
`
`Instituted Grounds in IPR2019-00516
`(a) Application to Grounds 2-5 of IPR2019-00516
`Grounds 2-5 in IPR2019-00516 were based on Zuckerberg in combination
`
`with either Rothmuller (Grounds 2-3) or Plotkin (Grounds 4-5). These grounds thus
`
`already include all the prior art needed with respect to both the original claims and
`
`the additional limitations introduced by corresponding substitute claims 21-32.
`
`For example, the Petition explained how Zuckerberg itself discloses a “tag
`
`list,” and how Zuckerberg in view of Rothmuller or Plotkin renders obvious a “tag
`
`type indicator for each tag appearing in the tag list, said tag type indicator being
`
`14
`
`
`
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`indicative of a tag source associated with the tag.” (Petition in IPR2019-00516 at
`
`24-27, 43-45, 52-53; Ex. 1002 in IPR2019-00516, ¶¶74-79, 80-83, 98-104, 111-
`
`114.) The explanation previously provided in connection with the combination of
`
`Zuckerberg and Rothmuller, and Zuckerberg and Plotkin, also fully addresses the
`
`new claim limitations. For example, Zuckerberg plainly discloses a “tag list” where
`
`“the tags in the tag list are displayed in a vertical list.” (Zuckerberg, e.g., Fig. 5.)
`
`A person of ordinary skill in the art would have been motivated to combine
`
`Rothmuller and Plotkin with Zuckerberg because doing so would have provided the
`
`benefit of a more flexibly-organized tag list, where tags could be displayed in any
`
`order, without the constraints of a separate list for each type. (Petition in IPR2019-
`
`00516 at 44-45; Ex. 1002 in IPR2019-00516, ¶102; see also id., ¶112.)
`
`For purposes of the specific combination of Zuckerberg with Rothmuller or
`
`Plotkin, the “first tag source” corresponds to either the collection of tags used to
`
`populate the friends list 546, or the collection of tags used to populate the text list
`
`544. (Petition in IPR2019-00516 at 24-26; Ex. 1002 in IPR2019-00516, ¶¶75-77.)
`
`For example, the combination of Zuckerberg with Rothmuller or Plotkin would have
`
`predictably resulted in a vertical tag list displaying three tags, e.g., a “first tag” from
`
`the friends list, a “second tag” from the text list and below the first tag, and a “third
`
`tag” from the friends list and sitting below the second tag, each tag displayed
`
`alongside an appropriate tag type indicator icon as taught in Rothmuller and Plotkin.
`
`
`
`
`
`15
`
`
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`It does not matter whether the “first tag source” is applied to the “friends list” tag
`
`collection or the “text list” tag collection of Zuckerberg; if the “friends list”
`
`collection is chosen as the “first tag source,” then the “text list” collection becomes
`
`the “second tag source,” and vice versa. (Ex. 1027, ¶11.)
`
`As noted, although both Rothmuller and Plotkin disclose the claimed vertical
`
`tag list and arrangement of first, second, and third tags, their disclosures also render
`
`these features obvious. A skilled artisan would have appreciated that the system
`
`created based on the combination of Zuckerberg with Rothmuller or Plotkin could
`
`have displayed three tags in the claimed vertical arrangement if the underlying tags
`
`used to populate the tag list happened to include a sequence of three tags from a first
`
`tag source, second tag source, and the first tag source, respectively. (Ex. 1027, ¶24.)
`
`For all of these reasons, the combination of Zuckerberg with Rothmuller, and
`
`Zuckerberg with Plotkin, thus renders obvious “wherein the tags in the tag list are
`
`displayed in a vertical list, and wherein: a first tag from a first tag source is displayed
`
`above a second tag from a second tag source; and the second tag from the second tag
`
`source is displayed above a third tag from the first tag source,” along with all of the
`
`other limitations in the substitute claims. (Id., ¶¶11, 28.)
`
`(b) Application to Grounds 6-7 of IPR2019-00516
`These grounds cited Rothmuller as the primary reference and did not include
`
`Zuckerberg, which was cited above with respect to new limitations relating to
`
`
`
`
`
`16
`
`
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`display of a photograph, user selection of a location, and tag association based on
`
`the user selection. Accordingly, in order to account for the new limitations of the
`
`substitute claims, Zuckerberg would be added to Grounds 6-7. The motivation to
`
`combine Rothmuller with Zuckerberg would have been straightforward, as shown
`
`below. As explained in the Petition, Rothmuller and Zuckerberg are analogous
`
`references in the field of computer-based systems for tagging content. (Petition in
`
`IPR2019-00516, at 43; Ex. 1002 in IPR2019-00516, ¶99.)
`
`It would have been obvious to a person of ordinary skill in the art to adapt
`
`Zuckerberg’s techniques to Rothmuller, predictably resulting in the system of
`
`Rothmuller in which the system displays a photograph comprising at least one
`
`subject or object, receives a user selection of a location in the photograph
`
`corresponding to the at least one subject or object, and in response, associates a tag
`
`with the at least one subject or object. (Ex. 1027, ¶30.) A person of ordinary skill
`
`in the art would have been motivated to combine Zuckerberg and Rothmuller in this
`
`fashion to achieve at least two clear benefits.
`
`First, the combination would have enhanced the system of Rothmuller by
`
`allowing the selection and association of tags with a particular object or subject in
`
`the photo rather than the photo as a whole. (Ex. 1027, ¶31.) A person of ordinary
`
`skill in the art would have appreciated that photographs commonly contain many
`
`different objects. For example, a photo could include multiple people, natural or
`
`
`
`
`
`17
`
`
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`architectural landmarks (such as trees or mountains), animals, and other things.
`
`Allowing a tag in Rothmuller to be associated with a particular subject or object in
`
`a photo makes the tag much more specific and useful than assignment to the photo
`
`as a whole, because it records where in the photo tagged item appears. (Id.)
`
`A second but related motivation to combine flows from the ability in both
`
`Zuckerberg and Rothmuller to assign multiple tags to a single photo. (Ex. 1027,
`
`¶32.) As explained in IPR2019-00516, Zuckerberg and Rothmuller both allow users
`
`to assign multiple tags to a single photo. (Petition in IPR2019-00516 at 22-23, 54-
`
`55; Ex. 1002 in IPR2019-00516, ¶¶72-73, 168-169.) Zuckerberg, for example,
`
`shows an example in which two people (i.e. “erin” and “betty jo”) were tagged in a
`
`single photo. (Zuckerberg, Figs. 6 & 7 (showing a “betty jo” tag being added to a
`
`photo that already has “erin” tag); see also id., 1:65 (“The media owner may select
`
`and tag multiple regions.”), 9:67-10:1.) Rothmuller similarly discloses an example
`
`in which a photo (“Lori on the road at Legoland”) was tagged with “Lori R” (a
`
`people tag) and “San Diego” (a places tag). (Rothmuller, 4:42-45, Fig. 2.) The
`
`common ability to assign multiple tags to a photo strengthens the motivation a
`
`skilled artisan would have had to add Zuckerberg’s region selection features to
`
`Rothmuller. For example, without the ability to granularly associate a tag with a
`
`particular subject or object in the photograph, the tag is much less useful because it
`
`cannot be used to identify where the tagged subject or object appears in the photo,
`
`
`
`
`
`18
`
`
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`or distinguish more significant objects from in the photo. (Ex. 1027, ¶32.) To take
`
`a tangible example, if the photograph “Lori on the road at Legoland” in Rothmuller
`
`included several other people in addition to Lori, the “Lori” people tag could not be
`
`used to determine which of the people in the photo was Lori. (Id.)
`
`A person of ordinary skill in the art would have perceived no technical
`
`obstacle in making this combination. (Ex. 1027, ¶33.) The addition to Rothmuller
`
`of the Zuckerberg user interface for displaying a photograph and selecting a region
`
`would have involved nothing more than routine and conventional programming
`
`techniques. The ability to display photographs and receive location input from the
`
`user (e.g. through a mouse, trackball, etc.) involved basic and well-known
`
`techniques. (Id.) A person of ordinary skill would have found the benefits of
`
`Zuckerberg’s image display and region selection techniques sufficiently compelling
`
`to adapt them to the user interface and tagging system in Rothmuller. (Id.)
`
`2.
`
`Instituted Grounds in IPR2019-00528
`(a) Application to Grounds 3-6 in IPR2019-00528
`These four instituted grounds cited MacLaurin as the primary reference and
`
`combined it with either Rothmuller (Grounds 3-4) or Plotkin (Grounds 5-6), but
`
`these grounds did not cite Zuckerberg. Accordingly, in order to account for the new
`
`limitations of the substitute claims, Zuckerberg would be added to these existing
`
`grounds, and added as Exhibit 1003 to IPR2019-00528. The motivation to combine
`
`
`
`
`
`19
`
`
`
`
`
`
`
`
`
`
`
`IPR2019-00516 & IPR2019-00528
`U.S. Patent No. 8,279,173
`
`MacLaurin with Zuckerberg would have been straightforward, as shown below.
`
`MacLaurin and Zuckerberg are analogous references in the field of computer-based
`
`systems for tagging content. (Ex. 1027, ¶34.)
`
`It would have been obvious to a person of ordinary skill in the art to adapt
`
`Zuckerberg’s techniques to MacLaurin, predictably resulting in the tagging system
`
`of MacLaurin further adapted to display a photograph comprising at least one subject
`
`or object, receives a user selection of a location in the photograph corresponding to
`
`the at least one subject or object, and in response, associates a tag with the at least
`
`one subject or object. (Id., ¶35.)
`
`The same clear benefits discussed above for the combination of Rothmuller
`
`and Zuckerberg would have applied equally to the combination of MacLaurin and
`
`Zuckerberg. (Ex. 1027, ¶¶36, 31-33.) The combination would have enhanced the
`
`system of MacLaurin by allowing the selection and association of tags with a
`
`particular object or subject in the photo rather than the photo as a whole, resulting in
`
`the benefits discussed above. MacLaurin allows – and in fact expressly
`
`encourages – assignment of multiple tags to a single photo. (Ex. 1002 in IPR2019-
`
`00528, ¶¶65-67.) MacLaurin explains that its tagging technique “excels in allowing
`
`items to have multiple tags.” (MacLaurin, 6:53-54.) By allowing an item to “belong
`
`to multiple groups and associations without requiring the item to be moved or copied
`
`into many different locations,” MacLaurin exp