US011017020B2
`
`a2) United States Patent
`US 11,017,020 B2
`(10) Patent No.:
`
` Desmondetal. (45) Date of Patent: May25, 2021
`
`
`(54) METHOD AND APPARATUS FOR
`MANAGINGDIGITAL FILES
`
`(71) Applicant: pemoryweb, LLC, Glen Ellyn, IL
`(US)
`(72)
`Inventors: Christopher J. Desmond, Glen Ellyn,
`IL (US); Nancy L. Desmond, Glen
`Ellyn, IL (US); L. Michael Taylor,
`Chicago, IL (US)
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`5,450,504 A
`5,694,514 A
`
`9/1995 Calia
`12/1997 Evans
`(Continued)
`FOREIGN PATENT DOCUMENTS
`102591922 B_
`10/2015
`2466869 A3
`2/2017
`
`CN
`EP
`
`(73) Assignee: us) LLC,Glen Ellyn, IL
`(*) Notice:
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`(21) Appl. No.: 17/079,208
`
`(Continued)
`OTHER PUBLICATIONS
`Yee et al., “Faceted Metadata for Image Search and Browsing,” CHI
`2003, pp. 401-408, 2003, ACM.
`(Continued)
`
`(22)
`
`Filed:
`
`Oct. 23, 2020
`
`Primary Examiner — Ashish Thomas
`Assistant Examiner — Mellissa M. Ohba
`
`(65)
`
`Prior Publication Data
`US 2021/0042349 Al
`Feb. 11, 2021
`Related U.S. Application Data
`(63) Continuation of application No. 16/536,300, filed on
`re
`.
`.
`.
`oe
`Aug. 8, 2019, which is a continuation of application
`:
`(Continued)
`
`(74) Attorney, Agent, or Firm — Nixon Peabody LLP
`(57)
`ABSTRACT
`A computer-implemented method of associating digital tags
`with digitalfiles comprises storing a plurality of digital files
`having embeddedtherein content data and metadata includ-
`.
`_
`.
`.
`:
`.
`ing tags; receiving, via a user interface device of a client
`device,a first tag label containing alphanumeric text created
`and inputted by a userofthe client device; modifying, using
`a controller device, a selected first one of the tags of the
`metadata in a first of the digital files to includethefirst tag
`label; receiving, via the user interface device or another user
`interface device, an instruction to searchfor all of the digital
`files havingat leastthefirst tag label; responsive to receiving
`(52) U.S. CL
`the instruction, automatically searching forall of the digital
`CPC veeseeeees G06F 16/5866 (2019.01); GO6F 16/51
`(2019.01); GO6F 16/901 (2019.01); G06F__files having at least the first tag label; and displaying, on a
`16/907 (2019.01), GO6F 3/0481 (2013.01)
`video display device associated with the client device, a first
`(58) Field of Classification Search
`indication ofthe first tag label.
`None
`
`(51)
`
`Int. CI
`G06F I6/58
`GO6F 16/51
`
`2019.01
`(2019.0Q
`01)
`(
`(Continued)
`
`59 Claims, 50 Drawing Sheets
`See application file for complete search history.
`
`
`Clinton Dewitt Firestone IV
`
`Birth:
`Death:
`
`July 12, 1896
`April 23, 1971
`
`
`
`Parents: Clinton Dewitt Firestone {Nand Viola Miller
`
`Comments: He was a WWHULS. Air force pitot and POW in WWHand veteran
`honorably discharged in December of 1947. He worked for 44 years far the
`Firestone Tire and Rubber Campanyin retail, wholesale and original equipment
`sales, marketing and management. He was barn in Akron, OH and ts buried in
`Caiumbiana, OH.
`
`Edit bio
`
`
`
`Recipes
`
`Petitioner Apple Inc. - Ex. 1014, p. 1
`
`Petitioner Apple Inc. - Ex. 1014, p. 1
`
`

`

`US 11,017,020 B2
`
`Page 2
`
`Related U.S. Application Data
`
`(51)
`
`Int. Cl.
`GO6F 16/901
`GO6F 16/907
`GO6F 3/0481
`
`(2019.01)
`(2019.01)
`(2013.01)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`...cccccecee GO06F 16/10
`715/839
`
`4/2012 Higgins
`8,166,016 B2
`4/2012 Hayashi
`8,166,168 B2
`5/2012 Zaltzman
`8,171,388 B2
`No. 15/375,927, filed on Dec. 12, 2016, now Pat. No.
`5/2012 Tsutsui
`8,175,340 B2
`10,423,658, which is a continuation of application
`
`No. 14/193,426, filed on Feb. 28, 2014, now Pat. No. 8,230,338 B2=7/2012 Dugan
`8,239,784 B2
`8/2012 Hotelling
`9,552,376, which is a continuation-in-part of appli-
`8,255,379 B2
`8/2012 Govindachetty
`cation No. 13/157,214, filed on Jun. 9, 2011, now Pat.
`8,264,570 B2
`9/2012 Karimoto
`8,271,506 B2
`9/2012 Martinez
`No. 9,098,531.
`8,281,027 B2
`10/2012 Martinez
`8,285,483 B2
`10/2012 Amer-Yahia
`8,307,029 B2
`11/2012 Davis
`315,
`eng
`8,326,000 B2
`12/2012 Jung
`8,332,402 B2
`12/2012 Forstall
`8,358,811 B2
`1/2013 Adams
`8,359,314 B2
`1/2013 Svendsen
`8,364,611 B2
`1/2013 Tendjoukian
`8,380,039 B2
`2/2013 Luo
`8,386,506 B2
`2/2013. Martinez
`8,390,702 B2
`3/2013. Bhatt
`11/1998 Lobo
`5,835,616 A
`
`
`5,850,470 A 12/1998_Kung 8,401,771 B2 3/2013. Krumm
`
`5,982,912 A
`11/1999 Fukui
`8,402,356 B2
`3/2013 Martinez
`6,002,401 A * 12/1999 Baker
`8,416,312 B2
`4/2013. Matsunaga
`8,429,156 B2
`4/2013 Buchmueller
`8,447,120 B2
`5/2013 Ji
`10/2000 Luo
`6,134,339 A
`8,458,115 B2
`6/2013. Cai
`6/2001 Fukui
`6,246,779 Bl
`8,463,931 B2
`6/2013 Evans
`10/2001 Steffens
`6,301,370 Bl
`8,484,223 B2
`7/2013 Ota —
`9/2003. Parulski
`6,629,104 Bl
`8,489,115 B2
`7/2013 Rodriguez
`2/2004 Luo
`6,697,502 B2
`8,490,011 B2
`7/2013 Stapleton
`3/2004 Flora
`6,714,215 BL
`
`
`6,728,401 Bl 8,493,495 B2=7/2013. D’Souza4/2004 Hardeberg
`6,940,545 Bl
`9/2005 Ray
`8,503,791 B2
`8/2013. Conwell
`6,965,693 BL
`11/2005 Kondo
`8,504,073 B2
`8/2013 Svendsen
`7,003,135 B2
`2/2006 Hsieh
`8,520,979 B2
`8/2013. Conwell
`7,190,829 B2
`3/2007 Zhang
`D689,079 S
`9/2013 Edwards
`7,372,976 B2
`5/2008 Rhoad
`D689,080 S
`9/2013 Edwards
`7,461,099 Bl
`12/2008 Sharpe
`D689,083 S
`9/2013 Pasceri
`7,A74,317 B2
`1/2009 Dolph
`D689,084 S
`9/2013 Pasceri
`7,475,060 B2
`1/2009 Toyama
`D689,085 S
`9/2013 Pasceri
`7,480,669 B2
`—-1/2009 Lo
`8,538,811 B2
`9/2013 Higgins
`7,522,701 B2
`4/2009 Jensen
`8,538,813 B2
`9/2013 Kakarla
`7,587,671 B2
`9/2009 Saft
`8,542,294 B2
`9/2013 Bhatt
`7,634,662 B2
`12/2009 Monroe
`8,554,623 B2
`10/2013 Higgins
`7,646,895 B2
`1/2010 Haupt
`8,560,390 B2
`10/2013 Higgins
`7,694,236 B2
`4/2010 Gusmorino
`8,560,517 B2
`10/2013 Yang
`7,702,185 B2
`4/2010 Keating
`8,583,620 B2
`11/2013 Govindachetty
`7,804,982 B2
`9/2010 Howard
`8,583,668 B2
`11/2013 Higgins
`7,822,746 B2
`10/2010 Svendsen
`8,584,015 B2
`11/2013 Osten |
`7,840,344 B2
`11/2010 Sloo
`8,589,389 B2
`11/2013 Bisdikian
`7,840,892 B2* 11/2010 Pyhalammi
`8,589,486 B2
`11/2013 Martinez
`81606021 B2
`122013 Conwell
`1
`3
`5
`onwe.
`8,626,699 B2
`1/2014.
`‘Xie
`32011 Yen
`4/2011. Ko
`8,649,604 B2
`2/2014 Steinberg
`5/2011 Stanton
`8,660,358 Bl
`2/2014 Bergboer
`6/2011 Howard
`8,671,154 B2
`3/2014 Davis
`6/2011 Hayashi
`8,676,001 B2
`3/2014 Bracher
`6/2011 Cha vcecsssscseceee GO6F 3/04817
`8,698,762 B2*
`4/2014 Wagner... GO6F 3/04883
`383/305
`345/173
`
`4/2014 Kalaboukis
`4/2014 Thota
`6/2014 Bachman
`6/2014 Martinez
`6/2014 Ganong
`6/2014 Davis |
`7/2014 Pasceri
`7/2014 Pasceri
`asce;rl
`7/2014 Kalaboukis
`7/2014 Abhyanker
`8/2014 Davis
`8/2014 Luo
`8/2014 Stapleton
`8/2014 Akiya
`8/2014 Chao we GO6F 16/54
`—
`382/305
`8/2014 Higgins
`9/2014 Raghuveer
`
`Petitioner Apple Inc. - Ex. 1014, p. 2
`
`7916908 B2
`7929-771 B2
`7,948,502 B2
`7,962,467 B2
`7,965,908 B2
`7.970.240 BL*
`
`7,991,283 B2
`8,001,124 B2
`8,015,144 B2
`8.024.317 B2
`8,032,508 B2
`055,
`8,055,675 B2
`8,060,492 B2
`8,069,142 B2
`8,073,461 B2*
`8,086,048 B2
`8,086,867 B2
`8,108,778 B2
`8,121,408 B2
`8,144,232 B2*
`
`*
`
`............. GO6F 16/58
`715/230
`
`8/2011 Chen
`8/2011 Svendsen
`9/2011 Zheng
`9/2011 Nair
`10/2011 Martinez
`gg
`11/2011 Higgins
`11/2011 Nair
`11/2011 Davis
`12/2011 Altman0... seoe
`12/2011 Naaman
`12/2011 Freeman
`1/2012 Athsani
`2/2012 Omori
`3/2012 Larson ........... HO4N 5/232935
`348/333.05
`
`8,706,406 B2
`8,712,192 B2
`8,743,411 B2
`8,745,133 B2
`8,750,574 B2
`8,762,285 B2
`D708,196 S
`D708,197 S$
`5
`8,769,099 B2
`8,769,393 Bl
`8.799.371 B2
`8,805,165 B2
`8,806,365 B2
`8,810,597 B2
`8,811,775 BL*
`
`8,150,844 B2
`8,150,967 B2
`8,165,352 Bl
`
`4/2012 Redstone
`4/2012 King
`4/2012 Mohanty
`
`8,813,107 B2
`8,825,472 B2
`
`Petitioner Apple Inc. - Ex. 1014, p. 2
`
`

`

`US 11,017,020 B2
`
`Page 3
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`3/2017 Hotelling
`9,606,668 B2
`4/2017 Martinez
`9,626,685 B2
`5/2017 Ganong ..........0 GO6K 9/00288
`9,639,740 B2*
`5/2017 Boyns
`9,646,025 B2
`5/2017 Bisdikian
`9,654,570 B2
`9/2014 Gao
`8,831,352 B2
`6/2017 Hartford
`9,674,650 B2
`9/2014 Hubaceketal.
`8,845,855 B2
`6/2017 Didjusto
`9,677,886 B2
`9/2014 Kakarla
`8,849,854 B2
`6/2017 East
`9,679,456 B2
`9/2014 Farmer
`8,849,909 B2
`
`
`D715,819 S 9,680,929 B2=6/2017 Tseng10/2014 Pasceri
`8,880,535 Bl
`11/2014 Agarwal
`9,683,858 B2
`6/2017 Zheng
`8,880,568 B2
`11/2014 Perezynski
`9,691,073 B2
`6/2017 Tseng
`8,890,888 B2
`11/2014 Lee
`9,703,873 B2
`7/2017 Fakeih
`8,892,495 B2
`11/2014 Hoffberg
`9,706,345 B2
`7/2017 Davis
`8,914,342 B2
`12/2014 Kalaboukis
`9,710,961 B2
`7/2017 Setlur
`8,923,889 B2
`12/2014 Svendsen
`9,715,366 B2
`7/2017 Bostick
`8,930,848 B2
`1/2015 Lim
`9,721,188 B2
`8/2017 Adam
`8,949,212 Bl
`2/2015 Dhandapani
`9,754,226 B2
`9/2017 Zheng ©
`8,954,425 B2
`2/2015 Xiao
`9,772,745 B2
`9/2017 Hasenei
`8,966,121 B2
`2/2015 Josefsberg
`9,787,799 B2
`10/2017 Grue
`8,972,177 B2
`3/2015 Zheng
`9,805,123 B2
`10/2017 Nair
`
`8,998,422 Bl 9,811,879 B2=11/2017 Miller4/2015 Snavely
`
`9,009,177 B2
`4/2015 Zheng
`9,836,183 Bl
`12/2017 Love
`9,014,511 B2
`4/2015 Bracher
`9,857,941 B2
`1/2018 Wagner
`9,015,617 B2
`4/2015 Stapleton
`9,858,348 Bl
`1/2018 Higgins
`9,015,633 B2
`4/2015 Takamura
`9,870,572 B2
`1/2018 Chapin
`9,020,247 B2
`4/2015 Adam
`9,881,179 B2
`1/2018 Patton
`9,031,953 B2
`5/2015 Rathnavelu
`9,882,994 B2
`1/2018 Bisdikian
`9,032,320 B2
`5/2015 Crawford
`9,916,075 B2
`3/2018 Chen
`9,047,847 B2
`6/2015 Hochmuth
`9,942,121 B2
`4/2018 Poletto
`9,055,037 B2
`6/2015 Evans
`10,001,917 B2
`6/2018 Kim
`9,063,226 B2
`6/2015 Zheng
`10,019,850 B2
`7/2018 Lindberg
`9,076,259 B2
`7/2015 Hourie
`10,037,327 B2
`7/2018 Thomee
`9,092,409 B2
`7/2015 Charaniya
`10,068,178 B2
`9/2018 van Zwol
`9,098,545 B2
`8/2015 Abhyanker
`10,073,584 B2
`9/2018 Miura
`9,104,729 B2
`8/2015 Dong
`10,074,093 B2
`9/2018 Higgins
`9,104,915 B2
`8/2015 Conwell
`10,083,533 B2
`9/2018 Bhatt
`9,110,903 B2
`8/2015 Martinez
`10,110,541 B2
`10/2018 Li
`
`9,135,751 B2 10,120,947 B2=11/2018 Kritt9/2015 Moore
`
`9,151,618 B2
`10/2015 Amer-Yahia
`10,139,989 B2
`11/2018 Shiroor
`9,152,849 B2
`10/2015 Ganong
`10,140,743 B2
`11/2018 Hochmuth
`
`9,158,794 B2 10,145,704 B2=12/2018 Lanza10/2015 Higgins
`
`9,160,802 B2
`10/2015 Svendsen
`10,147,215 B2
`12/2018 Lanza
`9,172,666 B2
`10/2015 Murdock
`10,187,543 B2
`1/2019 Lahcanski
`D742,405 S_
`11/2015 Choi
`10,223,701 B2
`3/2019 King
`9,202,200 B2
`12/2015 Stibel
`10,230,803 B2
`3/2019 Higgins
`9,218,328 B2
`12/2015 Stapleton
`10,235,444 B2
`3/2019 Poletto
`9,224,172 B2
`12/2015 Churchill
`10,242,051 B2
`3/2019 Shinn —
`9,235,766 B2
`1/2016 Yi
`10,282,752 B2
`5/2019 Athsani
`9,239,848 B2
`1/2016 Liu
`10,288,433 B2
`5/2019 Zheng
`9,245,041 B2
`1/2016 Pilskalns
`10,289,643 B2
`5/2019 Brucher
`9,261,376 B2
`2/2016 Zheng
`10,303,975 B2
`5/2019 Adam
`D751,597 S
`3/2016 Pasceri
`10,311,611 B2
`6/2019 Stoop
`9,311,396 B2
`4/2016 Meadow
`10,318,110 B2
`6/2019 Naaman
`9,323,855 B2
`4/2016 Hochmuth
`10,324,973 B2
`6/2019 Circlaeys
`9,336,240 B2
`5/2016 Bhatt
`10,331,863 B2
`6/2019 Patton
`9,372,931 B2
`6/2016 Capt
`10,360,352 B2
`7/2019 Patton
`9,390,104 B2
`7/2016 Thomee
`10,430,452 B2
`10/2019 Ross
`9,405,981 B2
`8/2016 Li
`10,445,346 B2
`10/2019 Govindachetty
`
`9,418,485 B2 10,489,980 Bl=11/2019 Canavor8/2016 Lindberg
`
`9,424,595 B2
`8/2016 Svendsen
`10,540,668 B2
`1/2020 Hoertz
`9,436,374 B2
`9/2016 Marr
`10,606,449 B2
`3/2020 Canavor
`9,460,116 B2
`10/2016 Pilskalns
`10,628,463 B2
`4/2020 Purumala
`9,462,054 B2
`10/2016 Poletto
`10,643,263 B2
`5/2020 Wormhoudt
`9,465,513 B2
`10/2016 sims
`10,650,039 B2
`5/2020 Mariner
`9,471,200 B2
`10/2016 Dellinger
`10,650,475 B2
`5/2020 Berg
`9,471,834 Bl
`10/2016 Filip
`2001/0030667 Al* 10/2001 Kelts 0.0.0... HOAN 21/4314
`9,483,500 B2
`11/2016 Blucher
`715/854
`9,495,583 B2
`11/2016 Gilley
`9,501,577 B2
`11/2016 Zheng
`9,507,778 B2
`11/2016 Jaffe
`9,419,682 B2
`12/2016 Pujara
`9,535,563 B2
`1/2017 Hoffberg
`9,536,146 B2
`‘1/2017 Zheng
`9,552,376 B2
`1/2017 Desmond
`9,557,162 B2
`‘1/2017 Rodriguez
`9,576,253 B2
`2/2017 Zaltzman
`9,582,546 B2
`2/2017 Hartford
`9,593,957 B2
`3/2017 Zheng
`9,600,484 B2
`3/2017 Davis
`
`11/2001 Cooper
`2/2002 Meyers
`6/2002 Rosenzweig ........... GO6F 16/58
`715/835
`7/2002 Staas .....cceeeen GOOF 16/58
`382/113
`8/2002 Kells... eee GO6F 16/954
`725/39
`
`2001/0043727 Al
`2002/0019224 Al
`2002/0075322 Al*
`
`2002/0097894 AL*
`
`2002/0112237 AL*
`
`2002/0191818 Al
`2003/0033296 A1*
`2003/0039380 Al
`2003/0053663 Al
`
`12/2002 Matsuo
`2/2003 Rothmiuller............. GO6F 16/40
`2/2003 Sukegawa
`3/2003 Chen
`
`Petitioner Apple Inc. - Ex. 1014, p. 3
`
`Petitioner Apple Inc. - Ex. 1014, p. 3
`
`

`

`US 11,017,020 B2
`
`Page 4
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2009/0132941 Al
`2009/0135438 Al
`2009/0185784 Al
`2009/0216704 Al
`2009/0222302 Al
`2009/0254867 Al
`2009/0265631 Al
`2009/0278806 Al
`2009/0279794 Al
`2009/0288005 Al
`2009/0290812 Al
`2009/0307618 Al
`2009/0324018 Al
`2009/0325602 Al
`2010/0030578 Al*
`
`2010/0041419 Al
`2010/0046842 Al
`2010/0053371 Al
`2010/0061631 Al
`2010/0064239 Al
`2010/0082427 Al
`2010/0083173 Al
`2010/0088641 Al
`2010/0107125 Al
`2010/0153348 Al
`2010/0162411 Al
`2010/0171763 A1l*
`
`5/2009 Pilskalns
`5/2009 Chopra
`7/2009 Hiroike
`8/2009 Zheng
`9/2009 Higgins
`10/2009 Farouki
`10/2009 Sigurbjornsson
`11/2009 Duarte
`11/2009 Brucher
`11/2009 Stapleton
`11/2009 Naaman
`12/2009 Lawler
`12/2009 Tell
`—
`12/2009 Higgins
`2/2010 Siddique ............. G06Q 40/12
`705/3
`
`2/2010 Svendsen
`2/2010 Conwell
`3/2010 Karimoto
`3/2010 Omori
`3/2010 Crawford
`4/2010 Burgener
`4/2010 Germann
`4/2010 Choi
`4/2010 Ockene
`6/2010 Perezynski
`6/2010 Chang
`7/2010 Bhatt 0. GO6F 16/9537
`345/660
`
`4/2003 Lee
`2003/0063669 Al
`6/2003 Lee
`2003/0103652 Al
`7/2003 Zimmerman
`2003/0122787 Al
`7/2003 Tian
`2003/0133599 Al
`9/2003 Ho
`2003/0179911 Al
`10/2003 Kee
`2003/0198368 Al
`11/2003. Chen
`2003/0210808 Al
`4/2004 Takenaka
`2004/0081338 Al
`6/2004 Lestideau
`2004/0109584 Al
`7/2004 Yokoi
`2004/0125991 Al
`7/2004 Meier
`2004/0135797 Al
`9/2004 Doi
`2004/0190758 Al
`10/2004 Phillips
`2004/0205504 Al
`11/2004 Harville
`2004/0218894 Al
`11/2004 Toyama
`2004/0225635 Al
`12/2004 Zhang
`2004/0264780 Al
`12/2004 Taugher
`2004/0264810 Al
`2/2005 Hwang
`2005/0031173 Al
`3/2005 Filley
`2005/0060299 Al
`5/2005 Sung
`2005/0094849 Al
`5/2005 Li
`2005/0100195 Al
`5/2005 Nagaoka
`2005/0105806 Al
`6/2005 Yonaha
`2005/0117802 Al
`6/2005 Nagahashi
`2005/0141766 Al
`8/2005 Yang
`2005/0180627 Al
`7/2010 Gilley
`2010/0172550 Al
`8/2005 Amano
`2005/0183026 Al
`7/2010 Lee
`2010/0182341 Al
`10/2005 Enomoto
`2005/0220347 Al
`7/2010 Higgins
`2010/0185509 Al
`11/2005 Takikawa
`2005/0251015 Al
`9/2010 Pisula
`2010/0231537 Al
`12/2005 Porter
`2005/0265603 Al
`9/2010 Davis
`2010/0241689 Al
`1/2006 Chiu
`2006/0001652 Al
`9/2010 Athsani
`2010/0241944 Al
`1/2006 Saito
`2006/0021027 Al
`9/2010 Matsunaga
`2010/0245614 Al
`2/2006 Kim
`2006/0029265 Al
`10/2010 Pilskalns
`2010/0268717 Al
`3/2006 Adam
`2006/0050933 Al
`10/2010 Bouget
`2010/0268766 Al
`4/2006 Kim
`2006/0078201 Al
`10/2010 Steinberg
`2010/0272363 Al
`6/2006 Li
`2006/0133672 Al
`11/2010 O'Sullivan
`2010/0280913 Al
`7/2006 Tanaka
`2006/0165380 Al
`11/2010 Coddington
`2010/0283743 Al
`9/2006 Steinberg
`2006/0204034 Al
`11/2010 Ganong
`2010/0287053 Al
`10/2006 Jung
`2006/0222215 Al
`11/2010 Athsani
`2010/0293035 Al
`10/2006 Kitamura
`2006/0222217 Al
`11/2010 Harrison
`2010/0293193 Al
`2006/0224738 Al* 10/2006 McIntyre ........... HOAN 1/00132
`709/225 2010/0302179 Al=12/2010 Aim
`
`2007/0110305 Al
`5/2007 Corcoran
`2010/0312596 Al* 12/2010 Saffarl oo... GO6F 3/0481
`2007/0118508 Al
`5/2007 Svendsen
`705/732
`2007/0152984 Al
`7/2007 Ording
`2007/0206834 Al
`9/2007 Shinkai
`2007/0211925 Al
`9/2007 Aoki
`2007/0239764 Al
`10/2007 Song
`2007/0271297 Al
`11/2007 Jaffe
`2007/0282908 Al
`12/2007 Van der Meulen
`2008/0040034 Al
`2/2008 Kanno
`2008/0051994 Al
`2/2008 Fisher
`2008/0052945 Al
`3/2008 Matas
`2008/0056580 Al
`3/2008 Okada
`2008/0080743 Al*
`4/2008 Schneiderman.
`
`..... GO8B 13/196
`382/118
`
`2/2011 Svendsen
`2011/0040779 Al
`3/2011 Aonuma
`2011/0063108 Al
`4/2011 Zheng
`2011/0093458 Al
`§/2011 Bhatt
`2011/0109769 Al
`5/2011 Govindachetty
`2011/0113064 Al
`6/2011 Kankainen
`2011/0145258 Al
`8/2011 Chin
`2011/0188713 Al
`8/2011 Feng
`2011/0191014 Al
`8/2011 Pilskalns
`2011/0191253 Al
`8/2011 Svendsen
`2011/0202267 Al
`8/2011 Zheng
`2011/0208426 Al
`11/2011 Zheng
`2011/0289031 Al
`12/2011 Zheng
`2011/0301832 Al
`5/2008 Zhang
`2008/0122944 Al
`12/2011 Cho
`2011/0307836 Al
`6/2008 Naaman
`2008/0148175 Al
`12/2011 Svendsen
`2011/0314016 Al
`7/2008 Lamiraux
`2008/0168349 Al
`2/2012 Park
`2012/0047457 Al
`7/2008 Blumenberg
`2008/0168402 Al
`4/2012 Osten
`2012/0096361 Al
`8/2008 Kimchi
`2008/0201302 Al
`5/2012 Sugiyama
`2012/0113475 Al
`9/2008 Gao
`2008/0212849 Al
`5/2012 Conwell
`2012/0114249 Al
`9/2008 Torii
`2008/0212879 Al
`6/2012 Gammill
`2012/0158755 Al
`9/2008 Steinberg
`2008/0220750 Al
`6/2012 Tsuda
`2012/0162249 Al
`9/2008 Noda
`2008/0232695 Al
`
`
`2008/0250398 Al 2012/0192110 Al=7/2012, Wu10/2008 Takahashi
`2008/0298766 A1* 12/2008 Wen ou... GO6F 16/5854
`2012/0204101 Al
`8/2012 Yoshida
`386/282
`2012/0210200 Al*
`8/2012 Berger... GO6F 3/0481
`715/202
`
`2008/0309632 Al
`2008/0317379 Al
`2009/0013041 Al
`2009/0019085 Al
`2009/0049408 Al
`2009/0106705 Al
`2009/0113350 Al
`2009/0132689 Al
`
`12/2008 Westerman
`12/2008 Steinberg
`1/2009 Farmer
`1/2009 Abhyanker
`2/2009 Naaman
`4/2009 Takamura
`4/2009 Hibino
`§/2009 Zaltzman
`
`2012/0218150 Al
`2012/0220311 Al
`2012/0251011 Al
`2012/0266090 Al
`2012/0278171 Al
`2012/0278767 Al
`2012/0329441 Al
`
`8/2012 Oyabu
`8/2012 Rodriguez
`10/2012 Gao
`10/2012 Nealer
`11/2012 Tang
`11/2012 Stibel
`12/2012 Tseng
`
`Petitioner Apple Inc. - Ex. 1014, p. 4
`
`Petitioner Apple Inc. - Ex. 1014, p. 4
`
`

`

`US 11,017,020 B2
`
`Page 5
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`2012/033 1091
`2013/0018881
`2013/0036165
`2013/0063613
`2013/0073202
`2013/0101157
`2013/0138685
`2013/0141612
`2013/0151597
`2013/0185676
`2013/0202198
`2013/0275536
`2013/0339440
`2014/0040774
`2014/0059477
`2014/0059492
`2014/007 1272
`2014/0088861
`2014/0089811
`2014/0101531
`2014/0101601
`2014/0112553
`2014/0143247
`2014/0149036
`2014/0161326
`2014/0181089
`2014/0188880
`2014/0193087
`2014/0207444
`2014/0258850
`2014/0354628
`2015/0039630
`2015/00669 19
`2015/0070165
`2015/0070397
`2015/0116540
`2015/0117713
`2015/013 1872
`2015/0156247
`2015/0186389
`2015/0213057
`2015/0213329
`2015/0244794
`2015/0244833
`2015/0358224
`2016/0048279
`2016/0092741
`2016/0162512
`2016/0247307
`2016/0253358
`2016/03 14187
`2016/032 1269
`2016/0328444
`2016/0344888
`2016/0357822
`2017/00244 15
`2017/0046565
`2017/0069123
`2017/0103081
`2017/0192645
`2017/0357672
`2018/0181281
`2018/0364872
`
`12/2012
`1/2013
`2/2013
`3/2013
`3/2013
`4/2013
`5/2013
`6/2013
`6/2013
`7/2013
`8/2013
`10/2013
`12/2013
`2/2014
`2/2014
`2/2014
`3/2014
`3/2014
`3/2014
`4/2014
`4/2014
`4/2014
`5/2014
`5/2014
`6/2014
`6/2014
`7/2014
`7/2014
`7/2014
`9/2014
`12/2014
`2/2015
`3/2015
`3/2015
`3/2015
`4/2015
`4/2015
`5/2015
`6/2015
`7/2015
`7/2015
`7/2015
`8/2015
`8/2015
`12/2015
`2/2016
`3/2016
`6/2016
`8/2016
`9/2016
`10/2016
`11/2016
`11/2016
`11/2016
`12/2016
`1/2017
`2/2017
`3/2017
`4/2017
`7/2017
`12/2017
`6/2018
`12/2018
`
`Tseng
`Bhatt
`Tseng
`Conwell
`Zheng
`Li
`Brucher
`Bhatt
`Akiya
`Cao
`Adam
`Murdock
`Balassanian
`Chartyoniuk
`Wong
`Hashida
`Rodriguez
`Nash
`Meadow
`Capt
`Tang
`Yamaguchi
`Rathnavelu
`Amer-Yahia
`Ganong
`Desmond.
`Abhyanker
`Conwell
`Heiman
`Carey
`Lindberg
`Thomee
`Park
`East
`Miller
`Gilman
`Zheng
`Ganong
`Hensel
`Zheng
`Brucher
`Adam
`Poletto
`Gru
`Poletto
`Han
`Li
`Battistini
`Stoop
`Bhatt
`Poletto
`Thomee
`Shinn
`Lahcanski
`Woods
`Brucher
`Gilley
`Hochmuth
`Jones
`Murray
`Circlaeys
`Suki
`Miura
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`WO
`WO
`WO
`
`2410414 Bl
`WO 2011/070225 Al
`WO 2013/019376 Al
`WO 2013/099704 Al
`
`10/2019
`6/2011
`2/2013
`7/2013
`
`OTHER PUBLICATIONS
`
`Kustanowitz et al., “Motivating Annotation for Personal Digital
`Phot Libraries: Lowering Barriers while Raising Incentives,” tech.
`Report HCIL-2004-18, U. Maryland, 2005 (10 pages).
`Jaffe et al., “Generating Summaries and Visualization for Large
`Collections of GeoReferenced Photogra[hs,” MIR ’06, pp. 89-98,
`2006 ACM.
`Snavely et al., “Photo Tourism: Exploring Photo Collection in 3D,”
`SIGGRAPH ’06 ACM Transactions on Graphics, 25(3): 835-846,
`2006 ACM.
`Ahern et al., “World Explorer: Visualizing Aggregate Data From
`Unstructured Text in Geo-Referenced. Collections,” JCDL ’07, pp.
`1-10, 2007, ACM.
`Ameset al., “Why wetag: motivations for annotation in mobile and.
`online media,” CHI-07 Proceedings of the SIGCHI Conference on
`Human Factors in Computing Systems, pp. 971-980, ACM, 2007
`(10 pages).
`Chen, Y-F. et al.; “GeoTracker: Geospatial and Temporal RSS
`Navigation”; At&T Labs—Research, Florham Park, NJ; WWW
`2007 / Track: Browsers and User Interfaces, Session: Smarter
`Browsing, pp. 41-50 (10 pages).
`Ferre, “CAMELIS: Organizing and Browsing a Personal Photo
`Collection with a Logical Information System,” Int. Conf. Concept
`Lattices and Their Applications, pp. 112-123, 2007, HAL.
`Goodman, E.; “Destination Services: Tourist media and networked
`places”; School of Information, UC Berkeley; Mar. 2, 2007 (11
`pages).
`Kanget al., “Capture, Annotate, Browse, Find, Share: novel Inter-
`faces for Personal Photo Management,” International Journal of
`Human-Computer Interaction, 23(3): 315-337, 2007, Lawrence
`Eribaum Associates, Inc.
`Kennedy, L. et al.; “How Flickr Helps Us Make Sense of the World:
`Context and Content in Community-Contributed Media Collec-
`tions”; MM ’07, Sep. 23-28, 2007; Augsburg, Bavaria, Germany;
`Copyright 2007; ACM 978-1-59593-70 1-/8/07/0009 (10 pages).
`Rattenbury, T. et al.; “Towards Automatic Extraction of Event and.
`Place Semantics from Flickr Tags”; SIGIR ’07, Jul. 23-27, 2007;
`Amsterdam, The Netherlands; ACM 978-1-59593-597-7/07/0007
`(8 pages).
`Slingsby, A. et al.; “Interactive tag maps and tag clouds for the
`multiscale exploration of large spatio-temporal datasets”; Informa-
`tion Visualization, pp. 497-504; 2007, IV ’07. 11” International
`Conference. ISSN: 1550-6037 (9 pages).
`Torniai et al., “Sharing, Discovering and Browsing Geotagged.
`Pictures on the Web,” 2007, Hewlett-Packard Development Com-
`pany, L.P., pp. 1-18.
`Amundsen, J.; “Using the Geographical Location of Photos in
`Mobile Phones”; Master of Science in Computer Science submis-
`sion; Norwegian University of Science and Technology; Jul. 2008
`(112 pages).
`Hollenstein, L.; “Capturing Vernacular Geography from Georeferenced
`Tags”; Master Thesis; Institute of Geography, University of Zurich;
`Nov. 2008 (139 pages).
`Kopf et al., “Deep photo: model-based photograph enhancement
`and viewing,” ACM Transactions on Graphics, vol. 27, No. 5,
`Article 116, Dec. 2008, ACM (10 pages).
`Miller et al., Give and take: a study of consumer photo-sharing
`culture and practice, CHI 07 Proceedings of the SIGCHI Confer-
`ence on Human Factors in Computing Systems, pp. 347-356, 2008
`(10 pages).
`Bartolini et al., “Integrating Semantic and Visual Facets for Brows-
`ing Digital Photo Collections,” SBED, pp. 65-72, 2009.
`Kisilevich et al., ““Event-based analysis of People’s Activities and.
`Behavior Using Flickr and Panoramio Geotagged Photo Collec-
`tions,” 14” International Conference Information Visualization, pp.
`289-296, 2010 IEEE.
`Gentile, L; “Using Flickr Geotags to Find Similar Tourism Desti-
`nations”; master thesis, 2011; Politecnico di Milano, Dept. of
`Computer Engineering (96 pages).
`Hoffman,A.; “Create Great iPhone Photos: Apps, Tips, Tricks, and.
`Effects”; copyright 2011; ISBN-10: 1-59327-285-5, ISBN-13: 978-
`1-59327-285-2 (216 pages).
`
`Petitioner Apple Inc. - Ex. 1014, p. 5
`
`Petitioner Apple Inc. - Ex. 1014, p. 5
`
`

`

`US 11,017,020 B2
`
`Page 6
`
`(56)
`
`References Cited
`OTHER PUBLICATIONS
`
`Tomasson et al., “PhotoCube: Effective and Efficient Multi-
`Dimensional Browsing of Personal Photo Collections,” ICMR’11,
`2011, ACM.
`Trattner et al., “Evaluating Tag-Based Information Access in Image
`Collections,” Proceedings of the 23’ ACM Conference on Hyper-
`text and Social Media, pp. 113-122, 2012 ACM.
`Kadar,B. et al.; “Where Do Tourists Go? Visualizing and Analysing
`the Spatial Distribution of Geotagged Photography”; Cartographica
`48:2, pp. 78-88; 2013; University of Toronto Press; doi: 10.3138/
`carto.48.2.1839 (11 pages).
`Nutanong, S. et al.; “An Efficient Layout Method for a Large
`Collection of Geographic Data Entries”; Center for Automation
`Research,Institute for Advanced Computer Studies, Dept. of Com-
`puter Science, University of Maryland; pp. 717-720 (4 pages).
`Richard,G.etal.; “Geotagging Photographs for Distribution on the
`Web”; Mineral Physics Institute, Earth and Space Sciences Build-
`ing, Stony Brook University, Stony Brook. NY; date unknown (9
`pages).
`Pogue, D. & Biersdorfer, J. D., “iPhoto ’09: The Missing Manual,”
`O’Reilly Media, Inc. (2009).
`
`* cited by examiner
`
`Petitioner Apple Inc. - Ex. 1014, p. 6
`
`Petitioner Apple Inc. - Ex. 1014, p. 6
`
`

`

`U.S. Patent
`
`May25, 2021
`
`Sheet 1 of 50
`
`US 11,017,020 B2
`
`
`
`FIG. 1
`
`Select None
`Select Aff
`Delete Selected
`
`IMG 123 change
`
`1 File(s) Selected
`File Attributes:
`September 2, 2010 change
`Date:
`Yellowstone, MT, USA change
`Location:
`Christopher Desmond, Nancy Desmond
`People:
`Fly Fishing in Yellowstane
`Event:
`Add
`Comments:
`Add
`Keywords:
`Add
`Album:
`eveve
`Star Ranking:
`Sharing Rights: Add
`Recipe:
`Add
`Document Type: Add
`File Name:
`
`
`
`Petitioner Apple Inc. - Ex. 1014, p. 7
`
`Petitioner Apple Inc. - Ex. 1014, p. 7
`
`

`

`U.S. Patent
`
`May25, 2021
`
`Sheet 2 of 50
`
`US 11,017,020 B2
`
`
`
`FIG. 2
`
`
`
`
`
`
`
`Comments:
`Suzanne and Anthony's Wedding Party where the cousins posed
`for a photo in the grass. Note, Jack with the lollipop and the
`photographer with his shoe in the photo
`
`Location:
`Historical Society
`Lisle, IL 60532
`
`People:
`Jack Wong
`C3 Wong
`Mary Firestone
`Zoe Peika
`Nick Persons
`
`Event: Suzanne & Anthony's Wedding Reception 2010
`
`Camera Details: more
`
`Petitioner Apple Inc. - Ex. 1014, p. 8
`
`Petitioner Apple Inc. - Ex. 1014, p. 8
`
`

`

`U.S. Patent
`
`May25, 2021
`
`Sheet 3 of 50
`
`US 11,017,020 B2
`
`
`
`FIG. 3
`
`Thumbnail | Table
`
`
`
`
`
`Mary Sader and Ciint Firestone
`Wedding October 11, 1945
`
`Capt. Clint Firestone
`1952
`
`Germany Trip 1955
`
`Father’s Day 1966
`
`Christopher wt
`Year 1972
`
`°
`LA Trip 1993
`
`mt
`:
`EIU College
`Graduation 1994
`
`Cairo, Egypt 1996
`
`"Europe
`Backpacking
`1996
`
`
` College Break 1992
` Nancy James and
`
`Christopher Desmond
`Wedding October 17, 1998
`
`.
`vas
`Amazon Fishing Tip
`November 2003
`
`Cousins Party 2040
`
`Fishing in Yellowstone
`Montana September 2010
`
`Petitioner Apple Inc. - Ex. 1014, p. 9
`
`Petitioner Apple Inc. - Ex. 1014, p. 9
`
`

`

`U.S. Patent
`
`May25, 2021
`
`Sheet 4 of 50
`
`US 11,017,020 B2
`
`
`
`FIG. 4
`
`Captions On | Off
`
`Christopher Desmond and Nancy Desmond Wedding — October 17, 19938
`
`Event &8
`
`
`
`Petitioner Apple Inc. - Ex. 1014, p. 10
`
`Petitioner Apple Inc. - Ex. 1014, p. 10
`
`

`

`U.S. Patent
`
`May25, 2021
`
`Sheet 5 of 50
`
`US 11,017,020 B2
`
`FIG. 5
`
`
`
`
`
`
`Petitioner Apple Inc. - Ex. 1014, p. 11
`
`Petitioner Apple Inc. - Ex. 1014, p. 11
`
`

`

`U.S. Patent
`
`May25, 2021
`
`Sheet 6 of 50
`
`US 11,017,020 B2
`
`
`
`FIG. 6
`
`Thumbnail | Table
`
`
`
`
`
`Steve Hre
`
`Suzanne Lucky
`
`Steve Fwah
`
`Ben Sherman
`
`Joe Schmo
`
`Clint Dewitt
`
`Chartotte Kittian
`
`ChrisJeep
`
`Peter LongHorn
`
`Tim Dog
`
` Tim Dog
` oSTNT Mary Virginia
` Suzanne Lucky
`
`Steve Fwah
`
`Ben Sherman
`
`Joe Schma
`
`Mary Virginia
`
`Clint Dewitt
`
`Chartotte Kittian
`
`Petitioner Apple Inc. - Ex. 1014, p. 12
`
`Petitioner Apple Inc. - Ex. 1014, p. 12
`
`

`

`U.S. Patent
`
`May25, 2021
`
`Sheet 7 of 50
`
`US 11,017,020 B2
`
`
`
`FIG. 7
`
`
`
`
`
`Clinton Dewitt Firestone IV
`
`July 12, 1896
`Birth:
`April 29,1971
`Death:
`Parents: Clinton Dewitt Firestone il and Viola Miller
`Comments: He was a WWIIULS. Air force pilot and POW in WWI and veteran
`honorably discharged in December of 1947. He worked for 44 years for the
`Firestone Tire and Rubber Companyin retail, wholesale and original equipment
`sales, marketing and management. He was born in Akran, OH and is buried in
`Columbiana, OH.
`
`Edit bio
`
`Locations
`
`Timeline
`
`Family Tree
`
`Recipes
`
`
`
`Petitioner Apple Inc. - Ex. 1014, p. 13
`
`Petitioner Apple Inc. - Ex. 1014, p. 13
`
`

`

`U.S. Patent
`
`May25, 2021
`
`Sheet 8 of 50
`
`US 11,017,020 B2
`
`
`
`FIG. 8
`
`
`
`Sees
`
`Descendants of Charlotte Killian Halter
`
`Tree | Table
`
`
`
`Petitioner Apple Inc. - Ex. 1014, p. 14
`
`Petitioner Apple Inc. - Ex. 1014, p. 14
`
`

`

`U.S. Patent
`
`May25, 2021
`
`Sheet 9 of 50
`
`US 11,017,020 B2
`
`
`
`FIG. 9
`
` ne
`
`Montana September 2016
`
`
`
`Petitioner Apple Inc. - Ex. 1014, p. 15
`
`Petitioner Apple Inc. - Ex. 1014, p. 15
`
`

`

`U.S. Patent
`
`May25, 2021
`
`Sheet 10 of 50
`
`US 11,017,020 B2
`
`
`
`FIG. 10
`
`
`
`
`
`Desmond’s Yellow Thai Chicken Curry
`
`Curry Mix
`+ Coconut milk (400 mi} - DO NOT SHAKE IT UP
`+
`800 gram of chicken (4 chicken breast}
`*
`fish sauce (Nam Pla} Thai 8amboo Garden — Bottle
`+ Garlic (2 cloves}
`*
`Broccoli { 2cups chopped}
`*«
`2 Peppers (chepped}
`+
`2 Carrots {chopped}
`
`* Zucchini (chopped)
`*
`Thai Basi! (8 leaves}
`+
`Lemon Grass{in jar} 1 teaspoon
`* Chinese Ginger Root {in jar} 1 teaspoon
`
`Rice
`«
`Thai Rice (something that only takes 2 cups of water}
`*
`Dice chicken in bow! and add two tablespoonsof fish
`sauce. Let marinate for 20 minutes.
`
`*
`
`Take thick part of coconut milk out into pan {about 4
`tablespoons}, Curry paste, 1 spoon of lemon grass, 2
`spoon of ginger and garlic. Heat over high with boil and
`THENstir for 1 minute. Add meat {uncooked} and fry
`unti! cooked over high heat
`* Add milk, brown sugar and salt, Bring back ta slight bolt
`and constantly stir. Add veggies and say sauce. Cook for
`about 10-14 minutes COVERED until veggies are cooked.
`Serve with a smile,
`
`Chef: Barry Desmond
`
`Video on How to Make it
`
`Original Handwritten Recipe
`
`Petitioner Apple Inc. - Ex. 1014, p. 16
`
`Petitioner Apple Inc. - Ex. 1014, p. 16
`
`

`

`U.S. Patent
`
`May25, 2021
`
`Sheet 11 of 50
`
`US 11,017,020 B2
`
`
`
`FIG. 11
`
`4
`
`LackMonk’sArrival|26-Dec-2003|chicagan|69|4|4|
`2
`i
`
`
`MikeTesty'sFistBirthday|13-Sep-1983|Minneapolis,wn|54|21|o__|
`
`Thumbnail | Table
`
`2
`
`
`
`HackMonk’sArival|29-Dec-2003|chicago,|69_|
`
`13-Sep-1983|o
`2
`i
`
`4|a|
`
`jackMonk’sArrival|29-Dec-2008|chicago|69|4|4|
`4
`
`5|oo|13-Sep-1983|MinneapolisMN|sa|2a|
`2
`i
`|2,109
`Cubs Beat Cards 1998
`5-Aug-1998
`Chicago,IL
`2,199
`ack Wrigley Monk's Arrival IL|6|a|aw|29-Dec-2003 Chicago,
`
`
`
`4
`9
`4
`go,
`6
`1
`4
`2
`.
`
`Mike Testy's 4th Birthda 43-Sep-1983_|Minneapolis, MN fsa|on|
`36__| 2,199_|
`
`Cubs Beat Cards 1998
`5-Aug-1998
`Chicago,
`iL
`2,199
`2
`Nancy Learns Howto Ride Bike] 2isu2978_| sttous,mo|76|2|0|
`
`5
`is,
`
`6|o
`o|2
`
`Petitioner Apple Inc. - Ex. 1014, p. 17
`
`Petitioner Apple Inc. - Ex. 1014, p. 17
`
`

`

`U.S. Patent
`
`May25, 2021
`
`Sheet 12 of 50
`
`US 11,017,020 B2
`
`
`
`FIG. 12
`
`4
`
`LackMonk’sArrival|26-Dec-2003|chicagan|69|4|4|
`2
`i
`
`
`MikeTesty'sFistBirthday|13-Sep-1983|Minneapolis,wn|54|21|o__|
`
`Thumbnail | Table
`
`2
`
`
`
`HackMonk’sArival|29-Dec-2003|chicago,|69_|
`
`13-Sep-1983|o
`2
`i
`
`4|a|
`
`jackMonk’sArrival|29-Dec-2008|chicago|69|4|4|
`4
`
`5|oo|13-Sep-1983|MinneapolisMN|sa|2a|
`2
`i
`|2,109
`Cubs Beat Cards 1998
`5-Aug-1998
`Chicago,IL
`2,199
`ack Wrigley Monk's Arrival IL|6|a|aw|29-Dec-2003 Chicago,
`
`
`
`4
`9
`4
`go,
`6
`1
`4
`2
`.
`
`Mike Testy's 4th Birthda 43-Sep-1983_|Minneapolis, MN fsa|on|
`36__| 2,199_|
`
`Cubs Beat Cards 1998
`5-Aug-1998
`Chicago,
`iL
`2,199
`2
`Nancy Learns Howto Ride Bike] 2isu2978_| sttous,mo|76|2|0|
`
`5
`is,
`
`6|o
`o|2
`
`Petitioner Apple Inc. - Ex. 1014, p. 18
`
`Petitioner Apple Inc. - Ex. 1014, p. 18
`
`

`

`U.S. Patent
`
`May25, 2021
`
`Sheet 13 of 50
`
`US 11,017,020 B2
`
`
`
`FIG. 13
`
`Thumbnail | Table
`
`pabers|eT|
`aSe
`beadedltCt|
`peacon|lt|
`pees|TtTt
`pfiffi
`sDanas
`+ Danas
`+ Danes
`
`
`
`
`
`Petitioner Apple Inc. - Ex. 1014, p. 19
`
`Petitioner Apple Inc. - Ex. 1014, p. 19
`
`

`

`U.S. Patent
`
`May25, 2021
`
`Sheet 14 of 50
`
`US 11,017,020 B2
`
`FIG. 14
`
`Thumbnail | Table
`
`
`
`
`
`
`JAibertsomn|cousin=~||co||
`
`[Kitatack|son|?||
`
`[Kiian,Brin|Nephew|os||
`
`[Kitiankevin|Nephew|es|oo|
`
`[Kitian,Saran|aughterinfow|7[|1|0
`
`[KilianMark|GreatNephew|s_|
`
`
`[Kitiaytovis|Greatrandson|7|3|2|
`Grandson
`14
`
`EE monk cs
`5
`Monk, Jack
`
`7
`3
`9
`
`
`
`[MooreBertha|Greatniece|a|6|
`[siythesarsh|sister=||
`
`
`
`3 1
`
`2
`
`Killian, Mike
`
`Brother
`
`Petitioner Apple Inc. - Ex. 1014, p. 20
`
`Petitioner Apple Inc. - Ex. 1014, p. 20
`
`

`

`U.S. Patent
`
`May25, 2021
`
`Sheet 15 of 50
`
`US 11,017,020 B2
`
`
`
`FIG. 15
`
`Thumbnail | Table
`
`
`
`rae]245ong]Srgapore|SegapowLn2Lo
`
`pom||cotogne||ermary|3|2|oo
`Lucilaobero|montana|[katy|os|a|
`listeome|aaawestst_|tsie|u|usa|a5|a2|2|
`cotege|sasmarket_|abron|on|usa|6a|2|oo|
`
`Amazontrip||Manus||brat]3s|os|
`cabin|g99pine[rakeGeneval_wi|usa|os|2|o|
`radschoot__|goariymouth|charteston||usa|azs6|az|a
`
`rtfithPark|298Glencarin|tostetiz|ca|usa|a2|o|o|
`LAtquestrianctr]s6storsedr|Glendale|ca[|usa|4[|4|o|
`
`DeiCoronado|12.Coronadoor|Coronado|_ca|usa|321|4|oo
`enwayPark|123Yawke|_soston_|_ma_|
`
`lWrigieyField[1190wAddison|chicago|nu_|
`Hone]aitainanyuhes|u|usa|uae?[as
`
`ecitrary|s2zsive|een{ox}sa|as}o_]Pike's
`
`market__|786Market_|seatte|wa|usa|as|a|oo
`
`Petitioner Apple Inc. - Ex. 1014, p. 21
`
`Petitioner Apple Inc. - Ex. 1014, p. 21
`
`

`

`U.S. Patent
`
`May25, 2021
`
`Sheet 16 of 50
`
`US 11,017,020 B2
`
`
`
`FIG. 16
`
`Category | Card | Table
`
`
`
`
`
`Petitioner Apple Inc. - Ex. 1014, p. 22
`
`Petitioner Apple Inc. - Ex. 1014, p. 22
`
`

`

`U.S. Patent
`
`May25, 2021
`
`Sheet 17 of 50
`
`US 11,017,020 B2
`
`
`
`FIG. 17
`
`
`
`Close {x}
`
`
`
`Petitioner Apple Inc. - Ex. 1014, p. 23
`
`Petitioner Apple Inc. - Ex. 1014, p. 23
`
`

`

`U.S. Patent
`
`May25, 2021
`
`Sheet 18 of 50
`
`US 11,017,020 B2
`
`FIG. 18
`
`Advanced Search Filter
`
`Date:
`Location:
`
`People:
`
`Event:
`
`Add
`Add
`
`Chicago, IL USA
`Yellowstone, MT USA
`Cologne, Germany
`
`Mike Cubbie
`Mary Lamb
`Christopher Monk
`Nancy Monk
`Dwight Schrut
`
` Keywords:
`
`
`
`Jack’s 1° Birthday
`Fly Fishing in Yellowstone
`Raking Leaves
`Christmas 201