I 1111111111111111
`
`1111111111 111111111111111 111111111111111 111111111111111111
`
`
`
`
`USO 10621228B2
`
`c12) United States Patent
`Desmond et al.
`
`(IO) Patent No.: US 10,621,228 B2
`* Apr. 14, 2020
`(45) Date of Patent:
`
`(54) METHOD AND APPARATUS FOR
`MANAGING DIGITAL FILES
`
`(71) Applicant: NCM IP Holdings, LLC, Glen Ellyn,
`IL (US)
`
`(72)
`
`Inventors: Christopher J. Desmond, Glen Ellyn,
`IL (US); Nancy L. Desmond, Glen
`Ellyn, IL (US); L. Michael Taylor,
`Chicago, IL (US)
`
`(73) Assignee: NCM IP Holdings, LLC, Glen Ellyn,
`IL (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by O days.
`
`This patent is subject to a terminal dis(cid:173)
`claimer.
`
`(21) Appl. No.: 16/578,238
`
`(22) Filed:
`
`Sep. 20, 2019
`
`(65)
`
`Prior Publication Data
`
`US 2020/0026727 Al
`
`Jan. 23, 2020
`
`(63)
`
`(51)
`
`(52)
`
`(58)
`
`Related U.S. Application Data
`
`Continuation of application No. 16/536,300, filed on
`Aug. 8, 2019, which is a continuation of application
`(Continued)
`
`Int. Cl.
`G06F 16158
`G06F 161901
`
`(2019.01)
`(2019.01)
`(Continued)
`
`U.S. Cl.
`CPC .......... G06F 1615866 (2019.01); G06F 16151
`(2019.01); G06F 161901 (2019.01); G06F
`161907 (2019.01); G06F 3/0481 (2013.01)
`Field of Classification Search
`CPC ... G06F 16/5866; G06F 16/907; G06F 16/901
`See application file for complete search history.
`
`Select None
`Select All
`Delete Selected
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,694,514 A
`6,629,104 Bl
`
`12/1997 Evans
`9/2003 Parulski
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`CN
`EP
`
`10/2015
`102591922 B
`2/2017
`2466869 A3
`(Continued)
`
`OTHER PUBLICATIONS
`
`Kustanowitz et al., "Motivating Annotation for Personal Digital
`Photo Libraries: Lowering Barriers while Raising Incentives," Tech.
`Report HCIL-2004-18, U. Maryland, 2005 (10 pages).
`(Continued)
`
`Primary Examiner - Loe Tran
`(74) Attorney, Agent, or Firm - Nixon Peabody LLP
`
`(57)
`
`ABSTRACT
`
`A computer-implemented method of associating digital tags
`with digital files comprises storing a plurality of digital files
`having embedded therein content data and metadata includ(cid:173)
`ing tags; receiving, via a user interface device of a client
`device, a first tag label containing alphanumeric text created
`and inputted by a user of the client device; modifying, using
`a controller device, a selected first one of the tags of the
`metadata in a first of the digital files to include the first tag
`label; receiving, via the user interface device or another user
`interface device, an instruction to search for all of the digital
`files having at least the first tag label; responsive to receiving
`the instruction, automatically searching for all of the digital
`files having at least the first tag label; and displaying, on a
`video display device associated with the client device, a first
`indication of the first tag label.
`
`19 Claims, 50 Drawing Sheets
`
`September 2, 2010 ~
`Yellowstone, MT, USA~
`Christopher Desmond, Nancy Desmond
`Fly Fishing in Yellowstone
`Add
`Add
`Add
`
`1 F1le(s)Selected
`File Attributes:
`Date:
`Location:
`People:
`Event:
`Comments:
`Keywords:
`Album:
`Star Ranking:
`Sharing Rights: Add
`Recipe:
`Add
`Document Type: Add
`File Name:
`IMG 123 ~
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 1 of 73
`
`

`

`US 10,621,228 B2
`Page 2
`
`Related U.S. Application Data
`
`No. 15/375,927, filed on Dec. 12, 2016, now Pat. No.
`10,423,658, which is a continuation of application
`No. 14/193,426, filed on Feb. 28, 2014, now Pat. No.
`9,552,376, which is a continuation-in-part of appli-
`cation No. 13/157,214, filed on Jun. 9, 2011, now Pat.
`No. 9,098,531.
`
`(51)
`
`Int. Cl.
`G06F 16151
`G06F 161907
`G06F 3/0481
`
`(2019.01)
`(2019.01)
`(2013.01)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`7,372,976 B2
`7,475,060 B2
`7,480,669 B2
`7,694,236 B2
`7,822,746 B2
`7,860,846 B2
`7,991,283 B2
`8,001,124 B2
`8,015,144 B2
`8,024,317 B2
`8,032,508 B2
`8,055,675 B2
`8,060,492 B2
`8,069,142 B2
`8,086,048 B2
`8,108,778 B2
`8,150,844 B2
`8,150,967 B2
`8,166,016 B2
`8,166,168 B2
`8,171,388 B2
`8,230,338 B2
`8,255,379 B2
`8,264,570 B2
`8,271,506 B2
`8,281,027 B2
`8,285,483 B2
`8,307,029 B2
`8,315,959 B2
`8,359,314 B2
`8,364,611 B2
`8,380,039 B2
`8,386,506 B2
`8,390,702 B2
`8,401,771 B2
`8,402,356 B2
`8,429,156 B2
`8,447,120 B2
`8,452,855 B2
`8,458,115 B2
`8,463,931 B2
`8,484,223 B2
`8,489,115 B2
`8,490,011 B2
`8,493,495 B2
`8,503,791 B2
`8,504,073 B2
`8,520,979 B2
`D689,079 S
`D689,080 S
`D689,083 S
`D689,084 S
`D689,085 S
`8,538,811 B2
`8,538,813 B2
`8,542,294 B2
`8,554,623 B2
`8,560,390 B2
`8,560,517 B2
`
`5/2008 Rhoad
`1/2009 Toyama
`1/2009 Lo
`4/2010 Gusmorino
`10/2010 Svendsen
`12/2010 Takahashi
`8/2011 Chen
`8/2011 Svendsen
`9/2011 Zheng
`9/2011 Nair
`10/2011 Martinez
`11/2011 Higgins
`11/2011 Nair
`11/2011 Davis
`12/2011 Naaman
`1/2012 Athsani
`4/2012 Redstone
`4/2012 King
`4/2012 Higgins
`4/2012 Hayashi
`5/2012 Zaltzman
`7/2012 Dugan
`8/2012 Govindachetty
`9/2012 Karimoto
`9/2012 Martinez
`10/2012 Martinez
`10/2012 Amer-Yahia
`11/2012 Davis
`11/2012 Zheng
`1/2013 Svendsen
`1/2013 Tendjoukian
`2/2013 Luo
`2/2013 Martinez
`3/2013 Bhatt
`3/2013 Krumm
`3/2013 Martinez
`4/2013 Buchmueller
`5/2013 Ji
`5/2013 Higgins
`6/2013 Cai
`6/2013 Evans
`7/2013 Ota
`7/2013 Rodriguez
`7/2013 Stapleton
`7/2013 D'Souza
`8/2013 Conwell
`8/2013 Svendsen
`8/2013 Conwell
`9/2013 Edwards
`9/2013 Edwards
`9/2013 Pasceri
`9/2013 Pasceri
`9/2013 Pasceri
`9/2013 Higgins
`9/2013 Kakarla
`9/2013 Bhatt
`10/2013 Higgins
`10/2013 Higgins
`10/2013 Yang
`
`8,583,620 B2
`8,583,668 B2
`8,589,389 B2
`8,589,486 B2
`8,594,702 B2
`8,606,021 B2
`8,626,699 B2
`8,671,154 B2
`8,676,001 B2
`8,706,406 B2
`8,745,133 B2
`8,762,285 B2
`D708,196 S
`D708,197 S
`D708,198 S
`8,769,099 B2
`8,769,393 Bl
`8,799,371 B2
`8,805,165 B2
`8,806,365 B2
`8,810,597 B2
`8,813,107 B2
`8,825,472 B2
`8,831,352 B2
`8,849,854 B2
`8,849,909 B2
`D715,819 S
`8,880,568 B2
`8,890,888 B2
`8,892,495 B2
`8,914,342 B2
`8,923,889 B2
`8,930,848 B2
`8,949,212 Bl
`8,954,425 B2
`8,966,121 B2
`8,972,177 B2
`8,998,422 Bl
`9,009,177 B2
`9,014,511 B2
`9,015,617 B2
`9,015,633 B2
`9,020,247 B2
`9,031,953 B2
`9,032,320 B2
`9,055,037 B2
`9,063,226 B2
`9,076,259 B2
`9,092,409 B2
`9,098,545 B2
`9,104,729 B2
`9,104,915 B2
`9,110,903 B2
`9,151,618 B2
`9,158,794 B2
`9,160,802 B2
`9,172,666 B2
`9,202,200 B2
`9,218,328 B2
`9,224,172 B2
`9,235,766 B2
`9,239,848 B2
`9,245,041 B2
`9,261,376 B2
`D751,597 S
`9,311,396 B2
`9,336,240 B2
`9,372,931 B2
`9,390,104 B2
`9,405,981 B2
`9,418,485 B2
`9,424,595 B2
`9,460,116 B2
`9,462,054 B2
`9,465,513 B2
`9,471,834 Bl
`9,483,500 B2
`9,501,577 B2
`9,507,778 B2
`
`11/2013 Govindachetty
`11/2013 Higgins
`11/2013 Bisdikian
`11/2013 Martinez
`11/2013 Naaman
`12/2013 Conwell
`1/2014 Xie
`3/2014 Davis
`3/2014 Brucher
`4/2014 Kalaboukis
`6/2014 Martinez
`6/2014 Davis
`7/2014 Pasceri
`7/2014 Pasceri
`7/2014 Pasceri
`7/2014 Kalaboukis
`7/2014 Abhyanker
`8/2014 Davis
`8/2014 Luo
`8/2014 Stapleton
`8/2014 Akiya
`8/2014 Higgins
`9/2014 Raghuveer
`9/2014 Gao
`9/2014 Kakarla
`9/2014 Farmer
`10/2014 Pasceri
`11/2014 Perczynski
`11/2014 Lee
`11/2014 Hollberg
`12/2014 Kalaboukis
`12/2014 Svendsen
`1/2015 Lim
`2/2015 Dhandapani
`2/2015 Xiao
`2/2015 Josefsberg
`3/2015 Zheng
`4/2015 Snavely
`4/2015 Zheng
`4/2015 Brucher
`4/2015 Stapleton
`4/2015 Takarnura
`4/2015 Adam
`5/2015 Rathnavelu
`5/2015 Crawford
`6/2015 Evans
`6/2015 Zheng
`7/2015 Hourie
`7/2015 Charaniya
`8/2015 Abhyanker
`8/2015 Dong
`8/2015 Conwell
`8/2015 Martinez
`10/2015 Amer-Yahia
`10/2015 Higgins
`10/2015 Svendsen
`10/2015 Murdock
`12/2015 Stibel
`12/2015 Stapleton
`12/2015 Churchill
`1/2016 Li
`1/2016 Liu
`1/2016 Pilskalns
`2/2016 Zheng
`3/2016 Pasceri
`4/2016 Meadow
`5/2016 Bhatt
`6/2016 Capt
`7/2016 Thomee
`8/2016 Li
`8/2016 Lindberg
`8/2016 Svendsen
`10/2016 Pilskalns
`10/2016 Poletto
`10/2016 Sims
`10/2016 Filip
`11/2016 Brucher
`11/2016 Zheng
`11/2016 Jaffe
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 2 of 73
`
`

`

`US 10,621,228 B2
`Page 3
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2009/0288005 Al
`2009/0290812 Al
`2009/0307618 Al
`2009/0325602 Al
`2010/0041419 Al
`2010/0046842 Al
`2010/0053371 Al
`2010/0064239 Al
`2010/0082427 Al
`2010/0107125 Al
`2010/0153348 Al
`2010/0162411 Al
`2010/0171763 Al
`2010/0182341 Al
`2010/0185509 Al
`2010/0241689 Al
`2010/0241944 Al
`2010/0268717 Al
`2010/0268766 Al
`2010/0280913 Al
`2010/0293035 Al
`2010/0293193 Al
`2011/0040779 Al
`2011/0093458 Al
`2011/0109769 Al
`2011/0113064 Al
`2011/0145258 Al
`2011/0191014 Al
`2011/0191253 Al
`2011/0202267 Al
`2011/0208426 Al
`2011/0289031 Al
`2011/0301832 Al
`2011/0314016 Al
`2012/0114249 Al
`2012/0158755 Al
`2012/0218150 Al
`2012/0220311 Al
`2012/0251011 Al
`2012/0266090 Al
`2012/0278171 Al
`2012/0278767 Al
`2012/0329441 Al
`2012/0331091 Al
`2013/0018881 Al
`2013/0036165 Al
`2013/0063613 Al
`2013/0073202 Al
`2013/0101157 Al
`2013/0138685 Al
`2013/0141612 Al
`2013/0151597 Al
`2013/0202198 Al
`2013/0275536 Al
`2013/0339440 Al
`2014/0059477 Al
`2014/0071272 Al
`2014/0089811 Al
`2014/0101531 Al
`2014/0101601 Al
`2014/0143247 Al
`2014/0149036 Al
`2014/0181089 Al
`2014/0188880 Al
`2014/0193087 Al
`2014/0354628 Al
`2015/0039630 Al
`2015/0070165 Al
`2015/0070397 Al
`2015/0116540 Al
`2015/0117713 Al
`2015/0156247 Al
`2015/0186389 Al
`2015/0213057 Al
`2015/0213329 Al
`2015/0244794 Al
`2015/0244833 Al
`2015/0358224 Al
`2016/0092741 Al
`
`11/2009 Stapleton
`11/2009 Naaman
`12/2009 Lawler
`12/2009 Higgins
`2/2010 Svendsen
`2/2010 Conwell
`3/2010 Karimoto
`3/2010 Crawford
`4/2010 Burgener
`4/2010 Ockene
`6/2010 Perczynski
`6/2010 Chang
`7/2010 Bhatt
`7/2010 Lee
`7/2010 Higgins
`9/2010 Davis
`9/2010 Athsani
`10/2010 Pilskalns
`10/2010 Bouget
`11/2010 O'Sullivan
`11/2010 Athsani
`11/2010 Harrison
`2/2011 Svendsen
`4/2011 Zheng
`5/2011 Bhatt
`5/2011 Govindachetty
`6/2011 Kankainen
`8/2011 Feng
`8/2011 Pilskalns
`8/2011 Amer-Yahia
`8/2011 Zheng
`11/2011 Zheng
`12/2011 Zheng
`12/2011 Svendsen
`5/2012 Conwell
`6/2012 Gammill
`8/2012 Oyabu
`8/2012 Rodriguez
`10/2012 Gao
`10/2012 Nealer
`11/2012 Tang
`11/2012 Stibel
`12/2012 Tseng
`12/2012 Tseng
`1/2013 Bhatt
`2/2013 Tseng
`3/2013 Conwell
`3/2013 Zheng
`4/2013 Li
`5/2013 Brucher
`6/2013 Bhatt
`6/2013 Akiya
`8/2013 Adam
`10/2013 Murdock
`12/2013 Balassanian
`2/2014 Wong
`3/2014 Rodriguez
`3/2014 Meadow
`4/2014 Capt
`4/2014 Tang
`5/2014 Rathnavelu
`5/2014 Amer-Yahia
`6/2014 Desmond
`7/2014 Abhyanker
`7/2014 Conwell
`12/2014 Lindberg
`2/2015 Thomee
`3/2015 East
`3/2015 Miller
`4/2015 Gilman
`4/2015 Zheng
`6/2015 Hensel
`7/2015 Zheng
`7/2015 Brucher
`7/2015 Adam
`8/2015 Poletto
`8/2015 Grue
`12/2015 Poletto
`3/2016 Li
`
`9,519,682 Bl
`9,535,563 B2
`9,536,146 B2
`9,552,376 B2
`9,557,162 B2
`9,576,253 B2
`9,582,546 B2
`9,593,957 B2
`9,600,484 B2
`9,626,685 B2
`9,646,025 B2
`9,654,570 B2
`9,674,650 B2
`9,679,456 B2
`9,680,929 B2
`9,683,858 B2
`9,691,073 B2
`9,706,345 B2
`9,710,961 B2
`9,721,188 B2
`9,754,226 B2
`9,772,745 B2
`9,787,799 B2
`9,805,123 B2
`9,811,879 B2
`9,858,348 Bl
`9,870,572 B2
`9,881,179 B2
`9,882,994 B2
`9,942,121 B2
`10,019,850 B2
`10,037,327 B2
`10,068,178 B2
`10,074,093 B2
`10,083,533 B2
`10,110,541 B2
`10,187,543 B2
`10,223,701 B2
`10,230,803 B2
`10,235,444 B2
`10,242,051 B2
`10,282,752 B2
`10,288,433 B2
`10,289,643 B2
`10,303,975 B2
`10,311,611 B2
`10,318,110 B2
`10,331,863 B2
`10,360,352 B2
`10,430,452 B2
`10,445,346 B2
`2005/0060299 Al
`2006/0165380 Al*
`
`2007/0118508 Al
`2007 /0282908 Al
`2008/0040034 Al
`2008/0051994 Al
`2008/0148175 Al
`2008/0201302 Al
`2008/0250398 Al
`2009/0013041 Al
`2009/0019085 Al
`2009/0049408 Al
`2009/0106705 Al
`2009/0113350 Al*
`
`2009/0132689 Al
`2009/0132941 Al
`2009/0135438 Al
`2009/0216704 Al
`2009/0222302 Al
`2009/0254867 Al
`2009/0265631 Al
`2009/0279794 Al
`
`12/2016 Pujara
`1/2017 Hollberg
`1/2017 Zheng
`1/2017 Desmond
`1/2017 Rodriguez
`2/2017 Zaltzman
`2/2017 Hartford
`3/2017 Zheng
`3/2017 Davis
`4/2017 Martinez
`5/2017 Boyns
`5/2017 Bisdikian
`6/2017 Hartford
`6/2017 East
`6/2017 Tseng
`6/2017 Zheng
`6/2017 Tseng
`7/2017 Davis
`7/2017 Setlur
`8/2017 Adam
`9/2017 Zheng
`9/2017 Hasenei
`10/2017 Grue
`10/2017 Nair
`11/2017 Miller
`1/2018 Higgins
`1/2018 Chapin
`1/2018 Patton
`1/2018 Bisdikian
`4/2018 Poletto
`7/2018 Lindberg
`7/2018 Thomee
`9/2018 van Zwol
`9/2018 Higgins
`9/2018 Bhatt
`10/2018 Li
`1/2019 Lahcanski
`3/2019 King
`3/2019 Higgins
`3/2019 Poletto
`3/2019 Shinn
`5/2019 Athsani
`5/2019 Zheng
`5/2019 Brucher
`5/2019 Adam
`6/2019 Stoop
`6/2019 Naaman
`6/2019 Patton
`7/2019 Patton
`10/2019 Ross
`10/2019 Govindachetty
`3/2005 Filley
`7/2006 Tanaka.
`
`GllB 27/34
`386/227
`
`5/2007 Svendsen
`12/2007 Van Der Meulen
`2/2008 Kanno
`2/2008 Fisher
`6/2008 Naaman
`8/2008 Kimchi
`10/2008 Takahashi
`1/2009 Farmer
`1/2009 Abhyanker
`2/2009 Naaman
`4/2009 Takarnura
`4/2009 Hibino .................... G06F 16/41
`715/853
`
`5/2009 Zaltzman
`5/2009 Pilskalns
`5/2009 Chopra
`8/2009 Zheng
`9/2009 Higgins
`10/2009 Farouki
`10/2009 Sigurbjornsson
`11/2009 Brucher
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 3 of 73
`
`

`

`US 10,621,228 B2
`Page 4
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2016/0162512 Al
`2016/0247307 Al
`2016/0253358 Al
`2016/0314187 Al
`2016/0321269 Al
`2016/0328444 Al
`2016/0344888 Al
`2017/0024415 Al
`2017/0103081 Al
`
`6/2016 Battistini
`8/2016 Stoop
`9/2016 Bhatt
`10/2016 Poletto
`11/2016 Thomee
`11/2016 Shinn
`11/2016 Lahcanski
`1/2017 Brucher
`4/2017 Jones
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`WO
`WO
`WO
`
`2410414 Bl
`WO 2011/070225 Al
`WO 2013/019376 Al
`WO 2013/099704 Al
`
`10/2019
`6/2011
`2/2013
`7/2013
`
`OTHER PUBLICATIONS
`
`Miller et al., "Give and take: a study of consumer photo-sharing
`culture and practice," CHI '07 Proceedings of the SIGCHI Confer(cid:173)
`ence on Human Factors in Computing Systems, pp. 347-356, 2007
`(10 pages).
`Ames et al., "Why we tag: motivations for annotation in mobile and
`online media," CHI '07 Proceedings of the SIGCHI Conference on
`Human Factors in Computing Systems, pp. 971-980, ACM, 2007
`(10 pages)l0083533.
`Yee et al., "Faceted Metadata for Image Search and Browsing", CHI
`2003, pp. 401-408, 2003, ACM.
`Ferre, "CAMELIS: Organizing and Browsing a Personal Photo
`Collection with a Logical Information System", Int. Conf. Concept
`Lattices and Their Applications, pp. 112-123, 2007, HAL.
`Tomasson et al., "PhotoCube: Effective and Efficient Multi(cid:173)
`Dimensional Browsing of Personal Photo Collections", ICMR '11,
`2011, ACM.
`Bartolini et al., "Integrating Semantic and Visual Facets for Brows(cid:173)
`ing Digital Photo Collections", SBED, pp. 65-72, 2009.
`Trattner et al., "Evaluating Tag-Based Information Access in Image
`Collections", Proceedings of the 23 rd ACM Conference on Hyper(cid:173)
`text and Social Media, pp. 113-122, 2012 ACM.
`Kang et al., "Capture, Annotate, Browse, Find, Share: Novel
`Interfaces for Personal Photo Management", International Journal
`of Human-Computer Interaction, 23(3), pp. 315-337, 2007, Law(cid:173)
`rence Eribaum Associates, Inc.
`Jaffe et al., "Generating Summaries and Visualization for Large
`Collections of GeoReferenced Photographs", MIR'06, pp. 89-98,
`2006 ACM.
`Torniai et al., "Sharing, Discovering and Browsing Geotagged
`Pictures on the Web", 2007, Hewlett-Packard Development Com(cid:173)
`pany, L.P., pp. 1-18 (19 pages).
`Snavely et al., "Photo Tourism: Exploring Photo Collection in 3D",
`SIGGRAPH '06 ACM Transactions on Graphics, vol. 25, Issue 3,
`pp. 835-846, 2006 ACM.
`
`Kisilevich et al., "Event-based analysis of People's Activities and
`Behavior Using Flickr and Panoramic Geotagged Photo Collec(cid:173)
`tions", 14th International Conference Information Visualization, pp.
`289-296, 2010 IEEE.
`Ahern et al., "World Explorer: Visualizing Aggregate Data From
`Unstructured Text in Geo-Referenced Collections", JCDL'07, pp.
`1-10, 2007, ACM.
`Kopf et al., "Deep photo: model-based photograph enhancement
`and viewing", ACM Transactions on Graphics, vol. 27, No. 5,
`Article 116, Dec. 2008, ACM (10 pages).
`Amundsen, J.; "Using the Geographical Location of Photos in
`Mobile Phones"; Master of Science in Computer Science submis(cid:173)
`sion; Norwegian University of Science and Technology; Jul. 2008
`(112 pages).
`Gentile, L; "Using Flickr Geotags to Find Similar Tourism Desti(cid:173)
`nations"; master thesis, 2011; Politecnico di Milano, Dept. of
`Computer Engineering (96 pages).
`Hollenstein, L.; "Capturing Vernacular Geography from Georeferenced
`Tags"; Master Thesis; Institute of Geography, University of Zurich;
`Nov. 2008 (139 pages).
`Nutanong, S. et al.; "An Efficient Layout Method for a Large
`Collection of Geographic Data Entries"; Center for Automation
`Research, Institute for Advanced Computer Studies, Dept. of Com(cid:173)
`puter Science, University of Maryland; pp. 717-720 ( 4 pages).
`Slingsby, A. et al.; "Interactive tag maps and tag clouds for the
`multiscale exploration of large spatio-temporal datasets"; Informa(cid:173)
`tion Visualization, pp. 497-504; 2007, IV '07. 11th International
`Conference. ISSN: 1550-6037 (9 pages).
`Hoffman, A.; "Create Great iPhone Photos: Apps, Tips, Tricks, and
`Effects"; copyright 2011; ISBN-10: 1-59327-285-5, ISBN-13: 978-
`1-59327-285-2 (216 pages).
`Chen, Y-F. et al.; "GeoTracker: Geospatial and Temporal RSS
`Navigation"; AT&T Labs-Research,
`Florham Park, NJ; WWW
`2007 I Track: Browsers and User Interfaces, Session: Smarter
`Browsing, pp. 41-50 (10 pages).
`Goodman, E.; "Destination Services: Tourist media and networked
`places"; School of Information, UC Berkeley; Mar. 2, 2007 (11
`pages).
`Rattenbury, T. et al.; "Towards Automatic Extraction of Event and
`Place Semantics from Flickr Tags"; SIGIR '07, Jul. 23-27, 2007;
`Amsterdam, The Netherlands; ACM 978-l-59593-597-7/07/0007
`(8 pages).
`Kadar, B. et al.; "Where Do Tourists Go? Visualizing and Analysing
`the Spatial Distribution of Geotagged Photography"; Cartographica
`48:2, pp. 78-88; 2013; University of Toronto Press; doi: 10.3138/
`carto.48.2.1839 (11 pages).
`Kennedy, L. et al.; "How Flickr Helps Us Make Sense of the World:
`Context and Content in Community-Contributed Media Collec(cid:173)
`tions"; MM '07, Sep. 23-28, 2007; Augsburg, Bavaria, Germany;
`Copyright 2007; ACM 978-l-59593-701-8/07/0009
`(10 pages).
`Richard, G. et al.; "Geotagging Photographs for Distribution on the
`Web"; Mineral Physics Institute, Earth and Space Sciences Build(cid:173)
`ing, Stony Brook University, Stony Brook. NY; date unknown (9
`pages).
`
`* cited by examiner
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 4 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 1 of 50
`
`US 10,621,228 B2
`
`FIG. 1
`
`Select None
`Select All
`Delete Selected
`
`September 2, 2010 change
`Yellowstone, MT, USA change
`Christopher Desmond, Nancy Desmond
`Fly Fishing in Yellowstone
`Add
`Add
`Add
`
`1 File(s) Selected
`File Attributes:
`Date:
`Location:
`People:
`Event:
`Comments:
`Keywords:
`Album:
`Star Ranking:
`Add
`Sharing Rights:
`Add
`Recipe:
`Document Type: Add
`File Name:
`IMG 123 change
`
`•••••
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 5 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 2 of 50
`
`US 10,621,228 B2
`
`FIG. 2
`
`Comments:
`Suzanne and Anthony's Wedding Party where the cousins posed
`for a photo in the grass. Note, Jack with the lollipop and the
`photographer with his shoe in the photo
`
`Location:
`Historical Society
`Lisle, IL 60532
`
`People:
`Jack Wong
`CJ Wong
`Mary Firestone
`Zoe Peika
`Nick Persons
`
`Event: Suzanne & Anthony's Wedding Reception 2010
`
`Camera Details: more
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 6 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 3 of 50
`
`US 10,621,228 B2
`
`FIG. 3
`
`Thumbnail I Table
`
`Mary Bader and Clint Firestone
`Wedding October 11, 1945
`
`Capt. Clint Firestone
`1952
`
`Germany Trip 1955
`
`Father's Day 1966
`
`Christopher pt
`Year 1972
`
`College Break 1992
`
`LA Trip 1993
`
`EIU College
`Graduation 1994
`
`Cairo, Egypt 1996
`
`Europe
`Backpacking
`1996
`
`\;:)rn)w]
`
`Nancy James and
`Christopher Desmond
`Wedding October 17, 1998
`
`Amazon Fishing Trip
`November 2009
`
`Cousins Party 2010
`
`Fishing in Yellowstone
`Montana September 2010
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 7 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 4 of 50
`
`US 10,621,228 B2
`
`FIG. 4
`
`Christopher Desmond and Nancy Desmond Wedding - October 17, 1998
`
`Event f}
`
`Captions On I Off
`
`◄
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 8 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 5 of 50
`
`US 10,621,228 B2
`
`FIG. 5
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 9 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 6 of 50
`
`US 10,621,228 B2
`
`FIG. 6
`
`Thumbnail I Table
`
`Clint Firestone
`
`Tim Dog
`
`Steve Fire
`
`Suzanne Lucky
`
`Steve Fwah
`
`Ben Sherman
`
`Joe Schmo
`
`◄
`
`►
`
`Mary Virginia
`
`Clint Dewitt
`
`Charlotte Killian
`
`Chris Jeep
`
`Peter LongHorn
`
`Tim Dog
`
`Steve Fire
`
`Suzanne Lucky
`
`Steve Fwah
`
`Ben Sherman
`
`Joe Schmo
`
`Mary Virginia
`
`Clint Dewitt
`
`Charlotte Killian
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 10 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 7 of 50
`
`US 10,621,228 B2
`
`FIG. 7
`
`Clinton Dewitt Firestone IV
`
`July 12, 1896
`Birth:
`April 29, 1971
`Death:
`Parents: Clinton Dewitt Firestone Ill and Viola Miller
`Comments: He was a WWII U.S. Air force pilot and POW in WWII and veteran
`honorably discharged in December of 1947. He worked for 44 years for the
`Firestone Tire and Rubber Company in retail, wholesale and original equipment
`sales, marketing and management. He was born in Akron, OH and is buried in
`Columbiana, OH.
`
`Edit bio
`
`Family Tree
`
`Recipes
`
`ftti,,~
`
`..................................................................................
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 11 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 8 of 50
`
`US 10,621,228 B2
`
`FIG. 8
`
`Descendants of Charlotte Killian Halter
`
`Tree I Table
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 12 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 9 of 50
`
`US 10,621,228 B2
`
`FIG. 9
`
`Year
`
`Month
`
`Day
`
`.._f ___________
`
`
`
`.....__'"-'"-'"-'"-"-'"-'"-'"-'"-'"-'"-'"-"-'"-'"-'"-'"-'"-'"--M,...._ ______
`
`__.
`
`Cousins Party 2010
`
`Fishing in Yellowstone
`Montana September 2010
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 13 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 10 of 50
`
`US 10,621,228 B2
`
`FIG. 10
`
`1 Zucchini (chopped)
`Thai Basil (8 leaves)
`Lemon Grass (in jar) 1 teaspoon
`Chinese Ginger Root (in jar) 1 teaspoon
`
`Rice
`Thai Rice (something that only takes 2 cups of water)
`Dice chicken in bowl and add two tablespoons of fish
`sauce. Let marinate for 20 minutes.
`Take thick part of coconut milk out into pan (about 4
`tablespoons), Curry paste, 1 spoon of lemon grass, 1
`spoon of ginger and garlic. Heat over high with boil and
`THEN stir for 1 minute. Add meat (uncooked) and fry
`until cooked over high heat
`Add milk, brown sugar and salt. Bring back to slight boil
`and constantly stir. Add veggies and soy sauce. Cook for
`about 10-14 minutes COVERED until veggies are cooked.
`Serve with a smile.
`
`Desmond's Yellow Thai Chicken Curry
`
`Curry Mix
`Coconut milk (400 ml)- DO NOT SHAKE IT UP
`800 gram of chicken (4 chicken breast)
`Fish sauce (Nam Pia) Thai Bamboo Garden - Bottle
`Garlic (2 cloves)
`Broccoli ( 2cups chopped)
`2 Peppers (chopped)
`2 Carrots (chopped)
`
`Chef: Barry Desmond
`
`Video on How to Make It
`
`Original Handwritten Recipe
`
`:r:::::.·•
`
`k(~~i . .:
`
`.....
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 14 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 11 of 50
`
`US 10,621,228 B2
`
`FIG. 11
`
`Thumbnail I Table
`
`Album/Event
`
`Date
`
`Location
`
`# Photos
`
`# Videos
`
`# Docs
`
`Jack Monk's Arrival
`
`26-Dec-2003
`
`Chicago, IL
`
`Mike Testy's First Birthday
`
`13-Sep-1983
`
`Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Jack Monk's Arrival
`
`29-Dec-2003
`
`Chicago, IL
`
`Mike Testv's 2nd Birthday
`
`13-Sep-1983
`
`Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Jack Monk's Arrival
`
`29-Dec-2003
`
`Chicago, IL
`
`Mike Testv's 3rd Birthday
`
`13-Sep-1983
`
`Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Jack Wrigley Monk's Arrival
`
`29-Dec-2003
`
`Chicago, IL
`
`Mike Testy's 4th Birthday
`
`13-Sep-1983
`
`Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Nancv Learns How to Ride a Bike
`
`21-Jul-1978
`
`St. Louis, MO
`
`69
`
`54
`
`36
`
`69
`
`54
`
`36
`
`69
`
`54
`
`36
`
`69
`
`54
`
`36
`
`76
`
`4
`
`21
`
`2,199
`
`4
`
`21
`
`2,199
`
`4
`
`21
`
`2,199
`
`4
`
`21
`
`2,199
`
`2
`
`4
`
`0
`
`2
`
`4
`
`0
`
`2
`
`4
`
`0
`
`2
`
`4
`
`0
`
`2
`
`0
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 15 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 12 of 50
`
`US 10,621,228 B2
`
`FIG. 12
`
`Thumbnail I Table
`
`Album/Event
`
`Date
`
`Location
`
`# Photos
`
`# Videos
`
`# Docs
`
`Jack Monk's Arrival
`
`26-Dec-2003
`
`Chicago, IL
`
`Mike Testy's First Birthday
`
`13-Sep-1983
`
`Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Jack Monk's Arrival
`
`29-Dec-2003
`
`Chicago, IL
`
`Mike Testv's 2nd Birthday
`
`13-Sep-1983
`
`Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Jack Monk's Arrival
`
`29-Dec-2003
`
`Chicago, IL
`
`Mike Testv's 3rd Birthday
`
`13-Sep-1983
`
`Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Jack Wrigley Monk's Arrival
`
`29-Dec-2003
`
`Chicago, IL
`
`Mike Testy's 4th Birthday
`
`13-Sep-1983
`
`Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Nancv Learns How to Ride a Bike
`
`21-Jul-1978
`
`St. Louis, MO
`
`69
`
`54
`
`36
`
`69
`
`54
`
`36
`
`69
`
`54
`
`36
`
`69
`
`54
`
`36
`
`76
`
`4
`
`21
`
`2,199
`
`4
`
`21
`
`2,199
`
`4
`
`21
`
`2,199
`
`4
`
`21
`
`2,199
`
`2
`
`4
`
`0
`
`2
`
`4
`
`0
`
`2
`
`4
`
`0
`
`2
`
`4
`
`0
`
`2
`
`0
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 16 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 13 of 50
`
`US 10,621,228 B2
`
`FIG. 13
`
`Thumbnail I Table
`
`Last Name
`
`# People
`
`# Photos
`
`#Videos
`
`# Docs
`
`+ Alberts
`
`+ Annex
`
`+ Bade
`+ Bacon
`
`+ Bates
`
`+ Boone
`
`+ Danas
`
`+ Danes
`
`- Monk (All)
`
`I Monk,CJ
`~ Monk,Jack
`+ Firestone
`
`+ Moore
`
`+ Slvthe
`+ Stein
`
`+ Testv
`
`2
`
`2
`
`3
`
`4
`
`5
`
`6
`
`7
`
`8
`
`2
`
`1
`
`1
`
`21
`
`1
`
`1
`
`2
`
`4
`
`8
`
`7
`
`8
`
`8
`
`7
`
`6
`
`5
`
`7
`
`499
`
`200
`
`199
`
`1249
`
`4
`
`9
`
`249
`
`788
`
`0
`
`0
`
`0
`
`0
`
`1
`
`2
`
`4
`
`3
`
`4
`
`2
`
`2
`
`17
`
`6
`
`0
`
`1
`
`2
`
`0
`
`0
`
`0
`
`0
`
`0
`
`2
`
`1
`
`2
`
`14
`
`7
`
`7
`
`39
`
`3
`
`9
`
`3
`
`12
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 17 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 14 of 50
`
`US 10,621,228 B2
`
`FIG. 14
`
`Last Name
`
`Relationship
`
`# Photos # Videos
`
`# Docs
`
`Thumbnail I Table
`
`Alberts, John
`
`Killian, Jack
`
`Killian, Brian
`
`Killian, Kevin
`
`Killian, Sarah
`
`Killian, John
`
`Killian, Mark
`
`Killian, Louis
`
`Killian, John
`
`I Monk,CJ
`[[l Monk, Jack
`Firestone, Mike
`
`Moore, Bertha
`
`Slvthe, Sarah
`
`Killian, John
`
`Killian, Mike
`
`Cousin
`
`Son
`
`Nephew
`
`Nephew
`
`Daughter-in-law
`
`Great Nephew
`
`Great Nephew
`
`Great Grandson
`
`Grandson
`
`Great Grandson
`
`Great Grandson
`
`Third Cousin
`
`Great Niece
`
`Sister
`
`Brother
`
`Brother
`
`8
`
`7
`
`8
`
`8
`
`7
`
`6
`
`5
`
`7
`
`499
`
`200
`
`199
`
`0
`
`0
`
`0
`
`0
`
`1
`
`2
`
`4
`
`3
`
`4
`
`2
`
`2
`
`1249
`
`17
`
`4
`
`9
`
`249
`
`788
`
`6
`
`0
`
`1
`
`2
`
`0
`
`0
`
`0
`
`0
`
`0
`
`2
`
`1
`
`2
`
`14
`
`7
`
`7
`
`39
`
`3
`
`9
`
`3
`
`12
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 18 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 15 of 50
`
`US 10,621,228 B2
`
`FIG. 15
`
`Thumbnail I Table
`
`Location Name
`
`Address
`
`Citv
`
`State
`
`Countrv
`
`# Photos
`
`# Videos # Docs
`
`Dom
`
`Lucilla & Roberto
`
`Lisle Home
`
`898 West St
`
`College
`
`545 Market
`
`Amazon Trip
`
`Cabin
`
`999 Pine
`
`Lake Geneva WI
`
`Grad School
`
`903 Plymouth
`
`Charleston
`
`IL
`
`Cologne
`
`Montalcino
`
`Lisle
`
`Akron
`
`Manus
`
`IL
`
`OH
`
`Germany
`
`ltalv
`
`USA
`
`USA
`
`Brazil
`
`USA
`
`USA
`
`USA
`
`3
`
`6
`
`45
`
`64
`
`235
`
`98
`
`1256
`
`12
`
`2
`
`1
`
`12
`
`2
`
`8
`
`2
`
`32
`
`0
`
`0
`
`0
`
`2
`
`0
`
`2
`
`0
`
`4
`
`0
`
`Griffith Park
`
`298 Glencarin
`
`Los Feliz
`
`LA Eauestrian Ctr
`
`568 Horse Dr
`
`Glendale
`
`Del Coronado
`
`12 Coronado Dr Coronado
`
`Fenwav Park
`
`123 Yawke
`
`Boston
`
`Wrigley Field
`
`1190 W Addison
`
`Chicago
`
`Home
`
`444 Main
`
`Anvwhere
`
`GA Grill Party
`
`321 Silver
`
`Pike's Market
`
`786 Market
`
`Macon
`
`Seattle
`
`CA
`
`CA
`
`CA
`
`MA
`
`IL
`
`IL
`
`GA
`
`WA
`
`USA
`
`USA
`
`USA
`
`USA
`
`USA
`
`USA
`
`USA
`
`Raffels
`
`345 Fong
`
`Singapore
`
`Singapore
`
`4
`
`321
`
`57
`
`498
`
`10,987
`
`15
`
`18
`
`23
`
`4
`
`4
`
`3
`
`7
`
`49
`
`0
`
`1
`
`2
`
`0
`
`0
`
`5
`
`3
`
`9
`
`0
`
`0
`
`0
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 19 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 16 of 50
`
`US 10,621,228 B2
`
`FIG. 16
`
`Category I Card I Table
`
`Recipe
`
`Chef
`
`Date
`
`Categorv
`
`Blacks Yellow Thai Chicken Curry
`
`Jack Black
`
`31 Jan 2010
`
`Dinner
`
`Skinnv Germans
`
`KFC in a Bag
`
`Shit on a Shingle
`
`Mrs. Fields Cookies
`
`Chicken Pot Pie
`
`Roll Your Own Dough
`
`Pizza ala Franciscan
`
`Meatball Delight
`
`Almond Cookies
`
`Jumpin Jack Flap Jacks
`
`Vicki's Chow Mein
`
`Fat Steak
`
`Mud Pie
`
`Caesar Salad
`
`Daddio Pancakes
`
`Gerda
`
`29 Dec 2003
`
`Breakfast
`
`The Kernal
`
`13 Sept 1988
`
`George James
`
`5 Aug 1998
`
`Mrs. Fields
`
`Jack Black
`
`21 July 1978
`
`31 Jan 2010
`
`Vito Spadavecchio
`
`29 Dec 2003
`
`Charles Faso
`
`13 Sept 1988
`
`Anytime
`
`Anytime
`
`Dessert
`
`Dinner
`
`Dinner
`
`Dinner
`
`Dinner
`
`Ben Delight
`
`Lori James
`
`Jack Jack
`
`5 Aug 1998
`
`21 July 1978
`
`Dessert
`
`31 Jan 2010
`
`Breakfast
`
`Vicki Firestone
`
`29 Dec 2003
`
`Barrv Monk
`
`Nancy Monk
`
`13 Sept 1988
`
`5 Aug 1998
`
`Dinner
`
`Dinner
`
`Dessert
`
`Christopher Monk
`
`21 Julv 1978
`
`Anvtime
`
`Barry Monk
`
`2 March 2011
`
`Breakfast
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 20 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 17 of 50
`
`US 10,621,228 B2
`
`FIG. 17
`
`Close (X)
`
`◄
`
`►
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 21 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 18 of 50
`
`US 10,621,228 B2
`
`FIG. 18
`
`Advanced Search Filter
`
`Keywords:
`Date:
`Location:
`
`Add
`Add
`
`People:
`
`Chicago, IL USA
`Yellowstone, MT USA
`Cologne, Germany
`
`Mike Cubbie
`Mary Lamb
`Christopher Monk
`Nancy Monk
`Dwight Sch rut
`
`Event:
`
`Jack's pt Birthday
`Fly Fishing in Yellowstone
`Raking Leaves
`Christmas 2010
`Thanksgiving 2010
`July 4th Parade
`Album:
`Add
`Add
`Star Ranking:
`Sharing Rights: Add
`DocumentType: Add
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 22 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 19 of 50
`
`US 10,621,228 B2
`
`FIG. 19
`
`Captain Phil's Memory-Webb
`
`Welcome, Captain Phil
`Last Login:
`11.18.2010
`
`My recent memories:
`123 Photos uploaded on 11.07.10
`2 albums created 11.17.10
`12 visitors since last login date
`123 Photos uploaded on 11.07.10
`2 albums created 11.17.10
`
`My recent Webb views:
`Captain Phil 2010 (photo album)
`Chicken Pot Pie (recipe)
`Captain Phil (Timeline)
`
`Updates and Alerts:
`License renewal due 1.15.2011
`
`Media
`# Photos
`
`#Videos
`# Documents
`
`Count Archive Status
`Count
`1,342
`·····························································································································
`80% complete
`61 % complete
`
`75
`173
`
`People Stats:
`
`Last Name
`
`# People
`
`# Photos
`
`#Videos
`
`# Docs
`
`Monk
`
`Firestone
`
`Testy
`
`Event Stats:
`
`Event
`
`7
`
`11
`
`4
`
`499
`
`1,249
`
`788
`
`4
`
`17
`
`1
`
`14
`
`39
`
`12
`
`Date
`
`Location
`
`# Media
`
`Mike Testv's 1st Birthdav
`
`13-Seot-1988 Minneapolis, MN
`
`21
`
`Cubs Beat Cards Aug 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Nancy Learns to Ride Bike
`
`21-July-1978
`
`St. Louis, MO
`
`2,199
`
`2
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 23 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 20 of 50
`
`US 10,621,228 B2
`
`FIG. 20
`
`0050
`
`I
`
`0101
`
`0401
`
`0501
`
`1201
`
`Third Party Media Provider
`
`Outside MW S stem
`Inside MW System
`
`System Reading
`Phase
`
`0100
`
`1200
`
`1300
`
`Application Export System
`
`0600
`
`0700
`
`0800
`
`0900
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 24 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 21 of 50
`
`US 10,621,228 B2
`
`System Reading Phase___/
`
`0100
`_____ ,
`
`System Interpreting and Adding Data to Relationship Table Phase _ /
`
`0200
`
`FIG. 21
`
`0401
`
`System Indexing Digital Files and
`Tag Data Blocks for a Digital File
`within User Relationship Table
`0300
`
`User Relationship Table
`
`User Settings
`
`Inside MW System
`
`Locate EX!F
`Image File
`Directories in
`Digital File
`
`0103
`
`File not
`uploaded
`
`0104
`
`L-----------------------------~
`
`UNIFIED PATENTS EXHIBIT 1001
`Page 25 of 73
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 22 of 50
`
`US 10,621,228 B2
`
`FIG. 22
`
`0320
`
`03eo
`
`MemoryWeb Tag
`
`MemoryWeb_Tag (Data Blocks)
`I
`- -o 361
`- -o 362
`
`MediaAsset.Caotion
`
`MediaAsset.StarRanklng
`
`MedlaAsset.DateCreated
`
`- -o 363
`
`-- 0364
`
`EXIF Tags version 2.3 Image File Directories /
`(Data Blocks)
`/
`Tae:Labels
`
`_/0322
`
`EXIF Family
`Grouo Name
`
`/0323
`Location
`
`0321
`
`IFD0
`
`IFD0
`
`IFD0
`
`0x9c9b or
`OxOlOe
`
`0x9c9f
`
`N/A
`
`0x9c9e
`
`IFD0
`
`IFD0
`
`ExiflFD ---0x9003 -
`
`0334
`
`0335
`
`0x9c9c
`
`0x9c9d
`
`N/A
`
`IFD0
`
`0x8298
`
`Multiple
`
`Descriotion Title
`
`Description Subject
`
`bescriotion Rating
`
`Description Tags
`
`Description Comments
`
`- -
`
`0324
`
`0325
`
`0326
`
`-
`
`O