a2) United States Patent
`US 10,621,228 B2
`10) Patent No.:
`
` Desmondetal. (45) Date of Patent: *Apr. 14, 2020
`
`
`US010621228B2
`
`(54) METHOD AND APPARATUS FOR
`MANAGINGDIGITAL FILES
`
`(72)
`
`(71) Applicant: NCM IP Holdings, LLC, Glen Ellyn,
`IL (US)
`Inventors: Christopher J. Desmond, Glen Ellyn,
`IL (US); Nancy L. Desmond, Glen
`Ellyn, IL (US); L. Michael Taylor,
`Chicago, IL (US)
`
`(*) Notice:
`
`(73) Assignee: NCM IP Holdings, LLC, Glen Ellyn,
`IL (US)
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C, 154(b) by 0 days.
`This patent is subject to a terminal dis-
`claimer.
`
`(21) Appl. No.: 16/578,238
`
`(22)
`
`Filed:
`
`Sep. 20, 2019
`
`(65)
`
`Prior Publication Data
`
`US 2020/0026727 Al
`
`Jan. 23, 2020
`.
`Related U.S. Application Data
`(63) Continuation of application No. 16/536,300, filed on
`Aug. 8, 2019, which is a continuation of application
`;
`;
`(Continued)
`onnnue
`Int. Cl
`ny
`GO6F 16/58
`GO6F 16/901
`
`(51)
`
`(2019.01)
`(2019.01)
`.
`(Continued)
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`5,694,514 A
`6,629,104 Bl
`
`12/1997 Evans ;
`9/2003 Parulski
`(Continued)
`FOREIGN PATENT DOCUMENTS
`102591922 B
`10/2015
`2466869 A3
`2/2017
`(Continued)
`
`CN
`EP
`
`OTHER PUBLICATIONS
`Kustanowitz et al., “Motivating Annotation for Personal Digital
`Photo Libraries: Lowering Barriers while Raising Incentives,” Tech.
`Report HCIL-2004-18, U. Maryland, 2005 (10 pages).
`(Continued)
`
`Primary Examiner — Loc Tran
`(74) Attorney, Agent, or Firm — Nixon Peabody LLP
`
`(57)
`
`ABSTRACT
`
`A computer-implemented method of associating digital tags
`with digital files comprises storing a plurality of digital files
`having embeddedtherein content data and metadata includ-
`ing tags; receiving, via a user interface device of a client
`device, a first tag label containing alphanumeric text created
`and inputted by a userofthe client device; modifying, using
`a controller device, a selected first one of the tags of the
`metadata in a first of the digital files to includethefirst tag
`a
`:
`.
`:
`label; receiving, via the user interface device or another user
`:
`:
`:
`:
`os
`interface device, an instruction to searchfor all of the digital
`files havingat leastthe first tag label; responsive to receiving
`the instruction, automatically searching for all of the digital
`files having at least the first tag label; and displaying, on a
`video display device associated with the client device, a first
`indication ofthe first tag label.
`
`(52) U.S. Cl.
`CPC veeeeeeees G06F 16/5866 (2019.01), GO6F 16/51
`(2019.01); GO6F 16/901 (2019.01), GO6F
`16/907 (2019.01); GO6F 3/0481 (2013.01)
`(58) Field of Classification Search
`CPC ... GO6F 16/5866; GO6F 16/907; GO6F 16/901
`19 Claims, 50 Drawing Sheets
`See application file for complete search history.
`
`
`Select None
`Select All
`Delete Selected
`
`1 File(s) Selected
`File Attributes:
`Date:
`Location:
`People:
`Event:
`Comments:
`Keywords:
`Album:
`Star Ranking:
`Sharing Rights:
`Recipe:
`DocumentType:
`File Name:
`
`September 2, 2010 change
`Yellowstone, MT, USA change
`Christopher Desmond, Nancy Desmond
`Fly Fishing in Yellowstone
`Add
`Add
`Add
`eocce
`Add
`Add
`Add
`IMG 123 change
`
`
`
`
`
`SAMSUNG 1001
`
`SAMSUNG 1001
`
`1
`
`

`

`US 10,621,228 B2
`
`Page 2
`
`Related U.S. Application Data
`
`No. 15/375,927, filed on Dec. 12, 2016, now Pat. No.
`10,423,658, which is a continuation of application
`No. 14/193,426, filed on Feb. 28, 2014, now Pat. No.
`9,552,376, which is a continuation-in-part of appli-
`cation No. 13/157,214, filed on Jun. 9, 2011, now Pat.
`No. 9,098,531.
`
`(51)
`
`Int. Cl.
`
`GO6F 16/51
`GO6F 16/907
`GO6F 3/0481
`
`(2019.01)
`(2019.01)
`(2013.01)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`7,372,976 B2
`7,475,060 B2
`7,480,669 B2
`7,694,236 B2
`7,822,746 B2
`7,860,846 B2
`7,991,283 B2
`8,001,124 B2
`8,015,144 B2
`8,024,317 B2
`8,032,508 B2
`8,055,675 B2
`8,060,492 B2
`8,069,142 B2
`8,086,048 B2
`8,108,778 B2
`8,150,844 B2
`8,150,967 B2
`8,166,016 B2
`8,166,168 B2
`8,171,388 B2
`8,230,338 B2
`8,255,379 B2
`8,264,570 B2
`8,271,506 B2
`8,281,027 B2
`8,285,483 B2
`8,307,029 B2
`8,315,959 B2
`8,359,314 B2
`8,364,611 B2
`8,380,039 B2
`8,386,506 B2
`8,390,702 B2
`8,401,771 B2
`8,402,356 B2
`8,429,156 B2
`8,447,120 B2
`8,452,855 B2
`8,458,115 B2
`8,463,931 B2
`8,484,223 B2
`8,489,115 B2
`8,490,011 B2
`8,493,495 B2
`8,503,791 B2
`8,504,073 B2
`8,520,979 B2
`D689,079 S
`D689,080 S
`D689,083 S
`D689,084 S
`D689,085 S
`8,538,811 B2
`8,538,813 B2
`8,542,294 B2
`8,554,623 B2
`8,560,390 B2
`8,560,517 B2
`
`5/2008 Rhoad
`1/2009 Toyama
`1/2009 Lo
`4/2010 Gusmorino
`10/2010 Svendsen
`12/2010 Takahashi
`8/2011 Chen
`8/2011 Svendsen
`9/2011 Zheng
`9/2011 Nair
`10/2011 Martinez
`11/2011 Higgins
`11/2011 Nair
`11/2011 Davis
`12/2011 Naaman
`1/2012 Athsani
`4/2012 Redstone
`4/2012 King
`4/2012 Higgins
`4/2012 Hayashi
`5/2012 Zaltzman
`7/2012 Dugan
`8/2012 Govindachetty
`9/2012 Karimoto
`9/2012 Martinez
`10/2012 Martinez
`10/2012 Amer-Yahia
`11/2012 Davis
`11/2012 Zheng
`1/2013 Svendsen
`1/2013 Tendjoukian
`2/2013 Luo
`2/2013 Martinez
`3/2013 Bhatt
`3/2013. Krumm
`3/2013 Martinez
`4/2013 Buchmueller
`5/2013 Ji
`5/2013 Higgins
`6/2013 Cai
`6/2013 Evans
`7/2013 Ota
`7/2013 Rodriguez
`7/2013 Stapleton
`7/2013 D’Souza
`8/2013 Conwell
`8/2013 Svendsen
`8/2013 Conwell
`9/2013 Edwards
`9/2013 Edwards
`9/2013 Pasceri
`9/2013 Pasceri
`9/2013 Pasceri
`9/2013 Higgins
`9/2013 Kakarla
`9/2013 Bhatt
`10/2013 Higgins
`10/2013 Higgins
`10/2013 Yang
`
`8,583,620
`8,583,668
`8,589,389
`8,589,486
`8,594,702
`8,606,021
`8,626,699
`8,671,154
`8,676,001
`8,706,406
`8,745,133
`8,762,285
`D708,196
`D708,197
`D708,198
`8,769,099
`8,769,393
`8,799,371
`8,805,165
`8,806,365
`8,810,597
`8,813,107
`8,825,472
`8,831,352
`8,849,854
`8,849,909
`D715,819
`8,880,568
`8,890,888
`8,892,495
`8,914,342
`8,923,889
`8,930,848
`8,949,212
`8,954,425
`8,966,121
`8,972,177
`8,998,422
`9,009,177
`9,014,511
`9,015,617
`9,015,633
`9,020,247
`9,031,953
`9,032,320
`9,055,037
`9,063,226
`9,076,259
`9,092,409
`9,098,545
`9,104,729
`9,104,915
`9,110,903
`9,151,618
`9,158,794
`9,160,802
`9,172,666
`9,202,200
`9,218,328
`9,224,172
`9,235,766
`9,239,848
`9,245,041
`9,261,376
`D751,597
`9,311,396
`9,336,240
`9,372,931
`9,390,104
`9,405,981
`9,418,485
`9,424,595
`9,460,116
`9,462,054
`9,465,513
`9,471,834
`9,483,500
`9,501,577
`9,507,778
`
`11/2013
`11/2013
`11/2013
`11/2013
`11/2013
`12/2013
`1/2014
`3/2014
`3/2014
`4/2014
`6/2014
`6/2014
`7/2014
`7/2014
`7/2014
`7/2014
`7/2014
`8/2014
`8/2014
`8/2014
`8/2014
`8/2014
`9/2014
`9/2014
`9/2014
`9/2014
`10/2014
`11/2014
`11/2014
`11/2014
`12/2014
`12/2014
`1/2015
`2/2015
`2/2015
`2/2015
`3/2015
`4/2015
`4/2015
`4/2015
`4/2015
`4/2015
`4/2015
`5/2015
`5/2015
`6/2015
`6/2015
`7/2015
`7/2015
`8/2015
`8/2015
`8/2015
`8/2015
`10/2015
`10/2015
`10/2015
`10/2015
`12/2015
`12/2015
`12/2015
`1/2016
`1/2016
`1/2016
`2/2016
`3/2016
`4/2016
`5/2016
`6/2016
`7/2016
`8/2016
`8/2016
`8/2016
`10/2016
`10/2016
`10/2016
`10/2016
`11/2016
`11/2016
`11/2016
`
`Govindachetty
`Higgins
`Bisdikian
`Martinez
`Naaman
`Conwell
`Xie
`Davis
`Brucher
`Kalaboukis
`Martinez
`Davis
`Pasceri
`Pasceri
`Pasceri
`Kalaboukis
`Abhyanker
`Davis
`Luo
`Stapleton
`Akiya
`Higgins
`Raghuveer
`Gao
`Kakarla
`Farmer
`Pasceri
`Perezynski
`Lee
`Hoffberg
`Kalaboukis
`Svendsen
`Lim
`Dhandapani
`Xiao
`Josefsberg
`Zheng
`Snavely
`Zheng
`Brucher
`Stapleton
`Takamura
`Adam
`Rathnavelu
`Crawford
`Evans
`Zheng
`Hourie
`Charaniya
`Abhyanker
`Dong
`Conwell
`Martinez
`Amer-Yahia
`Higgins
`Svendsen
`Murdock
`Stibel
`Stapleton
`Churchill
`Li
`Liu
`Pilskalns
`Zheng
`Pasceri
`Meadow
`Bhatt
`Capt
`Thomee
`Li
`Lindberg
`Svendsen
`Pilskalns
`Poletto
`Sims
`Filip
`Brucher
`Zheng
`Jaffe
`
`2
`
`

`

`US 10,621,228 B2
`
`Page 3
`
`(56)
`
`References Cited
`
`11/2009 Stapleton
`2009/0288005 Al
`11/2009 Naaman
`2009/0290812 Al
`12/2009 Lawler
`2009/0307618 Al
`12/2009 Higgins
`2009/0325602 Al
`2/2010 Svendsen
`2010/0041419 Al
`12/2016 Pujara
`9,519,682 Bl
`2/2010 Conwell
`2010/0046842 Al
`1/2017 Hoffberg
`9,535,563 B2
`3/2010 Karimoto
`2010/0053371 Al
`1/2017 Zheng
`9,536,146 B2
`3/2010 Crawford.
`2010/0064239 Al
`1/2017 Desmond
`9,552,376 B2
`4/2010 Burgener
`2010/0082427 Al
`1/2017 Rodriguez
`9,557,162 B2
`4/2010 Ockene
`2010/0107125 Al
`2/2017 Zaltzman
`9.576.253 B2
`6/2010 Perczynski
`2010/0153348 Al
`2/2017 Hartford
`9.582.546 B2
`6/2010 Chang
`2010/0162411 Al
`3/2017 Zheng
`9,593,957 B2
`7/2010 Bhatt
`2010/0171763 Al
`3/2017. Davis
`9,600,484 B2
`7/2010 Lee
`2010/0182341 Al
`4/2017 Martinez
`9.626.685 B2
`7/2010 Higgins
`2010/0185509 Al
`5/2017 Boyns
`9,646,025 B2
`9/2010 Davis
`2010/0241689 Al
`5/2017 Bisdikian
`9,654,570 B2
`9/2010 Athsani
`2010/0241944 Al
`6/2017 Hartford
`9.674.650 B2
`10/2010 Pilskalns
`2010/0268717 Al
`6/2017 East
`9.679.456 B2
`10/2010 Bouget
`2010/0268766 Al
`6/2017 Tseng
`9,680,929 B2
`11/2010 O’Sullivan
`2010/0280913 Al
`6/2017 Zheng
`9,683,858 B2
`11/2010 Athsani
`2010/0293035 Al
`6/2017 Tseng
`9,691,073 B2
`11/2010 Harrison
`2010/0293193 Al
`7/2017 Davis
`9,706,345 B2
`2/2011 Svendsen
`2011/0040779 Al
`7/2017 Setlur
`9,710,961 B2
`4/2011 Zheng
`2011/0093458 Al
`8/2017 Adam
`9.721.188 B2
`5/2011 Bhatt
`2011/0109769 Al
`9/2017 Zheng
`9,754,226 B2
`5/2011 Govindachetty
`2011/0113064 Al
`9/2017 Hasenei
`9,772,745 B2
`6/2011 Kankainen
`2011/0145258 Al
`10/2017 Grue
`9.787.799 B2
`8/2011 Feng
`2011/0191014 Al
`10/2017. Nair
`9.805.123 B2
`8/2011 Pilskalns
`2011/0191253 Al
`11/2017 Miller
`9.811.879 B2.
`8/2011 Amer-Yahia
`2011/0202267 Al
`1/2018 Higgins
`9,858,348 BL
`8/2011 Zheng
`2011/0208426 Al
`1/2018 Chapin
`9,870,572 B2
`11/2011 Zheng
`2011/0289031 Al
`1/2018 Patton
`9,881,179 B2
`12/2011 Zheng
`2011/0301832 Al
`1/2018 Bisdikian
`9,882,994 B2
`
`
`9.942.121 B2 2011/0314016 Al=12/2011 Svendsen4/2018 Poletto
`10,019,850 B2
`7/2018 Lindberg
`2012/0114249 Al
`5/2012 Conwell
`10,037,327 B2
`7/2018 Thomee
`2012/0158755 Al
`6/2012 Gammill
`10,068.178 B2
`9/2018 van Zwol
`2012/0218150 Al
`8/2012 Oyabu
`10,074,093 B2
`9/2018 Higgins
`2012/0220311 Al
`8/2012 Rodriguez
`10,083,533 B2
`9/2018 Bhatt
`2012/0251011 Al
`10/2012 Gao
`10,110,541 B2
`10/2018 Li
`2012/0266090 Al
`10/2012 Nealer
`10,187,543 B2
`1/2019 Lahcanski
`2012/0278171 Al
`11/2012 Tang
`10,223,701 B2
`3/2019 King
`2012/0278767 Al
`11/2012 Stibel
`10,230,803 B2
`3/2019 Higgins
`2012/0329441 Al
`12/2012 Tseng
`10,235.444 B2
`3/2019 Poletto
`2012/0331091 Al
`12/2012 Tseng
`(0.242.051 B2
`3/2019 Shinn
`2013/0018881 Al
`1/2013 Bhatt
`10.282.752 B2
`5/2019 Athsani
`2013/0036165 Al
`2/2013 Tseng
`10.288.433 B2
`5/2019 Zheng
`2013/0063613 Al
`3/2013 Conwell
`10.289.643 B2
`5/2019 Brucher
`2013/0073202 Al
`3/2013 Zheng
`10.303.975 B2
`5/2019 Adam
`2013/0101157 Al
`4/2013 Li
`10.311,611 B2
`6/2019 Stoop
`2013/0138685 Al
`5/2013. Brucher
`10,318,110 B2
`6/2019 Naaman
`2013/0141612 Al
`6/2013 Bhatt
`10,331,863 B2
`6/2019 Patton
`2013/0151597 Al
`6/2013 Akiya
`10.360,352 B2
`7/2019 Patton
`2013/0202198 Al
`8/2013 Adam
`10 430.452 B2
`10/2019 Ross
`2013/0275536 Al
`10/2013 Murdock
`10,445,346 B2
`10/2019 Govindachetty
`2013/0339440 Al
`12/2013 Balassanian
`
`2005/0060299 Al 2014/0059477 Al—2/2014 Wong3/2005. Filley
`
`2006/0165380 AL*
`7/2006 Tanaka oo... GLB 27/34
`2014/0071272 Al
`3/2014 Rodriguez
`386/207
`2014/0089811 Al
`3/2014 Meadow
`2014/0101531 Al
`4/2014 Capt
`5/2007 Svendsen
`2007/0118508 Al
`2014/0101601 Al
`4/2014 Tang
`12/2007 Van Der Meulen
`2007/0282908 Al
`2014/0143247 Al
`5/2014 Rathnavelu
`2/2008 Kanno
`5008/0040034 Al
`solaolgtoco Al
`OLA merYala
`2/2008 Fisher
`2008/0051994 Al
`2014/0181089 Al
`6/2014 Desmon
`6/2008 Naaman
`2008/0148175 Al
`2014/0188880 Al
`7/2014 Abhyanker
`8/2008 Kimchi
`2008/0201302 Al
`2014/0193087 Al
`7/2014 Conwell
`10/2008 Takahashi
`2008/0250398 Al
`2014/0354628 Al
`12/2014 Lindberg
`1/2009 Farmer
`2009/0013041 Al
`2015/0039630 Al
`2/2015 Thomee
`1/2009 Abhyanker
`2009/0019085 Al
`2015/0070165 Al
`3/2015. East
`2/2009 Naaman
`2009/0049408 Al
`2015/0070397 Al
`3/2015. Miller
`4/2009 Takamura
`2009/0106705 Al
`
`
`2009/0113350 AL* 2015/0116540 Al=4/2015 Gilman4/2009 Hibino ween GO6F16/41
`715/853
`2015/0117713 Al
`4/2015 Zheng
`2015/0156247 Al
`6/2015 Hensel
`2015/0186389 Al
`7/2015 Zheng
`2015/0213057 Al
`7/2015 Brucher
`2015/0213329 Al
`7/2015 Adam
`2015/0244794 Al
`8/2015 Poletto
`2015/0244833 Al
`8/2015 Grue
`2015/0358224 Al
`12/2015 Poletto
`2016/0092741 Al
`3/2016 Li
`
`5/2009 Zaltzman
`5/2009 Pilskalns
`5/2009 Chopra
`8/2009 Zheng
`9/2009 Higgins
`10/2009 Farouki
`10/2009 Sigurbjornsson
`11/2009 Brucher
`
`U.S. PATENT DOCUMENTS
`
`2009/0132689 Al
`2009/0132941 Al
`2009/0135438 Al
`2009/0216704 Al
`2009/0222302 Al
`2009/0254867 Al
`2009/0265631 Al
`2009/0279794 Al
`
`3
`
`

`

`US 10,621,228 B2
`Page 4
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`2016/0162512 Al
`2016/0247307 Al
`2016/0253358 Al
`2016/0314187 Al
`2016/0321269 Al
`2016/0328444 Al
`2016/0344888 Al
`2017/0024415 Al
`2017/0103081 Al
`
`6/2016 Battistini
`8/2016 Stoop
`9/2016 Bhatt
`10/2016 Poletto
`11/2016 Thomee
`11/2016 Shinn
`11/2016 Lahcanski
`1/2017 Brucher
`4/2017 Jones
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`WO
`WO
`WO
`
`2410414 B1
`WO 2011/070225 Al
`WO 2013/019376 Al
`WO 2013/099704 Al
`
`10/2019
`6/2011
`2/2013
`7/2013
`
`OTHER PUBLICATIONS
`
`Miller et al., “Give and take: a study of consumer photo-sharing
`culture and practice,” CHI ’07 Proceedings of the SIGCHI Confer-
`ence on Human Factors in Computing Systems, pp. 347-356, 2007
`(10 pages).
`Ameset al., “Why wetag: motivations for annotation in mobile and
`online media,” CHI ’07 Proceedings of the SIGCHI Conference on
`Human Factors in Computing Systems, pp. 971-980, ACM, 2007
`(10 pages)10083533.
`Yee et al., “Faceted Metadata for Image Search and Browsing”, CHI
`2003, pp. 401-408, 2003, ACM.
`Ferre, “CAMELIS: Organizing and Browsing a Personal Photo
`Collection with a Logical Information System”, Int. Conf. Concept
`Lattices and Their Applications, pp. 112-123, 2007, HAL.
`Tomasson et al., “PhotoCube: Effective and Efficient Multi-
`Dimensional Browsing of Personal Photo Collections”, ICMR ’11,
`2011, ACM.
`Bartolini et al., “Integrating Semantic and Visual Facets for Brows-
`ing Digital Photo Collections”, SBED, pp. 65-72, 2009.
`Trattner et al., “Evaluating Tag-Based Information Access in Image
`Collections”, Proceedings of the 23° ACM Conference on Hyper-
`text and Social Media, pp. 113-122, 2012 ACM.
`Kang et al., “Capture, Annotate, Browse, Find, Share: Novel
`Interfaces for Personal Photo Management”, International Journal
`of Human-Computer Interaction, 23(3), pp. 315-337, 2007, Law-
`rence Eribaum Associates, Inc.
`Jaffe et al., “Generating Summaries and Visualization for Large
`Collections of GeoReferenced Photographs”, MIR’06, pp. 89-98,
`2006 ACM.
`Tormiai et al., “Sharing, Discovering and Browsing Geotagged
`Pictures on the Web”, 2007, Hewlett-Packard Development Com-
`pany, L.P., pp. 1-18 (19 pages).
`Snavely et al., “Photo Tourism: Exploring Photo Collection in 3D”,
`SIGGRAPH ’06 ACM Transactions on Graphics, vol. 25, Issue 3,
`pp. 835-846, 2006 ACM.
`
`Kisilevich et al., ““Event-based analysis of People’s Activities and.
`Behavior Using Flickr and Panoramic Geotagged Photo Collec-
`tions”, 14” International Conference Information Visualization, pp.
`289-296, 2010 IEEE.
`Ahern et al., “World Explorer: Visualizing Aggregate Data From
`Unstructured Text in Geo-Referenced Collections”, JCDL’07, pp.
`1-10, 2007, ACM.
`Kopf et al., “Deep photo: model-based photograph enhancement
`and viewing”, ACM Transactions on Graphics, vol. 27, No. 5,
`Article 116, Dec. 2008, ACM (10 pages).
`Amundsen, J.; “Using the Geographical Location of Photos in
`Mobile Phones”; Master of Science in Computer Science submis-
`sion; Norwegian University of Science and Technology; Jul. 2008
`(112 pages).
`Gentile, L; “Using Flickr Geotags to Find Similar Tourism Desti-
`nations”; master thesis, 2011; Politecnico di Milano, Dept. of
`Computer Engineering (96 pages).
`Hollenstein, L.; “Capturing Vernacular Geography from Georeferenced
`Tags”; Master Thesis; Institute of Geography, University of Zurich;
`Nov. 2008 (139 pages).
`Nutanong, S. et al.; “An Efficient Layout Method for a Large
`Collection of Geographic Data Entries”; Center for Automation
`Research,Institute for Advanced Computer Studies, Dept. of Com-
`puter Science, University of Maryland; pp. 717-720 (4 pages).
`Slingsby, A. et al.; “Interactive tag maps and tag clouds for the
`multiscale exploration of large spatio-temporal datasets”; Informa-
`tion Visualization, pp. 497-504; 2007, IV °07. 11” International
`Conference. ISSN: 1550-6037 (9 pages).
`Hoffman,A.; “Create Great iPhone Photos: Apps, Tips, Tricks, and.
`Effects”; copyright 2011; ISBN-10: 1-59327-285-5, ISBN-13: 978-
`1-59327-285-2 (216 pages).
`Chen, Y-F. et al.; “GeoTracker: Geospatial and Temporal RSS
`Navigation”; AT&T Labs—Research, Florham Park, NJ; WWW
`2007 / Track: Browsers and User Interfaces, Session: Smarter
`Browsing, pp. 41-50 (10 pages).
`Goodman, E.; “Destination Services: Tourist media and networked
`places”; School of Information, UC Berkeley; Mar. 2, 2007 (11
`pages).
`Rattenbury, T. et al.; “Towards Automatic Extraction of Event and.
`Place Semantics from Flickr Tags”; SIGIR ’07, Jul. 23-27, 2007;
`Amsterdam, The Netherlands; ACM 978-1-59593-597-7/07/0007
`(8 pages).
`Kadar, B. et al.; “Where Do Tourists Go? Visualizing and Analysing
`the Spatial Distribution of Geotagged Photography”; Cartographica
`48:2, pp. 78-88; 2013; University of Toronto Press; doi: 10.3138/
`carto.48.2.1839 (11 pages).
`Kennedy, L. et al.; “How Flickr Helps Us Make Sense of the World:
`Context and Content in Community-Contributed Media Collec-
`tions”; MM ’07, Sep. 23-28, 2007; Augsburg, Bavaria, Germany;
`Copyright 2007; ACM 978-1-59593-701-8/07/0009 (10 pages).
`Richard, G.etal.; “Geotagging Photographs for Distribution on the
`Web”; Mineral Physics Institute, Earth and Space Sciences Build-
`ing, Stony Brook University, Stony Brook. NY; date unknown (9
`pages).
`
`* cited by examiner
`
`4
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 1 of 50
`
`US 10,621,228 B2
`
`
`
`FIG. 1
`
`Select None
`Select All
`Delete Selected
`
`IMG 123 change
`
`1 File(s) Selected
`File Attributes:
`September 2, 2010 change
`Date:
`Yellowstone, MT, USA change
`Location:
`Christopher Desmond, Nancy Desmond
`People:
`Fly Fishing in Yellowstone
`Event:
`Add
`Comments:
`Add
`Keywords:
`Add
`Album:
`eovc6e
`Star Ranking:
`Sharing Rights: Add
`Recipe:
`Add
`Document Type: Add
`File Name:
`
`5
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 2 of 50
`
`US 10,621,228 B2
`
`FIG. 2
`
`Comments:
`Suzanne and Anthony’s Wedding Party where the cousins posed
`for a photo in the grass. Note, Jack with the lollipop and the
`photographerwith his shoe in the photo
`
`Location:
`Historical Society
`Lisle, IL 60532
`
`
`
`
`
`People:
`Jack Wong
`CJ Wong
`MaryFirestone
`Zoe Peika
`Nick Persons
`
`Event: Suzanne & Anthony’s Wedding Reception 2010
`
`Camera Details: more
`
`6
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 3 of 50
`
`US 10,621,228 B2
`
`
`
`FIG. 3
`
`Thumbnail | Table
`
`Mary Bader and Clint Firestone
`Wedding October 11, 1945
`
`Capt. Clint Firestone
`1952
`
`GermanyTrip 1955
`
`Father’s Day 1966
`
`Christopher 1*
`Year 1972
`
`
` Europe
`College Break 1992 EIU College
`Graduation 1994 Nancy James and
`
`
`
`Cairo, Egypt 1996
`
`Backpacking
`
`Christopher Desmond
`Wedding October 17, 1998
`
`_ ;
`AmazonFishing Trip
`November 2009
`
`Cousins Party 2010
`
`Fishing in Yellowstone
`Montana September 2010
`
`7
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 4 of 50
`
`US 10,621,228 B2
`
`FIG. 4
`
`Captions On | Off
`
`
`
`Christopher Desmond and Nancy Desmond Wedding — October 17, 1998
`
`8
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 5 of 50
`
`US 10,621,228 B2
`
`FIG. 5
`
`9
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 6 of 50
`
`US 10,621,228 B2
`
`FIG. 6
`
`
`
`Thumbnail | Table
`
`Steve Fire
`
`Suzanne Lucky
`
`
`
`
`Clint Dewitt
`
`CharlotteKillian
`
`ChrisJeep
`
`Peter LongHorn
`
`Steve Fire
`
`SN
`
`Suzanne Lucky
`
`Steve Fwah
`
`Ben Sherman
`
`Joe Schmo
`
`MaryVirginia
`
`Clint Dewitt
`
`Charlotte Killian
`
`10
`
`10
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 7 of 50
`
`US 10,621,228 B2
`
`
`
`FIG. 7
`
`
`
`Clinton Dewitt Firestone IV
`
`July 12, 1896
`Birth:
`April 29, 1971
`Death:
`Parents: Clinton Dewitt Firestone Ill and Viola Miller
`Comments: He was a WWII U.S. Air force pilot and POW in WWII and veteran
`honorably discharged in December of 1947. He worked for 44 years for the
`Firestone Tire and Rubber Companyin retail, wholesale and original equipment
`sales, marketing and management. He was bornin Akron, OH and is buried in
`Columbiana, OH.
`
`Edit bio
`
`
`
`Locations
`
`Timeline
`
`Family Tree
`
`Recipes
`
`11
`
`11
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 8 of 50
`
`US 10,621,228 B2
`
`
`
`FIG. 8
`
`Descendants of Charlotte Killian Halter
`
`Tree | Table
`
`PWSoorsy
`
`
`SS.
`
`SIUENSS,
`
`ROE
`
`oy
`
`.
`
`HO AI Ne
`:
`_ YY seeeng
`: R .
`
`
`
`
`
`
`
`RRMNONAN AFA WO AK
`Sign
`RANG
`
`
`
`LIC Ry CLLWE
`ERE EAS
`
`RAY ACCORROC
`« 363 eosoe Hee,
`
`S
`
`Sane
`
`SS SESSA SRS
`
`NEES PEQSQGQ ¥
`
`
`SE NSS WIS ‘
`
`12
`
`12
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 9 of 50
`
`US 10,621,228 B2
`
`
`
`FIG. 9
`
`DayHeeeeeeeeeIR rcrncorrcorneernerereeeeaE
`
`~
`
`*
`
`*
`
`°
`
`.
`
`Montana September 2010
`
`Year
`Month
`
`:
`
`°
`
`"
`
`Cousins Party 2010
`
`Fishing in Yellowstone
`
`13
`
`13
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 10 of 50
`
`US 10,621,228 B2
`
`FIG. 10
`
` *
`
`Se
`
`
`
`Desmond's Yellow Thai Chicken Curry
`
`Curry Mix
`* Coconut milk (400 ml) - DO NOT SHAKE IT UP
`*
`800 gram ofchicken (4 chicken breast)
`*
`Fish sauce (Nam Pla) Thai Bamboo Garden — Bottle
`* Garlic (2 cloves)
`*
`Broccoli { 2cups chopped)
`*
`2 Peppers (chopped)
`*
`2Carrots (chopped)
`
`1 Zucchini (chopped)
`Thai Basil (8 leaves)
`*
`Lemon Grass(in jar) 1 teaspoon
`*
`* Chinese Ginger Root(in jar) 1 teaspoon
`
`Rice
`*
`Thai Rice (something that only takes 2 cups of water)
`*
`Dice chicken in bowl and add twotablespoons offish
`sauce. Let marinate for 20 minutes.
`
`*
`
`Take thick part of coconut milk out into pan (about 4
`tablespoons), Curry paste, 1 spoon of lemongrass, 1
`spoonofginger and garlic. Heat over high with boil and
`THEN stir for 1 minute. Add meat (uncooked) and fry
`until cooked over high heat
`¢ Add milk, brown sugar and salt. Bring back to slight boil
`and constantly stir. Add veggies and soy sauce. Cook for
`about 10-14 minutes COVERED until veggies are cooked.
`Serve witha smile.
`
`Chef: Barry Desmond
`
`Original Handwritten Recipe
`
`14
`
`14
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 11 of 50
`
`US 10,621,228 B2
`
`
`
`FIG. 11
`
`Thumbnail | Table
`
`
`Monk'sArrival|29-Dec-2003|_chicago,u_|69[|4|4| Ly¥$.$
`
`4&
`
`Alburn/Event
`Hack Monk's Arrival|26-Dec-2003|chicago|oo|4|4|
`MikeTesty'sfirst Birthday|13-Sep-1983|Minneapolismn|54|21|0
`Cubs Beat Cards 1998
`Jack Monk's Arrival|29-pec-2003|chicago|69|4|4
`Mike Testy's2nd Birthday|13-Sep-1983|Minneapolismn|5a|21|io
`Cubs Beat Cards 1998
`Hack Monk's Arrival|29-dec-2003|chicago|oo|4|4|
`Mike Testy's3rd Birthday|13-Sep-1983|Minneapolismn|5a|21|
`
`
`
`lackWrigley
`
`15
`
`15
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 12 of 50
`
`US 10,621,228 B2
`
`
`
`FIG. 12
`
`Thumbnail | Table
`
`
`Monk'sArrival|29-Dec-2003|_chicago,u_|69[|4|4| Le¥$tz
`
`4&
`
`Alburn/Event
`Hack Monk's Arrival|26-Dec-2003|chicago|oo|4|4|
`MikeTesty'sfirst Birthday|13-Sep-1983|Minneapolismn|54|21|0
`Cubs Beat Cards 1998
`Jack Monk's Arrival|29-pec-2003|chicago|69|4|4
`Mike Testy's2nd Birthday|13-Sep-1983|Minneapolismn|5a|21|io
`Cubs Beat Cards 1998
`Hack Monk's Arrival|29-dec-2003|chicago|oo|4|4|
`Mike Testy's3rd Birthday|13-Sep-1983|Minneapolismn|5a|21|
`
`
`
`lackWrigley
`
`16
`
`16
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 13 of 50
`
`US 10,621,228 B2
`
`FIG. 13
`
`
`
`eS+ Alberts
`
`Thumbnail | Table
`
`17
`
`17
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 14 of 50
`
`US 10,621,228 B2
`
`
`
`FIG. 14
`
`Thumbnail | Table
`
`Alberts,john
`
`pewfafafa
`
`
`Saran|sister=||to| 333333333333333333333333333333333333333333333333333333333332353
`G4eodeded
`POCCOOOOOOLELOLOOLELOLOOLELOLIOLELOLLOLELUDOOLELULOOLELODOOLEDE,Lo
`
`Killian, Jack
`Killian, Brian
`
`Killian, Kevin[kitian,Kevin|Nephew|
`[KitianSarah_|aughter-intow|7|1|oo|
`
`
`[Kilianohn|GreatNephew|6|2|2|
`
`
`[Kitian,Mark|GreatNephew|5|4|1|
`
`[Kilianjohn|Grandson|a9|a|ta
`
`pttenc)_}_steatanon_{_200_{2}7
`
`Monk, Jack|GreatGrandson|19|2|7|
`[Fivestone,Mike|Thirdcousin[249[a7[39|
`isostoniGeeatnecsa[6Pa
`[siythe,
`
`18
`
`18
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 15 of 50
`
`US 10,621,228 B2
`
`FIG. 15
`
`Thumbnail | Table
`
`
`
`e¥$tz
`
`4& L
`
`pom|cotogne||Gerrmany|3|2||
`Lucila&Roberto]|Montalcino||ttaly=|os|at|_|
`
`cotege|545Market_|Akron|on|usa|6a|2|o|
`mazontrip||Manus||erat|235|g|
`cabin|999PineflakeGeneval_wi|usa|os|2|o|
`radschool__|903plymouth|charleston]uw|usa|1256|32|a|
`
`critpark|298Glencarin|tosretz|ca|usa|a2|o|o|
`LAtquestrianctr|s68HorseDr_|Glendale|ca|usa[|4|a|o|
`Delcoronado_|12CoronadoDr|Coronado|_ca|usa[|321|a|o|
`
`Home|__aaamain|anywhere|uw|usa|10987_|a9|9|
`caGritParty|_32asiiver_|Macon|ca|usa|25|o_|
`
`pictwarist|a6nist|setle|wa[use[ae[a]0]
`fate[atsong—LSngagos||Sgn|2»P210
`
`19
`
`19
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 16 of 50
`
`US 10,621,228 B2
`
`FIG. 16
`
`
`
`
`Category | Card | Table FEte
`
`20
`
`20
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 17 of 50
`
`US 10,621,228 B2
`
`FIG. 17
`
`
`
`
`
`Close (X)
`
`21
`
`21
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 18 of 50
`
`US 10,621,228 B2
`
`
`
`FIG. 18
`
`
`
`
`
`AdvancedSearchFilter
`
`Keywords:
`Date:
`Location:
`
`Add
`Add
`
`Chicago, IL USA
`Yellowstone, MT USA
`Cologne, Germany
`
`Mike Cubbie
`Mary Lamb
`Christopher Monk
`Nancy Monk
`Dwight Schrut
`
`People:
`
`Event:
`
`Jack’s 1% Birthday
`Fly Fishing in Yellowstone
`Raking Leaves
`Christmas 2010
`Thanksgiving 2010
`July 4% Parade
`Add
`Album:
`Add
`Star Ranking:
`Sharing Rights: Add
`Document Type: Add
`
`22
`
`22
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 19 of 50
`
`US 10,621,228 B2
`
`
`
`FIG. 19
`
`Media
`# Photos
`# Videos
`
`Count Archive Status
`1,342
`75
`173
`
`61% complete # Documents
`
`Count
`80% complete
`
`Captain Phil’s Memory-Webb
`
`Welcome, Captain Phil
`Last Login:
`11.18.2010
`
`My recent Webb views:
`* Captain Phil 2010 (photo album)
`*
`Chicken PotPie (recipe)
`* Captain Phil (Timeline)
`
`
`
`My recent memories:
`*
`123 Photos uploaded on 11.07.10
`*
` 2albums created 11.17.10
`PeopleStats:
`*
`12 visitors since last login date
`*
`123 Photos uploaded on 11.07.10
`Lastname|People|photos|videos|aDocs_|
`*
` 2albums created 11.17.10
`
`
`St. Louis, MO
`
`EventStats:
`
`Updatesand Alerts:
`*
`License renewal due 1.15.2011
`vent|ate|tocation|#Media|
`Mike Testy’s 1%t Birthda 13-Sept-1988|Minneapolis, MN
`
`Cubs Beat Cards Aug 1998
`5-Aug-1998
`Chicago,IL
`2,199
`Nancy Learnsto Ride Bike
`21-July-1978
`
`
`
`23
`
`23
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 20 of 50
`
`US 10,621,228 B2
`
`FIG. 20
`
`0050
`
`1101
`
`Original Digital
`Third Party
`Third Party Facial
`
`
`
`file(s} upload to
`
`
`
`Geographical
`Recognition
`MemoryWeb
`
`
`
`Provider
`
`Mapping Provider
`
`
`merece eenecenereneeeetenen peraneeceeneereeeeed eeeSDeMYSYSeeeeens
`Inside MW System
`
`0501
`
`4201
`
`individual /
`Group of
`People
`
`MW Moditied File
`exported from
`MemoryWebto Storage
`System
`
`Third Party Media Provider
`
`Application Export System
`
`Third Party Facial
`
`Recognition
`
`Systern
`
`
`
` Systern Reading
`
`
`Phase
`
`0100
`
`System Interpreting and
`Adding Data to
`Relationship Table Phase
`
`
`
`0200
`
`Third Party Media
`
`0300
`System (Data
`Exchange}
`
`
`
`
`User Relationship
`Table
`
`System Indexing
`Digital Files and
`Tag Data Blocks
`for a Digital File
`within User
`
`Relationship Table
`
`9500
`
`
`
`
`
`
`
`Share to Social
`
`Network Provider
`
`System
`1100
`
`
`Third Party Geographical
`
`
`Location Mapping
`System
`
`Shareto in
`System
`
`‘Application Digital
`Tag Organizer
`
`System
`
`0600
`
`Dot-Tags System
`
`Continuous Link
`of Application
`
`Advanced Filters
`System
`
`0700
`
`0800
`
`Keyword Fast
`Search System
`
`9900
`
`24
`
`24
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 21 of 50
`
`US 10,621,228 B2
`
`0100
`
`0200
`
`FIG. 21
`
`Systern Interpreting and Adding Data to Relationship Table Phase
`
`J
`0101
`
`Outside MW System
`
`inside MW System
`
`
`
`System Reading Phase
`
`Original Digitalfile begins to
`upload to MemoryWeb
`
`
`
`102
`
`
`
`
`file Into MW
`
`
`Locate EXIF
`
`
`User Settings
`Read anditerate through
`Image File
`Duplicate
`.
`
`Directories in
`
`
`
`each EXIF tag item
`File?
`
`
`Create duplicates
`Digital File
`Storage of
`of originalfile in
`Original File
`differentfile sizes
`
`
`
`
`Save EXIF data to
`
`
`User Relationship
`Duplicate
`File
`File not
`
`
`
`Table for Digital
`Medium
`
`
`File
`Duplicate File
`
`uploaded
`0104
`
`Srnail
`
`Duplicate File
`Parse EXIF
`Thumbnail
`
`
`data into
`
`
`Duplicate File
`MemoryWeb
`
`Create new
`
`
`EXIF Tag (Data
`MW tag
`Blocks}
`data
`
`
`
`
`
`Microsoft
`Look up MW
`
`
`
`Windows Tag
`exists?
`reer
`tag data
`‘Data Blocks:
`‘ssociate
`
`
`
`existing Tag
`MemoryWeb
`
`gata
`Tag (Data Blocks)
`
`MemeryWeb
`Person Tag (Data
`
`Third Party Facial
`Recognition Provider
`
`0400
`Third Party Facial Recognition
`System
`9201
`
`Outside MW System
`Inside MW System
`
`Upload original
`
`
`
`System indexing Digital Files and
`Tag Data Blocksfor a Digital File
`within User Relationship Table
`0300
`
`
`User Relationship Table
`1900
`
`
`
`
`
`25
`
`25
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 22 of 50
`
`US 10,621,228 B2
`
`FIG. 22
`
`0320
`
`MemoryWebTag (Data Blocks)
`
`0360
`
`EXIF Tags version 2.3 ImageFile Directories é
`(Data Blocks)
`0322
`
`0324
`EXIF Family
`Group Name
`
`Tag Labels
`
`Description Title
`
`NEDO
`
`
`
`828
`Location
`0x9c9b or
`0x010e
`
`
`
`0324
`
`MemoryWeb Tag
`
`IMediaAsset.Caption
`
` 0361
`
`0362
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`fmagemagero.dimensions,wishneigeet|ute|
`
`
`
`0327
`
`
`IMediaAsset.Width
`waeeso
`
`
`0328
`IMediaAsset. Height
`fener
`Camera (Camera Maker, Camera Model, etc) |Ietuttipte|
`Advanced Photo (Lens Maker, Lens Model, etc, |Inuteiote|
`
`
`This is used to inject information that do not currently have
`0329
`EXIF standardized tags including Collection, People, Location
`Name, Recipe Name, Person Tag Data Blocks {0380}, etc.
`
`
`
`
`Q325
`MeciaAsset. StarRanking
`0363
`
`
`pesrpionang|a
`
`IMediaAsset.DateCreated
`
`0364
`
`
`
`
`
`
`
`
`
`
`
`Po
`
`0326
`
`0330
`0331
`0332
`0333
`
`ruinoaercouiesa
`
`0334
`
`0335
`
`ExifiED
`0x9286
`ic
`orsiaidefoes
`GPSLatitude Ref
`psloxoons
`
`IGPS Longitude Ref
`0x0005
`
`PSAltitude
`(0x0006
`
`
`
`26
`
`26
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 23 of 50
`
`US 10,621,228 B2
`
`FIG. 23
`
`0340
`
`0360
`
`Windows Imaging Component (WIC) Metadata
`(Data Blocks)
`
`MemoryWebTag (Data Blocks)
`
`MemoryWeb Tag
`
`MediaAsset. Filename
`
`MediaAsset. UploadBatch.BatchName
`
`
`
`
`
`0361
`
`0372
`
`
`
`
`
`27
`
`0342
`
`0343
`
`7 034l
`
`.
`File Name
`
`File item Type
`File Folder Path
`
`File Date Created
`
`File Date Modified
`
`File Size
`
`File Attributes
`File (Offline availability, Offline status, Shared with,
`Owner, Computer, etc}
`
`Other
`
`27
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 24 of 50
`
`US 10,621,228 B2
`
`FIG. 24
`
`0380
`
`0395
`
`0381
`0382
`
`0383
`0384
`0385
`0386
`0387
`
`0388
`0389
`0390
`0391
`0392
`0393
`0394
`0396
`
`MemoryWebPerson Tag (Data Blocks)
`Person Name
`Nickname
`Birthdate
`Date of Death
`
`Biography
`Mother
`Father
`
`Brother 1, Brother 2,...
`Sister 1, Sister 2, ...
`
`pets
`
`Daughter 1 , Daughter 2,...
`Son1,Son2....
`
`Spouse 1 , Spouse2,...
`
`Facial Recognition Data (Taylor?)
`Facebook ID
`
`28
`
`28
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 25 of 50
`
`US 10,621,228 B2
`
`FIG. 25
`
`Party Facial Recognition Pro
`
`0401
`
`0400
`
`Outside MW System
`
`inside MW System
`
`
`System parses
`
`System receives
`identified faces
`
`
`response from
`and corresponding
`Third Party Facial
`
`
`Facial Recognition
`
`
`System sends
`System request for
`
`
`thumbnails of all
`data for each
`Recognition
`Digital Files to
`
`
`
`
`Digital Files that have
`Third Party
`Provider
`Digital File
`Facial
`
`Recognition
`
`0408
`Provider
`
`System records and associates specific
`face for Digital Fe with a MemoryWeb
`is Face Recognition
`Existing (Trained)Face 1D
`
`Person ID and sends to MemoryWeb
`above score
`threshold?
`
`Relationship
`in User Relationship
`
`Person Tag (Data Blocks) in User
`
`Table?
`Table
` Retrieves
`
`Relationship Table
`Thumbnails ofali
`
`Digital Files that
`have uncorfirmed
`
`faces
`
`
`System generates facial
`System generates virtual
`
`
`recognition record for unknown
`unconfirmed facial
`
`recognition record for person
`person and then sends
`
`information to MemoryWeb
`and then sendsinformation to
`
`Person Tag (Data Blocks) in User
`MemoryWeb Person Tag
`
`Relationship Table
`{Data Blocks} in User
`Relationship Table
`
`
`
`29
`
`29
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 26 of 50
`
`US 10,621,228 B2
`
`FIG. 26
`
`0500
`
`0504
`
`
`
`Outside MW System
`Inside MW System
`0503
`0504
`Systems sends
`setrati
`registration
`confirmation
`
`System receives: contact na mes
`and IDs from Third Party Media
`i
`i
`Provider(daily)
`
`
`
`0502
`
`:
`Userselects sharing of
`their information with
`Third Party Media
`Provider with
`MemoryWeb
`
`Systems receives
`registration
`confirmation, Third
`y
`Party Media ID
`0505
`
`0506
`
`System sends
`registration
`information to
`UserSetting
`Table?
`
`System requests contact
`names and IDs from Third
`Party Media Provider
`(daily)
`
`0300
`
`
`User Relationship
`Table
`
`UserSettings [
`Table
`
`
`
`
`
`
`0508
`
`Yes
`
`0510
`
`‘g Person >
`Match
`No
`
`0509
`
`~
`systemassociates
`existing Block
`ag
`1 pate a oy
`f
`Rel ations! ip Table for
`DigitalFile
`
`System add new person to Tag
`Data Blockof Relationship
`Table for Digital File
`
`4000
`
`
`
`Share to Third Party Media
`Provider System
`
`30
`
`30
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 27 of 50
`
`US 10,621,228 B2
`
`FIG. 27
`
`1900
`
`1901
`
`1902
`1903
`1904
`
`1905
`1906
`1907
`
`1908
`1909
`1910
`1911
`1912
`1913
`
`Item
`
`User’s Name
`
`Payment ID
`Password
`
`Payment Date
`
`Account Type
`User’s email
`
`Language preference
`Date format
`Email notifications
`
`Contacts (with third Party Social Media)
`Facebook ID
`API Token
`
`31
`
`31
`
`

`

`U.S. Patent
`
`Apr. 14, 2020
`
`Sheet 28 of 50
`
`US 10,621,228 B2
`
`FIG. 28
`
`0601
`
`0600
`
`User selects
`Digital File in
`MW
`
`0602
`
`J
`
`
`
`
`
`32
`
`
`
`System sends
`request to
`Relationship
`Table for specific
`Digital Fle
`
`0608
`
`Continuous Link
`User selec