( 12 ) United States Patent
`Bederson et al .
`
`US 11,061,524 B2
`( 10 ) Patent No .:
`( 45 ) Date of Patent :
`* Jul . 13 , 2021
`
`US011061524B2
`
`( 54 ) TECHNIQUES TO MODIFY CONTENT AND
`VIEW CONTENT ON A MOBILE DEVICE
`( 71 ) Applicant : Zumobi , Inc. , Seattle , WA ( US )
`( 72 ) Inventors : Benjamin B. Bederson , Chevy Chase ,
`MD ( US ) ; John SanGiovanni , Seattle ,
`WA ( US )
`( 73 ) Assignee : Zumobi , LLC , Austin , TX ( US )
`Subject to any disclaimer , the term of this
`( * ) Notice :
`patent is extended or adjusted under 35
`U.S.C. 154 ( b ) by 299 days .
`This patent is subject to a terminal dis
`claimer .
`( 21 ) Appl . No .: 15 / 723,040
`( 22 ) Filed :
`Oct. 2 , 2017
`( 65 )
`Prior Publication Data
`Jan. 25 , 2018
`US 2018/0024703 A1
`
`Related U.S. Application Data
`( 63 ) Continuation of application No. 12 / 836,424 , filed on
`Jul . 14 , 2010 , now Pat . No. 9,778,810 .
`( 60 ) Provisional application No. 61 / 225,366 , filed on Jul .
`14 , 2009 .
`( 51 ) Int . Cl .
`G06F 3/0481
`( 52 ) U.S. CI .
`CPC
`
`( 2013.01 )
`
`GO6F 3/0481 ( 2013.01 ) ; G06F 3/04817
`( 2013.01 )
`
`( 58 ) Field of Classification Search
`GO6F 3/0481 ; G06F 3/04817
`CPC
`See application file for complete search history .
`
`( 56 )
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`5,949,428 A
`2004/0080670 A1 *
`
`9/1999 Toelle et al .
`4/2004 Cheatle
`
`2007/0245256 A1
`2008/0068507 A1
`2008/0122858 A1
`
`10/2007 Boss et al .
`3/2008 Krause et al .
`5/2008 Wilensky
`( Continued )
`FOREIGN PATENT DOCUMENTS
`
`G06T 11/60
`348/441
`
`WO
`WO
`
`WO - 2007091081 A2 *
`WO - 2007 / 091081 A2
`
`8/2007
`1/2008
`
`G11B 27/034
`
`OTHER PUBLICATIONS
`Good , L. et al . “ Zoomable User Interfaces as a Medium for Slide
`Show Presentations , ” Information Visualization , 2002 , vol . 1 , pp .
`35-49 , Palgrave Macmillan .
`( Continued )
`Primary Examiner Jennifer N TO
`Assistant Examiner — Ashley M Fortino
`( 57 )
`ABSTRACT
`A computer - based method for rendering modified graphical
`content is disclosed . The graphical content includes a plu
`rality of cells . The method includes receiving the graphical
`content with metadata modifying the graphical content ,
`rendering the graphical content upon a screen according to
`the metadata , and receiving user input . The method also
`includes in response to the user input , progressing through
`at least a subset of the cells , wherein the progressing
`includes moving a camera view from cell to cell in the at
`least a subset , each of the cells in the at least a subset being
`displayed as a focus point with surrounding area being
`adjusted in opacity according to the metadata .
`7 Claims , 19 Drawing Sheets
`
`124
`
`102
`
`Next
`
`EX2045
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`US 11,061,524 B2
`Page 2
`
`( 56 )
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`2008/0165141 A1 *
`
`2009/0060334 A1 *
`
`2010/0064223 Al
`
`7/2008 Christie
`3/2009 Rayner
`3/2010 Tilton
`
`G06F 3/044
`345/173
`G11B 27/034
`382/173
`
`OTHER PUBLICATIONS
`Quinn , A. , et al . , “ Readability of Scanned Books in Digital Librar
`ies , ” Proceedings of ACM CHI , CHI 2008 , Apr. 5-10 , 2008 , pp .
`705-714 , ACM Press .
`* cited by examiner
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul . 13 , 2021
`
`Sheet 1 of 19
`
`US 11,061,524 B2
`
`113
`
`Out
`
`131
`
`121
`
`122
`
`190
`
`133
`
`132
`
`W
`
`134
`
`135
`
`123
`
`101
`
`124
`
`125 9 2
`
`126
`
`138
`
`Prev
`
`112
`
`o
`
`FIG . 1
`
`136
`
`102
`
`137
`
`Next
`
`111
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul . 13 , 2021
`
`Sheet 2 of 19
`
`US 11,061,524 B2
`
`i
`
`Out
`
`101
`
`121
`
`::::
`
`.
`
`::
`
`.
`
`Prev
`
`Next
`
`FIG . 2
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul . 13 , 2021
`
`Sheet 3 of 19
`
`US 11,061,524 B2
`
`Out
`
`121
`
`...
`
`101
`
`122
`
`...
`
`123
`
`...
`
`.............
`
`Prev
`
`: . : . :
`
`.
`Next
`
`o
`
`FIG . 3
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul . 13 , 2021
`
`Sheet 4 of 19
`
`US 11,061,524 B2
`
`Out
`
`1. " .
`
`"
`
`135
`
`Prev
`1
`
`D
`
`: ) ; : * ..
`
`: **
`
`i
`
`:: .
`
`101
`
`:: , ...
`
`' :: .
`
`:: ..
`
`. :: .
`
`Next .
`
`o
`
`FIG . 4
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul . 13 , 2021
`
`Sheet 5 of 19
`
`US 11,061,524 B2
`
`( Out
`
`123
`
`101
`
`136
`
`. :: : "
`
`...
`? ?? ??
`
`...
`
`: *
`1.6 .
`***** : ..
`
`" Prev
`
`Next
`
`o
`
`FIG . 5
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul , 13 , 2021
`
`Sheet 6 of 19
`
`US 11,061,524 B2
`
`Out
`
`.
`
`'
`
`
`
` *** 2.3 :: .... :
`
`.
`
`.........
`
`124
`
`101
`
`.
`
`Prev
`
`Next
`
`FIG . 6
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul . 13 , 2021
`
`Sheet 7 of 19
`
`US 11,061,524 B2
`
`Out
`
`... ...
`
`, i ..
`
`...
`
`125
`
`101
`
`:::
`
`...
`
`: ...
`
`Prev
`
`Next
`
`o M
`
`FIG . 7
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul . 13 , 2021
`
`Sheet 8 of 19
`
`US 11,061,524 B2
`
`Out .
`
`" ...
`
`noen
`
`.
`
`"
`
`: . * . * ..
`
`101
`
`138
`
`; ; , ?
`
`. : . : . : . : ..
`
`.
`
`.
`
`Prev
`
`Next :
`
`?
`
`FIG . 8
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul . 13 , 2021
`
`Sheet 9 of 19
`
`US 11,061,524 B2
`
`Out
`
`..
`
`101
`
`126
`
`Prev
`
`Next
`
`O
`
`FIG . 9
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul . 13 , 2021
`
`Sheet 10 of 19
`
`US 11,061,524 B2
`
`Out
`
`WA
`
`1001
`
`1003
`
`1004
`
`1005
`
`3
`
`Prev
`
`o
`
`FIG . 10
`
`101
`
`102
`
`1002
`
`1006
`
`Next
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul . 13 , 2021
`
`Sheet 11 of 19
`
`US 11,061,524 B2
`
`Out
`
`1001
`
`1 .
`
`. :::::
`
`... ' * ? ..
`
`. : . :
`
`Prev
`
`Next
`
`FIG . 11
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul . 13 , 2021
`
`Sheet 12 of 19
`
`US 11,061,524 B2
`
`Out
`
`***
`
`. ::
`
`1002
`
`Prev
`
`Next
`
`o
`
`FIG . 12
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul . 13 , 2021
`
`Sheet 13 of 19
`
`US 11,061,524 B2
`
`Out
`
`. :: ,
`
`1003
`
`Prev
`
`.
`
`Next
`
`FIG . 13
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul . 13 , 2021
`
`Sheet 14 of 19
`
`US 11,061,524 B2
`
`Out
`
`S
`
`1004
`
`?? ?
`
`8 : Prev
`
`Next
`
`FIG . 14
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul . 13 , 2021
`
`Sheet 15 of 19
`
`US 11,061,524 B2
`
`Out
`
`: 7
`
`:: A
`
`1005
`
`:
`
`* Prev
`
`.
`
`Next
`
`FIG . 15
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul . 13 , 2021
`
`Sheet 16 of 19
`
`US 11,061,524 B2
`
`? ..
`Out
`
`* * *
`...
`:: . . ::: . , ..
`.. . : . :
`
`1006
`
`Prev
`
`...
`
`Next
`
`o
`
`FIG . 16
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul . 13 , 2021
`
`Sheet 17 of 19
`
`US 11,061,524 B2
`
`1700
`
`1701
`
`???
`
`1702
`
`1703
`
`CELL 1
`
`CELL 2
`
`CELL 3
`
`FIG . 17
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul . 13 , 2021
`
`Sheet 18 of 19
`
`US 11,061,524 B2
`
`RECEIVING INDICATIONS OF SALIENT VISUAL
`FEATURES WITHIN EACH OF THE CELLS
`
`1801
`
`ASSOCIATING METADATA WITH THE GRAPHICAL CONTENT
`WITH RESPECT TO THE SALIENT VISUAL FEATURES
`
`1802
`
`USING COMPUTER VISION ALGORITHMS TO PROCESS
`THE GRAPHICAL CONTENT AND ASSOCIATE ADDITIONAL
`METADATA ITEMS WITH THE GRAPHICAL CONTENT
`
`1803
`
`YES
`
`DOES
`THE USER INDICATE
`THAT THE ADDITIONAL
`METADATA ITEMS ARE
`ACCEPTABLE ?
`
`1804
`
`NO
`
`ALLOWING THE USER TO MANUALLY CHANGE ONE OR
`
`MORE OF THE ADDITIONAL METADATA ITEMS 1 1805
`
`ASSOCIATING THE ADDITIONAL METADATA ITEMS
`WITH THE GRAPHICAL CONTENT
`
`1806
`
`MAKING THE CONTENT AVAILABLE TO END USERS
`
`1807
`
`1800
`
`FIG . 18
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jul . 13 , 2021
`
`Sheet 19 of 19
`
`US 11,061,524 B2
`
`1900
`
`1906
`
`1912
`NETWORK
`
`CPU
`1901
`
`RAM
`1903
`
`ROM
`1904
`
`I / O
`ADAPTER
`1905
`
`COMMUNICATIONS
`ADAPTER
`1911
`
`1902
`
`USER
`INTERFACE
`ADAPTER
`1908
`
`DISPLAY
`ADAPTER
`1909
`
`im
`
`1910
`
`1913
`
`1907
`
`1915
`
`1914
`
`FIG . 19
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`US 11,061,524 B2
`
`1
`TECHNIQUES TO MODIFY CONTENT AND
`VIEW CONTENT ON A MOBILE DEVICE
`
`2
`above ) and renders the graphical content according to the
`metadata . For instance , for each portion that has a salient
`feature ( e.g. , each cell on a page of a comic book ) , there is
`a pan , a rotation , and a magnification associated therewith .
`CROSS - REFERENCE TO RELATED
`5 When a selected portion is the focus , other portions that may
`APPLICATIONS
`appear on the screen can be modified to increase focus on the
`first portion ( e.g. , by adjusting opacity of the portions that
`The present application is a continuation of U.S. patent
`are not the focus ) . A first screen may show many portions
`application Ser . No. 12 / 836,424 filed Jul . 14 , 2010 and
`entitled , “ TECHNIQUES TO MODIFY CONTENT AND from which a user can select a portion to view . When a user
`VIEW CONTENT ON MOBILE DEVICES , ” which claims 10 has selected , and is viewing , a particular portion , the user
`the benefit of U.S. Provisional Application No. 61 / 225,366 ,
`can move to a previous or subsequent portion by , e.g. , a
`filed Jul . 14 , 2009 and entitled , “ SYSTEMS AND METH-
`finger swipe on a display screen . A camera view on the first
`ODS PROVIDING TECHNIQUES TO MODIFY CON-
`portion then moves to the next selected portion as the next
`TENT AND VIEW CONTENT ON MOBILE DEVICES , ”
`selected portion is displayed according to the metadata .
`the disclosures of which are incorporated herein by refer- 15
`The foregoing has outlined rather broadly the features and
`technical advantages of the present invention in order that
`ence in their entirety .
`the detailed description of the invention that follows may be
`better understood . Additional features and advantages of the
`BACKGROUND
`invention will be described hereinafter which form the
`Comic books , graphic novels , and other graphic media are 20 subject of the claims of the invention . It should be appre
`ciated by those skilled in the art that the conception and
`quite popular among some readers . Some graphic media
`include sequential , rectangular cells where the story is told
`specific embodiment disclosed may be readily utilized as a
`as the sequence of cells progresses from right to left . More
`basis for modifying or designing other structures for carry
`modern forms of the media often include cells that have
`ing out the same purposes of the present invention . It should
`irregularly shaped boundaries and / or arrange the cells in 25 also be realized by those skilled in the art that such equiva
`irregular patterns upon the page that do not progress from
`lent constructions do not depart from the spirit and scope of
`right to left .
`the invention as set forth in the appended claims . The novel
`Recently , there have been attempts to adapt comic book
`features which are believed to be characteristic of the
`reader interfaces to handheld devices , such as the iPhoneTM ,
`invention , both as to its organization and method of opera
`available from Apple , Inc. Currently , most comic book 30 tion , together with further objects and advantages will be
`readers on the iPhoneTM do the same thing . They have a
`better understood from the following description when con
`series of sequential images that are cropped from a comic
`sidered in connection with the accompanying figures . It is to
`book , and the user reads the comics in the same way that a
`be expressly understood , however , that each of the figures is
`user browses photos in the photo library . In other words , a
`provided for the purpose of illustration and description only
`user drags his or her finger across the display screen to go 35 and is not intended as a definition of the limits of the present
`left or right to an adjacent square located to the left or right .
`invention .
`However , such a technique is not suitable for a page that has
`irregularly - shaped cells and / or an irregular arrangement of
`BRIEF DESCRIPTION OF THE DRAWINGS
`cells . Furthermore , the cropping and rearranging of cells
`often destroys the look and feel that was intended by the 40
`For a more complete understanding of the present inven
`author . A more intuitive and less destructive comic book
`tion , reference is now made to the following descriptions
`taken in conjunction with the accompanying drawings , in
`reading interface is , therefore , desirable .
`which :
`An example embodiment of a viewing application , being
`BRIEF SUMMARY
`45 run on a handheld device and rendering content upon a
`display screen , is shown in FIGS . 1-9 ;
`Various embodiments of the invention are directed to
`FIGS . 10-16 show a second example of the viewing
`systems , methods , and computer program products for edit-
`application using different graphical content ;
`ing existing content to be viewed on mobile device screens .
`FIG . 17 is an illustration of exemplary computer - based
`In one example , a computer - based tool allows a designer to
`edit content so that the content is more conveniently and 50 tool adapted according to one embodiment of the invention ;
`intuitively consumed on small screens ( e.g. , screens of
`FIG . 18 is an illustration of an exemplary method , accord
`mobile devices ) . In one embodiment , there is an application
`ing to one embodiment of the invention ; and
`for providing metadata to pre - existing media content where
`FIG . 19 illustrates an example computer system adapted
`the application allows a designer ( or other user ) to indicate
`according to one embodiment of the present invention .
`salient visual features for portions of visual content . The 55
`application uses vision algorithms to automatically generate
`DETAILED DESCRIPTION
`other kinds of metadata based on the positions of the salient
`features and other characteristics of the graphical content .
`An example embodiment of a viewing application , being
`The designer would then approve or modify the generated
`run on a handheld device and rendering content upon a
`metadata . Additionally or alternatively , any of the metadata 60 display screen , is shown in FIGS . 1-9 . FIG . 1 shows hand
`can be manually entered by the designer using the computer-
`held device 101 with display screen 102 ( in this case , a
`touch - screen that receives user input through user touching
`based tool .
`In another aspect , various embodiments are directed to
`of the screen ) . Control features 111 , 112 , 113 and 190 are
`rendered upon screen 102 and are described further below .
`systems , methods , and computer program products for view-
`ing graphic and / or textual media on small screens . In one 65 The view shown in FIG . 1 includes cells 121-126 , which in
`example , a viewing application receives graphical content
`this example are individually viewable portions of the page .
`and metadata ( such as that produced using the tool described
`Also included in the view are text boxes 131-138 .
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`US 11,061,524 B2
`
`4
`3
`thermore , the various cells 121-126 are selectable items
`An example of a handheld device that can be used in some
`where a user can select one of the cells , and the viewing
`embodiments is the iPhoneTM by Apple Inc. , though other
`application will take the user directly into that cell and still
`handheld devices can be used as well . However , not all
`do an appealing transition with a camera view that moves
`embodiments are limited to handheld devices , as some
`embodiments use a larger display screen for rendering 5 until the appropriate placement , size , rotation , and fading are
`graphical content . Furthermore , devices without touch
`rendered . So some users might actually choose to read the
`screens can be adapted for use in some embodiments , by ,
`comic in that way , by stepping in and out of the cells .
`e.g. , mapping keys to control features and frames . Addition-
`While not easily shown in FIG . 1-9 , various embodiments
`ally , some embodiments may be adapted for rendering
`support commonly accepted touch - based gestures . Similar
`graphical content upon a screen of a tablet computer , such as 10 to current viewers , the viewing application of FIGS . 1-9
`transitions back and forth between cells with a finger swipe
`an iPadTM from Apple , Inc.
`Typically it is difficult to come up with a pattern for
`from one side to the next . Thus , finger swipes may be used
`reading a page such as that shown in FIG . 1 on a mobile
`instead of , or in addition to , control features 111 and 112. In
`phone . In this view , the text is too small to view , and the cells
`the present case , a finger swipe from one side of screen 102
`are shaped irregularly and are arranged in a sequential 15 to the other is effective to cause a transition to an adjacent
`fashion that follows their irregular shapes . Various embodi-
`cell even though the page is arranged substantially vertically
`ments of the present invention are different from conven-
`( rather than horizontally ) . In this way , various embodiments
`tional approaches and provide a better way to render graphi-
`can be adapted for use with any of a variety of arbitrary
`shapes and arrangements . Furthermore , in many embodi
`cal content , as shown in FIGS . 2-9 .
`By selecting next 111 , the user can step through the comic 20 ments , the orientation of gestures are adjusted to correspond
`book . FIG . 2 shows a view after the user selects next 111 .
`to orientation of the screen . Thus , a side - to - side finger swipe
`The transition ( not shown ) includes an animated zoom and
`is still a side - to - side finger swipe whether the device is
`pan into the upper left hand corner of the page shown in FIG .
`arranged portrait - style or landscape - style .
`1. Cell 121 is the focus of FIG . 2 , and it is placed and
`FIGS . 10-16 show a second example using different
`zoomed according to metadata associated with the graphical 25 graphical content . In FIGS . 10-16 , the cells are rectangles
`content ( as explained in more detail below ) . When the user
`and are , therefore , more regular than the cells of FIGS . 1-9 .
`clicks next 111 again , the whole page pans and rotates to
`FIG . 10 shows the whole page , where a user can select a
`present the content shown in FIG . 3 , where cell 122 is the
`particular cell and where the user can return by selecting
`focus . In FIG . 3 , cells 121 and 123 , which are not the focus
`control 113. FIGS . 11-16 show a sequential transition among
`30 the cells , including at least one portrait - landscape - portrait
`of the view of FIG . 3 , are faded out slightly .
`In one example , the fading effect is implemented by
`transition , and extensive use of zooming .
`rendering a semi - transparent , or semi - opaque , mask on top
`Automatic portrait - landscape - portrait transitioning is
`of the cells that are not the focus . Applying a mask may be
`unconventional , but provides a good use of screen space .
`preferable in some embodiments , since it may not be nec-
`However , such automatic transitioning may not be preferred
`essary to modify the underlying existing content when a 35 by all users . Some embodiments include a control , such as
`mask can be applied on top of the content . However , some
`control 190 of FIG . 1 , allowing a user to disable automatic
`embodiments may include modifying the content itself .
`portrait - landscape - portrait transitioning . In such case , the
`As the user continues to go from cell - to - cell , similar
`application may split a landscape cell into multiple portrait
`positioning , zooming , panning , rotating ( if applicable ) and
`views , or split a portrait cell into multiple landscape views .
`fading are performed to give the user an appealing feel
`In another aspect , some embodiments include a computer
`one 40
`that is organic and intuitive . When the user selects next 111
`based tool that allows a designer , developer , author , artist , or
`again , the view shown in FIG . 4 is rendered . In FIG . 4 , the
`other user to add metadata to existing media content to
`text box 135 is enlarged and moved to make it easier to read .
`prepare the media content for display according to the
`FIGS . 2-9 show the views that are rendered as the user
`concepts discussed above . There are variety of different
`selects next 111 to view all cells on the page . The cell that 45 types of metadata that can be added to existing content , and
`is the focus of FIG . 7 is actually a landscape screen wider
`the present examples list a few . The original image has a
`than it is tall — and the application assumes that the user is
`sequence of views_cells in the case of comic books — that
`going to rotate the device so that the user will be able to view
`the end - user will perceive , and there is metadata associated
`cell 125 in a more natural , larger way . As the user selects
`with each of those viewpoints . One metadata item is referred
`next 111 again , the next view is portrait ( FIG . 8 ) , and the 50 to as a viewpoint , which includes a center point of that view ,
`application assumes that the user is going to rotate the device
`a magnification , and a rotation . Another metadata item
`back to its portrait view . As the user walks through a page
`includes a sequence of polygons to adjust opacity of items
`like this , the idea is that the user , as part of experiencing the
`that surround a given cell , referred to as polygonal overlays ,
`content , will rotate the phone back and forth as the user goes
`and they form the basis of the masks .
`Yet another example of metadata includes an indication of
`from cell to cell .
`The application shown in FIGS . 1-9 addresses a problem
`the visually most salient point in that cell . Often , the visually
`most salient point is determined by a developer or other user ,
`in current comic book readers . A comic is created as an
`artistic , creative expression often without awareness that it
`who uses intuition or other technique to decide which point
`might be consumed on a device with a small fixed - size
`is most likely the most salient to end - users . The viewing
`screen , so sometimes the content items are wider than they 60 application supports random access into any cell by tapping
`are tall and sometimes they are taller than they are wide .
`directly onto a portion of the cell to zoom right into it . So in
`Various embodiments of the present invention make an
`order to support that feature , viewing applications receive
`effective use of the screen so that the content is visually
`the indications of salient points because the cells can actu
`ally be overlapping . In order to know which cell an end - user
`perceivable and the text is readable .
`Referring back to FIG . 1 , there is control 113. At any 65 selects , it is assumed that users are most likely to tap on the
`point , a user can zoom out the entire page ( i.e. , go back to
`visually most salient characteristic of one of those cells ( e.g. ,
`the view shown in FIG . 1 ) by selecting control 113. Fur-
`the main figure's face ) . When a user selects a point on the
`
`55
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`US 11,061,524 B2
`
`5
`
`5
`6
`efficient . For more complex and difficult source images , the
`screen , the viewing application looks for the closest salient
`process might simply be manual in the worst - case scenario .
`point and goes to the cell associated with that closest salient
`FIG . 18 is an illustration of exemplary method 1800
`point . The computer based tool of various embodiments
`according to one embodiment of the invention . Method 1800
`allows a user to designate salient points in the various
`can be performed , for example , by a computer - based tool to
`portions of the page .
`add metadata to graphical content . The graphical content is
`FIG . 17 is an illustration of exemplary computer - based
`divided into a plurality of portions ( e.g. , cells ) . As explained
`tool 1700 , adapted according to one embodiment of the
`above , the user indicates the salient points of the cells , and
`invention . In the example embodiment , computer - based tool
`the tool makes a best estimate of other types of metadata
`1700 includes interface 1701 , where for each page a user
`defines at least the viewpoint , the polygonal overlay , and the 10 items , such as polygonal overlays , rotation , magnification ,
`and position ( at blocks 1801-1803 ) . The user can then accept
`salient points . In this example , the interface includes rect
`the computer's metadata , reject some , or reject all of the
`angle 1702 , which is sized to correspond to the screen of a
`metadata at block 1804. For metadata that is rejected , the
`given handheld device ( e.g. , an iPhoneTM ) . The user drags
`user is given an opportunity to modify the items manually at
`rectangle 1702 around on the screen , rotates it , and resizes 15 block 1805. The tool associates metadata with each of the
`it to define a given viewpoint . Interface 1701 also provides
`cells and makes the content available to end - users ( e.g. , by
`polygon tool 1703 to draw polygons to cover the portions of
`publishing to an Internet resource ) at blocks 1806-1807 .
`the view that are the parts that the user wants to de
`Other types of metadata may be indicated manually
`emphasize in order to define the polygonal overlay . Interface
`and / or generated automatically by the computer - based tool .
`1701 also supports a user's selection ( by mouse , touchscreen 20 An additional type of metadata can refer to pixels of a cell
`or otherwise ) in order to specify the salient portion of the
`that correspond to text , whether the text is by itself , in a
`screen that will be used for navigation . The user defines the
`balloon , or included in another arbitrary shape . Such data
`metadata for each of the views or cells manually in this way .
`can be used to “ pop - out ” the text , moving it and / or making
`One issue with the embodiment described above is that it
`it larger to increase readability . For instance , in some situ
`can be somewhat time consuming because these views can 25 ations , text ( at least in its original form ) may be too small to
`be arbitrarily shaped and arranged with different angles and
`read on a handheld device screen . Various embodiments
`different sizes . For instance , it might take a relatively long
`identify where the text is located and magnify the text . The
`time for a human user to draw the polygonal overlay to cover
`degree of magnification can be determined from a combi
`up one of these strangely shaped lines or triangular views
`nation of design and end - user preference . In one example , an
`that can be part of the source image . Another embodiment 30 end - user might prefer fourteen - point font , so the text is
`uses automated image processing techniques to increase
`marked to be magnified to fourteen - point font by the view
`efficiency . In such an example , interface 1701 allows the
`ing application . Additionally or alternatively , there may be a
`select the salient points for the viewpoints ( or for a
`default where the text is marked to magnify it ten or twenty
`user
`single viewpoint if the user prefers to go viewpoint - by-
`percent or thirty percent to make it still larger and still
`viewpoint ) . Tool 1700 includes computer vision to analyze 35 visually more salient .
`the image , looking for the boxes and lines and likely
`One variation of popping out the text is , instead of
`interesting areas and it makes a best estimation as to the
`popping the text out automatically at the viewing applica
`viewpoint metadata item and the polygonal overlay meta-
`tion , allowing the end - user to select the text to pop the text
`data item . Tool 1700 makes a best estimation as to the best
`out . Some embodiments may add other visual schemes to
`view and as to the polygons that should be generated in order 40 enhance aesthetic or artistic effect by , e.g. , enlarging text
`to hide the uninteresting parts . In many cases , the image
`with a bounce or other animated sequence make the expe
`processing has a particularly good chance of working
`rience more like video . In fact , any of a variety of visual
`because , for many kinds of content , such as comics and other
`effects can be added to the underlying content through use
`visually - oriented books , such content typically has white
`of metadata .
`backgrounds or solid or simple gradient backgrounds with 45
`In one example , the images are preprocessed to generate
`lines that typically follow the black lines that define the area
`views wherein the text , itself is the focus ( e.g. , as in FIG . 4 ) .
`as well . It is possible to do computer vision or image
`In other examples , the text is magnified within a graphic cell .
`processing algorithms that identify those lines with high
`In some instances where there are multiple text boxes ,
`accuracy . While the examples herein mention comic book
`magnifying them all in place would cause them to overlap .
`material for the underlying content , it is noted that any of a 50 One such solution is to add metadata that specifies parts of
`variety of graphical content can be modified by , or viewed
`the image that a given text box should not cover .
`When implemented via computer - executable instructions ,
`by , various embodiments of the invention .
`When the first estimate is acceptable , the user indicates
`various elements of embodiments of the present invention
`acceptance and moves the process on to the next task ( e.g. ,
`are in essence the software code defining the operations of
`moving on to the next cell ) . If the user does not think that 55 such various elements . The executable instructions or soft
`the computer's estimate is acceptable , then the user can
`ware code may be obtained from a readable medium ( e.g. ,
`manually manipulate the view and / or the polygons . In such
`a hard drive media , optical media , RAM , EPROM ,
`a case the computer has already identified the lines and the
`EEPROM , tape media , cartridge media , flash memory ,
`polygons and the underlying image , giving the human user
`ROM , memory stick , and / or the like ) . In fact , readable
`more to work with in defining the viewpoint and / or the 60 media can include any medium that can store information .
`polygonal overlay . In fact , some embodiments include a
`FIG . 19 illustrates an example computer system 1900
`snap - to feature to automatically snap views on polygons to
`adapted according to one embodiment of the present inven
`the next best item , so when a user drags a view , tool 1700
`tion . That is , computer system 1900 comprises an example
`snaps it to the next set of lines . In some instance , the
`system on which embodiments of the present invention may
`computer might get it all right , and the user might just verify 65 be implemented ( such as a viewing application or a tool to
`the computer's estimates . Thus , for simpler source images ,
`modify graphical content ) . Central processing unit ( CPU )
`the automatic embodiment has the potential to be quite
`1901 is coupled to system bus 1902. CPU 1901 may be any
`
`MemoryWeb Ex. 2045
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`US 11,061,524 B2
`
`7
`8
`particular embodiments of the process , machine , manufac
`general purpose or specialized purpose CPU . However , the
`ture , composition of matter , means , methods and steps
`present invention is not restricted by the architecture of CPU
`described in the specification . As one of ordinary skill in the
`1901 as long as CPU 1901 supports the inventive operations
`art will readily appreciate from the disclosure of the present
`as described herein . CPU 1901 may execute the various
`logical instructions according to embodiments of the present 5 invention , processes , machines , manufacture , compositions
`invention . For example , one or more CPUs , such as CPU
`of matter , means , methods , or steps , presently existing or
`1901 , may execute machine - level instructions according to
`later to be developed that perform substantially the same
`the exemplary operational flows described above regarding
`function or achieve substantially the same result as the
`the respective operation of the viewer application and / or the
`corresponding embodiments described herein may be uti
`10 li