USOO9466127B2
`
`(12) United States Patent
`Lindahl et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 9,466,127 B2
`Oct. 11, 2016
`
`(54)
`(75)
`
`(73)
`(*)
`
`(21)
`(22)
`(65)
`
`(51)
`
`(52)
`
`(58)
`
`(56)
`
`IMAGE ALTERATION TECHNIQUES
`
`Inventors: Aram Lindahl, Menlo Park, CA (US);
`Kelvin Chiu, Mountain View, CA (US)
`Assignee: Apple Inc., Cupertino, CA (US)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 317 days.
`Appl. No.: 12/895,725
`Filed:
`Sep. 30, 2010
`
`Prior Publication Data
`US 2012/OO81382 A1
`Apr. 5, 2012
`
`(2006.01)
`(2006.01)
`
`Int. C.
`G09G 5/00
`G06T II/00
`U.S. C.
`CPC ...................................... G06T II/00 (2013.01)
`Field of Classification Search
`CPC ............................. G06T 11/001: G06T 15/04
`USPC ......................................... 345/581, 619, 473
`See application file for complete search history.
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,758,043 A * 5/1998 Takizawa et al. ........... 358,116
`5,886,697 A * 3/1999 Naughton et al. ............ 345/473
`5,917,488 A
`6, 1999 Anderson et al.
`6,044,408 A * 3/2000 Engstrom et al. ..
`T19,328
`6,122,411 A * 9/2000 Shen et al. .................... 382,299
`6,369,822 B1
`4/2002 Peevers
`6,473,085 B1 * 10/2002 Brock et al. .................. 34.5/5O2
`6,847.467 B2
`1/2005 Obrador
`7,050,102 B1
`5, 2006 Vincent
`7,187.401 B2
`3/2007 Alhadef et al.
`7,412,360 B2 * 8/2008 Surazhsky et al. ............... 703/2
`7,586,032 B2
`9, 2009 Louis
`7,830,381 B2 * 1 1/2010 Lundstrom et al. .......... 345,428
`7.924,328 B2
`4/2011 Cazier
`7.969,447 B2 *
`6/2011 Scheepers et al. ........... 345,582
`
`9/2011 Milic-Frayling
`8,015,245 B2
`8,051,376 B2 11/2011 Adhikari
`8,073,203 B2 12/2011 Tang
`8,099.462 B2
`1/2012 Sheng
`2005/0273331 A1* 12/2005 Lu ................................. 704/246
`2006,0072019 A1
`4/2006 Stavely et al.
`2006, O181537 A1
`8, 2006 Vasan
`2007/0283269 A1* 12/2007 Obrador et al. .............. 71.5/723
`2008/0012856 A1
`1/2008 Yu et al. ....................... 345,424
`2008.0089.525 A1
`4/2008 Kauko
`2008/0298705 A1* 12/2008 Jeong ............................ 382,254
`2009, O163182 A1
`6, 2009 Gatti
`2009, O184849 A1
`7, 2009 Nasiri
`2009, 0207175 A1* 8, 2009 Warner ......................... 345/473
`(Continued)
`
`OTHER PUBLICATIONS
`
`Adobe, “After Effects / Effects and animation presets overview”.
`http://help.adobe.com/en US aftereffects/cs/using?
`WS387852.6689cb91655866c1103906c8 dea-7c30a.html.
`Adobe, “ADOBE(R) AFTER EFFECTS(R) CS4 help”.*
`“Photoshop Water Tutorial” http://www.secondpicture.com/tutori
`als/digital image processing/photoshop water tutorial.html.
`“Animated Spiral Tutorial'. http://www.hypnoticaddiction.com/tu
`torials/63-animated-spiral-tutorial.html.*
`Yufen Feng, “Research on Dynamic Water Surface and Ripple
`Animation', I.J. Information Technology and Computer Science,
`2010, 1, 18-24.*
`
`(Continued)
`Primary Examiner — Devona Faulk
`Assistant Examiner — Jin Ge
`(74) Attorney, Agent, or Firm — Blank Rome LLP
`(57)
`ABSTRACT
`Various techniques relating to the alteration of image data
`are provided herein. An electronic device capable of carry
`ing out such techniques may include data processing cir
`cuitry configured to receive image data and detect the
`occurrence of a device operation event (e.g., an audio event,
`a motion event, a location event, or an image event, and so
`forth). The data processing circuitry may alter the image
`data using a full-quality version or a reduced-quality version
`of an image alteration effect associated with the device
`operation event, depending on currently available resources.
`23 Claims, 18 Drawing Sheets
`
`
`
`STOREO
`ASEAA
`
`CRISINA
`MAGE
`AA
`
`NCAON OF
`ARASLE RESOURCES
`
`92
`
`30
`
`RCESSENGOs
`
`42
`
`FULREDUCED
`MiAEEFFECTS
`
`86
`
`DEWCE
`OPERATIONEVENT(s)
`
`84
`FURECE)
`ATERE)
`iAGEATA
`
`106
`ISDATA
`
`AUDIO
`ANAYSSOGC
`
`38
`TION
`SENSIKEGEWIE
`
`38
`3PS
`QCATION VICE
`
`
`
`28
`OSLAY
`
`98
`&OER
`
`98
`AEFIE
`
`108
`
`AIO
`NPSIGNAL
`
`AIO
`Pject
`
`AUDCAA
`
`Lightricks Ltd.
`EX1013
`Page 1 of 35
`
`

`

`US 9,466,127 B2
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2010/0303146 A1* 12/2010 Kamay .................... 375,240.02
`2010/0315423 A1* 12/2010 Ahn et al. ....
`... 345.426
`2012/0081382 A1* 4/2012 Lindahl et al. ............... 345,581
`
`
`
`9/2009 Loper
`2009/0219168 A1
`2009,0284541 A1 11, 2009 Liu
`2010/0045796 A1* 2/2010 Fraley et al. ................. 348/143
`2010/0066763 A1
`3/2010 MacDougall
`2010.0085379 A1
`4, 2010 Hishikawa
`2010/0134485 A1* 6/2010 Bhogal et al. ................ 345/419
`2010/0177247 A1* 7, 2010 Sekulovski et al.
`348,602
`2010, 0191459 A1* 7, 2010 Carter et al. .................. TO1,208
`2010/0214483 A1* 8, 2010 Gann et al. ................... 348,578
`2010/0286972 A1* 11/2010 Callegari ........................ TO3/10
`
`OTHER PUBLICATIONS
`GPU Gems Chapter 1.*
`GPU Gem Chapter 2.*
`Adobe, “After Effects Help f Managing and animating shape paths
`and masks'. http://helpx.adobe.com/after-effectsfusing/animating
`shape-paths-masks.html.
`U.S. Appl. No. 12/541,597, filed Aug. 14, 2009, Aram Lindahl et al.
`U.S. Appl. No. 12/556,380, filed Sep. 9, 2009, Lindahl.
`
`* cited by examiner
`
`Page 2 of 35
`
`

`

`U.S. Patent
`
`Oct. 11, 2016
`
`Sheet 1 of 18
`
`US 9,466,127 B2
`
`
`
`
`
`EanesONDIHOMLANpp
`
`*
`9}
`
`ANOVA
`
`0
`G
`
`AOVHOLS
`
`¥
`2f
`
`AVIASIC
`
`
`
`QIONOTONISSAGONd
`
`FOV
`
`ishyossao00ud
`
`ve
`
`SOV
`
`WALSASENS
`
`
`
`SOASONISNSS
`
`NOLLOW
`
`8
`
`SOASSad
`
`
`
`FOUNOSYSMOd
`
`SLuOdOf
`
`yi
`
`
`
`|(S}BUNLONUISLNANI
`
`
`
`SOASLAN!
`
`Olan
`
`|Ola
`
`Page 3 of 35
`
`Page 3 of 35
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Oct. 11, 2016
`
`Sheet 2 of 18
`
`US 9,466,127 B2
`
`
`
`WEAER
`
`cook collator
`
`Page 4 of 35
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 11, 2016
`Oct. 11, 2016
`
`
`
`Sheet 3 of 18
`Sheet 3 of 18
`
`US 9,466,127 B2
`
`US 9,466,127 B2
`
`Page 5 of 35
`
`Page 5 of 35
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 11, 2016
`Oct. 11, 2016
`
`Sheet 4 of 18
`Sheet 4 of 18
`
`US 9,466,127 B2
`US 9,466,127 B2
`
`
`
`ADIARONISNAS
`
`NOLLOW
`
`
`
`AQAANOLWOO7
`
`$d9
`
`06-
`
`
`
`
`INA
`
`
`ATNOUVHSdO/
`SIAC/
`
`90,7
`
`WiVGdSi
`
`bob?
`
`orony
`
`
`
`19071SISATYNY
`
`bg
`
`SSVAl
`
`fViv5fQaysiW
`/qsondsymins
`
`P|90ONISsa00ud|
`quof L_USHOLS
`ae¥6
`
`wo][vivasow
`
`JOVALfo
`
`98
`9g
`
`gigaddasvn||soinuas
`
`agondswring||zp|ONION!
`
`OfA
`
`26 iJONOWWOIGN
`
`SSOUNOSIAYFTVVAY
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`yOld
`
`WivdO1any
`
`|SOIASCLMI|ony|
`|olanya
`|TYNOISLN|
`
`Page 6 of 35
`
`Page 6 of 35
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Oct. 11, 2016
`
`Sheet 5 of 18
`
`US 9,466,127 B2
`
`801~oeiLf9017
`
`08?‘|SsOUunOsseyATWVAY
`
`40NOLLVORGNI
`
`wy,S0vilZee
`
`
`
`fF")“qayois|[CA]VivGffPivaapy|/ivASOV
`
`fasonaasmna,SIOOTONISSTOONd|TWNIOINO
`
`
`
`
`[Nei]:|S0iANaS
`quaiAprone|ONIDVINE
`Tinsf||
`
`IASONAG
`
`ACIASGONISNASWivddSI
`NOLLOW
`
`
`
`
`
`OWNOLYOOT
`
`Sd
`
`
`
`DIDO]SISATYNY
`
`Oony
`
`OP
`
`|SOIASGLAN|}oan|
`
`Page 7 of 35
`
`Page 7 of 35
`
`
`
`
`
`

`

`U.S. Patent
`
`Oct. 11, 2016
`
`Sheet 6 of 18
`
`US 9,466,127 B2
`
`
`
`Play lists
`X
`114 Genius
`NESTER U2
`Recently Added
`Recent Played
`125 mily
`
`
`
`
`
`
`
`:25
`
`-34
`
`Page 8 of 35
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 11, 2016
`Oct. 11, 2016
`
`Sheet 7 of 18
`Sheet 7 of 18
`
`US 9,466,127 B2
`US 9,466,127 B2
`
`
`
`
`
`Page 9 of 35
`
`Page 9 of 35
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 11, 2016
`Oct. 11, 2016
`
`Sheet 8 of 18
`Sheet 8 of 18
`
`US 9,466,127 B2
`US 9,466,127 B2
`
`
`
`oc
`
`é
`
`Ts)
`
`FG, 8
`FIG. 8
`
`Page 10 of 35
`
`Page 10 of 35
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 11, 2016
`Oct. 11, 2016
`
`Sheet 9 of 18
`Sheet 9 of 18
`
`US 9,466,127 B2
`US 9,466,127 B2
`
`
`
`
`
`CIOHSSMHLeg,—*||LOadd3
`
`SOIARG
`
`NOUVadad®
`
`
`
`
`
`1ALedsadOudINSAS
`
`6Old
`
`Page 11 of 35
`
`Page 11 of 35
`
`
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 11, 2016
`Oct. 11, 2016
`
`Sheet 10 of 18
`Sheet 10 of 18
`
`
`
`FIG.10
`
`US 9,466,127 B2
`US 9,466,127 B2
`
`Page 12 of 35
`
`Page 12 of 35
`
`

`

`
`
`
`
`U.S. Patent
`
`U.S. Patent md|NAG
`
`Oct. 11, 2016
`Oct. 11, 2016
`
`Sheet 11 of 18
`Sheet 11 of 18
`
`US 9,466,127 B2
`US 9,466,127 B2
`
`SOIAaG
`
`NOLWYAdO
`
`Page 13 of 35
`
`Page 13 of 35
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 11, 2016
`Oct. 11, 2016
`
`Sheet 12 of 18
`Sheet 12 of 18
`
`US 9,466,127 B2
`US 9,466,127 B2
`
`98h
`
`
`
`
`
`__|10ad43
`
`002
`
`-CIOHSSMHL
`
`SISVIVAY
`
`SSONNOSFY
`
`
`
`Page 14 of 35
`
`Page 14 of 35
`
`
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 11, 2016
`Oct. 11, 2016
`
`Sheet 13 of 18
`Sheet 13 of 18
`
`US 9,466,127 B2
`US 9,466,127 B2
`
`
`
`
` 146
`FIG.13
`
`EFFECTPosscattrtescnaeyense
` 146
`
`S.
`N
`xLene
`| WO*
`
`22 22
`
`NY
`
`
`
`
`
`206-~L
`
`Oo win
`3 a
`= Se
`“A as
`2 9>
`7 <i
`he
`aa
`
`971–
`
`Page 15 of 35
`
`Page 15 of 35
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 11, 2016
`
`Sheet 14 of 18
`
`US 9,466,127 B2
`US 9,466,127 B2
`
`
`
`4
`
`“
`
`4
`
`j
`
`4
`
`ViDid
`
`blo-
`Loads
`
`b(IOHSAYHL
`
`QL
`
`Losias
`
`A@CIOHSSMHL
`
`abe
`LOaads
`
`©(IOHSSYHL
`
`|SHOUNOSSY|FeVVAY
`
`Page 16 of 35
`
`Page 16 of 35
`
`

`

`U.S. Patent
`
`Oct. 11, 2016
`
`Sheet 15 of 18
`
`US 9,466,127 B2
`
`#3| ±| VIVG dS
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Page 17 of 35
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 11, 2016
`Oct. 11, 2016
`
`Sheet 16 of 18
`Sheet 16 of 18
`
`US 9,466,127 B2
`US 9,466,127 B2
`
`
`
`
`
`Page 18 of 35
`
`Page 18 of 35
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 11, 2016
`Oct. 11, 2016
`
`AW1dSI0
`
`re
`
`OE
`ÜÇ ·
`
`PINTS/aayarwfFiTina
`
`___fvwasow|_|amte||SOTONISSSOONdFFawaipriofT]YCOORC
`
`
`fee—
`
`AVM
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`be?
`
`Sheet 17 of 18
`Sheet 17 of 18
`
`So
`
`
`
`|NSGOONDbae--seeeeeees
`
`98
`
`Tin|
`
`isioaadagowmilbf|)AVOVLSY
`pcanL./j|VLVOVLSW|.|cocrercnemmnein
`
`|LOVELYG|osZ7.
`
`?v:Wik«!-----------------------------------------------------
`
`Boren:Dobeee
`
`US 9,466,127 B2
`US 9,466,127 B2
`
`801
`
`ZiOld
`
`Page 19 of 35
`
`Page 19 of 35
`
`
`
`
`
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 11, 2016
`Oct. 11, 2016
`
`
`
`Sheet 18 of 18
`Sheet 18 of 18
`
`FIG.18
`
`US 9,466,127 B2
`US 9,466,127 B2
`
`Page 20 of 35
`
`Page 20 of 35
`
`

`

`1.
`IMAGE ALTERATION TECHNIQUES
`
`US 9,466,127 B2
`
`BACKGROUND
`
`10
`
`15
`
`30
`
`35
`
`The present disclosure relates generally to image data
`processing and, more particularly, to altering image data in
`response to device operation events.
`This section is intended to introduce the reader to various
`aspects of art that may be related to various aspects of the
`present techniques, which are described and/or claimed
`below. This discussion is believed to be helpful in providing
`the reader with background information to facilitate a better
`understanding of the various aspects of the present disclo
`Sure. Accordingly, it should be understood that these state
`ments are to be read in this light, and not as admissions of
`prior art.
`Consumer electronics increasingly combine multiple
`functionalities into a single portable electronic device. For
`example, in many cases cell phones and media players are
`no longer merely distinct devices, each with their own
`unique capabilities. Rather, cell phone and media player
`functionalities are often now merged into one multimedia
`device with a multitude of capabilities. Indeed, many mod
`ern cell phones, digital media players, and even laptop
`25
`computers are often capable of providing for a number of
`additional functionalities, which may include: playing video
`games, providing for GPS location and navigation functions,
`providing for network communication functions (e.g.,
`e-mail, text messaging, web-surfing, etc.), playing audio
`files, and displaying image data, the latter of which may
`include displaying image data (e.g., pictures and/or video
`files) stored on the electronic device, captured by the elec
`tronic device (e.g., using an integrated camera), or streamed
`and/or downloaded over a network, such as the Internet.
`When a user captures or views an image on Such an
`electronic device, the user also may desire to alter Such
`images with their own creative and artistic initiatives. Such
`altered images may differ in appearance from the original
`image, but may provide the creator an aesthetically pleasing
`viewing experience. Most conventional graphics editing
`environments, however, may require users to manually
`select image alteration effects, potentially limiting the cre
`ative gamut of altered images that may be created. In
`addition, not all devices may have the same image process
`ing capabilities. Indeed, some electronic devices may have
`significantly more or significantly less processing bandwidth
`than others, which could limit the range of potential image
`alteration effects available to a user.
`
`40
`
`45
`
`SUMMARY
`
`A Summary of certain embodiments disclosed herein is set
`forth below. It should be understood that these aspects are
`presented merely to provide the reader with a brief summary
`of these certain embodiments and that these aspects are not
`intended to limit the scope of this disclosure. Indeed, this
`disclosure may encompass a variety of aspects that may not
`be set forth below.
`Embodiments of the present disclosure relate to the altera
`tion of image data. An electronic device capable of carrying
`out Such techniques may include data processing circuitry
`configured to receive image data and detect the occurrence
`of a device operation event (e.g., an audio event, a motion
`event, a location event, or an image event, and so forth). The
`data processing circuitry may alter the image data using a
`full-quality version or a reduced-quality version of an image
`
`50
`
`55
`
`60
`
`65
`
`2
`alteration effect associated with the device operation event,
`depending on currently available resources.
`Various refinements of the features noted above may exist
`in relation to various aspects of the present disclosure.
`Further features may also be incorporated in these various
`aspects as well. These refinements and additional features
`may exist individually or in any combination. For instance,
`various features discussed below in relation to one or more
`of the illustrated embodiments may be incorporated into any
`of the above-described aspects of the present disclosure
`alone or in any combination. Again, the brief Summary
`presented above is intended only to familiarize the reader
`with certain aspects and contexts of embodiments of the
`present disclosure without limitation to the claimed subject
`matter.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`Various aspects of this disclosure may be better under
`stood upon reading the following detailed description and
`upon reference to the drawings in which:
`FIG. 1 is a simplified block diagram depicting compo
`nents of an electronic device that may include image altera
`tion functions, in accordance with aspects of the present
`disclosure;
`FIG. 2 is a front-facing view showing the electronic
`device of FIG. 1 in the form of a portable handheld elec
`tronic device, in accordance with aspects of the present
`disclosure;
`FIG. 3 is a back-facing view of the electronic device
`shown in FIG. 2;
`FIG. 4 is a schematic block diagram depicting a technique
`for altering image data in response to various events that
`may be detected during operation of the electronic device of
`FIG. 1, in accordance with aspects of the present disclosure;
`FIG. 5 is a schematic block diagram depicting another
`technique for altering image data in response to various
`events that may be detected during operation of the elec
`tronic device of FIG. 1, in accordance with aspects of the
`present disclosure;
`FIG. 6 is a series of screens that may be displayed on the
`electronic device of FIG. 2 for video playback, in accor
`dance with aspects of the present disclosure;
`FIG. 7 is a series of screens that may be displayed on the
`electronic device of FIG. 2 for acquiring live image data, in
`accordance with aspects of the present disclosure;
`FIG. 8 is a series of screens that may be displayed on the
`electronic device of FIG. 2 for viewing images stored on the
`device of FIG. 2, in accordance with aspects of the present
`disclosure;
`FIG. 9 illustrates an embodiment of a water ripple image
`alteration technique, in accordance with aspects of the
`present disclosure;
`FIG. 10 illustrates altered image data stored in an image
`file resulting from the image alteration technique of FIG. 9.
`in accordance with aspects of the present disclosure;
`FIG. 11 illustrates an embodiment of a depth-changing
`image alteration technique, in accordance with aspects of the
`present disclosure;
`FIG. 12 illustrates an embodiment of a depth-changing
`image alteration technique, in accordance with aspects of the
`present disclosure;
`FIG. 13 illustrates an embodiment of a brush-stroke
`image alteration technique, in accordance with aspects of the
`present disclosure;
`
`Page 21 of 35
`
`

`

`US 9,466,127 B2
`
`3
`FIG. 14 is a plot modeling various levels of image
`alteration effect that may be applied depending on currently
`available device resources;
`FIG. 15 is a schematic block diagram depicting the
`association of various events that may be detected during 5
`operation of the electronic device of FIG. 1 with image data,
`in accordance with aspects of the present disclosure;
`FIG. 16 is a perspective view of an image transfer from
`the electronic device of FIG. 1 to an electronic device having
`higher image processing bandwidth to enable higher-band
`width image alteration, in accordance with aspects of the
`present disclosure;
`FIG. 17 is a schematic block diagram depicting a tech
`nique for altering image data based at least partly on
`previously recorded various device operation events that are
`associated with image data, in accordance with aspects of
`the present disclosure; and
`FIG. 18 illustrates an embodiment of a series of image
`alteration techniques that may be applied based on recorded
`device operation events associated with image data, in
`accordance with aspects of the present disclosure.
`
`10
`
`15
`
`DETAILED DESCRIPTION
`
`4
`assigned to Apple Inc. of Cupertino, Calif., which is incor
`porated herein by reference in its entirety.
`Because different electronic devices may have different
`image processing capabilities, when a user desires to apply
`Such image alteration effects to an image or video, the
`electronic device may consider its currently available pro
`cessing resources. Thus, in Some embodiments, the elec
`tronic device may apply a full-quality version of an image
`alteration effect only when the currently available device
`processing resources exceed a threshold. When the currently
`available device processing resources do not exceed the
`threshold, the electronic device may apply a reduced-quality
`version of the same image alteration effect. The electronic
`device may always apply a full- or reduced-quality version
`of the image alteration effect depending on its capabilities,
`or may vary between them as available processing resources
`change in real time. In this way, the same basic image
`alteration effect may be used across a variety of electronic
`devices and/or while the available processing resources of
`an electronic device increase or decrease over time.
`In certain embodiments, the image alteration effects
`applied to an image in response to a device operation event
`may take place at a later time, using a different electronic
`device or using the same electronic device when Sufficient
`device processing resources Subsequently have become
`available. For such embodiments, the electronic device that
`captures the image data may record and associate the device
`operation events (e.g., audio events, motion events, location
`events, imaging events, and so forth, as discussed further
`below) with the image data by, for example, storing the
`device operation events as image file metadata. At a later
`time, the stored image data and associated device operation
`events may be used to perform image alteration according to
`Such previously recorded device operation events.
`Before continuing, several of the terms mentioned above,
`which will be used extensively throughout the present
`disclosure, will be first defined in order to facilitate a better
`understanding of disclosed Subject matter. For instance, as
`used herein, the term "image alteration' or “image manipu
`lation” or the like may be understood to mean the application
`of an image alteration effect to a digital image, which may
`be a still image (e.g., picture) or a moving image (e.g.,
`video/movie). An "image alteration effect” may be under
`stood to mean any type of image effect (such as a filter or
`shape manipulation effect) that alters or otherwise changes
`the appearance of an original image to which the image
`alteration effect is applied.
`By way of example only. Such image alteration effects
`may include any one of the various effects provided in the
`Photo Booth R software program (available from Apple Inc.
`of Cupertino, Calif.), such as sepia, black and white, glow,
`colored pencil, thermal camera, X-ray, pop art, comic, bulge,
`dent, twirl, Squeeze, mirror, light tunnel, fisheye, or stretch
`effects, to name just a few. Image alteration effects may also
`include other types of effects, such as a water reflection
`effect, a water ripple effect, a spiral effect, a depth-changing
`(“Zooming) effect, a brush-stroke effect, a night vision
`effect, and more. Thus, it should be understood that “image
`alteration' does not necessarily mean that the original image
`data is permanently modified, but rather that an image
`alteration effect is applied to the image data, Such that the
`desired alteration effect is visible when the image data is
`displayed by an electronic device for viewing a by user.
`Additionally, it should be understood that the application of
`an “image alteration effect” means that the applied effect
`changes the appearance of the image data in Some way other
`
`25
`
`30
`
`35
`
`45
`
`One or more specific embodiments of the present disclo
`sure will be described below. These described embodiments
`are only examples of the presently disclosed techniques.
`Additionally, in an effort to provide a concise description of
`these embodiments, all features of an actual implementation
`may not be described in the specification. It should be
`appreciated that in the development of any such actual
`implementation, as in any engineering or design project,
`numerous implementation-specific decisions must be made
`to achieve the developers specific goals, such as compli
`ance with system-related and business-related constraints,
`which may vary from one implementation to another. More
`over, it should be appreciated that such a development effort
`might be complex and time consuming, but would never
`theless be a routine undertaking of design, fabrication, and
`40
`manufacture for those of ordinary skill having the benefit of
`this disclosure.
`When introducing elements of various embodiments of
`the present disclosure, the articles “a,” “an,” and “the are
`intended to mean that there are one or more of the elements.
`The terms “comprising,” “including,” and “having are
`intended to be inclusive and mean that there may be addi
`tional elements other than the listed elements. Additionally,
`it should be understood that references to “one embodiment’
`or “an embodiment of the present disclosure are not
`intended to be interpreted as excluding the existence of
`additional embodiments that also incorporate the recited
`features.
`As discussed above, the creativity of altered image results
`that may be achieved using conventional graphics editing 55
`applications may be limited, as such conventional graphics
`editing environments may require the manual application of
`Such effects. In addition, various electronic devices may
`have various image processing capabilities. Accordingly, the
`present disclosure provides various techniques for automati
`cally applying image alteration effects to image data in
`response to one or more device operation events and varying
`the complexity of such image alteration effects depending on
`the currently available device processing resources of the
`electronic device. Such image alteration effects may include,
`for example, those disclosed by U.S. patent application Ser.
`No. 12/541,597, “IMAGE ALTERATION TECHNIQUES.”
`
`50
`
`60
`
`65
`
`Page 22 of 35
`
`

`

`US 9,466,127 B2
`
`10
`
`15
`
`5
`than merely altering the orientation of the image (e.g.,
`Switching from a portrait to a landscape view), as displayed
`on an electronic device.
`Further, the term “device operation event' or the like shall
`be understood to refer to certain events that occur during
`operation of the electronic device mentioned above, which
`may be used to trigger the application of an image alteration
`effect to a particular image. For instance, as will be dis
`cussed below, device operation events may include audio
`events, which may include certain audio properties detected
`during audio playback (e.g., a music file or the audio portion
`of a video file) or by analyzing audio signals received
`through an audio input device (e.g., a voice recording
`received via a microphone). Device operation events may
`also include motion-related events detected by a motion
`sensing device. Such as an accelerometer. In further embodi
`ments, image alteration effects may also be triggered by
`location events (e.g., determined by global positioning sat
`ellite (GPS) coordinates) and/or by image capture events,
`which may include the detection of certain lighting condi
`tions, exposure values, sharpness data, etc., via an imaging
`Subsystem of the electronic device. An image capture event
`may also include detecting the presence of a new object or
`Subject entering the imaging acquisition frame. Accordingly,
`the term “image alteration rule' or the like shall be under
`stood to refer to a defined relationship (e.g., defined through
`user preference settings) by which a particular device opera
`tion event triggers a particular image alteration effect. Fur
`ther, it should be understood that a “device operation event
`may refer to the occurrence of a single discrete event, or may
`refer to a continuous change in a particular parameter over
`time, such that image alteration is based upon the change in
`the parameter. It should also be understood that “device
`operation events' are meant to exclude specific 'on-de
`mand user commands or inputs that directly request the
`application of an image alteration effect to image data, Such
`as those received directly through a graphics editing pro
`gram or setting, as discussed above.
`As noted above, embodiments of the present disclosure
`may involve applying varying levels of image alteration
`effects depending on the currently available device process
`ing resources. Thus, while a “full-quality version of an
`image alteration effect may be sometimes applied, a
`“reduced-quality version of the image alteration effect may
`be applied at other times. Accordingly, the term “full-quality
`45
`version' of an image alteration effect should be understood
`to refer to the application of a particular an image alteration
`effect in full (i.e., without purposeful reduction in complex
`ity). Likewise, the term “reduced-quality version' should be
`understood to refer to the application of an image alteration
`effect with reduced complexity from the “full-quality ver
`Sion.” That is, although a “full-quality version of an image
`alteration effect generally may be perceived by a user to be
`of higher quality (e.g., more aesthetically pleasing or more
`realistic), the “reduced-quality version' of Such an image
`alteration effect should require fewer device resources while
`producing a similar effect as the “full-quality version of the
`image alteration effect. By way of example, a “full-quality
`version' of a water ripple effect may involve water simu
`lation that includes randomization, attenuation, and/or cal
`culations of physics, while a “reduced-quality version' may
`involve simple sine waves.
`As noted above, the currently available device processing
`resources may affect whether a “full-quality version' or a
`“reduced-quality version of an image alteration effect is
`applied. As used herein, the terms “currently available
`device processing resources.” “processing bandwidth, and
`
`55
`
`6
`the like refer to a current capability of an electronic device
`to process data (e.g., a capability to perform a particular
`image alteration effect). By way of example, the current
`capability of an electronic device to process data may be
`impacted by a current CPU processing load (e.g., an amount
`of idle time), a current GPU processing load (e.g., a current
`workload), currently available memory resources (e.g., a
`total amount of memory or an available bandwidth of
`memory), and/or current available storage capacity.
`With the foregoing in mind, FIG. 1 is a block diagram
`illustrating an example of an electronic device 10, which
`may implement the image alteration techniques discussed
`herein. The electronic device 10 may be any type of elec
`tronic device that includes capabilities for displaying image
`data, which may include still images (e.g., pictures) or
`moving images (e.g., video). Such as a portable media
`player, a mobile phone, a laptop computer, or the like. By
`way of example only, the electronic device 10 may be a
`portable electronic device, such as a model of an iPodR),
`iPhone(R), or iPad(R) available from Apple Inc. In another
`embodiment, the electronic device 10 may be a desktop or
`laptop computer, including a MacBook(R), MacBook.(R) Pro,
`MacBook Air R, iMac(R), Mac(R) Mini, or Mac ProR), also
`available from Apple Inc. In further embodiments, the
`electronic device 10 may be a model of an electronic device
`from another manufacturer that is capable of displaying
`image data. As will be discussed further below, the elec
`tronic device 10 may include circuitry or logic (e.g., image
`processing logic 30) configured to process image data in
`response to one or more device operation events, which may
`include audio-related events, motion-related events, loca
`tion-related events, or image capture events, to name just a
`few.
`As shown in FIG. 1, the electronic device 10 may include
`various internal and/or external components which contrib
`ute to its function. Those of ordinary skill in the art should
`appreciate that the various functional blocks shown in FIG.
`1 may represent hardware elements (including circuitry),
`Software elements (including computer code stored on a
`computer-readable medium) or a combination of both hard
`ware and software elements. It should further be noted that
`FIG. 1 is merely one example of a particular implementation
`and is intended to illustrate the types of components that
`may be present in the electronic device 10. For example, in
`the presently illustrated embodiment, these components may
`include input/output (I/O) ports 12, input structures 14, one
`or more processors 16, memory 18, nonvolatile storage 20,
`one or more networking devices 24, a power source 26, a
`display 28, and image processing logic 30. The electronic
`device 10 may additionally include an imaging Subsystem
`34, a motion sensing device 36, a positioning device 38, and
`an audio input device 40, all of which may contribute to
`operation events that may be utilized by imaging processing
`logic 30 to trigger the application of one or more image
`alteration effects to certain image data.
`Before continuing, it should be noted that the image
`processing logic 30, the processor(s) 16, the GPU 42, and/or
`other data processing circuitry of the electronic device 10
`generally may be referred to herein as "data processing
`circuitry.” Such data processing circuitry may be embodied
`wholly or in part as Software, firmware, hardware, or any
`combination thereof. Furthermore, the data processing cir
`cuitry may be a single contained processing module or may
`be incorporated wholly or partially within any of the other
`elements within electronic device 10. Additionally or alter
`natively, the data processing circuitry may be partially
`
`25
`
`30
`
`35
`
`40
`
`50
`
`60
`
`65
`
`Page 23 of 35
`
`

`

`US 9,466,127 B2
`
`10
`
`15
`
`25
`
`30
`
`35
`
`7
`embodied within electronic device 10 and partially embod
`ied within another electronic device connected to the elec
`tronic device 10.
`With regard to each of the illustrated components, the I/O
`ports 12 may include ports configured to connect to a variety
`of external devices, such as headphones or other electronic
`devices. The input structures 14 may provide user input or
`feedback to processor(s) 16. For example, the input struc
`tures 14 may include buttons, sliders, Switches, and so forth.
`In certain embodiments, the input structures 14 may include
`a touch-sensitive mechanism of the display 28.
`In the electronic device 10 of FIG. 1, the processor(s) 16
`and/or other data processing circuitry may be operably
`coupled