( 12 ) United States Patent
`Leuthardt et al .
`
`US 11,042,623 B2
`( 10 ) Patent No .:
`( 45 ) Date of Patent :
`Jun . 22 , 2021
`
`US011042623B2
`
`( 54 )
`
`EXPRESSION RECOGNITION IN
`MESSAGING SYSTEMS
`( 71 ) Applicant : Face To Face Biometrics , Inc. , St.
`Louis , MO ( US )
`( 72 ) Inventors : Eric Leuthardt , St. Louis , MO ( US ) ;
`Scott Stern , St. Louis , MO ( US )
`( 73 ) Assignee : FaceToFace Biometrics , Inc. , St.
`Louis , MO ( US )
`Subject to any disclaimer , the term of this
`patent is extended or adjusted under 35
`U.S.C. 154 ( b ) by 0 days .
`( 21 ) Appl . No .: 16 / 831,432
`( 22 ) Filed :
`Mar. 26 , 2020
`( 65 )
`Prior Publication Data
`US 2020/0226239 A1
`Jul . 16 , 2020
`
`( * ) Notice :
`
`Related U.S. Application Data
`Continuation of application No. 16 / 397,787 , filed on
`Apr. 30 , 2019 , which is a continuation of application
`( Continued )
`
`( 56 )
`
`6,836,846 B1
`6,873,710 B1
`
`References Cited
`U.S. PATENT DOCUMENTS
`12/2004 Kanevsky et al .
`3/2005 Cohen Solal et al .
`( Continued )
`FOREIGN PATENT DOCUMENTS
`
`WO
`WO
`WO
`
`2014127065 A2
`2014127333 A1
`2014127065 A3
`
`8/2014
`8/2014
`10/2014
`
`OTHER PUBLICATIONS
`Keerthana Chintapudi ; Online Signature Verification for Secure
`Transactions ; International Journal of Scientific Engineering and
`Technology Research vol . 04 , Issue No. 35 , Aug. 2015 , pp . 7283
`7286 .
`( Continued )
`Primary Examiner Scott C Anderson
`( 74 ) Attorney , Agent , or Firm - Armstrong Teasdale LLP
`( 57 )
`ABSTRACT
`Some embodiments include a messaging system capable of
`expression - based communication and / or expression - based
`actions . The messaging system can run on a computing
`device . For example , the messaging system monitors a
`camera feed from a camera of the computing device to detect
`a biometric signature when a messaging interface of the
`messaging application is actively being used . The messaging
`system can match the detected biometric signature against a
`known profile utilizing
`facial recognition process to
`authenticate an operating user to use the messaging appli
`cation . The messaging system can determine a human
`expression based on the detected biometric signature utiliz
`ing an expression recognition process to associate a contex
`tual tag with an activity on the messaging interface . The
`messaging system can then communicate with a message
`server system to associate the contextual tag with content
`presented to the operating user or a conversation participated
`by the operating user via the messaging application .
`20 Claims , 8 Drawing Sheets
`
`( 63 )
`
`( 51 )
`
`( 58 )
`
`( 52 )
`
`U.S. CI .
`CPC
`
`Int . Cl .
`G06F 21/32
`G06Q 30/02
`
`( 2013.01 )
`( 2012.01 )
`( Continued )
`
`GO6F 21/32 ( 2013.01 ) ; G06F 3/00
`( 2013.01 ) ; G06F 3/011 ( 2013.01 ) ; G06F
`3/0304 ( 2013.01 ) ;
`
`( Continued )
`Field of Classification Search
`CPC ..... GO6F 3/00 ; G06F 3/011 ; GO6F 3/0304 ;
`G06F 21/629 ; G06F 2203/011 ;
`( Continued )
`
`IPR2023-00833
`Apple EX1001 Page 1
`
`

`

`US 11,042,623 B2
`Page 2
`
`Related U.S. Application Data
`No. 14 / 643,810 , filed on Mar. 10 , 2015 , now Pat . No.
`10,275,583 .
`( 60 ) Provisional application No. 62 / 051,031 , filed on Sep.
`16 , 2014 , provisional application No. 61 / 985,059 ,
`filed on Apr. 28 , 2014 , provisional application No.
`61 / 950,423 , filed on Mar. 10 , 2014 .
`( 51 ) Int . Ci .
`( 2018.01 )
`H04W 4/21
`( 2013.01 )
`G06F 21/62
`( 2006.01 )
`H04L 29/06
`( 2021.01 )
`H04W 12/06
`( 2006.01 )
`G06F 3/01
`( 2006.01 )
`G06F 3/03
`( 2006.01 )
`G06F 3/00
`( 2009.01 )
`H04W 4/12
`( 2006.01 )
`G06K 9/00
`( 2021.01 )
`H04W 12/65
`( 52 ) U.S. CI .
`G06F 21/629 ( 2013.01 ) ; G06Q 30/0271
`CPC
`( 2013.01 ) ; H04L 63/0861 ( 2013.01 ) ; H04W
`4/21 ( 2018.02 ) ; H04W 12/06 ( 2013.01 ) ; GOOF
`2203/011 ( 2013.01 ) ; G06K 9/00302 ( 2013.01 ) ;
`H04W 4/12 ( 2013.01 ) ; H04W 12/65 ( 2021.01 )
`( 58 ) Field of Classification Search
`CPC
`GO6F 21/32 ; G06Q 30/0271 ; H04W 4/12 ;
`HO4W 4/21 ; H04W 12/06 ; H04W
`12/00504
`See application file for complete search history .
`References Cited
`U.S. PATENT DOCUMENTS
`7,698,745 B2
`8,203,530 B2
`8,336,766 B1
`8,600,120 B2
`8,922,480 B1
`9,008,416 B2
`9,104,905 B2
`9,104,907 B2
`9,105,119 B2
`9,189,797 B2
`9,443,167 B2
`9,450,957 B1
`9,817,960 B2
`10,275,583 B2
`2002/0019852 A1
`2002/0116508 A1
`2003/0135740 A1
`
`4/2010 Pauker et al .
`6/2012 Chiang
`12/2012 Miller et al .
`12/2013 Gonion et al .
`12/2014 Freed et al .
`4/2015 Movellan et al .
`8/2015 Whitehill et al .
`8/2015 Whitehill et al .
`8/2015 Whitehill et al .
`11/2015 Ghosh et al .
`9/2016 Movellan et al .
`9/2016 Daniel
`11/2017 Leuthardt et al .
`4/2019 Leuthardt et al .
`2/2002 Bahar
`8/2002 Khan et al .
`7/2003 Talmor et al .
`
`( 56 )
`
`2003/02 14535 A1 11/2003 Hymel
`2005/0251688 A1 11/2005 Nanavati et al .
`2006/0282466 Al 12/2006 Yasukaga et al .
`2007/0208610 Al
`9/2007 Pisaris Henderson et al .
`G06F 3/015
`2008/0218472 A1 *
`9/2008 Breen
`345/156
`2009/0016617 Al
`1/2009 Bregman - Amitai et al .
`2009/0141895 A1
`6/2009 Anderson et al .
`2010/0014720 A1
`1/2010 Hoyos et al .
`2010/0125799 A1 *
`5/2010 Roberts
`2010/0141662 A1
`6/2010 Storey et al .
`2010/0205667 A1
`8/2010 Anderson et al .
`2011/0125844 A1
`5/2011 Collier et al .
`2011/0296324 A1 * 12/2011 Goossens
`
`G06F 3/011
`715/757
`
`G06F 3/04883
`715/763
`G06T 13/40
`345/473
`
`2011/0304629 A1 * 12/2011 Winchester
`
`2012/0011120 A1
`2012/0054039 Al
`2012/0054311 A1
`2012/0162495 A1 *
`
`1/2012 Burnand et al .
`3/2012 Patwa et al .
`3/2012 Mizuno et al .
`6/2012 Ogawa
`
`HO4N 5/232123
`348/345
`G06F 3/048
`715/744
`HO4N 7/147
`348/51
`G06K 9/00315
`348/239
`G06F 21/40
`382/118
`
`G06F 3/0484
`348/77
`
`1/2013 Kosta et al .
`2013/0005443 Al
`7/2013 Kim
`2013/0185648 A1 *
`9/2013 Herdy
`2013/0242064 Al *
`2013/0300900 A1 * 11/2013 Pfister
`2013/0322705 A1 * 12/2013 Wong
`2014/0085191 A1
`3/2014 Gonion et al .
`2014/0198958 Al
`7/2014 Nathan et al .
`2014/0201844 Al
`7/2014 Buck
`2014/0314284 Al 10/2014 Movellan et al .
`2014/0316881 A1 10/2014 Movellan et al .
`2014/0351163 A1
`11/2014 Tussy
`2014/0366158 Al 12/2014 Han et al .
`2014/0372301 A1 12/2014 Anamanamuri
`2015/0015690 A1 *
`1/2015 Roh
`2015/0033017 Al
`1/2015 Uliyar et al .
`2015/0046996 A1
`2/2015 Slaby et al .
`2015/0049953 Al
`2/2015 Movellan et al .
`2015/0254447 A1
`9/2015 Leuthardt et al .
`2015/0287054 A1 10/2015 Movellan et al .
`2015/0381575 Al 12/2015 Bhargav - Spantzel et al .
`
`OTHER PUBLICATIONS
`“ Google Scholar Search ; Shoulder Surfing Prevention ; available at
`http : 1 / scholar.google.corn / scholar ? hl = en & as_sdt = 0,47 & q = shoulder +
`surfing + prevention [ last visited Jun . 26 , 2017 9:59:33 AM ] . ”
`Alan Mislove ; A secure , resilient , cooperative messaging system ;
`researchgate.net : 2003 ; p . 1-6 .
`* cited by examiner
`
`IPR2023-00833
`Apple EX1001 Page 2
`
`

`

`U.S. Patent
`
`Jun . 22 , 2021
`
`Sheet 1 of 8
`
`US 11,042,623 B2
`
`FIG . IA
`
`IPR2023-00833
`Apple EX1001 Page 3
`
`

`

`U.S. Patent
`
`Jun . 22 , 2021
`
`Sheet 2 of 8
`
`US 11,042,623 B2
`
`0
`
`]
`
`FIG . 1B
`
`IPR2023-00833
`Apple EX1001 Page 4
`
`

`

`U.S. Patent
`
`Jun . 22 , 2021
`
`Sheet 3 of 8
`
`US 11,042,623 B2
`
`]
`
`FIG . 1C
`
`IPR2023-00833
`Apple EX1001 Page 5
`
`

`

`U.S. Patent
`
`Jun . 22 , 2021
`
`Sheet 4 of 8
`
`US 11,042,623 B2
`
`Client Device
`202
`
`Msg . Interface
`206
`Biometric
`Security
`Engine
`207
`
`Sensor ( s )
`214
`
`Exp . Recog .
`Engine
`216
`
`Client Device
`202
`
`Msg . Interface
`206
`Biometric
`Security
`Engine
`207
`
`Sensor ( s )
`214
`
`Exp . Recog .
`Engine
`216
`
`Client Device
`202
`
`Msg . Interface
`206
`Biometric
`Security
`Engine
`207
`
`Sensor ( s )
`214
`
`Exp . Recog .
`Engine
`216
`
`User Profiles DB
`208
`
`Messaging Platform
`204
`
`Authorization
`DB
`210
`
`Stimulus
`Response DB
`218
`
`FIG . 2
`
`Messaging System
`200
`
`IPR2023-00833
`Apple EX1001 Page 6
`
`

`

`U.S. Patent
`
`Jun . 22 , 2021
`
`Sheet 5 of 8
`
`US 11,042,623 B2
`
`Monitoring a video feed from a camera of the
`computing device to detect a biometric signature
`when a messaging interface of the messaging
`application is actively being used
`302
`
`300
`
`Matching the detected biometric signature against a
`known profile utilizing a facial recognition process to
`authenticate an operating user to use the messaging
`application
`304
`
`Determining a human expression based on the
`detected biometric signature utilizing an expression
`recognition process to associate a contextual tag
`with an activity on the messaging interface
`306
`
`Communicating with the message server system to
`associate the contextual tag with a conversation
`participated by the operating user via the messaging
`application
`308
`
`Activating or deactivating an interactive component
`of the messaging interface in response to detecting a
`specific expression via the expression recognition
`process
`310
`
`FIG . 3
`
`IPR2023-00833
`Apple EX1001 Page 7
`
`

`

`U.S. Patent
`
`Jun . 22 , 2021
`
`Sheet 6 of 8
`
`US 11,042,623 B2
`
`Monitoring a video feed from a camera to detect a
`biometric signature when a messaging interface of
`the messaging application is actively being used
`402
`
`400
`
`Matching the detected biometric signature against a
`known profile utilizing a facial recognition process to
`authenticate an operating user to use the messaging
`application
`404
`
`Determining a human expression based on the
`detected biometric signature utilizing an expression
`recognition process to associate a contextual tag
`with an activity on the messaging interface
`406
`
`Receiving an expression trigger condition associated
`with the targeted advertisement from a message
`server system or an advertisement service server
`408
`
`Presenting a targeted advertisement on the
`messaging interface based on the contextual tag and
`an identity of the operating user
`410
`
`FIG . 4
`
`IPR2023-00833
`Apple EX1001 Page 8
`
`

`

`U.S. Patent
`
`Jun . 22 , 2021
`
`Sheet 7 of 8
`
`US 11,042,623 B2
`
`Implementing a messaging service to facilitate a
`messaging application executing on a first computing
`device
`502
`
`500
`
`Authenticating an operating user of the first
`computing device based on a facial recognition
`process in conjunction with the messaging
`application when the operating user is actively using
`a messaging interface of the messaging application
`504
`
`Receiving an expression status from the messaging
`application based on an expression recognition
`process
`506
`
`Associating the expression status in a stimulus
`response database maintained by the message
`server system
`508
`
`Generating a query interface to the stimulus
`response database to provide associated expressions
`in response to a stimulus identifier
`510
`
`FIG . 5
`
`IPR2023-00833
`Apple EX1001 Page 9
`
`

`

`U.S. Patent
`
`Jun . 22 , 2021
`
`Sheet 8 of 8
`
`US 11,042,623 B2
`
`600
`
`650
`Storage Adapter
`
`To / From
`Persistent
`Storage
`
`610
`Processor ( s )
`
`Memory 620
`
`Code 670
`
`630
`
`640
`Network Adapter
`
`To / From
`clients
`and / or other
`nodes
`
`FIG . 6
`
`IPR2023-00833
`Apple EX1001 Page 10
`
`

`

`5
`
`25
`
`US 11,042,623 B2
`
`1
`EXPRESSION RECOGNITION IN
`MESSAGING SYSTEMS
`
`2
`FIG . 3 is a flow chart of a method of operating a
`messaging application on a computing device that imple
`ments an expression recognition process , in accordance with
`various embodiments .
`CROSS REFERENCE TO RELATED
`FIG . 4 is a flow chart of a method of operating a
`APPLICATIONS
`messaging application on a computing device capable of
`delivering advertisements , in accordance with various
`This application is a continuation of U.S. patent applica
`tion Ser . No. 16 / 397,787 , entitled “ EXPRESSION RECOG embodiments .
`FIG . 5 is a flow chart of a method of operating a message
`NITION IN MESSAGING SYSTEMS , ” which was filed on
`Apr. 30 , 2019 , which is a continuation of U.S. patent 10 server system that facilitates conversations between com
`puting devices , in accordance with various embodiments .
`application Ser . No. 14 / 643,810 , entitled “ EXPRESSION
`FIG . 6 is a block diagram of an example of a computing
`RECOGNITION IN MESSAGING SYSTEMS , ” which was
`device , which may represent one or more computing devices
`filed on Mar. 10 , 2015 , which claims the benefit of U.S.
`or servers described herein , in accordance with various
`Provisional Patent Application No. 61 / 950,423 , entitled 15 embodiments .
`“ BIOMETRIC FOR MOBILE ACCESS , " which was filed
`The figures depict various embodiments of this disclosure
`on Mar. 10 , 2014 ; U.S. Provisional Patent Application No.
`for purposes of illustration only . One skilled in the art will
`61 / 985,059 , entitled “ USE OF BIOMETRIC FOR ACCESS
`readily recognize from the following discussion that alter
`TO DATA DEVICE AND ASSOCIATED SOLUTIONS
`native embodiments of the structures and methods illustrated
`FOR DIFFICULT BIOMETRIC READING SCENARIOS , ” 20 herein may be employed without departing from the prin
`ciples of the invention described herein .
`which was filed on Apr. 28 , 2014 ; and U.S. Provisional
`Patent Application No. 62 / 051,031 , entitled " EXPRESSION
`DETAILED DESCRIPTION
`RECOGNITION IN MESSAGING SYSTEMS , ” which was
`filed on Sep. 16 , 2014 ; which are all incorporated by
`Disclosed is a technology that pertains to protecting
`reference herein in their entirety .
`messages through a messaging system via a biometric
`security mechanism . The messaging system includes a mes
`RELATED FIELD
`sage server system comprising one or more computer serv
`ers and messaging applications running on end - user devices .
`At least one embodiment of this disclosure relates gen-
`erally to an electronic messaging system , and in particular to 30 The end - user devices can include personal computers , smart
`privacy and security of an electronic messaging system .
`phones , tablets , wearable devices , gaming consoles , smart
`TVs , other electronic gadgets , or any combination thereof .
`The messaging applications act as independent agents for
`BACKGROUND
`the messaging system . For example , a messaging application
`With the wide availability of mobile devices , in some 35 may be installed on a general - purpose operating system
`( e.g. , Windows , android , iOS , etc. ) . The messaging appli
`areas / cultures , electronic messaging is becoming an integral
`part of a person's life . Because of this , privacy and security
`cation can also be implemented by client - side script ( e.g. ,
`concerns arise over the use of such systems . Conventional
`JavaScript ) that may be executed on a web browser of the
`technology protects against privacy violations by providing
`end - user devices .
`a screen lock on a mobile device whenever the mobile 40
`The biometric security mechanism is implemented via the
`device is not used . An authorized user can unlock the screen
`messaging application . Unlike the conventional security
`by typing a passcode into the mobile device . However , the
`mechanisms for a recipient user device that protect privacy
`passcode is knowledge that is transferable , and hence may
`only of the user of the recipient user device , the disclosed
`be stolen . Furthermore , the screen lock prevents access to
`biometric security mechanism also protects the privacy and
`other applications on the mobile device , making it unnec- 45 security of the sender of the message . This enables a new
`essarily inconvenient . Other solutions protect against viola-
`paradigm of protection for stakeholders ( e.g. , the message
`tion by scheduling the destruction of a message to ensure
`senders ) who are conventionally dependent solely on the
`message recipients to protect their interests .
`that its content therein does not survive indefinitely .
`In various embodiments , the disclosed technology uses
`BRIEF DESCRIPTION OF THE DRAWINGS
`50 sensors in end - user devices to monitor and profile an end
`user . Such profiling can serve to ensure security ( for
`FIG . 1A is a diagram illustrating a messaging application
`example , by utilizing biometric recognition , such as facial
`recognition or ear recognition ) . Such profiling can also serve
`of a mobile device revealing content of a message when a
`to enhance context mining ( for example , by utilizing expres
`facial profile associated with a recipient account is recog
`55 sion recognition ) . In some embodiments , the expression
`nized , in accordance with various embodiments .
`FIG . 1B is a diagram illustrating a messaging application
`recognition can be implemented without implementing the
`of a mobile device hiding content of a message when a facial
`biometric recognition . In some embodiments , the biometric
`profile associated with a recipient account is not detected , in
`recognition can be implemented without implementing the
`expression recognition .
`accordance with various embodiments .
`FIG . 1C is a diagram illustrating a messaging application 60 Security
`of a mobile device hiding content of a message when a facial
`The biometric security mechanism implements a biomet
`profile associated with a recipient account is detected and a
`ric recognition process to verify one or more identities as
`second unauthorized facial profile is detected , in accordance
`authorized by the message senders . For example , the bio
`metric recognition process may be based on facial recogni
`with various embodiments .
`FIG . 2 is a block diagram of a system environment of a 65 tion , ear recognition , silhouette recognition , speaker recog
`messaging system implementing a biometric security
`nition , fingerprint recognition , device motion pattern
`mechanism , in accordance with various embodiments .
`recognition , contact pattern recognition , etc. The biometric
`
`IPR2023-00833
`Apple EX1001 Page 11
`
`

`

`US 11,042,623 B2
`
`3
`4
`may send a message with a privacy setting indicating Nancy
`security mechanism can detect biometric patterns utilizing a
`as the recipient account . In some cases , the privacy setting
`sensor , such as a camera , a microphone , an accelerometer , a
`can add an attribute of the recipient profile being female .
`touch sensor , a gyroscope , etc. The biometric security
`The privacy shield may be implemented , for example , as
`mechanism can then profile ( e.g. , by recording patterns and
`then training one or more computational functions that 5 a screen lock , a scrambling of the message content , a blur of
`recognize one or more shared attributes or characteristic of
`the message content , a blackout of the message content , or
`the patterns ) the biometric patterns of one or more users into
`any combination thereof . In some embodiments , the bio
`profile model utilizing machine learning algorithms , such
`metric security mechanism continuously loops through the
`as support vector machines , Gaussian mixture models , hid-
`one or more types of biometric recognition processes . In
`den Marcov models , etc. Based on the profiles of the users , 10 some embodiments , the biometric security mechanism dis
`the biometric security mechanism can recognize a user in
`engages the privacy shield whenever an authorized biomet
`real - time by matching the biometric pattern detected by a
`ric profile is recognized and detected . In some embodiments ,
`the biometric security mechanism engages the privacy shield
`sensor with the profile model .
`The biometric security mechanism is instantiated when-
`whenever a non - authorized biometric profile is detected .
`ever the messaging interface of the message application is 15 A sender account or a recipient account can be associated
`actively opened and running on an end - user device ( e.g. ,
`with a single individual or a group of individuals . In the case
`such that the mechanism is active while the message appli-
`of a group of individuals , biometric profiles of every mem
`cation is open ) . The biometric security mechanism utilizes
`ber are associated with the sender account or the recipient
`one or more types of biometric recognition processes to
`account .
`determine whether the people who have access to the 20 Expression Recognition
`end - user device ( e.g. , looking at the screen of the end - user
`The use of the biometric security mechanism provides a
`device ) are consistent with the privacy settings of the sender
`dependable authentication process that improves or guaran
`tees privacy of the users of the messaging system . Accord
`account and the recipient account .
`To use the messaging application on a user device , a user
`ingly , the users can trust that the biometric recognition
`may have to identify a messaging system account associated 25 process is used for protection instead of exploitation .
`with the messaging system . The biometric security mecha-
`Because of this trust , an expression recognition process can
`nism uses the one or more types of biometric recognition
`be run concurrently to the biometric recognition process .
`processes to ensure that an operator using the messaging
`That is , a user , who otherwise would not use a messaging
`application matches a biometric profile of the messaging
`application with an expression recognition process , would
`system account . This procedure ensures that a third party 30 use that same messaging application if the biometric recog
`( e.g. , a brother or a classmate ) cannot send messages on
`nition process were there to guarantee security and privacy .
`behalf of the user . In the case that there are multiple
`The disclosed technology can implement expression rec
`operators for the same end - user device , this procedure also
`ognition process in addition to the biometric recognition
`ensures that independent private sessions of using the mes-
`process to provide additional contextual information asso
`saging application can be established based on automatic 35 ciated with a user's emotional / mood state when using a
`biometric recognition .
`messaging application . The expression recognition of a user
`The biometric security mechanism can provide customi-
`of the messaging system can provide several benefits . For
`zable privacy shields per message or conversation . These
`example , the message server system can maintain a stimulus
`privacy shields may be customized by privacy settings
`response database that maps associations between stimuli
`dictated by the sender of the message or the initiator of the 40 presented on the messaging interface and expressions rec
`conversation . The biometric security mechanism can use the
`ognized by the expression recognition process . The message
`one or more types of biometric recognition processes to
`server system can then generate and provide a query inter
`ensure that the operator using the messaging application
`face to present consumer data ( e.g. , anonymized consumer
`matches an authorized biometric profile in accordance with
`data ) based on the stimulus response database for advertis
`a privacy setting of an individual conversation or message . 45 ers , researchers , or business intelligence departments .
`For example , the default privacy setting may be that the
`For another example , the messaging application can use
`the expression recognition process to personalize the mes
`message or conversation is revealed when a biometric
`profile of the recipient account is recognized by the biomet-
`saging application . In some cases , based on the responding
`ric security mechanism . In another example , the default
`expression of a user when viewing a particular type of
`privacy setting may be that the message or conversation is 50 message ( e.g. , from a particular sender or during a particular
`revealed only when the biometric profile of the recipient
`time of day ) , the messaging application can determine
`account is recognized without the presence of other detected
`whether or not to present another message of the same type
`human beings in the proximate area of the recipient end - user
`via the messaging interface .
`For another example , the messaging application can use
`device .
`In some embodiments , the privacy setting can indicate a 55 the expression recognition process to add gesture control to
`the messaging application . In some cases , certain facial
`single authorized account or biometric profile . In some
`embodiments , the privacy setting can indicate a group of
`expressions can correspond to a gesture control to activate or
`authorized accounts or biometric profiles . In some embodi-
`deactivate an interactive component of the messaging appli
`ments , the privacy setting can indicate an attribute of a
`cation . In one specific example , a frown detected by the
`biometric profile , such as age , gender , facial gesture , facial 60 expression recognition process can cause the messaging
`expression , vocal pitch , etc. The attribute can be the sole
`application to suspend .
`variable to authorize a person . For example , the privacy
`For another example , the messaging application can use
`setting can indicate that any female face ( e.g. , in the case of
`the expression recognition process to customize targeted
`facial recognition ) or any female voice ( e.g. , in the case of
`advertisement . In some cases , an advertisement may be
`speaker recognition ) would be authorized . The attribute can 65 selected based on the current mood of the user . The current
`be an additional conditional on top of a specific account or
`mood of the user , in turn , can be estimated based on the
`biometric profile . For example , a sender account ( e.g. , John )
`expression recognized . In some cases , presentation of an
`
`IPR2023-00833
`Apple EX1001 Page 12
`
`

`

`US 11,042,623 B2
`
`10
`
`6
`5
`dance with various embodiments . FIG . 1B is a diagram
`advertisement may be triggered based on a user expression
`illustrating a messaging application of a mobile device
`condition . For example , an advertisement may be shown
`hiding content of a message when a facial profile associated
`only when the user is recognized to be smiling .
`with a recipient account is not detected , in accordance with
`For another example , the messaging application can use
`the expression recognition process to add context to con- 5 various embodiments . FIG . 1C is a diagram illustrating a
`versations between users of the messaging system . In some
`messaging application of a mobile device hiding content of
`cases , the recognized expression of a sender user can be
`a message when a facial profile associated with a recipient
`added as an emoticon to a message . In some cases , the
`account is detected and a second unauthorized facial profile
`recognized expression of a viewer user can be fed back to
`is detected , in accordance with various embodiments .
`FIG . 2 is a block diagram of a system environment of a
`the sender as a status update .
`messaging system 200 implementing a biometric security
`mechanism , in accordance with various embodiments .
`Alternative Embodiments
`The messaging system 200 can communicate with client
`The disclosure above pertains to the specific example of
`devices 202 ( e.g. , mobile phones , tablets , desktop comput
`an electronic messaging system for delivery of text , images , 15 ers , laptops , other network - enabled devices , or any combi
`audio clips , or video clips . However , it is within the con-
`nation thereof ) . The messaging system 200 can include a
`templation of this disclosure to implement the biometric
`messaging platform system 204 ( e.g. , one or more computer
`security mechanism to other similar systems that include
`servers ) configured to provide a service to facilitate human
`registration of a sender account and the receiver account . For
`understandable electronic communication between user
`example , the biometric security mechanism can apply to 20 accounts . The human - understandable electronic communi
`email , social network , dating network , event / meet up invi-
`cation can include emoticons , text , photos , audio clips ,
`tation , physical delivery of goods , enterprise messaging
`videos , links , images , or any combination thereof . The
`( e.g. , financial agent to customer messaging , real estate
`human - understandable content of the electronic communi
`agent to customer messaging , medical professional messag-
`cation may be part of an electronic message or can be
`ing , etc. ) , or any combination thereof . The biometric secu- 25 referenced in the electronic message ( e.g. , stored elsewhere
`rity mechanism is advantageous in enforcing privacy set-
`that is accessible through a network ) .
`tings of the sender at the device of the recipient . This is an
`In some embodiments , each of the client devices 202 can
`improvement over existing technology that only seeks to
`have its own instance of a messaging interface 206 and a
`protect against unauthorized exposure of content on behalf
`corresponding instance of a biometric security engine 207
`of the device owner but not on behalf of the sender .
`30 running thereon that communicates with the messaging
`The biometric security mechanism is also advantageous in
`platform system 204. In some embodiments , the messaging
`preventing impersonation attempts , such as for the purpose
`interface 206 and the biometric security engine 207 are part
`of cyber bullying , by verifying the identity of a message
`of a messaging application running and / or installed on the
`sender when composing a message . This improves the
`client device . In some embodiments , the messaging interface
`overall accountability of messaging within the messaging 35 206 is installed and / or running on a first client device and the
`system . Similarly , in an enterprise environment , the authen-
`biometric security engine 207 is installed and / or running on
`tication in both viewing and composition of the message
`a second client device . In these embodiments , the biometric
`enhances enterprise - level security and compliance ( e.g. ,
`security engine 207 on the second client device can control
`eHealth compliance when the message content involves
`a privacy shield implemented by the messaging interface
`medical information ) .
`40 206 .
`The disclosure above assumes that the messaging appli-
`For example , the messaging interface 206 and / or the
`cation implements both the biometric security mechanism
`biometric security engine 207 can be embodied as a mobile
`and a messaging interface ( e.g. , to compose and read mes-
`application running on operating systems of some of the
`sages ) . However , it is within the contemplation of this
`client devices 202. In another example , the messaging
`disclosure to implement the biometric security mechanism 45 interface 206 and / or the biometric security engine 207 can
`on a separate device or application from where the messag-
`be implemented as a web - based application running on web
`ing interface is implemented . For example , in the case of an
`browsers on some of the client devices 202 .
`enterprise email system , an email access interface may be
`The client devices 202 can be associated with user
`provided on a desktop computer while the biometric security
`accounts . In some embodiments , a user account of the
`mechanism may be implemented on a mobile device ( e.g. , a 50 messaging system 200 can have multiple client devices
`smart phone ) . In this example , the biometric security mecha-
`associated therewith . In some embodiments , a client device
`nism can require the mobile device to connect with the email
`can have multiple user accounts associated therewith . Con
`access interface on the desktop computer . The biometric
`versations between user accounts are tracked by the mes
`security mechanism can perform the biometric recognition
`saging system 200 such that the messaging system 200 can
`process to verify that there is a secure environment near the 55 deliver an electronic message from a client device of one
`desktop computer ( e.g. , no unauthorized user is around
`user account to a client device of another user account .
`and / or an authorized user is present ) . In response to verify-
`In some embodiments , the messaging system 200 can
`ing a secure environment , the biometric security mechanism
`include a user profile database 208. The user profile database
`can notify the email access interface to reveal content of a
`208 is configured to store user profiles of one or more user
`60 accounts . The user profiles may be associated with one or
`message .
`FIGS . 1A through 1C illustrate an example of the bio-
`more social networking systems ( e.g. , an affiliated social
`networking system , a social networking system integrated
`metric security mechanism that implements a facial recog-
`nition process to protect the privacy of both a sender of a
`with the messaging system 200 , or an external soci