`
`US009875647B1
`
`( 12 ) United States Patent
`Tannenbaum et al .
`
`( 10 ) Patent No . :
`( 45 ) Date of Patent :
`
`US 9 , 875 , 647 B1
`Jan . 23 , 2018
`
`( * ) Notice :
`
`( 54 ) SYSTEMS AND METHODS FOR
`PRESENTING SECURITY QUESTIONS VIA
`CONNECTED SECURITY SYSTEM
`( 71 ) Applicant : GOOGLE INC . , Mountain View , CA
`( US )
`( 72 ) Inventors : Harry Tannenbaum , San Francisco ,
`CA ( US ) ; Scott Hong , Palo Alto , CA
`( US )
`( 73 ) Assignee : GOOGLE INC . , Mountain View , CA
`( US )
`Subject to any disclaimer , the term of this
`patent is extended or adjusted under 35
`U . S . C . 154 ( b ) by 297 days .
`( 21 ) Appl . No . : 14 / 697 , 505
`Apr . 27 , 2015
`( 22 ) Filed :
`Int . Cl .
`( 2006 . 01 )
`G08B 29 / 18
`( 2006 . 01 )
`G08B 13 / 22
`( 2013 . 01 )
`G06F 21 / 34
`( 2006 . 01 )
`G08B 13 / 00
`( 2006 . 01 )
`H04L 12 / 24
`U . S . CI .
`CPC . . . . . . . . . . . G08B 29 / 185 ( 2013 . 01 ) ; G08B 13 / 22
`( 2013 . 01 ) ; G06F 21 / 34 ( 2013 . 01 ) ; G08B 13 / 00
`( 2013 . 01 ) ; H04L 41 / 22 ( 2013 . 01 )
`( 58 ) Field of Classification Search
`CPC . . . . . . GO8B 29 / 185 ; G08B 13 / 22 ; G08B 13 / 00 ;
`HO4L 41 / 22 ; G06F 21 / 34
`See application file for complete search history .
`
`( 52 )
`
`104
`
`150
`
`166
`
`120V
`
`( 56 )
`
`References Cited
`U . S . PATENT DOCUMENTS
`6 , 796 , 799 B1 *
`9 / 2004 Yoshiike . . . . . . . . . . . . G06F 19 / 3418
`434 / 236
`2007 / 0216764 Al *
`9 / 2007 Kwak . . . . . . . . . . . . . . . . . . . . . H04L 12 / 64
`348 / 14 . 06
`2012 / 0280783 A1 * 11 / 2012 Gerhardt . . . . . . . . . . . . G07C 9 / 00309
`340 / 5 . 6
`2013 / 0057695 Al *
`3 / 2013 Huisking . . . . . . . . . . . . . . . H04N 7 / 186
`348 / 156
`2015 / 0325067 A1 * 11 / 2015 Lee . . . . . . . . . . . . . . G07C 9 / 00079
`340 / 5 . 65
`* cited by examiner
`Primary Examiner — Mohamed Barakat
`( 74 ) Attorney , Agent , or Firm — Morgan , Lewis &
`Bockius LLP
`ABSTRACT
`( 57 )
`A computer system detects a trigger event , which includes
`detecting an unverified user within the premises . A notifi
`cation is provided regarding the detected trigger event to one
`or more authorized users distinct from the unverified user .
`Further , a first user input responsive to the notification and
`corresponding to instructions to execute a first security
`protocol is received from a first authorized user of the one
`or more authorized users . In response to receiving the first
`user input , the first security protocol is executed . Executing
`the first security protocol includes presenting to the unveri
`fied user an authentication request and monitoring the prem
`ises for a response to the authentication request . Further
`more , executing the first security protocol includes
`executing or declining to execute a second security protocol
`based on the response to the authentication request .
`20 Claims , 19 Drawing Sheets
`
`120V
`154
`
`120V .
`
`118
`
`100
`
`140
`
`142
`103
`
`HVAC
`
`102
`
`110
`
`114
`
`POOL
`HEATER
`
`IRRIGATION
`
`116 pomo los
`
`168
`
`198
`
`156
`108
`
`112
`
`wwws
`
`4
`
`4
`
`+
`
`H
`
`4 +
`
`W WiFi 180
`
`JO
`160
`
`NETWORKS )
`
`22 CD - 106 120
`APPLIANCE 1
`
`MISC .
`LEGACY
`+ APPLIANCE
`
`140
`
`«
`
`44
`
`162
`SMART
`HOME
`PROVIDER
`SERVER
`SYSTEM
`
`E
`
`164
`
`iApartments Ex. 1023
`
`Page 1 of 41
`
`
`
`U . S . Patent
`
`Jan . 23 , 2018
`
`Sheet 1 of 19
`
`US 9 , 875 , 647 B1
`
`OZN
`
`140
`
`100
`
`152
`
`- - - - - - - - - - - - - - - - - - - - -
`
`- -
`
`ROCK
`
`( 4 )
`154 154
`
`TOZT - - - - - - - - - . . . . . .
`
`)
`
`- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
`
`# # # ??? + ????????? # H # # ??????????????????? # # ???????????????? # # ?? " ?????????
`
`- - - - - - - - - - * * * * * * * * * OLL
`
`168
`
`158
`
`140
`
`APPLIANCE
`MISC . LEGACY APPLIANCE
`IT WiFi 180
`
`E
`164
`
`SMART HOME PROVIDER SERVER SYSTEM
`
`162
`
`( NETWORK ( S )
`
`Figure 1
`
`. . . .
`
`. . . . . .
`
`. . . .
`
`. . . . . . .
`
`. . . . . .
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`- -
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`- -
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`- -
`
`-
`
`-
`
`160
`
`112
`
`1
`
`)
`
`156 | T 168
`108
`108
`
`166
`
`150
`
`VOZL
`
`104
`
`I 110
`
`102 102
`
`POOL HEATER
`
`114
`
`HVAC
`
`IRRIGATION
`
`116
`
`- - - -
`
`9
`
`142 103
`
`iApartments Ex. 1023
`
`Page 2 of 41
`
`
`
`U . S . Patent
`
`Jan . 23 , 2018
`
`Sheet 2 of 19
`
`US 9 , 875 , 647 B1
`
`Network Architecture 200 _
`
`Smart Home Provider
`Server System
`164
`
`Network ( s )
`162
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`wwwwww
`
`w wwwww
`
`we
`
`-
`
`www www
`
`mwen
`
`weer Www
`
`wwwwwww
`
`-
`
`-
`
`- -
`
`-
`
`Device
`204 - 3
`
`Device
`204 - 8
`
`Smart Home Network 202
`—
`—
`—
`—
`—
`
`-
`
`-
`
`Smart Home Environment 100
`
`. .
`
`. .
`
`. .
`
`.
`
`_
`_
`_
`_
`_
`_
`Network Interface 160
`T
`-
`-
`-
`-
`-
`T
`
`Device
`204 - 1
`
`-
`
`-
`
`-
`
`Device
`204 - 2
`
`- - - - - - - -
`
`- - - - - - - - red - e
`
`Device
`204 - 5
`
`HUB
`180
`
`Device
`204 - 6
`
`-
`
`-
`
`-
`
`-
`
`Device
`204 - 7
`
`Device
`204 - 4
`
`Device
`204 - n
`
`1
`
`—
`
`-
`
`-
`
`- - - - -
`
`-
`
`-
`
`Figure 2
`
`iApartments Ex. 1023
`
`Page 3 of 41
`
`
`
`U . S . Patent
`
`Jan . 23 , 2018
`
`Sheet 3 of 19
`
`US 9 , 875 , 647 B1
`
`
`
`
`
`Application Programming Interfaces
`
`164
`SMART HOME
`PROVIDER
`SERVER SYSTEM
`302
`
`HOME DATA
`
`SERVICES
`
`ENGINES :
`- - STATISTICS
`- - INFERENCES
`- - INDEXING
`
`308
`
`wwwwwwwwwww
`
`DERIVED
`HOME
`DATA
`
`. . * . * . . *
`
`. . .
`
`. . . . . . .
`
`.
`
`310
`
`too
`
`CHARITIES
`
`314
`
`GOVERNMENTS - 316
`_ 318
`- 320
`
`ACADEMIC
`INSTITUTIONS
`
`BUSINESSES
`
`UTILITIES 1324
`
`NETWORK ( S )
`( E . G . , INTERNET M
`
`162
`
`DEVICES
`
`8100
`DEVICES
`HUB 180
`NETWORK !
`INTERFACE
`
`Figure 3
`
`iApartments Ex. 1023
`
`Page 4 of 41
`
`
`
`U . S . Patent
`
`Jan . 23 , 2018
`
`Sheet 4 of 19
`
`US 9 , 875 , 647 B1
`
`412
`
`EXTRINSIC
`INFORMATION ( e . g . ,
`FROM INTERNET )
`
`EX :
`- - WEATHER FORECAST
`- - PRICES
`- - NEIGHBORHOOD / HOME INFORMATION
`
`PROCESSING
`PARADIGMS
`
`MANAGED
`SERVICES
`
`ADVERTISING /
`COMMUNICATION
`
`EX :
`- - SECURITY
`- - DEMANDI
`RESPONSE
`410a
`
`410b
`
`410C
`
`SOCIAL
`
`CHALLENGES
`RULES /
`COMPLIANCE
`REWARDS
`
`Search 410d
`
`404
`
`402 SL posted 406
`. .
`
`408
`
`DS SS
`DCSON
`
`306
`
`PROCESSING
`
`pe r
`
`DS SS
`DCSC
`
`DS SS
`DC SC
`
`Doors
`
`DS SS
`DC SC
`
`Bed
`
`DEVICES OF SMART HOME NETWORK 202
`EX :
`DS = DATA SOURCE
`- - LIGHTS , HVAC , WATER CONTROLLERS /
`DC = DATA CONSUMER
`Devices
`SS = SERVICE SOURCE
`SC = SERVICE CONSUMER
`- - HOME APPLIANCES
`- - SMOKE / CO / HAZARD DEVICES / ALARMS
`
`Figure 4
`
`iApartments Ex. 1023
`
`Page 5 of 41
`
`
`
`U . S . Patent
`
`Jan . 23 , 2018
`
`Sheet 5 of 19
`
`US 9 , 875 , 647 B1
`
`500 -
`
`Server - Side
`Module 506
`
`Hub Server
`System 508
`
`514 39 Processor ( s ) 512
`
`Video
`Storage
`Database
`
`Device and
`Account
`Databases
`516
`
`1 / 0 Interface to
`Client ( s ) 518
`
`I / O Interface to
`Cameras 520
`
`Network ( s ) 162
`
`und
`
`h504 - 1
`- 502 - 1
`
`Client
`Side
`Module
`
`- -
`
`-
`
`-
`
`1620
`
`162A
`
`LAN
`160 _ ( Q )
`
`1620
`
`
`
`Prome more
`
`-
`
`-
`
`I - - - -
`522 - 7
`
`522 - 1
`- - 1
`zorno @ motoren
`voro
`
`162B LA
`162B
`400 180
`_ -
`
`-
`
`-
`
`- -
`
`-
`
`-
`
`. .
`
`.
`
`162A
`
`504 - m
`
`162A
`Ø e
`
`-
`
`-
`
`502 - m
`
`Client
`Side
`Module
`
`. . .
`. .
`
`. .
`. .
`
`. . .
`.
`. .
`
`. . .
`
`.
`. . .
`. . .
`
`. . .
`
`Figure 5
`
`iApartments Ex. 1023
`
`Page 6 of 41
`
`
`
`atent
`
`Jan . 23 , 2018
`
`Sheet 6 of 19
`
`US 9 , 875 , 647 B1
`
`Hub Device 180
`
`Radios
`640
`
`602
`CPU ( s )
`
`Input
`Device ( s )
`610
`
`Location
`Detection
`Device 614
`
`604 )
`Comm .
`Interface ( s )
`
`Memory 60614 Operating Logic 616
`Hub Communication Module 618
`Radio Communication Module 620
`User Interface Module 622
`Hub Database 624
`Sensor information 6240
`Device settings 6242
`Communication Protocol
`Information 6244
`
`608
`
`Output
`Device ( s )
`612
`
`Trigger detection module 626
`Security protocol module 628
`
`Figure 6
`
`iApartments Ex. 1023
`
`Page 7 of 41
`
`
`
`U . S . Patent
`
`Jan . 23 , 2018
`
`Sheet 7 of 19
`
`US 9 , 875 , 647 B1
`
`Operating System 710
`Network Communication Module 712
`Server - Side Module 714
`Data Receiving Module 7140
`Hub and Device Control Module
`7142
`Data Processing Module 7144
`Trigger detection module 7146
`Security protocol module 7148
`
`Hub Server System 508
`
`Memory 706
`
`702
`CPU ( s )
`
`708
`
`Network
`Interface ( s )
`
`_ 704
`
`Server Database 716
`Data Storage Database 7160
`
`. .
`
`.
`
`. . . . . . . . . . . . . . . . . . . . . . . .
`
`. . . . . .
`
`Account Database 7162
`Device Information Database 7164
`
`Figure 7
`
`iApartments Ex. 1023
`
`Page 8 of 41
`
`
`
`U . S . Patent
`
`Jan . 23 , 2018
`
`Sheet 8 of 19
`
`US 9 , 875 , 647 B1
`
`Client Device 504
`Memory 8064Operating System 818
`Network Communication Module 820
`Input Processing Module 822
`Application ( s ) 824
`User Interface Module 826
`Client - Side Module 828
`Hub and Device Control Module
`8280
`Data Review Module 8282
`
`804
`Network
`Interface ( s )
`
`8027
`CPU ( S )
`
`Client Data 830
`
`Account Data 8300
`Local Data Storage Database 8302
`
`Trigger detection module 832
`Security protocol module 834
`
`808
`
`User Interface 810
`Output
`Device ( s ) 812
`
`Input
`Device ( s ) 814
`
`Location Detection
`Device
`816
`
`Built - in Sensors
`890
`
`Figure 8
`
`iApartments Ex. 1023
`
`Page 9 of 41
`
`
`
`U . S . Patent
`
`Jan . 23 , 2018
`
`Sheet 9 of 19
`
`US 9 , 875 , 647 B1
`
`Smart Device 204
`Memory 906
`
`Image !
`Video
`Capture
`Device
`918
`
`CPU ( S )
`902
`
`908
`
`Radios
`940
`
`Operating Logic 920
`Device Communication Module 922
`Radio Communication Module 924
`Input Processing Module 926
`User Interface Module 928
`Application Module ( s ) 930
`Device - side Module 932
`- Command Receiving Module
`9320
`Data Processing Module 9322
`
`910 H - 12 Comm .
`
`User Interface
`
`Output
`Device ( s )
`912
`Input
`Device ( s )
`914
`
`Device Data 934
`Account Data 9340
`Local Data Storage Database
`9342
`
`Trigger detection module 936
`Security protocol module 938
`
`Interface ( s )
`904
`
`Location
`Detection
`Device
`916
`
`Built - in
`Sensors
`990
`
`Figure 9
`
`iApartments Ex. 1023
`
`Page 10 of 41
`
`
`
`U . S . Patent
`
`Jan . 23 , 2018
`
`Sheet 10 of 19
`
`US 9 , 875 , 647 B1
`
`Smart Home Provider
`Server System 164
`
`Memory 10062 Operating System 1010
`Network Communication Module 1012
`Server - Side Module 1014
`Data Receiving Module 10140
`Device Control Module 10142
`Data Processing Module 10144
`Trigger detection module 10146
`Security protocol module 10148
`
`1008
`
`10027
`
`CPU ( S )
`
`Network
`Interface ( s )
`
`1004 Server Database 1016
`Data Storage Database 10160
`Account Database 10162
`
`Figure 10
`
`iApartments Ex. 1023
`
`Page 11 of 41
`
`
`
`… . … . … . … . …
`EMERGENCY RESPONDERS
`
`U . S . Patent
`
`Jan . 23 , 2018
`
`Sheet 11 of 19
`
`US 9 , 875 , 647 B1
`
`504 - 2
`
`100 - 3 100 - 3
`
`?
`
`100 - 2
`
`? ? ?
`
`504 - 3
`
`- - - - - - - - - -
`
`1104
`
`-
`
`-
`
`- - -
`
`1100
`
`164
`
`?
`
`HOME PROVIDER SERVER
`SYSTEM
`SMART
`
`- -
`-
`
`???? ?
`| ????
`|
`|
`/
`| |
`|
`11
`| |
`| |
`|
`| | 11i 11 / |
`/
`/
`
`? /
`
`DEVICES
`
`! NETWORK ; INTERFACE ] Figure 11
`
`c100 - 1
`
`NETWORK ( S )
`
`? ( E . G . INTERNET
`
`162
`
`- - - - - - - - - -
`
`- - - - - - - - - - A? ,
`
`-
`
`1102
`
`
`
`SECURITY CALL CENTER
`
`504 - 1
`
`iApartments Ex. 1023
`
`Page 12 of 41
`
`
`
`U . S . Patent
`
`Jan . 23 , 2018
`
`Sheet 12 of 19
`
`US 9 , 875 , 647 B1
`
`DO 118 - 1
`
`Drb 106 m 106 120
`
`132
`c
`
`100
`
`1200 - 3
`
`118 - 2
`
`1202
`1182 , 202
`saw )
`
`180
`WiFi
`
`
`
`
`
`
`
`
`
`??? ?? ??? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??? ??? ??? ??? ?? ?? ?? ?? ??? ??? ?? ?? ?? ??
`
`162
`
`( NETWORK ( S )
`
`Figure 12
`
`N
`
`120V
`
`* * * *
`
`160
`
`?? ?? ?? ?? ?? ?? ??? ?? ?? ?? ?? ??? ??? ??? ??? ?? ??
`
`504 - 1
`
`6 . 0 .
`corar
`
`150
`
`104
`
`1200 - 2 1200 - 2
`
`G2 - 108
`
`RC
`
`1200 - 1 1200 - 1
`
`110
`
`102
`
`iApartments Ex. 1023
`
`Page 13 of 41
`
`
`
`U . S . Patent
`
`Jan . 23 , 2018
`
`Sheet 13 of 19
`
`US 9 , 875 , 647 B1
`
`1330
`
`1318 - 2
`
`- 1318 - 1
`
`- 1318 - 3
`
`_ 1318 - 4
`
`1318 - 5
`
`
`
`
`
`
`
`What question would you like to present ?
`
`1300
`
`- 1310
`
`would you like to do ? detected in Room 1200 - 3 at 1 : 00 AM . What Suspicious activity has been
`
`
`
`
`
`
`
`
`
`View Live Video Feed
`
`
`
`What is my dog ' s name ?
`
`
`
`What college did I attend ?
`
`What is my wife ' s name ?
`
`
`
`What ' s your name ?
`
`Connect to speaker
`
`
`
`Figure 13B
`
`c
`
`1311 M41312 1314 1316
`
`Ignore
`View Clip / Photo
`
`
`
`
`
`Defer Action
`
`
`
`Sound Alarm
`
`1318
`
`1320 1322
`
`
`
`
`
`Present Security Question
`
`Figure 13A
`
`Alert Security Call Center Alert
`
`
`
`Police
`
`. . .
`
`ww . . . . . . . .
`
`w
`
`w
`
`504
`_
`
`504
`
`
`
`iApartments Ex. 1023
`
`Page 14 of 41
`
`
`
`atent
`
`Jan . 23 , 2018
`
`Sheet 14 of 19
`
`US 9 , 875 , 647 B1
`
`1360
`
`1314 - 1
`
`1314 - 2
`
`L1314 - 3
`
`
`
`like to defer this action ?
`How would you
`
`
`
`
`Contact Neighbor
`
`
`
`Contact Brother
`
`
`
`Contact Wife
`
`
`
`Figure 13D
`
`1350 1350
`
`1352
`
`1312 1314
`- 1316
`
`1320
`
`1322
`
`N
`
`41340
`
`
`
`Response received : " Pepper "
`
`504
`
`504
`
`
`
`
`
`What would you like to do ?
`
`
`
`Listen to Recorded Audio
`
`
`
`
`
`Present Another Question
`
`Ignore
`
`
`
`Defer Action
`
`
`
`Sound Alarm
`
`
`
`Alert Police
`
`
`
`Figure 13C
`
`
`
`Alert Security Call Center
`
`
`
`
`
`iApartments Ex. 1023
`
`Page 15 of 41
`
`
`
`U . S . Patent
`
`Jan . 23 , 2018
`
`Sheet 15 of 19
`
`US 9 , 875 , 647 B1
`
`1400
`
`-
`
`-
`
`-
`|
`-
`
`1
`
`-
`
`-
`
`-
`
`-
`
`L
`4
`
`1402
`1404
`
`1406
`
`n
`
`1408
`
`-
`
`-
`
`—
`
`—
`-
`
`—
`-
`
`—
`-
`
`—
`-
`
`—
`-
`
`—
`-
`
`—
`
`—
`
`—
`
`—
`
`U
`
`Trommel
`
`-
`-
`
`-
`
`Detect a trigger event , including detecting an unverified user within the premises
`-
`-
`-
`-
`-
`-
`-
`-
`-
`- -
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`Detect the opening of a door
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`Receive inputs from one or more distinct devices of a computer system
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`Obtain an activity pattern of an authorized user , of one or more authorized users i
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`| The activity pattern for the authorized user indicates , for a respective day !
`I of the week and / or a respective range of time : a time spent in an area of
`the premises , and / or the number of instances in which the authorized user
`entered the area of the premises
`-
`-
`-
`-
`-
`- -
`-
`-
`-
`-
`comune
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`—
`-
`-
`- -
`- . - - - -
`-
`-
`-
`,
`1
`The activity pattern is determined over a predefined period of time N
`i !
`TL IT
`ILI I
`- - -
`-
`-
`.
`-
`-
`-
`-
`- -
`—
`L
`- - - - -
`-
`-
`-
`-
`-
`- -
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`—
`—
`Identify an activity pattern of the unverified user
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`- -
`-
`-
`-
`-
`-
`- - - - - - - -
`-
`Determine a current date and time
`seguimen mene me never seen -
`-
`-
`antes e
`n
`e
`estado e
`
`-
`
`L
`
`-
`
`-
`
`-
`
`e
`
`-
`
`e
`
`-
`
`-
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`!
`
`-
`-
`- -
`
`n
`
`e
`
`se
`
`u even more
`
`M
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`A
`
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`Determine that the activity pattern of the unverified user is at least partially
`distinct from the activity pattern of the authorized user
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`Il Determining that the activity pattern of the unverified user is at least II
`even
`Ti partially distinct from the activity pattern of the authorized user is with W
`T
`respect to the current date and time
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`I T
`- -
`- - - -
`-
`- - - -
`- - - -
`-
`- - -
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`
`- - - - - - - an d
`
`J
`
`II
`
`11 Il IT
`
`Figure 14A
`
`1410
`
`1 412
`
`1416
`
`1418
`
`1420
`
`iApartments Ex. 1023
`
`Page 16 of 41
`
`
`
`U . S . Patent
`
`Jan . 23 , 2018
`
`Sheet 16 of 19
`
`US 9 , 875 , 647 B1
`
`Provide a notification regarding the detected trigger event to one or more authorized to 1422
`users distinct from the unverified user
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`i
`The notification includes information identifying the trigger event
`i
`n
`n contra
`ste come across wel - -
`- -
`-
`-
`- -
`-
`-
`-
`- - ve se
`s - - - - men se
`Ben nemen
`average
`made
`| Providing the notification is in accordance with detecting that the one or more
`|
`authorized users are not within the premises
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`
`-
`
`-
`
`-
`
`Ó
`
`Ó
`
`Figure 14B
`
`Ó
`
`-
`
`-
`
`-
`
`1424
`
`1426
`
`iApartments Ex. 1023
`
`Page 17 of 41
`
`
`
`U . S . Patent
`
`Jan . 23 , 2018
`
`Sheet 17 of 19
`
`US 9 , 875 , 647 B1
`
`Receive , from an authorized user of the one or more authorized users , a user input
`responsive to the notification and corresponding to instructions to execute a first
`security protocol
`
`41428
`
`In response to receiving the user input , execute the first security protocol
`
`-
`
`-
`
`Present to the unverified user an authentication request
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`The authentication request includes a security question , and the response
`comprises an answer responsive to the question
`- - - - -
`-
`- -
`- - - - - - -
`-
`-
`-
`- - -
`- - - - - - -
`- - - - - - - - - - - -
`Instructions include a selection of the security question , of a plurality L
`of security questions , by the authorized user
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`. . . . . .
`. .
`. . . . .
`.
`.
`Play a pre - recorded audio file
`-
`-
`-
`-
`
`. .
`
`. . .
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`N
`-
`
`-
`
`-
`
`Bowen -
`-
`
`Monitor the premises for a response to the authentication request
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`The response to the authentication request is a recorded audio file
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`—
`—
`—
`—
`—
`—
`—
`—
`—
`—
`—
`—
`—
`—
`—
`The response to the authentication request includes a personal
`identification number
`
`-
`
`-
`
`1430
`Th1432 1432
`
`1434
`
`1436
`
`1438
`
`1440
`
`1442
`
`1444
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`.
`-
`-
`en a
`L
`e
`den werden . .
`- -
`-
`-
`Determine whether the unverified user is an authorized user based on the
`response to the authentication request
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`Apply audio processing techniques to process the recorded audio file in 1448
`-
`-
`-
`-
`-
`-
`-
`-
`- -
`-
`-
`-
`-
`- -
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`Present to the authorized user the response to the authentication request
`-
`-
`-
`-
`-
`
`- -
`
`.
`
`1446
`
`-
`-
`-
`- -
`-
`Receive , from the first authorized user , a user input responsive to the
`user input indicating whether the unverified user is authorized il 1452
`presented response to the authentication request , wherein the second
`Il
`- -
`-
`- J
`- J
`T1
`I
`
`- -
`
`-
`-
`
`-
`-
`
`-
`-
`
`-
`-
`
`-
`-
`
`-
`-
`
`-
`-
`
`I
`
`-
`.
`
`-
`
`I
`
`-
`
`-
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`I I I
`
`- -
`-
`
`Figure 140
`
`iApartments Ex. 1023
`
`Page 18 of 41
`
`
`
`U . S . Patent
`
`Jan . 23 , 2018
`
`Sheet 18 of 19
`
`US 9 , 875 , 647 B1
`
`In response to receiving the user input , execute the first security protocol
`
`1430
`
`Execute or decline to execute a second security protocol based on the response
`to the authentication request
`-
`-
`-
`-
`-
`-
`-
`- -
`-
`-
`-
`-
`- -
`-
`-
`-
`-
`-
`-
`-
`-
`-
`| Executing the second security protocol includes sounding an alarm device
`-
`-
`-
`-
`-
`-
`-
`-
`-
`- - -
`- -
`- - -
`-
`-
`- -
`-
`-
`-
`-
`-
`- -
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`- -
`-
`-
`-
`- -
`-
`-
`- - -
`-
`Executing the second security protocol includes providing a notification to
`one or more other authorized users
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`! Executing the second security protocol includes sending instructions to W
`other devices
`o see -
`-
`-
`-
`-
`- - -
`Les seves
`-
`-
`-
`-
`-
`- as e
`s mesures - - -
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`If the unverified user is not an authorized user , execute the second
`1
`| security protocol , and if the unverified user is an authorized user , decline N
`execution of the second security protocol
`-
`-
`www
`-
`-
`- -
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`1454
`
`1456
`
`1458
`
`1460
`
`1462
`
`h
`
`T
`
`-
`
`-
`
`-
`
`-
`
`- -
`
`Figure 14D
`
`iApartments Ex. 1023
`
`Page 19 of 41
`
`
`
`U . S . Patent
`
`Jan . 23 , 2018
`
`Sheet 19 of 19
`
`US 9 , 875 , 647 B1
`
`-
`
`|
`
`-
`
`a
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`Receive , from an authorized user , a user input responsive to the notification and
`corresponding to instructions to execute the second security protocol
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`In response to receiving the user input , execute the second security protocol
`-
`- - - umelecom -
`are reunescu -
`comment avec une seule
`seconda e
`con
`contre
`
`was
`
`1466
`
`-
`
`-
`
`?
`????
`????
`???
`???
`???
`??
`??
`???
`????
`???
`????
`?????
`? ?
`?
`??
`????
`?????
`???
`???? ??? ????
`????
`??? ???
`??
`???
`???
`| Receive a user input corresponding to instructions to access other devices , distinct
`from one or more devices of the computer system
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`?
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`Menu -
`
`-
`
`- - sarees - -
`
`- -
`
`-
`
`-
`-
`-
`-
`-
`-
`-
`Retrieve data from the other devices
`- - -
`-
`- -
`T
`- -
`- -
`-
`-
`- - - - - -
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`- es mesures
`
`meet - Menu
`
`1470
`
`-
`
`-
`
`-
`
`-
`
`-
`
`camente desconocerse e
`
`n
`
`- -
`-
`- -
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`Provide the data to the one or more authorized users
`ntre em como
`de
`comedy scenes comments are more common comment commentare
`com a c
`como
`
`-
`
`- I -
`-
`
`- 1
`
`1472
`1472
`
`-
`
`- - -
`
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`i In the absence of receiving a user input responsive to the notification , execute the first 1
`security protocol .
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`-
`
`1474
`
`Figure 14E
`
`iApartments Ex. 1023
`
`Page 20 of 41
`
`
`
`SYSTEMS AND METHODS FOR
`PRESENTING SECURITY QUESTIONS VIA
`CONNECTED SECURITY SYSTEM
`
`US 9 , 875 , 647 B1
`Thus , computer systems are provided with more effective
`and efficient methods for detecting trigger events and
`executing security protocols , thereby increasing the effec
`tiveness and efficiency of such devices and systems .
`TECHNICAL FIELD
`BRIEF DESCRIPTION OF THE DRAWINGS
`This relates generally to security systems , including but
`For a better understanding of the various described imple
`not limited to methods and systems for detecting trigger
`mentations , reference should be made to the Description of
`events and executing security protocols .
`10 Implementations below , in conjunction with the following
`drawings in which like reference numerals refer to corre
`BACKGROUND
`sponding parts throughout the figures .
`FIG . 1 is an example smart home environment , in accor
`Security systems have traditionally lacked granularity
`dance with some implementations .
`with respect to identifying and classifying a detected threat .
`An armed system that detects movement on a premises , for * 15
`FIG . 2 is a block diagram illustrating an example network
`architecture that includes a smart home network , in accor
`example , will trigger an alarm regardless of whether the
`dance with some implementations .
`cause truly warrants an alarm . Consequently , isolating false
`FIG . 3 illustrates a network - level view of an extensible
`alarm scenarios from legitimate threats within an environ
`devices and services platform with which the smart home
`ment has long been a challenge with typical security sys - 20 environment of FIG . 1 is integrated , in accordance with
`some implementations .
`tems .
`FIG . 4 illustrates an abstracted functional view of the
`SUMMARY
`extensible devices and services platform of FIG . 3 , with
`reference to a processing engine as well as devices of the
`Accordingly , there is a need for methods , systems , and 25 smart home environment , in accordance with some imple
`mentations .
`interfaces for detecting trigger events and executing security
`protocols . By utilizing information and data gathered by one
`FIG . 5 is a representative operating environment in which
`or more devices or systems in a connected network , precise
`a hub device server system interacts with client devices and
`and accurate context is provided with respect to events that
`hub devices communicatively coupled to
`local smart
`trigger an alarm in an environment . Furthermore , by execut - 30 devices , in accordance with some implementations .
`ing predefined security protocols in response , such as pre -
`FIG . 6 is a block diagram illustrating a representative hub
`senting unverified users with security questions that they
`device , in accordance with some implementations .
`must correctly answer , false alarm scenarios can be better
`FIG . 7 is a block diagram illustrating a representative hub
`distinguished from legitimate threats and proper actions can
`server system , in accordance with some implementations .
`be taken in response .
`FIG . 8 is a block diagram illustrating a representative
`In accordance with some implementations , a method is
`client device associated with a user account , in accordance
`performed at a computer system ( e . g . , one or more smart
`with some implementations .
`devices in a smart home network ) with one or more pro -
`FIG . 9 is a block diagram illustrating a representative
`cessors and memory storing instructions for execution by the
`smart device , in accordance with some implementations .
`one or more processors . The method includes detecting a 40
`FIG . 10 is a block diagram illustrating a representative
`trigger event , including detecting an unverified user within
`smart home provider server system , in accordance with
`the premises . A notification is provided regarding the
`some implementations .
`detected trigger event to one or more authorized users
`FIG . 11 is an example smart security network , in accor
`distinct from the unverified user . The method further
`dance with some implementations .
`includes receiving , from a first authorized user of the one or 45
`FIG . 12 is an example smart home environment in a smart
`more authorized users , a first user input responsive to the
`security network , in accordance with some implementations .
`notification and corresponding to instructions to execute a
`FIGS . 13A - 13D illustrate examples of graphical user
`first security protocol . In response to receiving the first user
`interfaces for displaying notifications and executing opera
`input , the first security protocol is executed . Executing the
`tions responsive to notifications , in accordance with some
`first security protocol includes presenting to the unverified 50 implementations .
`user an authentication request and monitoring the premises
`FIGS . 14A - 14E are flow diagrams illustrating a method of
`for a response to the authentication request . Furthermore ,
`detecting trigger events and executing security protocols , in
`executing the first security protocol includes executing or
`accordance with some implementations .
`declining to execute a second security protocol based on the
`Like reference numerals refer to corresponding parts
`55 throughout the several views of the drawings .
`response to the authentication request .
`In accordance with some implementations , a computer
`system ( e . g . , one or more smart devices in a smart home
`DESCRIPTION OF IMPLEMENTATIONS
`network ) includes one or more processors , memory , and one
`Reference will now be made in detail to implementations ,
`or more programs ; the one or more programs are stored in
`the memory and configured to be executed by the one or 60 examples of which are illustrated in the accompanying
`more processors . The one or more programs include instruc -
`drawings . In the following detailed description , numerous
`tions for performing the operations of the method described
`specific details are set forth in order to provide a thorough
`above . In accordance with some implementations , a non -
`understanding of the various described implementations .
`transitory computer - readable storage medium has stored
`However , it will be apparent to one of ordinary skill in the
`therein instructions that , when executed by the computer 65 art that the various described implementations may be
`system , cause the computer system to perform the opera
`practiced without these specific details . In other instances ,
`well - known methods , procedures , components , circuits , and
`tions of the method described above .
`
`35
`
`iApartments Ex. 1023
`
`Page 21 of 41
`
`
`
`US 9 , 875 , 647 B1
`
`may again be the landlord with respect to remote control
`networks have not been described in detail so as not to
`functionality . Importantly , while the identity of the person
`unnecessarily obscure aspects of the implementations .
`performing the action may be germane to a particular
`It will also be understood that , although the terms first ,
`second , etc . are , in some instances , used herein to describe
`advantage provided by one or more of the implementations ,
`various elements , these elements should not be limited by 5 such identity should not be construed in the descriptions that
`these terms . These terms are only used to distinguish one
`follow as necessarily limiting the scope of the present
`element from another . For example , a first type of security
`teachings to those particular individuals having those par
`protocol could be termed a second type of security protocol ,
`ticular identities .
`and , similarly , a second type of security protocol could be
`FIG . 1 is an example smart home environment 100 in
`termed a first type of security protocol , without departing 10 , accordance with some implementations . The Smart home
`from the scope of the various described implementations .
`environment 100 includes a structure 150 ( e . g . , a house ,
`The first type of security protocol and the second type of
`office building , garage , or mobile home ) with various inte
`security protocol are both types of security protocols , but
`grated devices . It will be appreciated that devices may also
`they are not the same type of security protocol .
`The terminology used in the description of the various 15 be integrated into a smart home environment 100 that does
`not include an entire structure 150 , such as an apartment ,
`described implementations herein is for the purpose of
`describing particular implementations only and is not
`condominium , or office space . Further , the smart home
`intended to be limiting . As used in the description of the
`environment 100 may control and / or be coupled to devices
`various described implementations and the appended claims ,
`outside of the actual structure 150 . Indeed , several devices
`the singular forms “ a ” , “ an ” and “ the ” are intended to 20 in the smart home environment 100 need not be physically
`include the plural forms as well , unless the context clearly
`within the structure 150 . For example , a device controlling
`indicates otherwise . It will also be understood that the term
`a pool heater 114 or irrigation system 116 may be located
`" and / or ” as used herein refers to and encompasses any and
`outside of the structure 150 .
`all possible combinations of one or more of the associated
`The depicted structure 150 includes a plurality of rooms
`listed items . It will be further understood that the terms 25 152 , separated at least partly from each other via walls 154 .
`“ includes , ” “ including , ” “ comprises , " and / or “ comprising , ”
`The walls 154 may include interior walls or exterior walls .
`when used in this specification , specify the presence of
`Each room may further include a floor 156 and a ceiling 158 .
`stated features , integers , steps , operations , elements , and / or
`Devices may be mounted on , integrated with and / or sup
`components , but do not preclude the presence or addition of
`ported by a wall 154 , floor 156 or ceiling 158 .
`one or more other features , integers , steps , operations , 30
`In some implementations , the integrated devices of the
`elements , components , and / or groups thereof .
`smart home environment 100 include intelligent , multi
`As used herein , the term " if " is , optionally , construed to
`sen