W0 93,077‘, 3
`
`PCI‘/US92/08316
`
`ll
`
`the electronic device automatically dials
`electronic device),
`the designated database at predetermined (either by the user or
`by the database in configuration)
`times, downloads "mail",
`terminates the telephone connection and turns on the mail-
`5 waiting light on the front of the electronic device. Through
`the use of the auto—poll function,
`the electronic device can be
`kept automatically ‘updated. with. timely,
`fresh. .»i'.nformat:»ion- from
`the services selected by the user or by the database.
`The mail
`message, can for example,
`indicate when a particular service
`has been activated and is available to the user, or confirm
`shipment of goods purchased through use of the electronic
`
`10
`
`15
`
`20
`
`25
`
`:
`
`30
`
`35
`
`4 is a flow chart illustrating general operation
`Fig.
`of the electronic device 20 of Fig. 2.
`The electronic device
`20 begins, after initialization, at step 150 and displays a
`menu of items available to the user.
`
`the user
`View information screens on the electronic device,
`turns his television on, selects channel
`3 or 4, and the
`electronic device main screens are displayed on the television
`screen.
`
`The device, at step 152, next polls the IR detector
`132 to determine if the user selected a displayed menu item by
`operating the remote keypad 52. Absent a selection from the
`remote keypad control 52,
`the program.continues»with~steps 154
`through 162, and returns to step 150.
`If the user selected a
`menu item corresponding to an available service of a remote
`facility,
`the program flow branches to steps 164 through 170.
`If the menu selection signals indicate the user desires a
`printout of the screen,
`the program flow branches to steps 172
`through 176.
`
`Steps 154 through 162 describe performance of the
`electronic device 20 for receiving and displaying inbound data
`calls- At step 154,
`the electronic device 20 checks the modem
`110 for an incoming data call.
`If there is no incoming data
`
`SDOCID -:WOf 9307713Al
`
`Iv‘)
`
`
`
`NTFX-1002 / Page 411 of 1867
`
`

`
`
`
`W0 93/07713
`
`PCT/US92/08316
`
`12
`
`call,
`
`the program returns to step 150 to display the user menu-
`
`If a call is present,
`
`the program executes steps 156 through
`
`162. Step 156 answers the modem, step 158 establishes a
`
`U‘
`
`connection with a remote facility calling the electronic device
`
`20, step 160 downloads (receives) the data from the calling
`
`facility, and step 162 displays any information from the
`
`v.
`
`the program
`download. After displaying the information;
`returns to step 150.
`The incoming data may modify operation of
`
`the electronic device 20 by reconfiguring the system- The
`
`incoming data call may provide the user with an additional menu
`
`item if the facilities expand, or if the user requests
`
`additional options.
`
`Steps 164-through 170 access and acquire desired
`
`information responsive to the selected menu item.
`
`The program
`
`retrieves stored information relating to accessing the desired
`
`facility from the.memory 102. This information can include
`
`modem settings, node addresses, passwords, etc.
`
`Step 166
`
`initiates and dials the modem 110, and step 168 establishes a
`
`connection with the host computer 30 providing the desired
`service.
`
`After connection,
`
`the electronic device 20 initiates
`After
`
`a download of the desired information at step 170-
`
`downloading the information,
`
`the electronic device branches to
`
`step 162 to display the information for the selected service.
`
`After displaying necessary information desired by the selected
`
`service,
`
`the program returns to step 150.
`
`Steps 172 through 176 produce a printout of the
`
`current television image.
`
`Step 172 captures data for printing,
`
`either in text or graphics form.
`printer 56 and step 176 sends the captured print data to the
`
`Step 174 initializes the
`
`printer 56 for printing.
`
`The program returns to step 150 after
`
`step 176.
`
`Fig-
`
`5 is a top level software flow diagram
`
`illustrating a typical operation for a preferred embodiment of
`
`the present invention as implemented in the source code
`
`included in the appendix.
`
`The source code appendix includes
`
`five software modules to implement the functions of the
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`soocuo. <wo, 9307713A1_I_:-
`
`
`
`NTFX-1002 / Page 412 of 1867
`
`

`
`
`
`W0 93/07713
`
`PCT!‘/US92/083116
`
`electronic device 20. These modules include ED.C, EDDATA.C,
`ED.H, ED.ASM, and DRAWSUBS.C.
`
`kl
`
`The ED.C module is the main program which calls the
`
`13
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`electronic device 20, or loaded remotely from~a»configuration» I
`facility, for example.
`The ED.C includes the C-code main
`definition.
`ED.C includes some functions not found in the
`other modules that are specific to text manipulation, such as
`for example, scrolling.
`
`ED.H is a C—program header file which includes the
`global definitions,
`tables and equates used in other modules.
`
`EDDATA.C are strictly menu data in ASCII format.
`Each menu screen includes a name, such as "MAIN MENU" for
`example.
`The ED.C program retrieves particular screens by
`reference to the appropriate name.
`
`ED.ASM includes primitives for low—level operation of
`particular features or common tasks.
`For example,
`these
`primitives decode one key press from the IR receiver, send one
`byte to the modem, receive one byte from the modem, control the
`mail status light toggling, write one pixel on a screen, etc.
`Other features control graphics screens,
`the pointing device,
`card reader, and speakerphone for example. These may be
`described as generic, universal routines to perform the
`selected functions.
`
`DRAwSUBS.C.are“higher.level.pnimitivesv.writtenmin’a~---
`C-language, for performing a next higher level of functional
`control.
`For example,
`
`the primitives in DRAwSUBS.C include
`code to use the primitives of the ED.ASM module to draw boxes
`or lines, or to set a mode between graphics and text, for
`example.
`
`5 are found in the appropriate one of
`Details of Fig.
`6A includes a set of steps for
`Figs. 6A through 6C.
`Fig.
`initialization and active monitoring for input from a user.
`Step 200 includes a power on/reset initialization and self-test
`functions. After step 200,
`the device displays its main menu
`
`SDOCID <WO§93077I3A1 I
`
`:-
`
`
`
`NTFX-1002 / Page 413 of 1867
`
`

`
`
`
`W0 93/077 13
`
`PCI"/US92/08316
`
`‘
`
`14
`
`at step 202.
`
`Step 204 is a main polling loop, waiting for
`
`different activities.
`
`These activities include detecting no
`
`activity (step 206), a manual reset (208), a ringing phone
`(step 220), a credit card swipe (step 260 in Fig. 6B) or an IR
`
`In
`
`key activation (step 262 in Fig. 6B).
`
`Detection of no activity at step 206 branches the
`
`program back to step 202.
`
`A manual reset at step 208“returns
`
`the program to step 200 for reset initialization and self-test.
`A ringing phone at step 220 results in the electronic
`
`device answering the phone (step 222) and a subsequent
`
`determination of a call type at step 224.
`
`The call may be
`
`either a voice call (step 226), an incoming fax (step 228), a
`
`mail message waiting call (step 230), or a configuration call
`
`(step 232).
`
`The voice call type results in allowing the user to
`
`normally use the telephone to conduct a voice conversation
`
`(step 234) and return to display the main menu (step 202).
`
`An
`
`incoming fax results in the electronic device logging into an
`
`appropriate facility (step 236) and receiving printing the fax
`
`document (step 238)
`
`from the facility.
`
`Subsequently,
`
`the
`
`program flow returns to display of the main menu step 202.
`
`A message waiting causes the electronic device to log
`
`into a mail facility (step 240), receive the message (step
`Thereafter,
`
`242), and turn on the message light (step 244).
`
`the system returns to step 202 to display the main menu.
`
`The call type may be a configuration call, which
`
`causes the device to log into a configuration facility (step
`
`246), and receive configuration information at step 248.
`
`The
`
`configuration information may be menu configuration (step 250),
`
`program configuration (252) or security configuration (step
`
`254). After receiving the configuration information,
`
`the
`
`program returns to reset, step 200.
`
`Fig.
`
`6A connects to Fig.
`
`6B at the indicated markings
`
`of A, B and C- Fig. 6B illustrates two other activities,
`
`credit card swiping in the card reader (step 260) and IR key
`activation (step 262). Credit card swiping results in the
`
`device reading the card information (step 264), dialing the
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`ISDOCID -:WO__9307713A1_I_;.
`
`
`
`NTFX-1002 / Page 414 of 1867
`
`

`
`
`
`W0 93/07713
`
`PCT/US92/08316
`
`15
`
`phone (step 266),
`
`obtaining any nece
`
`sending the card information (step 268) and
`ssary authorization (step 270).
`returns to step 202, Fig.
`6A,
`thereafter.
`
`The system
`
`u
`
`(I
`
`10
`
`15
`
`20
`
`25
`
`-
`
`30
`
`in a menu format (step 280).
`purposes-of«exp&anation~only,
`(step 282),
`
`y displayed on the television,
`Representative functions, for
`include speakerphone operation ““W“
`get voicemail (step 284),
`send a fax (step 286) or
`do stock quotes (step 288).
`other functions,
`
`296),
`
`After step 282,
`the device dials the phone (step
`logs into a voicemail facility (298) and activates DTMF
`signalling to retrieve voicemail (step 300), if necessary.
`program returns to step 202 after obtaining v
`oice messages,
`which may be presented on the television spea
`ker.
`To send a
`fax, after step 286,
`the devices presents a
`drawing screen,
`recording cursor movements recorded through use
`of the pointing device (step 290), dials the phone and sends a
`password (step 292),
`and sends the image data to a fax facility
`(step 294).
`The image is typically transmitted in PCX format.
`Fig.
`6C connects to Fig.
`6B at points D, B,
`If a decoded IR key
`activation routes the program to s
`themdevicewdisplays»
`306).
`
`The
`
`menu .
`
`Steps 310,
`312, 314 and 316 advance to step 320 which
`dials the phone and sends the password to the
`facility.
`
`35
`
`upon the particular desired option,
`display of the main menu.
`
`depending
`and returns to step 202 for
`
`:DOClD' -:WO, , 9307713111 l_ >
`
`
`
`NTFX-1002 / Page 415 of 1867
`
`

`
`
`
`W0 93/07713
`
`PCI'/ US92/08316
`
`16
`
`The configuration of the preferred embodiment of the
`present invention does not limit itself solely to accessing and
`receiving requested information.
`The present invention is able
`to access remote facilities and initiate desired activities on
`
`behalf of the user using the electronic device.
`
`The present
`
`invention includes an ability to store profile information
`
`relating to the user, such as cable_service provider; etc: and”
`initiate calls, at the users behalf,
`to authorize the cable
`
`service provider to provide “pay-per-view" service or
`
`10
`
`interactive television, for example.
`
`By using the credit card
`
`reader, or credit card information stored in the memory 102,
`the electronic device may quickly provide access and
`authorization for particular events the user desires.
`
`In this
`
`fashion,
`
`the electronic device 20 provides a simple interface
`
`to the electronic networks available to conventional computer
`
`users, without the expense and complexity of learning to
`
`operate a personal computer and associated hardware and
`
`software. The electronic device has the ability to capture
`
`displayed television images and convert them to PCX format, a
`
`widely used graphic standard.
`
`The PCX format is useful in many
`
`forms,
`
`including facsimile transmission.
`
`In conclusion,
`
`the present invention provides a
`
`simple, efficient solution to a problem of accessing, using and
`
`paying for the many electronic facilities presently available.
`
`Complex and time-consuming acquisition and configuration, as
`
`well as training in use of the computer components becomes
`
`unnecessary with the present device. While the above is a
`
`complete description of the preferred embodiments of the
`
`invention, various alternatives, modifications, and equivalents
`
`may be used. Therefore,
`
`the above description should not be
`
`taken as limiting the scope of the invention which is defined
`
`by the appended claims.
`
`15
`
`20
`
`25
`
`30
`
`um
`
`JSDOCID -:WO_ 93077 13A 1_|, >
`
`
`
`NTFX-1002 / Page 416 of 1867
`
`

`
`
`
`\V()93/07713
`
`PCT/US92/08316
`
`WHAT IS CLAIMED IS:
`
`17
`
`ll
`
`1.
`
`An electronic device,
`
`for use with a
`
`telecommunications network and independent computer systems
`
`5
`
`providing facilities to a user, wherein one facility is a
`configuration facility comprising:
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`autelexisionap
`
`T
`
`a digital communications device for transmission and
`
`receipt of digital signals;
`
`a remote keypad for transmitting menu selection
`
`signals; and
`
`a controller, coupled to said television and to said
`
`digital communications device, for:
`
`displaying a menu on said television, said menu
`
`including first and second items, each said item
`
`identifying a facility;
`
`responding to said menu selection signals to
`
`select a particular one item from said first and second
`
`items;
`
`operating said digital communications device to
`
`access the telecommunications network and connect to a
`
`chosen one of the independent computer systems providing
`said service corresponding to said particular one item
`
`selected from said menu displayed on said television;
`
`initiating said service from said independent
`computer system; and
`
`ddsplayinguinformationron.saidutelevision»
`
`responsive to said service provided by said connected
`
`independent computer system.
`
`2.
`
`The electronic device of claim 1 further
`
`comprising:
`
`means, coupled to said controller, for initiating a
`
`self-
`
`configuration data call to a configuration facility; and
`
`means, coupled to said controller, for self-
`
`configuring the electronic device,
`
`including identification of
`
`particular menu items available to a user.
`
`SDOCID -.WO_
`
`9C507713Al
`
`I
`
`:-
`
`
`
`NTFX-1002 / Page 417 of 1867
`
`

`
`
`
`wo 93/07'/"13
`
`PCfl7[fi392/08316
`
`18
`
`An electronic device, for use with a television,
`3.
`and a communication network coupled to a remote da tabase,
`comprising:
`
`means for interfacing to the television;
`means for interfacing to the communications net
`a keypad remote control; and
`contro1ler“means7*c
`
`work;
`
`'0
`
`10
`
`control, for:
`
`displaying, on the television, a menu having a
`plurality of options;
`
`15
`
`20
`
`35
`
`5.
`
`In
`
`>
`JSDOCID <WO___9307713A1_l
`/tmmmmm——
`
`
`
`NTFX-1002 / Page 418 of 1867
`
`

`
`
`
`W0 93/077 I 3
`
`' PCT/US92/08316
`
`10
`
`20
`
`for
`
`sDOC|D' <WO__93077l3A1_l
`
`:-
`
`
`
`NTFX-1002 / Page 419 of 1867
`
`

`
`
`
`W0 93/07713
`
`PCT/ US92/08316
`
`20
`
`responding to a menu selection signal
`
`from a remote
`
`keypad control to identify a particular one of said plurality
`of user selectable facilities;
`
`initiating a data call by the electronic device to a
`
`host computer system servicing said particular facility;
`
`‘ix
`
`connecting the electronic device to said host
`computer system;
`_.
`.
`downloading data associated with said particular
`
`facility to the electronic device; and
`
`displaying said downloaded data associated with said
`
`particular facility.
`
`15.
`
`A speakerphone for a television, comprising:
`
`a remote keypad including a plurality of keys and a
`
`transmitter for issuing data relating to keypress events;
`
`a telephone signal receiver for receiving an incoming
`
`telephone call;
`
`a microphone;
`
`an audio converter circuit, coupled.to the television,
`
`for
`
`converting incoming telephone audio signals into television
`
`audio signals;
`
`a remote receiver for receiving said data relating to
`
`keypress events and converting them into control signals; and
`
`a speakerphone controller,,coupled to said telephone
`
`receiver, said audio converter circuit and said remote
`
`receiver, for controlling telephone operation responsive to
`
`said control signals, with a speaker of the television
`
`producing audio signals of an incoming telephone call and said
`
`microphone providing input for the speakerphone.
`
`16.
`
`The speakerphone of claim 15 wherein said
`
`speakerphone controller includes a dialing mechanism,
`
`responsive to said control signals, for placing a phone call.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`ISDOCID <WO
`
`5'3077lJA1 I
`
`;.
`
`
`
`NTFX-1002 / Page 420 of 1867
`
`

`
`.B3
`
`on
`
`
`
`33:;2.5:
`
`..3._\.
`
`g_:§__._.__9_--5%:3/SWW;
`--1wa_223nNM2§_w_:§.WWWxué5.5s5
`MT,3ME.3E53..
`1
`.L2283
`
`mNwtW2m32.28W:22
`
`NW5:
`
`NTFX-1002 / Page 421 of 1867
`
`
`
`

`
`WO 93/07713
`
`PCI‘/US92/083 16
`
`.«.
`
`Nwfi
`
`3&5.
`
`._..23.59
` :_EHmmm§§;_,_Mm§a_:_.E“Un3_“Em
`
`
`
`H
`
`
`
`llllllllIIII...II...IlIIllu_.8&18
`
`_.
`
`E
`
`54.1
`
`8 E$ 0..
`Z
`
`9-
`
`E3O
`
`.
`
`to
`ID
`
`lSDOClD_ <WO__9307713ALl_:»
`
`NTFX-1002 / Page 422 of 1867
`
`
`
`

`
`W0 93/07713
`
`PC!"/US92/08316
`
`3/7
`
`ozamozmmzozNsq:@m;mz.m
`
`mEt<w§;
`
`m~moam.N
`
`o»»o4.m
`
`oz2zqm.¢
`
`'.DOClD: -:WO___9307713A1_|
`
`>
`
`NTFX-1002 / Page 423 of 1867
`
`
`
`
`

`
`WO 93/07'/"13
`
`PCT/US92/0831 6
`
`4/7
`
`
`Y I
`
`ESTABLISH
`CONNECTION
`
`y
`
`[70
`
`«
`
`DOWNLOAD
`DATA
`
`‘
`
`
`ESTABLISH
`CONNECTION
`
`N
`
` JSDOCID. <WOi9307713A1_l _>
`
`T-
`
`
`
`
`
`POLL
`l R DETECTOR
`
`mo
`
`POLL monsm
`roa CALL
`
`» /W C
`
`NTFX-1002 / Page 424 of 1867
`
`

`
`3%n_/mm0
`
`Mm0V.w.umP
`
`W
`
`.O.Mm.
`
`.9..
`
`.,
`
`.m.
`.S
`
`aLEE.m3ETWMnu»8A.,_
`
`NTFX-1002 / Page 425 of 1867
`
`

`
`:9
`
`L
`
`ISDOCID: -.WO__
`
`35.:
`
`Vm
`
`7:T.WU.W§_§§
`
`
`5&3”2.25..3
`-My
`
`.3mm.5eWE
`
`0..
`
`r
`
`a-
`
`Tu
`
`._%m@E
`
`NTFX-1002 / Page 426 of 1867
`
`
`
`

`
`“ “ mmwx mmufl‘ mm “ HM ‘HmMWMMHHVHWHMHXMWA Wu
`
`shfl
`
`W0 93/0771 3
`
`PCT/ US92/08316
`
`7/7
`
`ululu
`
`
`
`“J;m‘,‘x‘““5‘x‘w|nuIl‘!.:.Z.,..:...I,,_....IlI
`
`
`
`1.233..can
`
`
`
`~22:.23
`
`3DOClD <WO}93077l3A1_I_:»
`
`A‘-v-u-_u-n
`1.» .u—‘
`3
`.
`.‘
`
`UTE c23%~e"
`
`EE
`
`NTFX-1002 / Page 427 of 1867
`
`
`

`
`
`
`HNTERMATIONAL SEARCH REPORT
`
`
`
`
` A. C fl0N OF SUBJlECE'MA'!'|l'lER
`
`IPC(5)
`:H¢<!>.*/ilxz
`US CL :33’
`
`
`
`8
`
`
`According In lmzauminnnl Pmcm Clmsnificmion (IPC) or to both national classification and [PC
` Minimum documentation watched (clnnaificafion nynuan followed by clnnnificnlion aymboln)
`B. m: cm-:1»
`
`
`u.s.
`
`.
`
`
`
`3791105386-98,93,%).l10,53,54;l78/l8,l9;358/84,86,141-147
`
`i
`
`'
`
`
`
`
`
`
`
`
`IIOIIC
`
`c.
`
`nocummms consxnmmn To an RELEVANT
`
`8
`
`Z
`
`8
`
`8 W
`
`8
`
`Citation ofdocument. with indication. when appropriate. ofthe nelevcmz pannnga:
`US,A, 4,456,925 (Skerlos ct al) 26 June 1934, Ice abstract, ass 1§,1§
`Figures.
`8-10
`
`
`
`
`US,A, 4,987,486 (Johnson et a1) 22 January 1991, we abstract.
`
`[-4, [4
`5,6-13
`
`X Y
`
`X Y
`
`
`
`
`
`
`
`
`
`
`5,6
`
`13
`
`7
`
`
`
`US,A, 4,329,684 (Monteath et al.) 11 May 1982 See abstract.
`
`US,A, 4,695,880 (Johnson et al) 22 September 1987, see abstract.
`
`US,A, 4,916,737 (Chomet ct al) 10 April 1990, m abstract, Figure
`2.
`
`Y Y
`
`Y
`
`
`
` D lfiumm documalm an listed in the continuation or B0): c.
`
`dr.':aoa'pn':nn':y
`
`dzzaiunbanlfimaibcacpnuhndadhzzkzizzfléo
`
`
`
`
`‘A’
`_
`
`5'
`*1:
`
`_
`.
`.
`damnsddbmtnaczdnndlbmmmhemocudad
`toba|::nofp:.|%b:rn:.‘_uv::::a
`.
`.
`.
`.
`.
`“’'$’d°°'"=';3P'bi'n“°&='m':=:’$"==''m‘-‘3'‘§
`4oaummwua:pmymu74g=¢=upeu=aycui;(o)_w=bad:b
`
`‘X’
`
`
`ec.=E£=1=d|:vvc5cn.:i.hocn::‘dau:|un1:rvnivo::i:IQ:Iivoc:q:
`°*==|¢=-4°='====='="—"==°1'3'
` cu::."I!::n:! to bvcbo m iavfivocsp urban :3 doa5j_'::_i3
`
`
`
`'0'
`
`d¢ntduri:Iatnc:crnldi:::£n:-_-m.|cs.uh&é‘k'aac:'o1k=1
`fig
`
`oca'!3:dc1thu::a¢rc:uoa:f:::'nm:£aa£.anchcu:nb:z:cn:1
`hoimcbvixzzacnnwunckiltdizltul
`
`
`
`A
`
`'7'
`
`Duzeofthe adamlcompletion oftbc immmsonnl search
`
`c=:>p:‘.-_:fl:$3y
`as
`_
`_ an ,
`daip:I$li:ipr?Iovm&Ez=n'%7:lfiIiz3d:mb::Ii& -3- cfi
` Dulcofmnilingofthennlnenrchrqaon
`98 FEB 1933
`
`04 JANUARY 1993
`
`
`
`
`
`Nnndmniling addn=unofIhelSAIUS
`fiflmfizndoffina
`Conzmi:-.;.-'s:n:::ofPnIau1aundTr1#ca:arirn
`BoxlPCl'
`
`
`Wm-Liana. D.C. mm:
`
`
`Fncaimilc No. No? AP‘FL!CA.BLE
`
`
`Fonn PCT/ISAIZIO (nccomi uhnaxluly 1971):)
`
`WING F. CHAN
`
`/1/.7,”
`
`JSDOCID, <WO___9307713A1_|_>
`
`
`
`
`
`NTFX-1002 / Page 428 of 1867
`
`

`
`
`
`WORLD INTELLECTUAL PROPERTY ORGANIZATION
`International Bureau
`
`
`
`
`H04N 5/225, A63F 9%
`
`A1
`
`-
`
`:.~.‘.‘.’
`
`
`
`
`
`
`
`
`
`
`_
`INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT)
`(51) International Patent Classification 6 :
`(11) International Publication Number:
`WO 97/02699
`
`(43) International Publication Date:
`23 January 1997 (23.01.97)
`
`(72) Inventors: HONEY, Stanley, K.', 827 Lytton Avenue, Palo
`
`Alto, CA 94301 (US). CAVALLARO, Richard, H.; 1516
`
`Wildrose Way, Mountain View, CA 94043 (US). HILL,
`David, Blyth; 2240 Mandeville Canyon Road, Los Angeles,
`CA 90049 (US). SETOS, Andrew, G.; 14807 Bestor Boule-
`vard, Pacific Palisades, CA 90272 (US). GEPNER, Jerry,
`Neil; 527 Trappc_I.‘_ane,_Langhorn, PA 19047 (US). HEI-
`DMANN, Thimothy,” Paul; 96U‘E'astwood P1ace,"Los"Altos,
`CA 94024 (US). OLSEN, Patrick, Wade; 1106 Viscaino
`Avenue, Sunnyvale, CA 94086 (US). I-IEINZMANN, Fred,
`Judson; 820 Vista Grande Avenue, Los Altos, CA 94024
`(US). PHILLIPS, Alan, C.; 1015 Border Road, Los AI-
`tos, CA 94024 (US). GUT}-IART, Harold; 897 Stagi Lane,
`Los Altos, CA 95113 (US). BURNS, Alan, Alexander: 108
`Ramona Road, Portola Valley, CA 94028 (US). RENO,
`Charles, Lawrence; 471 Claremont Way, Menlo Park, CA
`94025 (US). EVANS, Philip, Calvin; 430 La Mesa Drive,
`Portola Valley, CA 94028 (US).
`
`(21) International Application Number:
`
`PCT/US96l11 122
`
`(22) International Filing Date:
`
`28 June 1996 (28.06.96)
`
`
`
`
`(30) Priority Data:
`08/493,009 \
`08/585,145
`
`10 January 1996 (l0.0l.96)
`
`(71) Applicant: FOX SPORTS PRODUCTIONS. INC. [US/US];
`10201 West Pico Boulevard, Los Angeles, CA 90064 (US).
`
`
`
`
`
`
`US
`
`
`
`
`
`
`
`
`
`
` (74) Agents: FLIESLER, Martin, C. et al.; Fliesler, Dubb, Meyer &
`
`Lovejoy L.I...P., Suite 400, Four Embarcadero Center, San
`
`
`Francisco, CA 94111-4156 (US).
`
`
`
`
`
`
`(81) Designated States: AL. AM. AT, AU, AZ, BB, BG, BR, BY.
`CA, CH, CN, CZ, DE, DK, EE, ES, FI, GE, GE, HU, IL,
`
`
`
`IS. JP. KE. KG, KP, KR, KZ, LK, LR, LS, LT, LU, LV,
`MD, MG, MK, MN, MW, MX, NO, Z, PL, PT, RO, RU,
`
`
`SD, SE, SG, SI, SK, TJ,
`, TR, TI‘, UA, UG, UZ, VN,
`ARIPO patent (KE, LS, MW, SD, SZ, UG), Eurasian patent
`
`
`(AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European patent
`(AT, BE, CH, DE, DK, ES, FI, FR, GB, GR. IE, IT, LU.
`MC, NL, PT, SE), OAPI patent (BF, BI, CF, CG, CI, CM,
`
`GA, GN, lvfl., MR, NE, SN, TD, TG).
`
`
`
`Published
`
`With international search report.
`With amended claims.
`
`A SYSTEM FOR ENHANCING THE TELEVISION PRESENTATION OF AN OBJECT AT A SPORTING EVENT
`
`__/(57) Abstract
` A system (200) for enhancing the television presentation of an object at a sporting
`.event includes a sensor (210, 212, 214, 216), which determines the location of the object.
`
`Based on the location of the object and the field of view of a broadcast camera (201,
`
`202, 203, 204), a processor (302) detennines the position of the object in the video frame
`
`of the broadcast camera. Once knowing where the object is positioned within the video
`frame, the television signal can be edited or augmented to enhance the presentation of
`
`the object.
`
`(54) Title:
`
`
`
`
`
`38882833 E‘.
`
`illflfilll mm 'i7fl?fi.“h‘lfll
`
`I 1
`
`
`
`NTFX-1002 / Page 429 of 1867
`
`

`
`
`
`FOR THE PURPOSES OF INFORMATION ONLY
`
`Codes used to identify States party to the PCT on the front pages of pamphlets publishing international
`applications under the PCI‘.
`
`AM
`AT
`AU
`BB
`BE
`BF
`BG
`BJ
`BR
`BY
`CA
`CF
`CG
`CH
`CI
`CM
`CN
`
`CZ
`DE
`DK
`EE
`ES
`Fl
`FR
`GA
`
`'
`
`Armenia
`Austria
`Australia
`Barbados
`Belgium
`Buxkina Faso
`Bulgaria
`Benin
`Brazil
`Belarus
`Canada
`Central African Republic
`Congo
`Switzerland
`Cote d‘lvoire
`Cameroon
`China
`Czrchmlnvakia
`Czech Republic
`Germany
`Denmark
`Estonia
`Spain
`Finland
`France
`Gabon
`
`United Kingdom
`Georgia
`Guinea
`Greece
`Hungary
`Ireland
`Italy
`Japan
`Kenya
`Kyrgystan
`Democratic People‘s Republic
`of Korea
`Republic of Korea
`Kaukhsmn
`Liechtenstein
`Sri Lanka
`Liberia
`Lithuania
`Luxembourg
`Latvia
`Monaco
`Republic of Moldova
`Madagascar
`Mali
`Mongolia
`Mauritania
`
`Malawi
`Mexico
`Niger
`Netherlands
`Norway
`New Zealand
`Poland
`Portugal
`Romania
`Russian Federation
`Sudan
`Sweden
`Singapore
`Slovmia
`Slovakia
`Senegal
`Swaziland
`Chad
`Togo
`Tajikistan
`Trinidad and Tobago
`Ukraine
`Uganda
`United States of America
`Uzbekistan
`Vic! Nam
`
`JSDOCID -:WO
`
`97026Q9A1_l_ >
`
`
`
`NTFX-1002 / Page 430 of 1867
`
`

`
`wo 97/02599
`
`IPCTIUS96/1 1 1122
`
`-1-
`
`A SYSTEM FOR ENHANCING THE TELEVISION PRESENTATION
`OF AN OBJECT AT A SPORTING EVENT
`
`BACKGROUND OF THE INVENTION
`
`Field of the Invention
`
`10
`
`The present
`
`invention is directed to a system for enhancing the
`
`television presentation of an object at a sporting event.
`
`Description of the Related Art
`
`The television presentation of sporting events needs to be improved.
`
`15
`
`Because of the size and speed of some objects and the distance of the
`
`television camera from the playing field, some objects at a sporting event are
`
`hard to see on a television screen. One example of an object at a sporting
`
`event is the game object, which is defined as the article or thing which is the
`
`focus of the game, for example, a hockey puck, baseball, football, soccer ball,
`shotput, basketball, tennis ball, etc. To compensate for objects that are hard
`
`20
`
`to see on television, broadcasters will use zoom lenses. However, the limited
`
`field of view of a zoomed camera prevents the object from being viewed in
`
`relation to the playing field and prevents the viewer from seeing other objects
`
`that are part of the sporting event. Additionally, even with zoom lenses some
`
`25
`
`objects remain difficult to see on television.
`
`'
`
`_,
`
`For example, television viewing of a hockey game is hampered by
`
`poor visibility and distortion of the hockey puck. The puck is small, and is
`
`passed, shot and deflected at high speeds. A standard hockey puck is three
`
`inches in diameter and one inch high. A television viewer sees only a limited
`
`30
`
`portion of the ice and the scene being viewed changes rapidly as the camera
`
`moves to follow the action.
`
`In order to be able to view all areas of the
`
`hockey rink, cameras must be located far from the playing surface. Thus, on
`
`:DOC|D <WO__970269£-‘A1 I
`
`:-
`
`NTFX-1002 / Page 431 of 1867
`
`

`
`W9 97/92699
`
`lP(C’ll'/lUS96I]1lIl22
`
`-2-
`
`a standard television screen a hockey puck tends to appear as a small dot or
`
`a smear, and sometimes the puck is not perceptible by a viewer. As a result,
`
`it is difficult to follow the puck from player to player, and it is especially
`
`difficult to"f6ll'ow the ‘puck as it is shot toward the goal‘; For most viewers,
`
`5
`
`recognizing that a goal has been scored probably comes after the fact when a
`
`signal light is turned on or the announcer informs the viewer that a goal has
`
`been scored. Because viewers cannot follow the puck, they do not sustain
`
`interest in the game. Although hockey is a popular sport, telecasts of hockey
`
`games would have wider appeal and be more fully enjoyed if the movement
`
`10
`
`of the puck could be more closely followed.
`
`In golf and baseball it is hard to see the ball when the baH is hit in the
`
`air for a long distance (e. g. a home run in baseball or a tee shot in golf). To
`
`compensate, cameras will zoom in on the baseball or golf ball which allows
`
`the viewers to see the ball against the sky. Such a view of the ball does not
`
`15
`
`give the viewer a true perspective of the ball’s path. That is, the viewer
`
`cannot determine how far the ball was hit or where the ball is in the relation
`
`to the playing field. At times a viewer of a baseball game cannot distinguish
`
`between a home run and a routine fly ball. One way to show a baseball or a
`
`golf ball in relation to the playing field is to zoom out and show the entire
`
`20
`
`baseball field or golf course. However, such a zoom-out will make the ball
`
`difficult or impossible to see. Other sporting events present similar visibility
`
`problems.
`
`Furthermore,
`
`television viewers do not have access to the same
`
`infonnation that is available to spectators at the event (e.g. game programs,
`
`25
`
`messages on the scoreboard, etc.). Thus, broadcasters may be able to sustain
`
`greater viewer interested by presenting the viewers with additional desired
`
`information and provide for the use of that information in an exciting way.
`
`a
`
`Hflflilil rW’W
`
`'l7fl"fi99lll
`
`|
`
`‘I
`
`NTFX-1002 / Page 432 of 1867
`
`

`
`
`
`W/(U9 97102699
`
`lPClI'I1US96/ll M22
`
`-3-
`
`Thus,
`
`there is a need for enhancing ‘the television presentation of
`
`objects at sporting events. However, any solution to the problem should not
`
`significantly effect the players or fans who attend the sporting event.
`
`
`SUMMARY OF THE INVENTION
`
`The present invention is directed to overcome the disadvantages of the
`
`prior art. Thus, the present invention provides for a system for enhancing the
`
`television presentation of an object at a sporting event. The system, roughly
`
`described, determines an object’s location, captures a video image which
`
`includes a field of view encompassing the location of the object, determines
`
`the position of the object
`
`in the field of View of the broadcast camera
`
`capturing the video image and enhances the television presentation of the
`
`object.
`
`One embodiment
`
`includes one or more sensors and at
`
`least one
`
`processor. The sensors are used to detect the object. Many different types
`
`of sensors can be used; therefore, the invention is not limited to any particular
`
`sensor. One exemplar sensor is an infrared sensor which detects infrared
`
`signals from an object. One infrared sensor can be used to determine the
`
`object's position in a video image from the broadcast camera. Two or more
`
`infrared sensors can be used to detemiine the three dimensional location of the
`
`object. The processor communicates with the sensors and is programmed to
`
`determine the object’s position in the video image from the broadcast camera.
`
`If the sensor or the broadcast camera can be moved, the system includes one
`
`or more field of view sensors which provide the processor with information
`
`about the broadcast camera’s (and the sensor’s) field of view. A field of view
`
`sensor can include one or more of the following: a pan sensor,
`
`tilt sensor,
`
`zoom sensor, beacon or anything else used to determine the field of view of
`
`a camera or sensor. If the processor knows the three dimensional location of
`
`10
`
`15
`
`20
`
`25
`
`-DOCID -WC
`
`Q7U?fi<40AI
`
`I
`
`:-
`
`
`
`NTFX-1002 / Page 433 of 1867
`
`

`
`W0 97/02599
`
`PCT/US96/11122
`
`-4-
`
`the object, the processor can determine the position of the object in a video
`
`frame of any broadcast camera whose field of view is known.
`
`Various means can be used to enhance the television presentation. For
`
`example, the system can add a graphic" to the video image at the positionof ‘
`
`5
`
`the object in the video frame, a graphic near the position of the object in the
`
`video frame, a graphic in a location logically related to the position of the
`
`object in the video frame, statistical information or announcements related to g
`
`the object’s location, audio data or non-video data.
`
`In one alternative, the
`
`processor manipulates the video image.
`
`In another alternative, the processor
`
`10
`
`creates a second video image and combines the video image created by the
`
`processor and the video image from the broadcast camera. Other suitable
`
`methods of manipulating television data are also within the spirit of the
`
`present invention.
`
`These and other objects and advantages of the invention will appear
`
`15
`
`more clearly from the following description in which the preferred
`
`embodiment of the invention has been set forth in conjunction with the
`
`drawings.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`20
`
`Figure 1 depicts a perspective view of a hockey puck according to the
`
`present invention.
`
`Figure 2 depicts a cut—away plan view of the hockey puck of Figure 1.
`
`Figure 3 depicts a cross-sectional view of the_hockey puck of Figure
`
`25
`
`Figure 4 shows a first embodiment electronics package for the infrared
`
`transmitter of a hockey puck.
`
`Figure 5 depicts one exemplar shock sensor.
`
`.
`
`Figure 6 shows a second embodiment electronics package for the
`
`hockey puck of Figure 1.
`
`ISDOCID <WC)¥ 9702699A l_ |_ >
`
`NTFX-1002 / Page 434 of 1867
`
`

`
`W9 97/02599
`
`IPCII‘/EUS96Il111Il22
`
`-5-
`
`Figure 7 is a block diagram of a system used for enhancing the
`
`television presentation of the hockey puck of Figure 1.
`
`Figure 8 is a block diagram of a camera location for the system of
`
`Figure 7'.“ A“
`
`' "
`
`5
`
`Figure 9 is a block diagram of the front end processor electronics of
`
`Figure 8.
`
`Figure 10 is a flow chart describing some of the operations in the
`
`camera location and related hardware.
`
`Figure 11
`
`is a graphical representation of data from the infrared
`
`10
`
`sensors.
`
`Figure 12 is a block diagram of the pulse detection and strobe detection
`
`system.
`
`Figure 13 is a block diagram of a pulse detector.
`
`Figure 14 is a block diagram of the synchronizer box of Figure 12.
`
`15
`
`Figure 15 is a partial block diagram of the graphics center and
`
`production center.
`
`Figure 16 is a flow chart describing some of the operation of the
`
`graphics center and production center.
`
`Figures 17A and B are flow charts describing two exemplar methods
`
`20
`
`for determining the pucl<’s three dimensional location.
`
`Figure 18 is a flow chart describing the detennination of the puck’s
`
`position in a video frame and one alternative for enhancing the television
`
`_
`
`presentation of the hockey puck.
`
`Figure 19 shows the hockey puck in a video fra