`United States Patent
`[19J
`DeLeeuw et al.
`
`US006088018A
`[11]Patent Number:
`6,088,018
`[45]Date of Patent:
`Jul. 11, 2000
`
`I 1111111111111111 11111 1111111111 111111111111111 1111111111 111111111111111111
`
`[54] METHOD OF USING VIDEO REFLECTION
`
`"Human-Object Reactor Technology", Reality Fusion, Inc.;
`
`
`
`
`
`IN PROVIDING INPUT DATA TO A
`
`
`http://www.realityfusion.com/techl.htm; 1997, 1 page.
`COMPUTER SYSTEM
`
`"Known-Object Reactor Technology", Reality Fusion, Inc.;
`
`
`
`
`
`
`
`http://www.realityfusion.com/tech2.htm; 1997, 1 page.
`William C. DeLeeuw, Hillsboro;
`
`
`[75] Inventors:
`
`
`Kenneth L. Knowlson, Portland;
`"Dynamic Layer Transformer Technology", Reality Fusion,
`
`
`
`
`Bradford H. Needham, Hillsboro, all
`
`
`Inc.; http://www.realityfusion.com/tech3.htm; 1997, 1 page.
`of Oreg.
`
`
`[73]Assignee:
`
`
`
`"RFI Press Release-Feb. 3, 1998", Reality Fusion, Inc.;
`
`
`
`
`Intel Corporation, Santa Clara, Calif.
`
`
`
`http://www.realityfusion.com/release_020398.htm; 1997, 2
`pages.
`
`[21]Appl. No.: 09/096,282
`
`
`
`[22]Filed: Jun. 11, 1998
`
`"Products", Reality Fusion, Inc.; http://www.realityfusion­
`
`
`
`
`
`
`.com/products.htm; 1997, 2 pages.
`....................................................... G09G 5/00 [51]Int. Cl.7
`
`
`
`
`
`
`"Corporate Profile", Reality Fusion, Inc.; http://www.reali­
`
`345/113; 345/158 [52]U.S. Cl. ........................... 345/156;
`
`
`
`tyfusion.com/corprofile.htm; 1997, 1 page.
`[58]Field of Search .....................................
`345/156, 157,
`
`
`
`
`"Developers", "Products", Reality Fusion, Inc.; http://ww­
`
`345/158, 145, 146, 113, 114, 115, 343,
`
`w.realityfusion.com/developers.htm; 1997, 1 page.
`344, 345
`
`Krueger, Myron W., "Artificial Reality", 2nd ed., c. 1991, 14
`
`
`
`
`
`pages.
`
`Foley, James D., "Computer Graphics: Principles and Prac­
`
`
`
`tice", 2nd ed. in C., c. 1997, 2 pages.
`
`[56]
`
`
`
`References Cited
`
`
`
`U.S. PATENT DOCUMENTS
`
`
`4,686,521 8/1987 Beaven et al. .
`
`4,833,462 5 /1989 Gover et al. .
`
`4,988,981 1/1991 Zimmerman et al. .
`
`
`
`5,270,688 12/1993 Dawson et al. ......................... 345/145
`
`Primary Examiner-Regina Liang
`
`
`
`
`5,283,560 2/1994 Bartlett .................................... 345/113
`
`
`
`Attorney, Agent, or Firm-Steven P. Skabrat
`
`
`
`5,436,639 7/1995 Arai et al. ............................... 345/156
`ABSTRACT
`
`5,463,728 10/1995 Blahut et al. .
`[57]
`
`
`
`
`5,528,263 6/1996 Flatzker et al. ......................... 345/156
`
`
`
`Providing input signals to a computer system having a
`
`
`
`
`5,594,469 1/1997 Freeman et al. ........................ 345/158
`
`
`5,616,078 4/1997 Oh ........................................... 345/156
`
`
`
`display, the computer system being coupled to a video
`
`
`
`camera or other video source, is accomplished by capturing
`FOREIGN PATENT DOCUMENTS
`
`
`
`video data signals generated by the video camera, the video
`
`
`
`
`data signals representing a scene, rendering the scene on the
`
`
`
`
`display such that the scene is reflected and transparently
`
`
`
`
`visible on the display, analyzing the video data signals to
`
`
`
`
`detect an object in the scene, and generating an input signal
`"Headlines-Feb. 1998", Reality Fusion, Inc.; http://www­
`
`
`
`
`
`
`for the computer system in response to the detected object.
`
`.realityfusion.com/; 1997, 2 pages.
`
`
`"Technology", Reality Fusion, Inc.; http://www.realityfu­
`
`27 Claims, 16 Drawing Sheets
`
`sion.com/technology.htm; 1997, 3 pages.
`
`8-139994 5/1996 Japan .
`
`OTHER PUBLICATIONS
`
`IPR2021-00922
`Apple EX1007 Page 1
`
`

`

`U.S. Patent Jul. 11, 2000
`6,088,018
`
`Sheet 1 of 16
`
`FlG. l
`
`IPR2021-00922
`Apple EX1007 Page 2
`
`

`

`00
`= �
`
`....
`
`00
`= 00
`
`....
`
`...., '"""'
`
`O'I
`
`0
`
`N
`....
`�
`
`
`
`=­�
`
`'JJ.
`
`0
`0
`0
`N
`
`
`:-'"""'
`
`'"""'
`
`=
`�
`
`28
`
`BUFFER
`DISPLAY
`VISIBLE
`
`/
`
`... STOCK TICKER,
`...
`
`WINDOW
`SECOND
`TEXT IN
`THIS IS
`
`WINDOW
`ANOTHER
`
`WINDOW
`TEXT IN A �__....,... __ ...,.__,
`HIS IS SOME
`
`!
`FRAME BUFFER
`
`FIG. 2
`18
`
`10
`!
`
`GRAPHICS
`
`TRANSPARENT
`
`OUTPUT FRAME BUFFER
`
`OPERATING SYSTEM
`
`I -..
`
`STOCK TICKER ... 1
`
`I_
`22
`
`20
`
`J
`
`-
`
`L
`
`16
`
`WINDOW
`SECOND
`
`WINDOW
`ANOTHER TEXT IN
`WINDOW THIS IS
`TEXT IN A
`
`12
`
`,HIS IS SOME
`
`'
`
`-
`
`INTERLEAVING
`
`......
`
`......
`�
`�
`
`•
`
`•
`
`26
`
`I-------�
`
`MIXING
`COLOR
`
`24
`
`,----------.i
`
`Apple EX1007 Page 3
`IPR2021-00922
`
`

`

`U.S. Patent Jul. 11, 2000
`6,088,018
`
`Sheet 3 of 16
`
`102
`
`I
`
`MICROPROCESSOR 104
`
`100
`/
`
`CACHE
`
`MEMORY
`
`BRIDGE
`
`PROCESSOR
`BUS I
`105
`HOST 1)106
`1J8 HIGH PERFORMANCE
`
`1/0 BUS
`
`)10
`
`)12
`
`)14
`
`11 6
`I
`
`MAIN
`VIDEO
`1/0 BUS
`VIDEO
`BRIDGE MEMORY MEMORY DISPLAY
`
`)18 STANDARD 1/0 BUS
`)22
`
`)20
`
`J01
`
`MASS
`
`KEYBOARD AND
`
`
`
`STORAGE POINTING DEVICES
`
`VIDEO
`CAMERA
`
`FIG. 3
`
`IPR2021-00922
`Apple EX1007 Page 4
`
`

`

`U.S. Patent Jul. 11, 2000
`6,088,018
`
`Sheet 4 of 16
`
`212
`I
`
`200
`I
`
`OTHER
`
`APPLICATION
`
`PROGRAMS
`
`
`
`APPLICATION PROGRAMS
`
`USING TRANSPARENCY
`
`(
`204
`
`GRAPHICS API
`
`DISPLAY DRIVER
`208
`
`202
`TRANSPARENCY SUPPORT �
`
`
`
`VIDEO CONTROL API
`
`206
`
`VIDEO CARD
`
`VIDEO MEMORY
`
`210
`lJ
`
`114
`l/
`
`FIG. 4
`
`IPR2021-00922
`Apple EX1007 Page 5
`
`

`

`00
`
`= �
`
`....
`00
`= 00
`
`....
`0--,
`
`0 ...., '"""'
`
`Ul
`
`
`��....
`
`0
`0
`0
`N
`
`= :-'"""''"""'�
`�
`
` ......
`
` =
`
`�
`�
`•
`r:JJ.
`d •
`
`......�
`
`r300
`
`FRAME BUFFER
`FIRST WORKING
`
`I
`
`.
`I MEMORYi
`
`I
`
`I
`I
`
`OUTPUT FRAME
`
`BUFFER I
`SYSTEM
`
`OPERATING
`
`'
`I
`'
`
`FIG. 5
`
`� 10
`I
`
`SECOND WORKING
`
`FRAME BUFFER
`
`302
`
`�
`
`I
`
`I .
`
`I
`I
`
`I
`
`I
`
`FRAME BUFFER 2
`GRAPHICS
`TRANSPARENT
`
`�
`19
`
`I
`I
`I
`'
`
`I VIDEO
`I I
`I
`114
`
`MEMORY
`I MAIN
`112
`
`GRAPHICS
`
`I
`18
`
`304
`
`M/
`
`I N
`FRAME BUFFER 1 . 306
`TRANSPARENT I
`
`Apple EX1007 Page 6
`IPR2021-00922
`
`

`

`U.S. Patent
`6,088,018
`
`
`
`Sheet 6 of 16
`
`Jul. 11, 2000
`
`T+OS
`OS
`MIX
`
`T+OS
`OS
`MIX
`
`T+OS
`OS
`MIX
`
`• • •
`
`OS
`
`T+OS
`OS
`MIX
`
`T+OS
`OS
`MIX
`
`T+OS
`
`MIX
`
`T+OS
`OS
`MIX
`
`T+OS
`OS
`MIX
`
`T+OS
`OS
`MIX
`
`OS
`
`T+OS
`OS
`MIX
`
`T+OS
`OS
`MIX
`
`T+OS
`
`• • •
`MIX
`
`•
`
`•
`
`•
`
`•
`
`•
`
`•
`
`FIG. 6
`
`IPR2021-00922
`Apple EX1007 Page 7
`
`

`

`6,088,018
`U.S. Patent Jul. 11, 2000
`
`Sheet 7 of 16
`
`400
`
`DETERMINE OPERATING SYSTEM
`DISPLAY OUTPUT INFORMATION
`
`, '
`
`ALLOCATE TWO WORKING FRAME
`
`BUFFERS IN VIDEO MEMORY
`
`I
`
`COPY DATA FROM VISIBLE OPERATING
`
`SYSTEM OUTPUT FRAME BUFFER TO
`
`ONE OF THE ALLOCATED WORKING
`
`FRAME BUFFERS
`
`MAKE OPERATING SYSTEM
`
`OUTPUT FRAME BUFFER
`
`NON-VISIBLE
`
`,,
`
`MAKE SELECTED WORKING
`
`FRAME BUFFER VISIBLE
`
`FIG. 7
`
`IPR2021-00922
`Apple EX1007 Page 8
`
`

`

`6,088,018
`U.S. Patent Jul. 11, 2000
`
`Sheet 8 of 16
`410
`
`I
`
`START
`
`J12
`
`COPY OPERATING SYSTEM OUTPUT
`FRAME BUFFER TO NON-VISIBLE
`FIRST WORKING FRAME BUFFER
`
`4)4
`
`PERFORM MIXING OPERATION ONTO
`
`FIRST WORKING FRAME BUFFER WITH FIRST
`
`WORKING FRAME BUFFER AND SELECTED
`
`ONE OF THE TRANSPARENT
`
`GRAPHICS FRAME BUFFERS
`
`MAKE FIRST WORKING FRAME BUFFER
`
`)416
`
`
`
`VISIBLE AND SECOND WORKING
`
`FRAME BUFFER NON-VISIBLE
`
`1
`
`J18
`
`COPY OPERATING SYSTEM OUTPUT
`
`
`
`FRAME BUFFER TO NON-VISIBLE SECOND
`
`WORKING FRAME BUFFER
`
`J20
`
`PERFORM MIXING OPERATION ONTO
`
`SECOND WORKING FRAME BUFFER WITH SECOND
`
`WORKING FRAME BUFFER AND A SELECTED
`
`
`
`TRANSPARENT GRAPHICS FRAME BUFFER
`
`J22
`
`MAKE SECOND WORKING FRAME BUFFER
`
`VISIBLE AND FIRST WORKING
`
`FRAME BUFFER NON-VISIBLE
`
`FIG. 8
`
`IPR2021-00922
`Apple EX1007 Page 9
`
`

`

`U.S. Patent Jul. 11, 2000
`6,088,018
`
`Sheet 9 of 16
`
`426
`;
`DETERMINE MEMORY LOCATION IN
`WORKING FRAME BUFFER FOR PLACEMENT
`OF POINT M OF TRANSPARENT
`GRAPHICS FRAME BUFFER
`
`I
`
`'
`
`READ PIXEL FROM WORKING FRAME BUFFER J 428
`
`AND DETERMINE CORRESPONDING PIXEL FROM
`
`CURRENTLY SELECTED TRANSPARENT
`GRAPHICS FRAME BUFFER
`
`PERFORM WEIGHTED AVERAGE OF WORKING J 430
`FRAME BUFFER PIXEL AND TRANSPARENT
`GRAPHICS FRAME BUFFER PIXEL
`432
`I
`PLACE RESULT IN WORKING FRAME
`BUFFER AT SAME LOCATION
`434
`,
`I
`DETERMINE NEXT WORKING
`
`FRAME BUFFER LOCATION, TAKING INTO
`
`ACCOUNT INTERLEAVING PATTERN
`
`'
`
`,
`436
`MORE
`YES
`PIXELS
`?
`
`NO
`
`J38
`END
`
`FIG. 9
`
`IPR2021-00922
`Apple EX1007 Page 10
`
`

`

`U.S. Patent Jul. 11, 2000
`16 6,088,018
`Sheet 10 of
`
`502
`l
`
`APPLICATION
`PROGRAM
`
`5JB
`5J6
`HANDLE
`FILTER
`INPUT
`INITIALIZATION EVENTS
`
`I
`
`701
`VIDEO I
`CAMERA_,
`
`COM
`INTERFACES
`
`512
`
`04
`J
`J10
`
`FILTER
`GRAPH
`MANAGER
`
`5JD
`VIDEO
`DATA
`
`FILTER
`GRAPH
`
`FIG. 10
`
`IPR2021-00922
`Apple EX1007 Page 11
`
`

`

`U.S. Patent Jul. 11,
`16 6,088,018
`Sheet 11 of
`2000
`
`ll
`
`0
`
`c::
`�-----
`C:: LU
`0N
`...J >-
`0 ...J
`C.) �
`<(
`
`N
`
`�----- c::
`0
`co I-
`0 u
`...J LU
`co I-LU
`Cl
`
`I
`
`co
`��
`Cl<C LU
`z a.. I-
`�-----
`0 (/) �
`UC:: LU
`LUO� (/) ...J 0Ou u
`
`>-
`
`co
`c:: u zLU LU
`�-----
`0 C:: I C::
`LU LU 1-<(Cl Cl - a..
`-z �(/)>LU z
`� �
`I-
`
`v
`
`LU C:: �LU
`�'
`I-a.. I-
`(/) (/) c::
`c::� LU
`u.o�...J 0Ou u
`
`N
`
`LULU
`�'
`'
`
`0
`N
`
`LU
`I{)'
`0�LU :::>
`Cl I-
`-c..
`> <(u
`
`'
`
`�ILU a..
`��
`u::: (.')
`
`IPR2021-00922
`Apple EX1007 Page 12
`
`

`

`I
`
`'
`
`U.S. Patent Jul. 11, 2000
`16 6,088,018
`Sheet 12 of
`540
`CREATE A FILTER �
`GRAPH
`J42
`LOCATE VIDEO
`RENDERER WITH
`
`TRANSPARENCY FILTER
`
`IN THE FILTER GRAPH
`J44
`SET DESIRED LEVEL OF
`OPACITY INTO THE VIDEO
`RENDERER WITH
`
`TRANSPARENCY FILTER
`5J6
`LOCATE COLOR
`ANALYZER FILTER
`IN THE FILTER GRAPH
`' J48
`SET COLOR RANGES TO BE
`TRACKED AND CORRESPONDING
`ALPHA VALUES INTO THE
`COLOR ANALYZER FILTER
`J50
`FIG. 12
`LOCATE BLOB DETECTOR
`FILTER IN THE FILTER GRAPH
`J52
`SET CONTROL WINDOW ID,
`ALPHA VALUES AND CORRESPONDING
`
`CONTROL MESSAGES INTO THE
`BLOB DETECTOR FILTER
`
`'
`
`I
`
`START DATA FLOW l,/55 4
`THROUGH THE FILTER GRAPH
`
`IPR2021-00922
`Apple EX1007 Page 13
`
`

`

`U.S. Patent Jul. 11, 2000
`16 6,088,018
`Sheet 13 of
`
`560
`
`WINDOW
`MESSAGE
`READY?
`
`NO
`
`YES
`
`NO
`
`566
`
`BLOB TYPE 2
`DETECTED
`?
`
`562
`
`BLOB TYPE 1
`DETECTED
`?
`
`YES
`
`568
`
`NO RECORD BLOB LOCATION
`AND SIZE IN TYPE 2 LIST
`
`YES
`
`564
`
`RECORD BLOB LOCATION
`AND SIZE IN TYPE 1 LIST
`
`NO
`
`YES
`
`572
`
`---------i
`
`DETECT VALID
`INPUT EVENT
`
`574
`
`APPLICATION
`CLOSE MESSAGE
`
`NO
`
`578
`
`FIG. 13
`
`580
`
`DELETE THE
`FILTER GRAPH -------1
`
`STOP THE
`FILTER GRAPH
`
`EXIT THE APPLICATION
`PROGRAM
`
`IPR2021-00922
`Apple EX1007 Page 14
`
`

`

`U.S. Patent Jul. 11, 2000
`16 6,088,018
`Sheet 14 of
`
`START
`
`600
`I
`
`END OF BLOB
`TYPE 1 LIST?
`
`NO
`
`YES
`
`602
`
`FIND CLOSEST BLOB IN
`BLOB TYPE 2 LIST AND
`RECORD INDEX OF BLOB
`TYPE 2 IN BLOB TYPE 1
`LIST
`
`604
`
`GO TO NEXT ENTRY
`IN BLOB TYPE 1 LIST
`
`NO
`
`YES
`
`606
`
`FIND CLOSEST TYPE 1/TYPE 2
`
`PAIR IN BLOB TYPE 1 LIST
`
`608
`
`ISTHE
`CLOSEST PAIR LESS
`THAN THRESHOLD
`NO
`DISTANCE APART?
`
`YES
`
`610
`
`SUBMIT INPUT EVENT TO
`OPERATING SYSTEM
`
`611
`
`SUBMIT OTHER INPUT
`EVENT TO OPERA TING
`SYSTEM
`
`--, 612
`-------'-----
`
`RESET TYPE 1 BLOB LIST
`AND TYPE 2 BLOB LIST
`
`��__,614
`
`EXIT
`
`FIG. 14
`
`IPR2021-00922
`Apple EX1007 Page 15
`
`

`

`U.S. Patent Jul. 11, 2000
`6,088,018
`
`Sheet 15 of 16
`
`,,--- - --._700
`START
`
`702
`
`CREATE VIDEO CONTROL API OBJECT,
`THREE VIDEO FRAME BUFFERS AND
`SECONDARY THREAD
`
`704
`
`IF TRANSPARENT GRAPHICS FRAME
`
`BUFFERS DO NOT EXIST OR THEIR SIZES
`HAVE CHANGED, CREATE NEW TRANSPARENT
`GRAPHICS FRAME BUFFERS AND MARK THE FIRST
`
`TRANSPARENT GRAPHICS FRAME BUFFER AS ACTIVE
`
`-�-----------,
`.---------
`706
`CREATE A GDI DISPLAY CONTEXT (DC) FOR THE
`
`ACTIVE TRANSPARENT GRAPHICS FRAME BUFFER
`
`.-------�--------,
`708
`USE GDI TO DRAW VIDEO ON THE
`
`TRANSPARENT DISPLAY CONTEXT
`
`MARK INACTIVE TRANSPARENT GRAPHICS 710
`
`
`FRAME BUFFER AS ACTIVE AND MARK ACTIVE
`
`TRANSPARENT GRAPHICS FRAME BUFFER AS INACTIVE
`
`712
`�-------L--------�
`
`DELETE TRANSPARENT DISPLAY CONTEXT
`
`�-------L--------�
`714
`SET SYSTEM EVENT TO INDICATE THAT
`
`NEW FRAME BUFFER IS READY FOR
`MIXING
`
`716
`
`718
`
`720
`
`MORE VIDEO
`FRAMES TO
`PROCESS? NO
`
`DESTROY
`WAIT FOR SECONDARY
`FRAME
`THREAD TO TERMINATE
`BUFFERS
`
`722
`
`FIG. 15 END
`
`IPR2021-00922
`Apple EX1007 Page 16
`
`

`

`U.S. Patent Jul. 11, 2000
`Sheet 16 of 16
`
`6,088,018
`
`740
`
`SECONDARY THREAD CREATED
`
`742
`
`WAIT FOR A TRANSPARENT GRAPHICS
`
`FRAME BUFFER TO BE COMPLETED
`
`744
`
`BLOCK TRANSFER CONTENTS OF OPERATING
`SYSTEM OUTPUT FRAME BUFFER TO BOTH
`WORKING FRAME BUFFERS
`
`746
`
`SET OPERATING SYSTEM OUTPUT FRAME
`
`
`
`BUFFER TO NON-VISIBLE STATE AND SET
`
`
`
`ONE WORKING FRAME BUFFER TO VISIBLE STATE
`
`WAIT PREDETERMINED TIME OR FOR A
`
`
`TRANSPARENT GRAPHICS FRAME BUFFER
`TO BE COMPLETED OR FOR INDICATION
`
`748
`
`THAT NEW FRAME BUFFER IS READY FOR MIXING
`
`750
`
`MIX CONTENTS OF INACTIVE TRANSPARENT GRAPHICS
`
`
`FRAME BUFFER WITH THE NON-VISIBLE WORKING FRAME BUFFER
`
`752
`
`FLIP THE TWO WORKING FRAME BUFFERS SO THE
`
`
`
`VISIBLE ONE BECOMES NON-VISIBLE AND THE
`
`NON-VISIBLE ONE BECOMES VISIBLE
`
`754
`
`YES
`
`756
`
`TERMINATE SECONDARY
`
`THREAD
`
`FIG. 16
`
`IPR2021-00922
`Apple EX1007 Page 17
`
`

`

`1
`
`6,088,018
`
`
`
`IN PROVIDING INPUT DATA TO A
`
`COMPUTER SYSTEM
`
`BACKGROUND
`
`2
`FIG. 6 is a diagram illustrating an alternating pixel
`
`
`
`METHOD OF USING VIDEO REFLECTION
`
`
`
`
`technique mixing between the transparent graphics frame
`
`
`buffer and the operating system output frame buffer accord­
`
`
`ing to an embodiment of the present invention;
`
`
`
`A portion of the disclosure of this patent document
`
`
`FIG. 7 is a flow diagram for initializing a system to
`5
`
`
`
`
`contains material which is subject to copyright protection.
`
`
`
`
`
`provide transparent graphics according to an embodiment of
`The copyright owner has no objection to the facsimile
`
`
`the present invention;
`reproduction by anyone of the patent document or the patent
`
`
`
`
`
`
`FIG. 8 is a flow diagram showing double buffering control
`
`
`
`
`
`disclosure, as it appears in the Patent and Trademark Office
`
`
`
`processing according to an embodiment of the present
`
`
`
`patent file or records, but otherwise reserves all copyright
`
`10 invention; and
`
`rights whatsoever.
`
`FIG. 9 is a flow diagram of color mixing and interleaving
`
`
`
`processing according to an embodiment of the present
`invention.
`1. Field
`FIG. 10 is a diagram illustrating components of an appli­
`
`
`
`15
`
`
`
`
`The present invention relates generally to computer sys-
`
`
`
`
`cation program providing a video reflection input technique
`
`
`
`tems and more specifically to a method for providing input
`
`
`
`according to an embodiment of the present invention;
`
`data to a computer system.
`
`
`
`to an graph according FIG. 11 is a diagram of a filter
`2.Description
`
`embodiment of the present invention;
`
`
`
`FIG.12 is a flow diagram of filter initialization processing
`
`
`
`Many techniques and devices for communicating input 20
`data from a user to a computer system have been developed.
`
`
`
`
`
`
`according to an embodiment of the present invention;
`Keyboards, cursor movement devices (such as a mouse,
`
`
`
`
`
`
`
`FIG. 13 is a flow diagram of handle input events process­
`
`
`
`trackball, and joystick, for example), pens and tablets, bar
`
`
`
`ing according to an embodiment of the present invention;
`
`
`
`code scanners, and other devices have all been used to direct
`
`
`
`
`for detecting valid FIG. 14 is a flow diagram of processing
`
`
`
`a computer to perform selected tasks. With the growing use
`
`
`25 input events according to an embodiment of the present
`of video cameras coupled to computer systems such as
`
`
`
`
`invention;
`
`
`
`
`
`personal computers (PCs), for example, opportunities for
`FIG. 15 is a flow diagram describing a processing thread
`
`
`
`
`
`using real-time video data in new and interesting ways have
`
`
`
`of a video renderer with transparency according to an
`
`
`
`
`
`arisen. However, many recent applications for video data in
`
`
`embodiment of the present invention; and
`
`
`
`PCs have been focused on various display features, without
`
`
`
`FIG. 16 is a flow diagram describing a processing thread
`
`
`regard for the possibilities of new input paradigms. What
`
`
`
`of a transparency mixer core according to an embodiment of
`
`
`
`may be valuable are input techniques that take advantage of
`
`the present invention.
`
`the capabilities of video cameras to enrich the PC user's
`
`
`experience and make PCs easier to use.
`DETAILED DESCRIPTION
`
`30
`
`35
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`
`
`SUMMARY
`In the following description, various aspects of the
`
`
`
`
`
`
`
`present invention will be described. For purposes of
`
`
`An embodiment of the present invention is a method of
`
`
`
`
`explanation, specific numbers, systems and configurations
`providing input signals to a system having a display, the
`
`
`
`
`
`are set forth in order to provide a thorough understanding of
`
`
`system being coupled to a source of video data signals. The
`
`
`
`
`
`
`40 the present invention. However, it will also be apparent to
`
`method includes capturing video data signals generated by
`
`
`
`
`
`
`one skilled in the art that the present invention may be
`
`
`
`the video source, the video data signals representing a scene,
`
`
`
`
`
`
`practiced without the specific details. In other instances, well
`
`
`rendering the scene on the display such that the scene is
`
`known features are omitted or simplified in order not to
`
`
`
`
`
`
`transparently visible on the display, analyzing the video data
`obscure the present invention.
`
`
`
`
`
`signals to detect an object in the scene, and generating an
`
`
`An embodiment of the present invention is a method for
`
`
`
`
`input signal for the system in response to the detected object.
`45
`
`
`
`providing user input data signals to a computer system such
`
`
`as a PC, for example. Real-time video images may be
`
`
`
`
`captured by a video camera coupled to a PC or received from
`
`
`
`
`
`The features and advantages of the present invention will
`
`
`
`any other source of video data signals and rendered to the
`
`
`
`
`
`become apparent from the following detailed description of
`
`
`
`
`video display such that the captured 50 entire screen of the PC's
`
`
`the present invention in which:
`
`
`images may be made transparent and shown on the display.
`
`
`
`FIG. 1 is a diagram of a sample display portion illustrating
`
`
`The video data signals may be mixed with the normal
`
`
`
`video reflection according to an embodiment of the present
`
`
`
`
`application program and operating system software display
`invention;
`
`
`
`output data signals so as not to obscure the normal fore-
`
`
`FIG. 2 is a diagram illustrating an example of transparent
`
`
`55 ground display images. The resulting effect is that when the
`
`
`
`
`graphics data displayed with operating system output graph­
`
`
`
`video camera is pointed toward the user, the user may see a
`
`
`
`ics data according to an embodiment of the present inven­
`
`
`
`reflected image of himself or herself on the computer screen
`tion;
`
`
`
`while the user may also be able to see and interact with other
`
`
`
`FIG. 3 is a diagram illustrating a sample computer system
`
`
`
`
`display elements such as desktop icons or application pro-
`
`
`
`suitable to be programmed with a transparency method
`
`
`60 gram displays. Because the reflected image is displayed
`
`
`
`
`according to an embodiment of the present invention;
`
`
`
`transparently, it does not interfere with the display of the
`
`
`
`
`
`FIG. 4 is a diagram of a software and hardware stack for other display elements.
`
`
`
`
`implementing transparent graphics according to an embodi­
`Although some embodiments of the present invention are
`
`
`
`
`
`ment of the present invention;
`
`
`
`described herein with reference to a video camera, the
`
`
`
`
`
`
`FIG. 5 is a diagram illustrating multiple frame buffers 65 invention is not so limited in this respect. In alternate
`
`
`
`
`
`
`
`
`used for providing transparent graphics according to an embodiments, the source of the video data signals may be
`(VCR),
`
`
`
`
`embodiment of the present invention; any video source, such as a video cassette recorder
`
`IPR2021-00922
`Apple EX1007 Page 18
`
`

`

`6,088,018
`
`4
`3
`broadcast television signals, or previously stored video data
`
`
`finger and thumb and the images of the index finger and
`
`
`signals, for example.
`
`
`
`
`thumb are detected in the captured video images, a mouse
`
`down event may be introduced to the operating system
`
`
`
`One of the features of some embodiments of the present
`
`
`
`software. When the user releases the pinched hand position
`
`
`invention is that the user may see his or her reflection or
`
`
`
`and these images of the index finger and thumb are detected
`
`
`
`image on the computer screen after it is captured by the 5
`
`
`in the captured video images, a mouse up event may be
`
`
`video camera. Using this video reflection, the user may
`
`
`
`
`introduced. Other movements signifying specific input sig­
`
`
`
`
`interact with application program and operating system
`
`
`
`nals and events may also be defined in alternate embodi­
`
`
`
`
`display elements of the computer system's graphical user
`ments.
`
`
`
`
`interface (GUI) by physically moving real objects (such as
`FIG. 1 is a diagram of a sample display portion illustrating
`
`
`
`
`
`
`
`
`the user's hands and fingers, for example) that are in the field
`10
`
`
`
`video reflection according to an embodiment of the present
`
`
`
`of view of the video camera. The input capability of some
`
`
`
`invention. In this example, the image of the user is captured
`
`
`
`embodiments of the present invention may be provided
`
`
`
`by the video camera and rendered in a transparent manner to
`
`
`
`
`without any modification to existing application programs.
`
`
`
`
`
`the display. The user then interacts with display objects, such
`
`
`
`In embodiments of the present invention, these movements
`
`
`
`
`as application program icons, for example, to provide input
`
`
`
`
`may be used to direct actions within the computer system 15
`
`
`signals and events to the system.
`
`
`such as grasping or selecting icons and other application
`An underlying capability of some embodiments of the
`
`
`
`
`program elements, much as the user now does with a mouse
`
`
`present invention is a method of providing a transparent
`
`
`
`
`or other selection device. This physical movement activity to
`
`
`layer of display data signals (such as video data signals
`
`
`
`
`
`create a desired result while observing the user's reflected
`
`
`
`communicated by a video camera, for example) over the top
`
`
`
`image is very natural and intuitive for users. For example,
`20
`
`
`
`of another layer of display data signals on a computer
`
`
`
`
`users frequently look at reflected images of themselves in a
`
`
`
`display so that the user may see both layers clearly and
`
`
`
`
`mirror, for example, while manipulating objects with their
`
`
`
`substantially simultaneously. This capability will be
`
`
`
`hands without thinking very much about it. Additionally, it
`
`
`
`described first for general use and further below for use with
`
`
`
`
`is typically easy for users to use either a dominant or
`
`
`an input detection method of some embodiments of the
`
`non-dominant hand or both hands to perform operations
`
`
`
`
`present invention. The capability to display transparent
`25
`
`
`while viewing their reflected image. In alternate
`
`
`windows doubles, in essence, the maximum screen area
`
`
`
`
`embodiments, specific predetermined props may be used as
`
`
`
`available on a display for use by application programs. One
`
`
`the objects to direct actions within the computer. These
`
`
`
`embodiment is a method for producing transparent computer
`
`
`
`props may be active or inactive. Active props contain
`
`graphics layers by interleaving ( or alternating in a pattern)
`
`
`
`mechanical or electrical mechanisms such as light emitting
`
`
`the pixels from one video frame buffer with the pixels from
`30
`
`
`
`
`diode (LED) lights. Inactive props are items such as colored
`
`another video frame buffer. In this embodiment, selected
`
`
`
`
`sticks or paddles, fabric swatches, and colored gloves, for
`
`
`pixels from a first frame buffer are mixed by color averaging
`
`
`
`
`
`example, although the present invention is clearly not lim­
`
`
`with corresponding pixels from a second frame buffer to
`ited in this respect.
`
`
`reduce the "checkerboard" effect created by the use of
`
`
`
`
`spatial multiplexing alone. Additionally, because the degree
`
`
`
`While some embodiments of the present invention may 35
`
`
`
`of interleaving is adjustable and the color averaging may be
`
`
`
`not replace the use of a computer mouse entirely, it is a
`
`
`weighted, the degree of transparency of the displayed
`
`
`
`
`powerful new paradigm for user input in a computer system.
`images may be controlled.
`
`
`
`In particular, it presents a natural and direct method for
`
`
`manipulating computer display objects by moving real
`
`In this embodiment, an output frame buffer used by
`
`
`objects. Various types of application programs may be
`
`
`
`
`operating system software is not affected by provision of the
`40
`
`
`extended to make beneficial use of this input paradigm.
`
`
`
`
`transparency feature and the operating system is unaware of
`
`
`
`Computer games, for example, may employ user input
`
`
`
`the transparency operations. Hence, the transparency effect
`
`through video reflection for increased user interaction.
`
`
`
`
`
`provided by embodiments of the present invention does not
`
`
`
`
`Video presentation systems wherein a user points to selected
`
`
`
`
`require modifications to application programs for transpar-
`
`
`areas of video display data may be made easier to use.
`
`
`ency to work over them. Furthermore, input operations to
`45
`
`
`
`Video-conferencing applications may be extended to pro­
`
`
`application program and operating system windows are not
`
`
`
`vide additional input features. A combination PC/television
`
`
`
`affected by transparent foreground effects.
`
`
`
`(TV) may make use of this input capability, especially where
`An embodiment of the present invention operates by
`
`
`
`
`
`
`
`
`
`the distance between the user and display is relatively large.
`
`
`
`combining at least two frame buffers of computer graphics
`
`
`
`For many users, such as children, for example, the manipu-50
`
`
`output data or video data in the form of electrical signals.
`
`
`
`lation of real objects (such as their hands) to send input to
`
`
`The pixels of the output, or visible, frame buffer are created
`
`
`
`a computer system is more intuitive than other typical input
`
`
`by spatially interleaving the contents of two input frame
`devices.
`
`
`
`buffers. The interleaving in this embodiment is accom­
`
`
`In one embodiment, the colors of pixels in the video data
`
`
`
`plished by combining pixels of one frame buffer with those
`
`
`
`
`signals sent to the PC for processing and display may be 55
`
`
`
`
`of the other frame buffer. This results in the visual illusion
`
`
`
`analyzed to track the movement of real objects such as a
`
`
`
`
`of two displays of images layered one on another. As the
`
`
`
`
`user's hands. This may be assisted through the use of colored
`
`pixels are being interleaved, the pixels of the first frame
`
`
`
`
`
`dots attached to the user's hands or fingers or through other
`
`
`buffer are color averaged with the pixels of the second frame
`
`
`
`computer vision methods. Optionally, distinctive colors may
`
`
`buffer that they are about to replace. Color averaging is
`
`
`
`be placed on specific objects (like fingers and thumbs, for
`
`
`
`performed on the pixels of one frame buffer by averaging
`60
`
`
`
`
`example, or other props), to help the computer vision
`
`
`them with the corresponding pixels of the other frame buffer
`
`
`
`components of embodiments of the present invention isolate
`
`prior to, or during, interleaving them into the output frame
`
`
`
`and identify objects to be recognized as input event genera­
`
`
`
`
`buffer. The result comprises multiple overlapping images
`tors.
`
`
`
`
`being substantially simultaneously visible on a display such
`
`65 as a computer monitor, for example.
`
`
`
`In one embodiment, the cursor of the PC's mouse may be
`
`
`
`
`
`
`
`
`moved in accordance with the movement of a user's index FIG. 2 is a diagram illustrating an example of transparent
`
`
`
`
`
`
`finger and thumb. When the user "pinches" his or her index graphics data displayed with operating system output graph-
`
`IPR2021-00922
`Apple EX1007 Page 19
`
`

`

`6,088,018
`
`6
`5
`ics data according to an embodiment of the present inven­
`
`
`
`strikes to the input window (such as a mouse entry or text
`
`
`
`
`tion. Operating system output frame buffer 10 is an area of
`
`
`entry) without substantial interference with the display of
`
`
`memory used to store the current display data of the com­
`
`
`the transparent display components.
`
`
`puter system shown below in FIG. 3. The operating system
`
`
`
`In accordance with some embodiments of the present
`
`
`output frame buffer may be allocated in any memory avail­
`
`
`
`effects transparency 5 invention, a method for producing
`
`
`
`able to the operating system. A frame buffer is a set of
`
`
`
`
`
`employs minimal mixing of display contents. Instead, it
`
`
`
`storage locations to store a two-dimensional array of pixel
`
`
`
`
`relies on the human eye's inability to distinguish between
`
`data. The operating system output frame buffer may be
`
`
`
`the color of adjacent pixels on a computer monitor (in
`
`
`
`associated with operating system software of the computer
`
`
`
`essence, the human eye averages each pixel with its
`10 neighbor).
`
`
`
`
`
`system, which controls the generation and display of the data
`
`
`Some mixing is employed, because large com­
`
`
`
`
`signals on a computer monitor (not shown). In one
`
`
`
`
`
`puter monitors and low display resolutions may result in a
`
`
`
`
`embodiment, the operating system software comprises the
`
`
`
`"checkerboard" effect when pixels are interleaved in this
`
`
`Windows 95® or Windows NT® operating system software
`
`
`
`
`manner. In one embodiment, one half of the pixels from a
`
`
`
`available from Microsoft Corporation, although other oper­
`
`
`
`first frame buffer (such as the operating system output frame
`
`
`
`ating system software supporting graphical user interfaces
`
`
`
`half of the15 buffer) are averaged with the corresponding one
`
`
`
`
`may also be employed. In this example, the operating system
`
`
`
`pixels from a second frame buffer (such as the transparent
`
`
`
`output frame buffer 10 contains application program display
`
`
`
`graphics frame buffer) as the pixels of the two frame buffers
`
`
`
`data signals for three overlapping windows shown pictori­
`
`
`
`are interleaved into a third display buffer. By averaging a
`
`ally in FIG. 2 and labeled 12, 14, and 16, respectively.
`
`
`
`
`
`fraction of the pixels, there may be a decrease in the
`
`
`the transparency 20 processing power used when providing
`
`
`
`Transparent graphics frame buffer 18 is an area of
`
`
`
`
`effect. In alternate embodiments, different percentages of
`
`
`memory used to store the display data of transparent graph­
`
`
`
`
`pixels may be averaged (e.g., one fourth of the pixels, one
`
`
`
`
`ics for substantially simultaneous display with the display
`
`
`
`
`
`eighth of the pixels, one sixteenth of the pixels, one thirty­
`
`
`
`data signals of the operating system output frame buffer.
`
`
`
`second of the pixels, or any one Nth of the pixels where N
`
`
`This area of memory may be allocated in any memory
`
`
`is a positive integer), and the percentages may be changed
`
`
`
`
`
`available in the system. Transparent graphics frame buffer
`25
`dynamically
`
`
`18 may be used to store frames of video data received from
`
`
`
`a video camera or other source of video data signals. In the
`FIG. 3 is a diagram illustrating a sample computer system
`
`
`
`
`
`
`example shown in FIG. 2, display components such as a
`
`
`
`
`suitable to be programmed according to an embodiment of
`
`
`clock 20 and stock ticker 22 are shown as sample application
`
`
`
`a method for producing transparency displays in conjunction
`
`
`program display features which illustrate the use of
`
`
`
`30 with video reflection to obtain user input signals in accor­
`
`
`
`
`transparency, although generally any display components
`
`
`
`dance with the present invention. Sample system 100 may be
`
`
`
`may be made transparent through the use of embodiments of
`
`
`
`used, for example, to execute the processing for the methods
`
`
`
`
`the present invention. More specifically, in embodiments of
`
`
`described herein. Sample system 100 is representative of
`
`
`the present invention, captured video frames may be shown
`
`
`computer systems based on the PENTIUM®, PENTIUM®
`35 Pro, and PENTIUM® II microprocessors available from
`
`
`
`
`as a sequence of transparent images on the display.
`
`
`
`
`Intel Corporation, although other systems (including per­
`
`
`
`The display components of the operating system output
`
`
`sonal computers (PCs) having other microprocessors, engi­
`
`
`
`frame buffer and the transparent graphics frame buffer may
`
`
`
`
`neering workstations, set-top boxes and the like) may also be
`
`
`be combined in block 24 by color mixing selected corre­
`
`
`used. Sample system 100 includes microprocessor 102 and
`
`sponding pixels of each buffer while interleaving the result­
`
`
`
`
`
`40 cache memory 104 coupled to each other through processor
`ing pixels of the color mixing operation with the operating
`
`
`
`
`bus 105. Sample system 100 also includes high performance
`
`
`
`
`system output frame buffer's pixels to form the display
`
`
`1/0 bus 108 and standard 1/0 bus 118. Processor bus 105 and
`
`
`
`
`components of visible display b