(12) United States Patent
`Roseman
`
`111111
`
`1111111111111111111111111111111111111111111111111111111111111
`US006608636Bl
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,608,636 Bl
`Aug. 19,2003
`
`(54) SERVER BASED VIRTUAL CONFERENCING
`
`(75)
`
`Inventor: Robert D. Roseman, Kettering, OH
`(US)
`
`(73) Assignee: NCR Corporation, Dayton, OH (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21) Appl. No.: 07/883,623
`
`(22) Filed:
`
`May 13, 1992
`
`Int. Cl? ................................................ G06F 15/16
`(51)
`(52) U.S. Cl. ....................................................... 345/753
`(58) Field of Search ............................................ 379/53
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`4,400,724 A * 8/1983 Fields
`......................... 358/85
`4,650,929 A * 3/1987 Boerger et a!.
`............... 358/86
`4,893,326 A * 1!1990 Duran et a!. .................. 379/53
`4,965,819 A * 10/1990 Kannes ........................ 379/53
`4,995,071 A * 2/1991 Weber et a!.
`................. 379/53
`5,099,510 A * 3/1992 Blinken, Jr. eta!. ........ 379/202
`5,208,912 A * 5/1993 Nakayama et a!.
`......... 395/200
`5,220,657 A * 6/1993 Bly et a!.
`................... 395/425
`5,315,633 A * 5/1994 Champa ...................... 348/16
`5,363,507 A * 11/1994 Nakayama et a!.
`......... 395/800
`5,611,038 A * 3/1997 Shaw eta!. ................. 395/806
`
`OTHER PUBLICATIONS
`
`NEC Research and Development-"Multimedia Desktop
`Conferencing System:MERMAID" by Watabe, Sakata,
`Maeno, Fukuoka & Ohmori-vol. 32 No.1 pp 158-167 Jan.
`1991.
`I.E.E.E. 1991-"Teams for Tomorrow" by Robert Johansen
`pp 521-534.
`Management Review-"Computer Conferencing: Brain(cid:173)
`stormingg Across Time and Space" by Marlene Piturro Aug.
`1989 pp 43-50.
`Presentation Products Magazine-"The Electronic Meet(cid:173)
`ing" by Robert Moskowitz Sep. 1990 pp 24--32.
`
`Gesellschaft Fur Mathematik und Datenverarbeitung
`MBH-"The Object Oriented Approach in CSCW" No. 498
`pp 1-60 Jan. 1991 Igor T. Hawryszkiewycz.
`Interacting with Computer vol. 3 No.3 (1991) "A Survey of
`CSCW Systems" by Tom Rodden pp 319-353.
`ShareVision Technology Inc., 1992; Preliminary Informa(cid:173)
`tion from ShareVision Technology "The Need to Bring
`People Together".
`IBM Corporation 1991; Ultimedia Multimedia Solutions
`From IBM; Person to Person; "Personal Conferencing Com(cid:173)
`munications That Work"; 10-91.
`SMART Technologies Inc.; "SMART 2000 Conferencing
`System for Multimedia Interactive Meetings"; 11!92.
`Fujitsu Networks Industry, Inc.; Product Description; Desk(cid:173)
`top Conferencing "Novell Local Area Networks"; 2/93.
`Fujitsu Networks Industry, Inc.; Desk Top Conferencing for
`Windows; "Real Life. Real Time. Real Power"; BR-001.
`Northern Telecom; VISIT Video; "VISIT Video Product
`Highlights".
`
`(List continued on next page.)
`
`Primary Examiner-Christopher B. Shin
`(74) Attorney, Agent, or Firm-Gregory A Welte
`
`(57)
`
`ABSTRACT
`
`The invention concerns remote conferencing systems of the
`multi-media type. An example of a "single-medium" con(cid:173)
`ference system can be found in presently available telephone
`conferencing systems. The single medium is sound, or voice.
`In the conference, multiple parties, at different locations, are
`linked together by the telephone system and can speak to
`each other.
`
`With "multi-media" conferencing, multiple parties are
`linked by both video and audio media: the parties can see,
`as well as hear, each other. Further, the invention shows the
`parties a common display, which functions as a "conference
`table." Still further, any party can modify the display, by
`"writing" on it, or by pointing to different parts of it; the
`other parties can see the modifications. Further still, the
`invention allows parties to "whisper" to each other, without
`the knowledge of other parties, as in an actual conference.
`
`8 Claims, 22 Drawing Sheets
`
`SURVEY
`CAMERA
`
`Facebook Inc.'s Exhibit 1003
`
`001
`
`

`
`US 6,608,636 Bl
`Page 2
`
`01HER PUBLICATIONS
`
`ES•F2F; The Window of Opportunity, "Videoconferencing
`On Any Wire", April Streeter.
`Business Week/Jun. 11, 1990; Information Processing "At
`These Shouting Matches, No One Says A Word", Jim
`Bartimo; p. 78.
`The New York Times; Sun., Jun. 23, 1991; Technology;
`"Plugging The Gap Between E-Mail And Video Conferenc(cid:173)
`ing", Edmund Andrews; pp. 8-9.
`Elsevier Science Publishers B.V., 1988; Application of Dis(cid:173)
`tributed Systems "Distributed Computer Conferencing",
`Jacob Palme (Computer Networks and ISDN Systems 14)
`1987; pp. 137-145.
`Patricia Seybold's Office Computing Group, 1988; vol. 11,
`No. 9; Technology That Supports Meetings "We Can't Go
`On Meeting Like This!"; 9/88.
`American Management Assoc.; Information Management
`FORUM; "Group Support Systems And Electronic Collabo(cid:173)
`ration", Bye Wynne; 5/91.
`American Management Assoc.; Information Management
`FORUM; "Groupware And Your Future", Bye Wynne;
`11!89.
`Campus News, MMR •Summer 1990; "Texaco Contributes
`to Group Decision Support Services Project"; p. 23.
`THINK; No. 1!1990; "How To Have An "Electronic" Brain(cid:173)
`strom" Mason Southworth; pp. 12-13.
`Technology; "Software Catches The Team Spirit", Louis
`Richman; Reprinted from FORTUNE, 1987 TIME Inc.
`Info World, vol. 11, Issue 5; Resolution Evolution; Man(cid:173)
`agement, "Workgroup Idea Still Unclear To Users", Alice
`LaPlante; Dec. 11, 1989.
`
`"Meeting Support-An Emerging Market"; p. 69.
`Creative Classroom Corp, 1991; ENTENTE' Turnkey Elec(cid:173)
`tronic Classroom; "It's Hard To Get Lost In This Class(cid:173)
`room".
`3M Meeting Management Institute; Meeting Management
`News, vol. 3, No. 1; "The Year 2000: Expect Meetings To
`Change, Not Decline"; 4/91.
`Info World; vol. 11, Issue 49; Case Tools; Workgroup
`Computing; IBM Study: PCs Aid In Decision Process; pp. 1,
`8.
`Raymond Panko, Patterns of Organizational Communica(cid:173)
`tion; pp. 1-3; Tables: Use of Study Time; pp. 1-4.
`The New York Times, Sun. Mar. 24, 1991; "Brainstorming
`By Modem".
`Hitachi Review, vol. 40 (1991), No. 3; "Broadband ISDN
`Multimedia Workstations And Group Tele-Working Sys(cid:173)
`tems", T. Hoshi, F. Nakamura, and S. Nakamura; pp.
`217-222.
`3M Meeting Management Institute; "Meeting Management
`Bibliography"; Mar. 7, 1991.
`The University of Arizona, Dept. of Management Informa(cid:173)
`tion Systems; "Bibliography Group Decision Support Sys(cid:173)
`tems".
`3M Meeting Management Institute; Meeting Management
`News, vol. 3, No. 3; "Study Focuses On Use Of Rules In
`Meetings"; 12/91.
`Share Vision Technology Inc.; Feb. 1993; "Create A Meeting
`of the Minds".
`
`* cited by examiner
`
`Facebook Inc.'s Exhibit 1003
`
`002
`
`

`
`U.S. Patent
`
`Aug. 19,2003
`
`Sheet 1 of 22
`
`US 6,608,636 Bl
`
`CLOSE- UP CAMERA
`
`FIG.l
`
`SURVEY
`CAMERA
`
`~----
`'
`I I
`
`CONFERENCE
`ROOM 17L
`(DATE, TIME)
`
`I
`
`\ I \ \ ._____------
`
`FIG. 8
`
`------
`
`-------
`
`I
`
`__)
`
`Facebook Inc.'s Exhibit 1003
`
`003
`
`

`
`----
`XSTHER ICONS
`1 I I
`
`\ \_ ---
`
`FIG. 2
`
`VIRTUAL CONFERENCE
`
`----
`
`-----
`1 ! .. .1.~ . I
`~. . I
`l '
`\
`'
`.
`I
`\_
`-----
`_)
`
`HOW DO YOU WISH TO IDENTIFY INVITEES?
`D SEND LIST TO HOST AS DATA FILE?
`D PICK INVITEES FROM DIRECTORY?
`D WALK THE HALLS ?
`
`-
`
`~
`
`\ \ I l I
`____ )
`
`I I
`
`d •
`\Jl
`•
`~
`~ ......
`~ = ......
`
`~
`~
`'"""' ~~
`N c
`8
`
`'JJ. =(cid:173)~
`~ .....
`N
`0 ......,
`N
`N
`
`e
`rJ'l
`-..a-..
`a-..
`Q
`00
`~ a-..
`
`~
`1--"
`
`Facebook Inc.'s Exhibit 1003
`
`004
`
`

`
`d •
`\Jl
`•
`~
`~ ......
`~ = ......
`
`> = ~
`
`"""' ~~
`N c
`8
`
`'JJ. =(cid:173)~
`
`~
`~
`0 ......,
`N
`N
`
`e
`
`rJ'l
`0'1
`'0-,
`Q
`00
`
`~
`0'1
`~
`1--"
`
`-----
`------
`\ I THERE IS A MEETING AT 10 A.M. IN
`
`THE OAKWOOD CONFERENCE ROOM
`THE INVITEES ARE :
`STAN NETILES
`BILL PHILLIPS
`NORM FLETCHER
`BEN STANAFORD
`FRANK CASTILLE
`SUBLECT:
`
`1
`
`(
`
`I
`
`,
`
`\
`I
`
`'\
`
`I
`
`\
`I
`
`CONSULTANT'S REPORT ON WIDER SWEET J
`
`\
`SPOT IN GOLF CLUBS
`REMARKS FROM GUS :
`\__ BRING YOUR PROTOTYPES OF THE
`NEW 7 - IRON
`-- ~-
`-----
`
`----
`----
`
`-
`
`I
`
`1
`__.1
`
`FIG. 3
`
`----
`I I
`
`WHO IS DELEGATE ?
`
`(
`
`-----
`
`SHALL HOST SEND INIVATION?
`
`I
`
`I
`
`\
`' \__ __
`----
`I I
`(
`\
`·\__ ___ _
`
`I
`
`I
`
`J
`--- """
`1
`__.1
`~-·
`
`1 \
`
`
`
`I
`
`J
`
`ou WANT TO SEE MARQUI
`D DO YOU WANT TO SEND DELEGATE?
`D DO YOU WANT TO DELIVER MESSAGE
`TO CONFERENCE ?
`D MESSAGE IS VOICE?
`D MESSAGE IS VIDEO?
`D MESSAGE IS TEXT?
`
`I
`
`I
`
`J \
`'\__-::: ~S:ESSAGEFILE ?
`
`- - - __;'
`
`-- __;'
`
`Facebook Inc.'s Exhibit 1003
`
`005
`
`

`
`FIG.4
`
`NORTH
`DAKOTA
`
`SOUTH
`DAKOTA
`
`COLORADO
`
`KANSAS
`
`NEW
`MEXICO
`
`d •
`\Jl
`•
`~
`~ ......
`~ = ......
`
`~
`~
`'"""' ~~
`N c
`8
`
`'JJ. =(cid:173)~
`~ .....
`
`~
`0 ......,
`N
`N
`
`e
`rJ'l
`-..a-..
`a-..
`Q
`00
`~ a-..
`
`~
`1--"
`
`Facebook Inc.'s Exhibit 1003
`
`006
`
`

`
`U.S. Patent
`
`Aug. 19, 2003
`
`Sheet 5 of 22
`
`US 6,608,636 Bl
`
`FIG. 5
`
`®
`TOLEDO
`
`0
`MASSILLON
`
`SANDUSKY
`
`0 0RANGE
`•
`LAKEWOOD
`
`YOUNGSTOWN
`0
`
`AKRONO
`
`CANTON
`0
`MASSILLON
`
`COLUMBUS
`@
`
`SPRINGFIELD
`0
`
`DAYTON
`0
`0 XENIA
`
`HAMILTON
`0
`
`Facebook Inc.'s Exhibit 1003
`
`007
`
`

`
`U.S. Patent
`
`Aug. 19,2003
`
`Sheet 6 of 22
`
`US 6,608,636 Bl
`
`COMPANY HEADQUARTERS
`
`FIG. 6
`
`SECOND FLOOR
`
`JACKSON , M. P.
`
`Facebook Inc.'s Exhibit 1003
`
`008
`
`

`
`U.S. Patent
`
`Aug. 19, 2003
`
`Sheet 7 of 22
`
`US 6,608,636 Bl
`
`----
`
`------
`
`r-
`,
`I
`I I
`I I
`-
`I I ""---------,/
`
`-
`
`I
`
`I I
`I I I
`\
`\
`I \
`~\ . ~ ... ----
`
`I
`
`~
`~
`~
`
`--------- ... .....
`
`-
`
`-
`
`\ I
`\ I
`\ I
`I I
`l I
`
`I
`
`I I
`
`I
`
`I
`
`I
`
`I
`I
`
`----
`
`------
`
`------
`
`-----
`
`Facebook Inc.'s Exhibit 1003
`
`009
`
`

`
`U.S. Patent
`
`Aug. 19, 2003
`
`Sheet 8 of 22
`
`US 6,608,636 Bl
`
`----~
`
`I
`
`------
`FIG. 9 ----
`-----
`/ ~~rel~~~ r.:::l\
`I~~~~~~~\
`I
`I c:!>
`¢"::7- POINTERS
`~
`)
`I
`Jl
`
`I
`
`\*
`.
`\
`-----
`----- --
`FIG. 10 -------- --- -----
`------
`/ 01
`D D
`I I
`
`TABLE
`
`-----
`----\
`DOCUMENT Q
`'\
`TO BE PLACED j
`ON TABLE
`I
`
`I
`
`I
`
`\~0
`\ ;,________,
`\ ----- ------ --- --- --------
`
`I
`
`I
`
`I I _ _j
`
`Facebook Inc.'s Exhibit 1003
`
`010
`
`

`
`U.S. Patent
`
`Aug.19, 2003
`
`Sheet 9 of 22
`
`US 6,608,636 Bl
`
`~---\
`\
`l
`}
`
`I
`
`I
`
`\ i
`, _____ j
`\ -----
`----- --- --- -------
`-- ----- ...
`-------- --
`
`FIG. 11
`
`-------
`I 01
`I I
`
`--
`D D
`
`:-----:
`
`1 _____ 1
`
`~-----~
`
`: _____ :
`
`,-----,
`rr=;;;;;;;;;;;::::;:;;;:~===n I_--- _I
`
`I
`
`I
`
`I Di~o
`\
`:------,
`;g
`
`/
`
`_ _)
`
`FIG. 12
`
`-
`
`-
`
`<>
`9
`·~
`\
`----- --
`\,
`
`1
`
`l
`)
`j
`JJ
`
`-----
`
`Facebook Inc.'s Exhibit 1003
`
`011
`
`

`
`---- -----
`
`FIG. 13
`--~~ ---
`
`t
`
`WHISPER
`
`t
`
`I c::::>
`
`¢
`
`-
`
`I
`
`I
`
`I
`
`U.S. Patent
`
`Aug.19, 2003
`
`Sheet 10 of 22
`
`US 6,608,636 Bl
`
`----
`<>
`\*
`}
`I
`\
`\ ....___ _ _ ____ JI
`------
`- - - - - - --
`FIG. 14
`~---- --
`f<>DDDDDR~T B\
`<>
`I
`c:>
`\¢
`'
`\
`I
`_w
`\ ----------
`--- -
`
`----
`
`-
`
`-
`
`-- ----~
`
`.---~"-..
`
`IDENTIFY RECIPIENT
`
`I
`}
`
`-
`
`NOTE
`
`----------
`
`Facebook Inc.'s Exhibit 1003
`
`012
`
`

`
`U.S. Patent
`
`Aug. 19,2003
`
`Sheet 11 of 22
`
`US 6,608,636 Bl
`
`FIG. 15
`
`FIG. 15A
`
`FIG. 158
`
`FIG. 15C
`
`FIG. 150
`
`FIG. 15E
`
`FIG.15A
`
`INITIATE CONFERENCE
`
`INITIATOR ACTIVATES CONFERENCE ICON
`
`HOST REQUESTS:
`A) HOW MANY PARTICIPANTS?
`B) PARTICIPANT LOCATION?
`C) PARTICIPANT IDENTIFICATION ?
`D) TIME AND DATE OF CONFERENCE?
`
`INITIATOR ACTIVATES:
`A) HOW MANY PARTICIPANTS
`
`,
`INITIATOR ENTERS A NUMBER LESS
`THAN OR EQUAL TO THE SYSTEM MAXIMUM
`
`INITIATOR ACTIVATES:
`B) PARTICIPANT LOCATION
`
`HOST PROVIDES LARGE AREA MAP
`
`INITIATOR SELECTS ZONES ( 1- N)
`ON LARGE AREA MAP
`
`I HOST DISPLAYS ZONE ( 1- N}' SUCCESSIVELY I
`
`Facebook Inc.'s Exhibit 1003
`
`013
`
`

`
`U.S. Patent
`
`Aug. 19,2003
`
`Sheet 12 of 22
`
`US 6,608,636 Bl
`
`FIG.15B
`
`INITIATOR ACTIVATES SPECIFIC SITES
`( 1 - M) IN EACH ZONE ( 1 - N) SELECTED
`
`HOST STORES SPECIFIC SITE I. D. FOR SITES
`( 1 - M ) SELECTED IN EACH ZONE ( 1 - N )
`
`INITIATOR ACTIVATES:
`C) PARTICIPANT IDENTIFICATION
`
`HOST REQUESTS:
`1) LIST AVAILABLE?
`2 ) PROVIDE SITE LISTINGS ?
`3) WALK HALLS AT SITES?
`
`HOST REQUESTS :
`I) TYPE IN LIST?
`II) SCAN LIST ?
`
`HOST PROVIDES A
`LIST FOR EACH SITE
`(1-M) SELECTED
`
`Facebook Inc.'s Exhibit 1003
`
`014
`
`

`
`U.S. Patent
`
`Aug. 19,2003
`
`Sheet 13 of 22
`
`US 6,608,636 Bl
`
`N
`
`INITIATOR SELECTS
`PARTICIPANTS ON
`EACH SITE LIST
`
`FIG. 15C
`
`y
`
`REQUESTOR
`TYPES A LIST
`AND SENDS
`DOCUMENTS
`TO HOST
`
`REQUESTOR
`SCANS LIST
`WITH CAMERA
`AND SENDS
`DATA
`
`HOST PROVIDES A HALLWAY FOR EACH SITE (1-M)
`
`INITIATOR SCROLLS UP I DOWN EACH
`HALLWAY ACTIVATING NAMEPLATE ON WALL
`
`Facebook Inc.'s Exhibit 1003
`
`015
`
`

`
`U.S. Patent
`
`Aug. 19,2003
`
`Sheet 14 of 22
`
`US 6,608,636 Bl
`
`FIG.15D
`
`HOST PROVIDES EACH DOORWAY
`EITHER OPEN , CLOSED OR PARTIALLY OPEN
`
`INITIATOR LEAVES
`MESSAGE WITH
`HOST TO REQUEST
`PARTICIPATION
`WHEN AVAILABLE
`
`INITIATOR
`REQUESTS
`PARTICIPATION
`
`INITIATOR
`PEEKS THROUGH
`DOORWAY
`
`HOST SIGNALS
`PARTICIPANT
`
`VOICE CONTACT ESTABLISHED
`
`PARTICIPANT I. D. ACTIVATED
`BY INITIATOR IF PARTICIPANT
`WILL ATTEND CONFERENCE
`
`INITIATOR ACTIVATES:
`D) TIME AND DATE OF CONFERENCE
`
`HOST REQUEST TIME AND DATE
`
`Facebook Inc.'s Exhibit 1003
`
`016
`
`

`
`U.S. Patent
`FIG.15E
`
`Aug. 19,2003
`
`Sheet 15 of 22
`
`US 6,608,636 Bl
`
`INITIATOR TYPES IN DATE AND TIME
`
`HOST ACTIVATES CONFERENCE ICON
`ON EACH PARTICIPANT'S SCREEN
`WITH FLASHING TIME AND DATE
`
`HOST REQUEST CONFERENCE TYPE:
`1) OPEN?
`2) RESTRICTED?
`3) SECURE?
`
`HOST ACTIVATES
`CONFERENCE
`ICON WITHOUT
`SPECIFIC INVITATION
`ON ALL CONNECTED
`TERMINALS
`
`HOST PROVIDES A
`KEY ICON FOR EACH
`PARTICIPANT AND
`SPECIFIES RULES
`FOR KEY TRANSFER
`IF PARTICIPANT WISHES
`TO SEND A SUBSTITUTE
`
`HOST PROVIDES A KEY ICON AND A STORAGE SAFE FOR
`THE KEY TO EACH PARTICIPANT. A CODE KNOWN ONLY TO
`THE INDIVIDUAL PARTICIPANT IS REQUIRED TO OPEN
`THE SAFE AND USE THE CONFERENCE KEY
`
`Facebook Inc.'s Exhibit 1003
`
`017
`
`

`
`U.S. Patent
`
`Aug. 19,2003
`
`Sheet 16 of 22
`
`US 6,608,636 Bl
`
`CONFERENCE ROOM
`
`FIG. 16A
`
`FIG.16B
`
`DATA FILE
`PRESENTED ON
`TABLE BY HOST
`
`HOST SENDS
`OPEN FILE TO ALL
`PARTICIPANT'S TABLE
`
`DATA FILE
`TRANSMITTED
`TO HOST
`
`HOST TRANSMITS
`DATA FILE TO
`TABLE OF EACH
`PARTICIPANT
`
`FIG. 16C
`
`ICON REMAINS ON
`TABLE AND ON SCREEN
`
`ACTIVATING ICON ON SCREEN PRESENTS
`DATA FILE TO INDIVIDUAL PARTICIPANT
`
`Facebook Inc.'s Exhibit 1003
`
`018
`
`

`
`U.S. Patent
`
`Aug. 19,2003
`
`Sheet 17 of 22
`
`US 6,608,636 Bl
`
`ICON FOR PARTICIPANT
`
`FIG. 17A
`
`HOST TRANSMITS VIDEO
`OF PARTICIPANT TO A SMALL
`WINDOW ICON ON EACH
`PARTICIPANT'S SCREEN
`
`FIG. 17B
`
`IF
`PARTICIPANT
`ICON ACTIVATED BY
`II WHISPER MODE II
`ICON BY ANOTHER
`PARTICIPANT
`
`HOST PROVIDES AN
`EXCLUSIVE VOICE LINK
`BETWEEN THE TWO
`PARTICIPANTS
`
`FIG.17C
`
`IF NOTE
`DRAGGED FROM
`PARTICIPANT
`SCREEN PAD TO
`OTHER
`PARTICIPANT'S
`ICON
`
`HOST TRANSMITS NOTE TO
`IDENTIFIED PARTICIPANT ONLY
`
`Facebook Inc.'s Exhibit 1003
`
`019
`
`

`
`U.S. Patent
`
`Aug. 19,2003
`
`Sheet 18 of 22
`
`US 6,608,636 Bl
`
`POINTER FOR PARTICIPANT
`
`FIG. 18
`
`HOST TRANSMITS MOVE TO
`EACH PARTICIPANT'S SCREEN
`
`PENCIL ICON FOR WRITING
`
`FIG. 19
`
`HOST EXCLUSIVELY READS DATA
`FROM THAT PARTICIPANT ONTO TABLE
`
`OTHER PARTICIPANTS ARE DISABLED FROM
`THE PENCIL UNTIL THE CONTROLLING
`PARTICIPANT DEACTIVATES THE PENCIL
`
`Facebook Inc.'s Exhibit 1003
`
`020
`
`

`
`U.S. Patent
`
`Aug. 19,2003
`
`Sheet 19 of 22
`
`US 6,608,636 Bl
`
`NOTEPAD ICON
`
`FIG. 20A
`
`HOST TRANSMITS DATA TO
`ALL PARTICIPANT NOTEPADS
`
`FIG. 20B
`
`HOST MOVES NOTEPAD SHEET
`AND DATA TO WALL FOR STORAGE
`
`NOTEPAD ICON CAN BE ACTIVATED BY ANY PARTICIPANT;
`HOST WILL NOT ACT ;
`DATA ON NOTEPAD WILL BE DISPLAYED ON INDIVIDUAL'S SCREEN
`
`Facebook Inc.'s Exhibit 1003
`
`021
`
`

`
`U.S. Patent
`
`Aug. 19,2003
`
`Sheet 20 of 22
`
`US 6,608,636 Bl
`
`CONFERENCE ROOM DOORS
`
`FIG. 21A
`
`FIG. 21B
`
`HOST ASKS IF
`PARTICIPANT WISHES
`TO WALK HALLWAY
`
`IF
`PARTICIPANT
`DRAGS HIS ICON
`AND OTHER ICONS
`THROUGH DOORWAY
`INTO OTHER
`CONFERENCE
`ROOM
`
`HOST PROVIDES
`CONFERENCE ROOM FOR
`II DRAGGED IN II PARTICIPANTS
`
`Facebook Inc.'s Exhibit 1003
`
`022
`
`

`
`U.S. Patent
`
`Aug. 19,2003
`
`Sheet 21 of 22
`
`US 6,608,636 Bl
`
`FIG. 22A
`
`ENDING CONFERENCE
`
`INITIATOR REQUESTS END CONFERENCE
`
`HOST RESPONDS :
`A) MINUTES?
`8 ) KEEP ROOM ?
`
`INITIATOR ACTIVATES :
`A) MINUTES
`
`HOST SENDS LATEST
`MODIFICATION OF
`EACH DATA FILE TO
`EACH PARTICIPANT
`
`y
`
`HOST
`QUERIES
`MORE
`?
`
`N
`
`HOST SENDS
`LISTING OF FILE
`MODIFICATIONS TO
`INITIATOR
`
`HOST
`DESTROYS
`INTERMEDIATE
`FILES
`
`Facebook Inc.'s Exhibit 1003
`
`023
`
`

`
`U.S. Patent
`
`Aug. 19,2003
`
`Sheet 22 of 22
`
`US 6,608,636 Bl
`
`INITIATOR
`CHOOSES FILES
`TO KEEP
`
`HOST SENDS
`THOSE FILES
`INITIATOR
`
`N
`
`y
`
`HOST DESTROYS
`CONFERENCE ROOM
`ATTRIBUTES AND FILES
`
`HOST STORES CONFERENCE ROOM
`ATTRIBUTES AND SPECIFIED DATA FILES
`
`HOST TERMINATES ACTIVITES
`
`FIG. 22B
`
`Facebook Inc.'s Exhibit 1003
`
`024
`
`

`
`US 6,608,636 Bl
`
`1
`SERVER BASED VIRTUAL CONFERENCING
`
`The invention concerns conferencing systems in which
`several people at different locations can communicate with
`each other.
`
`5
`
`10
`
`BACKGROUND OF THE INVENTION
`Systems are presently available wherein a person can both
`see and speak with another at a distance. The well-knownAT
`& T® Picturephone® is an example: it combines the tele(cid:173)
`phone with television.
`There also exist systems which allows computers to
`provide a feature known as "screen sharing." With screen
`sharing, one person, at a remote location, can control a
`second person's computer. For example, the remote person 15
`can manipulate a pointing device which moves a cursor on
`the screen of the second person's computer. To an extent, the
`two people share a common screen.
`The invention carries these concepts further. The inven(cid:173)
`tion allows multiple persons, at different locations, to hold a 20
`conference, by providing many of the conveniences which
`the participants would have if present together in the same
`physical room.
`
`OBJECTS OF THE INVENTION
`It is an object of the invention to provide a virtual
`conferencing system.
`It is a further object of the invention to provide a virtual
`conferencing system which allows multiple persons to view,
`and also manipulate, a common video display, which 1s 30
`simultaneously displayed at their different locations.
`
`2
`FIG. 2 illustrates computer displays showing menu
`choices which allow a user to initiate a virtual conference.
`FIG. 3 illustrate further menu choices.
`FIGS. 4, 5, and 6 illustrate a type of "telephone directory,"
`in which the persons listed are not listed alphabetically, but
`are listed by geographic location.
`FIG. 7 illustrates a computer display showing a virtual
`hallway. Behind each door is a person's office. Each office
`contains the equipment shown in FIG. 1. The person viewing
`the hallway can look into each office (if allowed to do so) via
`a connection made with the cameras shown in FIG. 1.
`FIG. 8 shows key-icons which can be used to gain entry
`to a virtual conference which is not open to the general
`public.
`FIG. 9 shows one type of video screen which is generated
`by the host computer, and which is distributed to all par(cid:173)
`ticipants in the conference.
`FIG. 10 shows a screen, of the type given in FIG. 9, but
`further including a "Document to be Placed on Table,"
`which was transmitted to the host by a participant, and
`which the participant will show to the other participants by
`placing it onto the "conference table."
`FIG. 11 shows the sequence of events by which the
`25 "Document ... " of FIG. 10 will become displayed.
`FIGS. 12 and 14 illustrate display screens which can
`appear when one participant passes a note to another.
`FIG. 13 illustrates one display screen which can appear
`when one participant "whispers" to another.
`FIG. 15 et seq. provide flow charts which correspond to
`the Pseudo-code contained in the Appendix.
`
`SUMMARY OF THE INVENTION
`An example will illustrate one form of the invention. Two
`(or more) parties each operate their own local computers. 35
`The computers have associated video cameras, speaker-type
`telephones, and pointing devices (such as "mouses"). When
`a conference is established, the local computers become
`connected to a host computer, via commercially available
`Local Area Networks (LANs) and Wide Area Networks 40
`(WANs).
`The parties send the information which they want
`displayed, such as drawings, to the host computer. The host
`computer generates a common video screen, which it dis(cid:173)
`tributes to the parties: they see the drawings at their own 45
`local computers. Each party can move a pointer on the
`display, and point to features on the drawings. The tele(cid:173)
`phones and video cameras allow the parties to see and speak
`with each other.
`The host controls many of the events occurring during the 50
`conference, as well as those occurring both during initiation
`of the conference and after termination of the proceedings.
`For example, the host provides a system which allows a
`person to invite participants to the conference in a
`convenient manner, including some automated invita- 55
`tion features which will be described later.
`As another example, the host provides some unique
`stenographic and recording features.
`As yet another example, the host can maintain the virtual
`conference room in existence after the conference 60
`terminates, so that the records of the proceedings
`remain available for persons who wish to examine
`them.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`FIG. 1 illustrates equipment which enables a person to
`remotely attend a virtual conference.
`
`65
`
`DETAILED DESCRIPTION OF THE
`INVENTION
`
`Overview
`
`In the invention, the participants share a common virtual
`conference table. Each participant can
`(1) place a document onto the table electronically,
`(2) write on the document, draw on it, and otherwise
`manipulate it, and
`(3) move a pointer to different positions on the document,
`to point to specific parts of it.
`All other participants see the the preceding three events as
`they occur.
`The invention further provides the following features:
`The participants can privately whisper or pass notes to
`each other, without the knowledge of the others.
`A person wishing to invite participants to a conference
`can "walk a hallway" shown on that person's computer
`display. Doors shown in the hallway are virtual doors,
`through which the person can look into offices of
`invitees, via a video camera system.
`The invitation. process itself can be viewed as a confer(cid:173)
`ence between two parties, namely, the inviting person
`and the person (or office generally) behind the door
`with which the inviting person is communicating at that
`time.
`
`Hardware Requirements
`
`Every office is equipped with the following equipment:
`a computer (termed a "local computer" herein),
`one or two video cameras, and
`a telephone having "speaker-phone" capability.
`
`Facebook Inc.'s Exhibit 1003
`
`025
`
`

`
`US 6,608,636 Bl
`
`3
`One office is shown in FIG. 1.
`One of the video cameras (a "survey camera") surveys the
`entire office from a point near the door, as though the camera
`were looking into the office. The second camera (a "close(cid:173)
`up" camera) is located next to the computer, and provides a 5
`close-up view of the computer user. The second camera
`gives the "computer's view" of the user.
`One of the cameras can serve a second purpose: it can be
`used as an optical scanner, for capturing images of docu(cid:173)
`ments and drawings. The computer is equipped with a frame 10
`grabber for converting the image seen by the camera into a
`bit-mapped file. Alternately, an optical scanner can be used
`for capturing these images.
`These individual systems are located at different geo(cid:173)
`graphic locations, and, when a virtual conference is to be 15
`held, become connected to a central, host, computer (or
`multiplicity of host computers) via the proper combination
`of Local Area Networks (LANs) and Wide Area Networks
`(WANs).
`
`4
`The Requester may have previously prepared a list of
`participants. If the list takes the form of a data file, he (or
`she) can send it to the host computer directly. If not, he can
`convert the list into a data file and sent the data file to the
`host in several ways.
`One, he can type the list into his own local computer,
`thereby generating a data file.
`Two, he can show the physical, printed list to a digitizing
`device, which generates the data file. One such digitizing
`device is an optical scanner, which generates a bit-mapped
`image. The scanner sends the bit-mapped image to the
`Requester's local computer, which contains Optical Char(cid:173)
`acter Recognition (OCR) software. The OCR software trans-
`lates the bit-mapped image into a data file written in ASCII
`code, which is sent to the host.
`Another digitizing device is the close-up video camera, if
`equipped with the proper lens. The local computer contains
`a frame-grabber, which captures a bit-mapped image of the
`list, and sends it to the OCR software as above.
`Once the host obtains the list, the host sends an invitation
`to each invitee, telling the date, time, place, subject, and any
`other relevant information about the meeting. One way to
`send the invitations is by electronic mail, which is known in
`25 the art. A second way is for the host to leave an "invitation
`card," such as that shown in FIG. 3, on the screen of each
`local computer. Such cards are later discussed.
`
`20
`
`30
`
`Initiation of Conference
`A person requesting a conference informs the host com(cid:173)
`puter by appropriate means. For example, the Requester can
`"click" a pointing device onto a "conference icon," such as
`that shown in FIG. 2, on his display screen. In response, his
`local computer sends the proper signal to the host. Now, the
`host must be given several items of information, which can,
`for convenience, be divided into two groups.
`One group relates to the participants:
`Who are the participants?
`Can others participate without invitations?
`Can participants send substitutes?
`The second group relates to the conference room:
`Where does it exist? That is, where does the data structure 35
`reside from which the conference room is constructed.
`The location must be known to a participant who
`wishes to find the conference room.
`What are the room attributes?
`What is the room decor? What paintings are on the 40
`walls?
`What equipment is present in the room? For example,
`the host computer may provide the option for any
`participant to record the entire proceedings of a
`conference, or any selected part of the proceedings.
`This option may be represented by an icon in the
`shape of a tape recorder, which is contained in the
`conference display screen. Similar options can be
`present in the form of icons in the shape of
`telephones, notepads, library books, tables, etc.
`The third relates to conference procedures:
`What rules govern the conduct of the meeting?
`Does the Requester have absolute control of the voice
`and message interaction among the participants? Or
`Is the meeting a brainstorming free-for-all, where 55
`numerous people can speak at once?
`What is the decor of the room?
`What happens to the room when the conference 1s
`adjourned? Are minutes of the conference kept?
`These issues will be considered in detail.
`How Does Requester Identify Participants to the Host Com(cid:173)
`puter?
`
`If Requester Has No List
`
`If the requester has no list, the host can give the option of
`picking names from a directory of names, akin to a "tele(cid:173)
`phone directory." Another possibility is for the host to
`present the "telephone directory" in a more graphical format,
`such as the following.
`(The inventor points out that the following invitation
`procedure can also serve as an actual conference, but held
`between the inviting person and the person invited. After the
`inviting person successfully establishes communication with
`the invitee, the inviting person, if desired, can order the host
`computer to provide an entire conference room, or selected
`equipment from the room, such as a recorder, for the
`convenience of the two parties. Further, the two parties,
`during their conference, can "walk the halls," and invite
`other participants, thereby expanding the conference to
`45 include additional participants.)
`Assume that the Requester wishes to invite fellow
`employees to the conference. The host computer first dis(cid:173)
`plays a map of all locations of facilities of the company, such
`as in FIG. 4. The Requester picks a state. The host displays
`the state, showing all company facilities, as in FIG. 5. The
`Requester picks a facility. The host shows the facility, in
`exploded form, as in FIG. 6.
`Now the Requester can "walk the halls." The host shows
`an image of a hallway, as in FIG. 7. Each door bears a name.
`The Requester scrolls along the hallway, until he reaches an
`invitee's door. The door can be open, closed, or ajar. The
`status of the door is under the control of the person occu(cid:173)
`pying the office in question, via that person's local computer.
`
`50
`
`60
`
`Door Closed
`
`Requester May Have List of Participants in
`Advance
`The Requester's local computer gathers the information
`indicated by the menu shown in FIG. 2.
`
`If the office occupant does not wish to be interrupted by
`other people, the occupant instructs the conferencing system
`to close the door. Under such an instruction, the conferenc-
`65 ing system shows a closed door to all Requesters who are
`"walking the hallway." In a similar way, the occupant
`controls whether the door is fully open or ajar.
`
`Facebook Inc.'s Exhibit 1003
`
`026
`
`

`
`US 6,608,636 Bl
`
`5
`Door Open or Ajar
`
`If the door is open, then the Requester sees the image
`which the survey camera sees of that person's office. That is,
`the Requester can look into the Invitee-Occupant's office.
`If the door is ajar, the Requester is given the option of
`"peeking" into the office. That is, the Requester is given
`momentary access to the survey camera, for a period such as
`five seconds.
`
`5
`
`6
`is used by several Invitees at once during the conference.
`That is, the marquis constantly advertises information about
`the upcoming conference, somewhat as a "reserved" sign
`does for a physical conference room.
`The invitation card icon provides access to other options,
`as will be explained below.
`Host Computer Knows Identities of Invitees at this Time
`
`Host Computer Needs Additional Information
`
`Gaining Invitee's Attention
`
`10
`
`At this time, the host knows the identities of the invitees.
`Several additional details must be resolved before the meet(cid:173)
`ing begins.
`Can Invitee Send Delegate? It is possible that, after an
`Invitee has accepted an invitation, something may prevent
`the Invitee from attending the forthcoming conference. That
`is, the Invitee will be an Absentee. Ordinarily, this would
`create a small problem: the host computer is planning to
`connect the Absentee's local computer with the others for
`the conference, but the Absentee will be not be present. If the
`Requester has allowed Invitees to appoint Delegates to
`attend in their places, the Invitees may appoint one, in the
`following ways. (The menu used for this purpose is shown
`in FIG. 2.)
`Row to Appoint Delegate. The Absentee can click onto
`the invitation card left on the display, and obtain access to
`the host computer. The host computer then presents several
`options to the Absentee. The Absentee can designate a
`Delegate, and assign to the host the task of transmitting an
`30 invitation to the Delegate.
`Alternately, the Absentee himself informs the Delegate of
`the new appointment to attend the meeting in customary
`ways, such as by telephone.
`
`25
`
`Where Does the Delegate Attend the Conference?
`
`Requester Pops his Head onto Invitee's Display. Once the
`Requester has located the Invitee's office, the Requester
`must gain the. attention of the Invitee. (The reader is
`reminded that the Requester is not physically present at the 15
`Invitee's office door; the Requester is "virtually present."
`The Requester is physically present at a computer present in
`another office. The computer is linked to the survey camera
`located at the Invitee's office.)
`One option is to allow the Requester to instruct the host 20
`computer to place a small picture of the Requester onto the
`Invitee's computer display. The picture beckons the Invi(cid:173)
`tee's attention. This procedure is somewhat analagous to real
`life: in real life, the Requester can pop his head into the
`Invitee's office and announce the conference.
`The image placed onto the Invitee's screen is that seen by
`the Requester's close-up camera, and occupies a small
`portion of the invitee's display. The Invitee has the option of
`blocking such beckoning, by instructing the host computer
`to block such i