US007920623B2
`
`(12) United States Patent
`Stone et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,920,623 B2
`Apr. 5, 2011
`
`(54)
`
`(75)
`
`(73)
`
`(*)
`
`(21)
`(22)
`(65)
`
`(51)
`
`(52)
`(58)
`
`METHOD AND APPARATUS FOR
`SIMULTANEOUS DISPLAY OF MULTIPLE
`AUDIO/VIDEO PROGRAMS TRANSMITTED
`OVER A DIGITAL LINK
`
`Inventors: Christopher J. Stone, Newtown, PA
`(US); Stephen A. Allinson, Langhorne,
`PA (US); Christopher S. Del Sordo,
`Souderton, PA (US); Brad T. Howard,
`Lawrenceville, GA (US); Terry L.
`Ziegler, Lansdale, PA (US)
`Assignee: General Instrument Corporation,
`Horsham, PA (US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 1878 days.
`
`Notice:
`
`Appl. No.: 10/714,236
`
`Filed:
`
`Nov. 14, 2003
`
`Prior Publication Data
`|US 2005/01 08778 A1
`May 19, 2005
`
`Int. Cl.
`(2006.01)
`H04N 7/12
`(2006.01)
`PH04N 7/173
`U.S. Cl. ..................................... 375/240.01; 725/90
`Field of Classification Search .................. 382/240,
`382/232: 386/125, 126, 111, 112, 46, 68,
`386/96, 83; 375/240.26, 240.24, 240.16,
`375/240.12, 240.1, 240; 348/564, 565, 567,
`348/568, 588, 598, 724, 469, 705, 723, 743;
`725/90, 89, 102, 88
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`|U.S. PATENT DOCUMENTS
`5,586,264 A * 12/1996 Belknap et al. ............... 7.25/115
`5,625,410 A * 4/1997 Washino et al. ...
`... 348/154
`5,796,945. A * 8/1998 Tarabella ...........
`... 709/219
`5,847,771 A * 12/1998 Cloutier et al. ....
`... 348/564
`5,877,821. A * 3/1999 Newlin et al. ................. 348/724
`6,035,037 A
`3/2000 Chaney
`6,510,553 B1* 1/2003 Hazra ............................. 725/87
`4/2003 Yuzawa .............
`... 370/345
`6,549,528 B2 *
`.
`6,671,290 B1*
`... 370/486
`12/2003 Murayama et al.
`.
`7,072,948 B2 *
`7/2006 Yen et al. .......
`... 709/218
`5/2003 Demas et al. ................... 725/90
`2003/0093800 A1 *
`11/2005 Howard et al. .......... 375/240.26
`2005/0259.751 A1*
`* cited by examiner
`Primary Examiner – Behrooz Senfi
`(74) Attorney, Agent, or Firm – Larry T. Cullen
`(57)
`ABSTRACT
`Simultaneous display of multiple audio/video (AV) programs
`transmitted over a digital linkis described. At a source device,
`at least one non-composited digital transport stream is gen
`erated from the plurality of AV programs. The at least one
`non-composited digital transport stream is augmented with
`control information. The control information is operative to
`invoke simultaneous display of the plurality of AV programs
`on a display device. The at least one non-composited digital
`transport stream is then transmitted as augmented over the
`digital link. At a sink device, at least one non-composited
`digital transport stream having the plurality of AV programs is
`received over the digital link. Control information is extracted
`from the at least one non-composited digital transport stream.
`The plurality of AV programs are identified within said non
`composited digital transport stream in response to the control
`information. The identified AV programs are then simulta
`neously displayed on the display device.
`
`28 Claims, 4 Drawing Sheets
`
`PIP COMMAND
`
`2:02__
`SOURCE DEVICE
`
`
`
`
`
`IUNER
`
`108
`AUDIO/WISUAL
`SOURCE
`
`snº prior TT T TT T T TT
`STREAM
`DECODER
`
`
`
`HTC EXHIBIT 1017
`
`Page 1 of 11
`
`

`

`U.S. Patent
`
`Apr. 5, 2011
`
`Sheet 1 of4
`
`US 7,920,623 B2
`
`asmE:
`
`o_
`
`»<._n_mS
`
`bmaoumo_os:<3ss<
`
`»~_:=o~_8Nos:SEs<
`
`8S>\oS=<
`
`
`E2:~Emzs=
`zozsseaEs
`
`Es8»
`
`Es$53:.EsSNsax;
`zszéesEs
`
`men
`
`<55HEEn_EI
`
`
`
`sEssEsassss
`
`
`
`
`
`ssnszss§E_8sxsszss
`
`_I||
`SHEQEsa
`
`NewEssaE
`
`Page 2 of 11
`
`
`
`
`
`
`
`.sm_zssmss4<pHoHo<E<ssz<=sos
`
`Page 2 of 11
`
`
`
`
`

`

`U.S. Patent
`
`Apr. 5, 2011
`
`Sheet 2 of 4
`
`US 7,920,623 B2
`
`OBTAIN NON-COMPOSITED DIGITAL TRANSPORT
`STREAM{S) HAVING MULTIPLE AW PROGRAMS
`
`402
`
`AUGMENT THE NON-COMPOSITED DIGITAL
`TRANSPORT STREAMS) WITH CONTROL INFORMATION | 404
`OPERATIVE TO INVOKE SIMULIANEOUS DISPLAY OF
`THE MULTIPLE AW PROGRAMS ON A DISPLAY DEVICE
`
`TRANSMIT THE NON-COMPOSITED DIGITAL
`TRANSPORT STREAM(S) OVER A DIGITAL LINK HA06
`IN COMMUNICATION WITH THE DISPLAY DEVICE
`
`400
`
`
`
`
`
`
`
`
`
`
`
`RECEIVE NON-COMPOSITED DIGITAL TRANSPORT
`STREAM(S) HAVING AW PROGRAM(S
`
`502
`
`EXTRACT CONTROL INFORMATION FROM THE
`NON-COMPOSITED DIGITAL TRANSPORT STREAM(S
`
`504
`
`
`
`
`
`
`
`PIP
`SESSION REQUEST?
`
`506
`
`NO
`
`510
`YES |DISPLAY SINGLE AW PROGRAM ON DISPLAY DEVICE
`
`RECOVER MULTIPLE AW PROGRAMS FROM THE
`NON–COMPOSITED DIGITAL TRANSPORT STREAM
`USING THE CONTROL INFORMATION
`
`SIMULIANEOUSLY DISPLAY THE MULTIPLE
`AW PROGRAMS ON THE DISPLAY DEVICE
`
`508
`
`512
`
`500
`FI G. 5
`
`Page 3 of 11
`
`

`

`U.S. Patent
`
`Apr. 5, 2011
`
`Sheet 3 of 4
`
`US 7,920,623 B2
`
`WIDEO |
`AUDIO |
`WIDEO 2
`AUDIO 2
`
`602
`
`
`
`608
`
`606
`IDENTIFICATION
`DATA
`
`PMI
`
`TO IEEE
`1394 BUS
`
`F IG 6
`
`
`
`WIDEO |
`AUDIO |
`
`WIDEO 2
`
`AUDIO 2
`
`804
`
`702
`
`
`
`WIDEO |
`AUDIO |
`
`WIDEO 2
`AUDIO 2
`
`T0 IEEE
`1394 BUS
`
`708
`
`PMT2
`
`IDENTIFICATION
`DATA
`
`PSI
`
`PAT
`
`F I G. 7
`
`TO IEEE
`1394 BUS
`
`812
`
`COMMAND
`
`F I G 8
`
`
`
`WIDEO |
`AUDIO |
`
`WIDEO 2
`AUDIO 2
`
`902
`
`
`
`COMMAND
`
`964 *
`
`T0 IEEE
`1394 BUS
`
`910
`F I G. 9
`
`Page 4 of 11
`
`

`

`U.S. Patent
`
`Apr. 5, 2011
`
`Sheet 4 of 4
`
`US 7,920,623 B2
`
`1000
`- - - -, -º _?" —
`COMPUTER
`
`SUPPORT
`CIRCUITS
`
`1004
`
`
`
`I/O
`INTERFACE
`
`100.5
`
`MEMORY
`
`|
`|
`|
`|
`|
`|
`|
`|
`|
`|
`
`F
`
`
`
`1011
`
`
`
`OUTPUT
`DEVICE(S)
`
`INPUT
`DEVICE(S)
`
`
`
`
`
`
`
`1012
`
`Page 5 of 11
`
`

`

`1
`METHOD AND APPARATUS FOR
`SIMULTANEOUS DISPLAY OF MULTIPLE
`AUDIO/VIDEO PROGRAMS TRANSMITTED
`OVER A DIGITAL LINK
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`The present invention generally relates to audio/video sys
`tems and, more particularly, to simultaneous display of mul
`tiple audio/video programs transmitted over a digital link.
`2. Description of the Related Art
`Recently, high definition televisions (HDTVs) with digital
`interfaces, such as an IEEE 1394 bus interface, have appeared
`on the market. The IEEE 1394 bus (also referred to as
`FireWire, iDink, or DTV-Link) is a high-speed serial bus for
`transmitting digital data. In a typical configuration, a user
`employs an external tuning device, such as a set-top box
`(STB), to receive audio/video (AV) signals from a cable or
`satellite operator, or from a terrestrial broadcast. The STB, or
`“source device”, tunes to a particular AV signal to receive an
`AV stream, which is coupled to the digital bus. For example,
`the AV stream may include compressed AV data in accor
`dance with the MPEG (Moving Pictures Expert Group) stan
`dard and the source device may output an MPEG-2 transport
`stream to the digital bus. The HDTV, or “sink device”,
`receives the AV stream over the digital bus, decodes the AV
`data therein, and displays the decoded AV data. By employing
`a digital bus, a source/sink system reduces or eliminates noise
`and other deleterious effects typically associated with analog
`transmission.
`Use of an IEEE 1394 bus to transmit an AV stream between
`a source device and sink device is in its infancy and there are
`several drawbacks in its operation. One such drawback is the
`inability to properly display multiple AV streams simulta
`neously, known as picture-in-picture (PIP), or picture-over
`picture (POP). PIP/POP functionality includes the ability to
`display a first selected AV stream in a larger portion of a
`television screen and a second selected AV stream in a smaller
`portion of the television screen. PIP/POP functionality allows
`the user to watch a primary AV stream in the full screen of the
`television, while monitoring a secondary AV stream in the
`smaller PIP/POP window, which typically overlays a portion
`of the full screen view.
`Currently, there is no defined manner for a sink device
`(e.g., HDTV) receiving multiple AV streams over an IEEE
`1394 bus to know that the user is requesting a PIP/POP
`session or that the multiple AV streams are intended for a
`PIP/POP session. This leaves the user with expensive state
`of-the-art equipment that does not include simple PIP/POP
`functionality. Accordingly, there exists a need in the art for a
`mechanism to simultaneously display multiple AV streams
`transmitted over a digital link between a source device and a
`sink device.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`SUMMARY OF THE INVENTION
`
`55
`
`A method and apparatus for simultaneous display of mul
`tiple audio/video (AV) programs transmitted over a digital
`link is described. One aspect of the invention relates to a
`method of encoding a plurality of AV programs for simulta
`neous display on a display device. In one embodiment, at least
`one non-composited digital transport stream is generated
`from the plurality of AV programs. The at least one non
`composited digital transport stream is augmented with con
`trol information. The control information is operative to
`invoke simultaneous display of the plurality of AV programs
`
`60
`
`65
`
`US 7,920,623 B2
`
`2
`on the display device. The at least one non-composited digital
`transport stream is then transmitted over the digital link.
`Another aspect of the invention relates to a method of
`decoding a non-composited digital transport stream having a
`plurality AV programs configured for simultaneous display at
`a display device. In one embodiment, at least one non-com
`posited digital transport stream having the plurality of AV
`programs is received over the digital link. Control informa
`tion is extracted from the at least one non-composited digital
`transport stream. The plurality of AV programs are identified
`within said non-composited digital transport stream in
`response to the control information. The identified AV pro
`grams are then simultaneously displayed on the display
`device.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`So that the manner in which the above recited features of
`the present invention can be understood in detail, a more
`particular description of the invention, briefly summarized
`above, may be had by reference to embodiments, some of
`which are illustrated in the appended drawings. It is to be
`noted, however, that the appended drawings illustrate only
`typical embodiments of this invention and are therefore not to
`be considered limiting of its scope, for the invention may
`admit to other equally effective embodiments.
`FIG. 1 is a block diagram depicting an exemplary embodi
`ment of an audio/video (AV) system;
`FIG. 2 is a block diagram depicting an exemplary embodi
`ment of a stream encoder for use with the AV system shown in
`FIG. 1:
`FIG. 3 is a block diagram depicting an exemplary embodi
`ment of a stream decoder for use with the AV system shown in
`FIG. 1:
`FIG. 4 is a flow diagram depicting an exemplary embodi
`ment of a process for encoding AV programs for simultaneous
`display on a display device;
`FIG. 5 is a flow diagram depicting an exemplary embodi
`ment of a process for decoding one or more digital transport
`streams produced by the encoding process of FIG. 4;
`FIG. 6 is a data flow diagram depicting an example of the
`encoding process of FIG. 4 for two separate AV programs
`each having a video component and an audio component;
`FIG. 7 is a data flow diagram depicting another example of
`the encoding process of FIG. 4 for two separate AV programs
`each having a video component and an audio component;
`FIG. 8 is a data flow diagram depicting another example of
`the encoding process of FIG. 4 for two separate AV programs
`each having a video component and an audio component;
`FIG.9 is a data flow diagram depicting yet another example
`of the encoding process of FIG. 4 for two separate AV pro
`grams each having a video component and an audio compo
`ment;
`FIG. 10 is a block diagram depicting an exemplary
`embodiment of a computer suitable for implementing pro
`cesses and methods described herein.
`To facilitate understanding, identical reference numerals
`have been used, wherever possible, to designate identical
`elements that are common to the figures.
`
`DETAILED DESCRIPTION OF THE INVENTION
`
`Simultaneous display of multiple audio/video (AV) pro
`grams transmitted over a digital link is described. One or
`more aspects of the invention are described with respect to an
`MPEG (Moving Pictures Expert Group) transport carrying
`AV programs over an IEEE 1394 bus. Those skilled in the art
`
`Page 6 of 11
`
`

`

`3
`will appreciate that the invention may be used with other
`types of digital transport streams comprising time-division
`multiplexed (TDM) or packet division multiplexed (PDM)
`data. In addition, those skilled in the art will appreciate that
`the invention may be used with other types of digital buses,
`such as a universal serial bus (USB) and the like.
`FIG. 1 is a block diagram depicting an exemplary embodi
`ment of an audio/video (AV) system 100. The AV system 100
`comprises a source device 102, a sink device 104, and a digital
`link 106 connecting the source device 102 to the sink device
`104. The source device 102 is coupled to an AV source 108 for
`receiving AV signals therefrom. Each AV signal includes an
`AV program or service comprising one or more of audio,
`video, and data (generally referred to herein as an “AV pro
`gram”). The AV source 108 may comprise an interface to any
`type of audio/video/data signal transmission source, such as
`land-based radio-frequency type broadcast networks, cable
`networks, space satellite signal transmission networks,
`broadband telephone networks, and the like. The transmitted
`AV programs may be in any type of digital transport stream
`20
`format suitable for transmission purposes, such as the MPEG
`format, including MPEG-2 as defined in ISO/IEC Standard
`13818, the digital satellite systems (DSS) format, the asyn
`chronous transfer mode (ATM) format, and the like.
`The source device 102 comprises tuner circuitry 110, base
`band processing circuitry 111, a stream or transport encoder
`(“stream encoder 112”), control circuitry 114, and interface
`circuitry 116. For example, the source device 102 may be a
`set-top box (STB), video cassette recorder (VCR), receiver,
`or like type tuning device known in the art. An input terminal
`of the tuner circuitry 110 is coupled to the AV source 108 for
`receiving AV signals. The tuner circuitry 110 selects and
`tunes N of the AV signals in a well known manner, where N is
`an integer greater than zero. For example, the tuner circuitry
`110 may include dual tuners fortuning a first AV signal and a
`second AV signal of the AV signals provided by the AV source
`108 (i.e., N=2). An output bus of the tuner circuitry 110
`provides the N tuned AV signals.
`Another input terminal of the tuner circuitry 110 is coupled
`to the control circuitry 114 for receiving commands there
`from. The control circuitry 114 controls the both the selection
`and the number of AV signals that are tuned by the tuner
`circuitry 110 in response to commands from the user. Nota
`bly, in response to a picture-in-picture (PIP) command from
`the user, the control circuitry 114 commands the tuner cir
`cuitry 110 to simultaneously tune a plurality of AV signals
`(e.g., two AV signals for a dual program PIP session).
`An input bus of the baseband processing circuitry 111 is
`coupled to the output bus of the tuner circuitry 110 for receiv
`ing the N tuned AV signals. The baseband processing circuitry
`50
`111 recovers an AV program from each of the tuned AV
`signals in a well known manner. The baseband processing
`circuitry 111 provides N recovered AV programs correspond
`ing to the N tuned AV signals as output. As described above,
`the recovered AV programs may be in any known format,
`including known digital transport stream formats, such as
`those complying with the MPEG-2 systems standard.
`An input bus of the stream encoder 112 is coupled to the
`output bus of the baseband processing circuitry 110 for
`receiving the N recovered AV programs. The stream encoder
`112 encodes the AV programs to generate one or more non
`composited digital transport streams as output for transmis
`sion to the sink device 104. As used herein, the term “non
`composited” means that given digital transport stream
`comprises a plurality of separate AV streams that are multi
`plexed, rather than a single video stream that contains a single
`video image formed by layering multiple video images and a
`
`30
`
`35
`
`40
`
`45
`
`55
`
`60
`
`65
`
`US 7,920,623 B2
`
`10
`
`15
`
`25
`
`4
`single audio stream that contains a single audio track formed
`by mixing multiple audio tracks. The digital transport
`stream(s) produced by the stream encoder 112 may be any
`type of digital transport or program stream(s) known in the
`art. For purposes of clarity by example, embodiments of the
`invention will be described with respect to MPEG-2 transport
`StreamS.
`Another input terminal of the stream encoder 112 is
`coupled to the control circuitry 114. In response to a PIP
`command from the user, the control circuitry 114 commands
`the stream encoder 112 to encode the AV programs in a
`manner establishing a PIP session. Notably, the stream
`encoder 112 augments the data stream(s) with control infor
`mation operative to invoke simultaneous display of the N
`recovered AV programs on the sink device 104. An embodi
`ment of the stream decoder 112 is described below.
`An input bus of the interface circuitry 116 is coupled to the
`output bus of the stream encoder 112 for receiving the digital
`transport stream(s). The interface circuitry 116 processes the
`digital transport stream(s) for transmission over the digital
`link 106 in a well known manner. For example, in one
`embodiment, the digital link 106 is an IEEE 1394 bus and the
`interface circuitry 116 processes the digital transport
`stream(s) for transmission in accordance with the IEEE 1394
`protocol.
`The sink device 104 comprises interface circuitry 118, a
`stream or transport decoder (“stream decoder 120”), decoder/
`display circuitry 122, and a display 124. For example, the sink
`device 104 may be a television (e.g., high-definition televi
`sion (HDTV)), monitor, or like type display device known in
`the art. An input bus of the interface circuitry 118 is coupled
`to the digital link 106 for receiving the digital transport
`stream(s) generated by the source device 102. The interface
`circuitry 118 provides the digital transport stream(s) to an
`input bus of the stream decoder 120. The stream decoder 120
`extracts the control information within the digital transport
`stream(s). Notably, the stream decoder 120 uses the control
`information to identify an invocation of a PIP session by the
`user and to recover the AV programs within the digital trans
`port stream(s) for the PIP session. An output bus of the stream
`decoder 120 provides the N recovered AV programs, as well
`as the control information. An embodiment of the stream
`decoder 120 is described below.
`An input bus of the decoder/display circuitry 122 is
`coupled to the output bus of the stream decoder 120 for
`receiving the control information and the N recovered AV
`programs. The decoder/display circuitry 122 decodes the AV
`data within each AV program in a well known manner. For
`example, if an AV program comprises compressed AV data in
`accordance with the MPEG standard, the decoder/display
`circuitry 122 comprises an MPEG decoder for decoding the
`audio, video, and/or data within each AV program. The
`decoder/display circuitry 122 uses the control information to
`display the AV programs on the display 124. The display 124
`may be defined by N regions 126, through 126, in which each
`of the N recovered AV programs are to be displayed. For
`example, if a dual PIP session has been invoked by the user,
`the control information recovered by the stream decoder 120
`is used to determine which of the two AV programs is to be
`displayed in a primary region, and which of the two AV
`programs is to be displayed in a secondary region.
`FIG. 2 is a block diagram depicting an exemplary embodi
`ment of the stream encoder 112 shown in FIG.1. For purposes
`of clarity by example, the stream encoder 112 is described as
`having two separate AV programs as input, where a first AV
`program comprises first video data (“video 1”) and first audio
`data (“audio 1"), and a second AV program comprises second
`
`Page 7 of 11
`
`

`

`US 7,920,623 B2
`
`5
`
`15
`
`20
`
`25
`
`30
`
`5
`video data (“video 2’) and second audio data (“audio 2’).
`Those skilled in the art will appreciate that the stream encoder
`112 may have any number of AV programs as input, such AV
`programs comprising video, audio, and/or data. The stream
`encoder 112 illustratively comprises a bus 204 coupled to a
`memory 202, a multiplexer unit 206, a control information
`unit 208, and a transport stream transfer unit 210. An input
`bus of the memory 202 is coupled to receive the AV programs.
`For example, at least a portion of the memory 202 may com
`prise a buffer 203, such as a first-in-first-out (FIFO) buffer, for
`10
`buffering the data within each of the AV programs. An output
`bus of the memory 202 is coupled to the bus 204.
`An input/output bus of the multiplexer unit 206 is coupled
`to the bus 204. The multiplexer unit 206 may be used to
`multiplex the video and audio data of the AV programs, as
`well as the AV programs themselves, to form one or more
`non-composited digital transport streams. An input/output
`bus of the control information unit 208 is coupled to the bus
`204. Another input terminal of the control information unit
`208 is coupled to receive PIP command data. In response to a
`PIP command, the control information unit 208 may augment
`the non-composited digital transport stream(s) generated by
`the multiplexer unit 206 with control information in order to
`identify a PIP session and to distinguish among the different
`AV programs for display. An input bus of the transport stream
`transfer unit 210 is coupled to the bus 204. The transport
`stream transfer unit 210 is configured to provide the non
`composited digital transport stream(s) as output for the
`stream encoder 112. Embodiments of operation of the stream
`encoder 112 are described below.
`FIG. 3 is a block diagram depicting an exemplary embodi
`ment of the stream decoder 120 of FIG. 1. For purposes of
`clarity by example, the stream decoder 120 is described as
`having two separate AV programs as output, where a first AV
`program comprises first video data (“video 1”) and first audio
`35
`data (“audio 1"), and a second AV program comprises second
`video data (“video 2’) and second audio data (“audio 2’).
`Those skilled in the art will appreciate that the stream decoder
`120 may output any number of AV programs depending on
`the number of AV programs transmitted within the digital
`transport stream(s), such AV programs comprising video,
`audio, and/or data. The stream decoder 120 illustratively
`comprises a bus 304 coupled to a memory 302, a demulti
`plexer unit 306, a control information analyzer 308, and an
`AV transfer unit 310. An input bus of the memory 302 is
`coupled to receive one or more digital transport streams. For
`example, at least a portion of the memory 302 may comprise
`a buffer 303, such as a FIFO buffer, for buffering the data
`within each of the digital transport streams. An output bus of
`the memory 302 is coupled to the bus 304.
`An input/output bus of the control information analyzer
`308 is coupled to the bus 304. The control information ana
`lyzer 308 extracts control information from the digital trans
`port stream(s) stored within the memory 302. An input/output
`bus of the demultiplexer unit 306 is coupled to the bus 304.
`The demultiplexer unit 306 may use the control information
`extracted by the control information analyzer 308 to recover
`the AV programs from the non-composited digital transport
`stream(s). An input bus of the AV transfer unit 310 is coupled
`to the bus 304. The AV transfer unit 310 is configured to
`provide the AV programs as output for the stream decoder
`120. Embodiments of operation of the stream decoder 120 are
`described below.
`FIG. 4 is a flow diagram depicting an exemplary embodi
`ment of a process or method (“process 400”) for encoding AV
`65
`programs for simultaneous display on a display device. The
`process 400 may be performed by the stream encoder 112
`
`45
`
`40
`
`50
`
`55
`
`60
`
`6
`described above. The process 400 begins at step 402, where at
`least one non-composited digital transport stream is obtained.
`In one embodiment, the non-composited digital transport
`stream(s) is generated having a plurality of AV programs. For
`example, a single non-composited digital transport stream is
`generated having a single program stream, where the single
`program stream is generated by multiplexing the plurality of
`AV programs. Alternatively, a single non-composited digital
`transport stream is generated having a plurality of program
`streams, where each program stream represents one of the AV
`programs. In yet another example, multiple non-composited
`digital transport streams may be generated, each of which
`comprises a single program stream representing one of the AV
`programs. Examples illustrating these embodiments are
`described below with respect to FIGS. 6-9. In another
`embodiment of the invention, the non-composited digital
`transport stream(s) having multiple AV programs are recov
`ered from received AV signals.
`At step 404, the non-composited digital transport stream(s)
`is augmented with control information operative to invoke
`simultaneous display of the AV programs on a display device.
`The control information is configured to allow a display
`device to determine that the user has requested a PIP session.
`In addition, the control information allows the display device
`to distinguish among the different AV programs within the
`non-composited digital transport stream(s) for display in their
`respective regions defined by the display device. In one
`embodiment, the control information is disposed within con
`trol packets associated with a program stream within the
`transport stream (“program control packets”). For example, if
`the transport stream is an MPEG transport stream, then the
`control information may be disposed within a program map
`table (PMT). In another embodiment, the control information
`is disposed within control packets associated with the trans
`port stream (“stream control packets”). For example, if the
`transport stream is an MPEG transport stream, then the con
`trol information may be disposed within a program associa
`tion table (PAT). In yet another embodiment, the control
`information comprises a command that is multiplexed with
`the digital transport stream(s). Examples illustrating these
`embodiments are described below with respect to FIGS. 6-9.
`At step 406, the non-composited digital transport stream(s)
`are transmitted over a digital link in communication with the
`display device.
`FIG. 5 is a flow diagram depicting an exemplary embodi
`ment of a process or method (“process 500”) for decoding one
`or more digital transport streams produced by the process
`400. The process 500 may be performed by the stream
`decoder 120 described above. The process 500 begins at step
`502, where at least one non-composited digital transport
`stream having a plurality of AV programs is received. At step
`504, control information is extracted from the non-compos
`ited digital transport stream(s). As described above, the con
`trol information may be disposed in particular control packets
`(e.g., stream control packets or program control packets), or
`may be part of a command multiplexed with the non-com
`posited digital transport stream(s). At step 506, a determina
`tion is made as to whether a PIP session has been requested.
`If the control information indicates that the non-composited
`digital transport stream(s) contain a plurality of AV programs,
`then the process 500 proceeds to step 508. Otherwise, the
`process proceeds to step 510, where an AV program is recov
`ered from the transport stream and displayed on the display
`device. At step 508, the plurality of AV programs are recov
`ered from the non-composited digital transport stream(s)
`
`Page 8 of 11
`
`

`

`US 7,920,623 B2
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`7
`using the control information. At step 512, the AV streams are
`simultaneously displayed within regions defined by the dis
`play device.
`FIG. 6 is a data flow diagram depicting an example of the
`encoding process 400 for two separate AV programs each
`having a video component and an audio component. A first
`AV program comprises first video data (“video 1”) and first
`audio data (“audio 1"), and a second AV program comprises
`second video data (“video 2”) and second audio data (“audio
`2”). The video and audio components of each of the AV
`10
`programs are multiplexed at step 602 to produce a single
`non-composited program stream (PS). The non-composited
`program stream is multiplexed with program specific infor
`mation (PSI) at step 604 to produce an MPEG-2 transport
`stream. The PSI defines control information for the transport
`stream. In particular, the PSI includes a single PMT associ
`ated with the non-composited stream comprising the two AV
`programs. At step 606, the MPEG-2 transport stream is pro
`cessed for transmission over an IEEE 1394 bus.
`In order for the display device to distinguish among the two
`AV programs, the PMT is augmented at step 608 to include
`identification data. In one embodiment, the identification data
`is disposed in a program level descriptor of the PMT. The
`descriptor may comprise a plurality of fields, such as a
`descriptor tag for identification, a descriptor length to store
`the length of the descriptor in bits, the identification data, and
`one or more reserved fields. The identification data may com
`prise packet identifiers (PIDs) associated with packets defin
`ing the different AV programs. For example, the identification
`data may comprise a PID for video 1, a PID for audio 1, a PID
`for video 2, and a PID for audio 2. In this manner, the display
`device will be able to determine an invocation of a PIP session
`and distinguish among the different AV programs within the
`MPEG transport stream.
`FIG. 7 is a data flow diagram depicting an example of the
`process 400 for two separate AV programs each having a
`video component and an audio component. A first AV pro
`gram comprises first video data (“video 1”) and first audio
`data (“audio 1"), and a second AV program comprises second
`video data (“video 2’) and second audio data (“audio 2'). The
`video and audio components of the first AV program are
`multiplexed at step 702 to produce a first program stream. The
`video and audio components of the second AV program are
`multiplexed at step 704 to produce a second program stream.
`The first and second program streams are multiplexed with
`PSI at step 706 to produce an MPEG-2 transport stream. The
`PSI defines control information for the transport stream. In
`particular, the PSI includes a single PAT associated with the
`transport stream, a first PMT associated with the first program
`stream, and a second PMT associated with the second pro
`gram stream. At step 708, the MPEG-2 transport stream is
`processed for transmission over an IEEE 1394 bus.
`In order for the display device to distinguish among the two
`AV programs, the PAT is augmented at step 710 to include
`identification data. In one embodiment, the identification data
`is private data disposed in an adaptation field of the PAT. The
`adaptation field may comprise a plurality offields, such as an
`adaptation field length for storing the length of the adaptation
`field in bits, one or more private data fields, one or more
`reserved data fields, and the identification data. The identifi
`cation data may comprise PIDs associated with packets defin
`ing the first and second PMTs. In this manner, the display
`device will be able to determine an invocation of a PIP session
`and distinguish among the different AV programs within the
`MPEG transport stream.
`FIG. 8 is a data flow diagram depicting another example of
`the process 400 for two separate AV programs each having a
`
`45
`
`50
`
`55
`
`60
`
`65
`
`8
`video component and an audio component. A first AV pro
`gram comprises first video data (“video 1”) and first audio
`data (“audio 1"), and a second AV program comprises second
`video data (“video 2”) and second audio data (“audio 2'). The
`video and audio components of the first AV program are
`multiplexed at step 802 to produce a first program stream. The
`video and audio components of the second AV program are
`multiplexed at step 804 to produce a second program stream.
`The first program stream i