United States Patent [191
`Tahara
`
`HllllIlllliilllllllllll|l|llllllllllllllllllll||||l|l|||illlllllllllllllll
`US005473380A
`[11] Patent Number:
`5,473,380
`[45] Date of Patent:
`Dec. 5, 1995
`
`[54] PICTURE SIGNAL TRANSMITTING
`METHOD AND APPARATUS
`
`5,260,783 11/1993 Dixit ..................................... .. 348/401
`FOREIGN PATENT DOCUMENTS
`
`[75] Inventor: Katsumi Tahara, Kanagawa, Japan
`
`0349347 1/1990 European Pat. 01f. .
`
`[73] Assignee: Sony Corporation, Tokyo, Japan
`
`[21] Appl. No.: 219,472
`[22] Filed:
`Mar. 29, 1994
`_
`_
`‘V
`_
`_
`Forelgn Apphcatlon Pnonty Data
`
`[30]
`
`OTHER PUBLICATIONS
`AT&T Technical Journal, vol. 72, No. 1, Feb. 1993, New
`York, pp. 67-89, Aravind et al., “Image and Video Coding
`St (1 d ”.
`an at S
`Primary Examiner-Howard W. Britton
`Attorney, Agent, or Firm-—Wi1lia.m S. Frommer; Alvin
`
`Mar. 29, 1993
`
`[JP]
`
`Japan
`
`. .. .
`
`...... .. 5-069829
`
`Smderbrand
`
`'
`
`[51] Int. Cl.6 ..................................................... .. H04N 7/50
`[52] US. Cl. ......................... .. 348/423; 358/335; 358/339
`[58] Field of Search ................................... .. 348/423, 402,
`348/416; 370/1101; 358/335, 339; H04N 7/133,
`7/137
`
`[56]
`
`_
`References Clted
`Us PATENT DOCUMENTS
`
`ABSTRACT
`[571
`A picture type identi?er, indicating one of intra-picture
`coding (an I-picture), forward or backward predictive cod
`ing (a P-picture) and bi-directionally predictive coding (a
`B-picture), is included with a picture signal when the signal
`is encoded and when the signal is decoded. Each of initial
`and subsequent encoding and decoding is a function of the
`included picture type.
`
`5,148,272
`
`9/1992 Acampora .......................... .. 370/1101
`
`20 Claims, 18 Drawing Sheets
`
`INPUT
`PICTURE
`SIGNAL
`
`PFIE-
`PROCESSING
`CIRCUIT
`
`LUMINANCE
`SIGNAL
`COLQR
`
`,2
`3
`ND
`
`[O
`
`SEPARATION —_>I_T%_I1
`(
`ll
`,3
`
`38
`r/
`PRE-
`PROCESSING
`
`Y/C
`SYNTHESIS
`
`OUTPUT
`2:21.112
`
`3
`O
`
`“£13191? E
`$2505
`IF
`D S'GNA'"
`
`36
`S
`
`D/A
`
`D/A
`
`S
`37
`
`FRAME MEMORY
`I5
`
`I4
`'/
`
`—-'
`
`I6
`,
`COLOR DIFFERENCE
`FRAME MEMORY
`
`3
`
`[7 .
`~
`co?l'é'g?m
`
`,
`
`FRAME
`l
`
`B"
`STREAM
`
`I
`
`RECORDING
`MEDIUM
`BIT
`
`E
`
`FRAM
`'
`
`34
`J
`1_-
`
`_
`
`DECODER
`
`3/ 2
`
`FORMAT
`609.1911“
`BLOCK
`
`‘A \35 32 \_
`coLoR DIFFERENCE 33
`FRAME MEMORY N/
`
`FRAME
`
`Apple Exhibit 1012
`Page 1 of 29
`
`

`
`US. Patent
`
`Dec. 5, 1995
`
`Sheet 1 of 18
`
`5,473,380
`
`maul 1k¥
`
`P01)
`
`Apple Exhibit 1012
`Page 2 of 29
`
`

`
`US. Patent
`
`Dec. 5, 1995
`
`5,473,380
`
`$
`
`,
`
`if
`
`T
`
`m
`E
`5“ I
`
`'1'
`
`5E
`
`*-
`
`E
`
`1*
`
`at
`
`“E
`
`Apple Exhibit 1012
`Page 3 of 29
`
`

`
`U.S. Patent
`
`Dec. 5, 1995
`
`Sheet 3 of 18
`
`5,473,380
`
`@X
`
`OX
`
`<m.0_..._
`
`Apple Exhibit 1012
`Page 4 of 29
`
`Apple Exhibit 1012
`Page 4 of 29
`
`

`
`U.S. Patent
`
`Dec. 5, 1995
`
`Sheet 4 of 18
`
`5,473,380
`
`10E
`
`Apple Exhibit 1012
`Page 5 of 29
`
`

`
`US. Patent
`
`Dec. s, 1995
`
`Sheet 5 of 18
`
`5,473,380
`
`H DOTS
`
`SLICE 1
`
`SLICE 2
`
`SLICE 3
`
`SLICE N
`
`(A) FRAME
`
`8 DOTS 8 DOTS
`
`8 DOTS
`
`Y [ 1 ]
`
`Y [ 2 ]
`
`“DOTS
`
`Y[31
`
`Y[4l
`
`VLINES
`
`SDQTS Cb[5]
`
`CI'[6]
`
`xN
`
`XM
`
`(C) MACRQ BLOCK
`
`16
`
`1612345678
`
`(B) SLICE
`
`Apple Exhibit 1012
`Page 6 of 29
`
`

`
`U.S. Patent
`
`Dec. 5, 1995
`
`Sheet 6 of 18
`
`5,473,380
`
`._.Dm._.DO
`
`mm_rs_<mm»m
`
` _I _s_<mEm:m4_Iv_o<mammu_
`
`zo_mm__2
`
`mmumam
`
`_____
`
`
`
`:=om_ozo_m_omo
`
`
`
`mmtamVmzoFo_nmm._wm:oI_mm_.._1._<zoFomm_n_m\momm=._<>_mmnm<zE9=<Ez_
`mS._omm<_nosawus.__
`
`____
`
`mm:5_a
`
`
`
`_IImo_>mo5%.IIIIIIL.IIIIIIIIIIII_wn_>._.
`
`Ben
`
`>mos_m_2m_2<E
`
`em
`
`2m
`
`._<2_O_mO
`
`mmD._.O_..._
`
`._<2_0EO
`
`mmaoa
`
`III;w<.EEn__o._m:m_:<E_
`
`nE<_sv_o<m
`
`zo_._.o_nmmu
`
`
`
`nmm'lmE:.o_._zo=<m-
`
`
`
`P8II..,n_.,__$_,m._.,.__._vmmmmm>mw_wm<_._o
`>mo_zms_ms_<mu_muImmapofi-
`
`
`
`zmn_s_oo
`zofiozh
`
`G_.:._
`
`zo:.os_
`
`_o._m_u_
`
`
`
`
`
`n._mEms_<E
`
`zo:o_omm._
`
`IIIL
`
`_.0<._u_zo_»o_omEa
`
`_n._u_n=m_2<Efl_
`
`NDOE
`
`:_._om_o
`
`IE».wM___ommn___w¢._<&%a
`.
`mm>o-
`mm>omoz<:oND
`
`zofiowpmn
`
`._.Dn_z_
`
`$.25...
`
`_______
`
`._._=O_..=O
`
`on
`
`mv
`
`Apple Exhibit 1012
`Page 7 of 29
`
`Apple Exhibit 1012
`Page 7 of 29
`
`
`
`
`
`
`
`
`
`
`
`

`
`US. Patent
`
`Dec. 5, 1995
`
`Sheet 7 of 18
`
`5,473,380
`
`
`
`2:6 I! llll |- 111:!!! $30
`
`
`
`woos wZBEmE 3mm
`
`
`
`
`
`21> HHHHHHHM. |||||||l.1||||||l. 72> 21> ||||||||||||||| I Z:
`
`
`
`
`
`
`
`Z: I: Z; |||||||||||||||| 1:;
`
`
`
`HHMHHHH, Hnnnnmn 2:5 ||||||||||||||| E718
`
`
`
`
`
`
`
`Emu-m _
`
`Apple Exhibit 1012
`Page 8 of 29
`
`

`
`US. Patent
`
`Dec. 5, 1995
`
`Sheet 8 of 18
`
`5,473,380
`
`21>
`
`Z;
`Q:
`
`.llllnllllllllll
`
`[l‘lllll'll
`
`.Illllllll
`
`5;
`
`3:
`3:
`
`I:
`
`2:5
`
`$30
`H36
`
`$30
`
`
`
`MODE PUO DAME
`
`
`
`MODE FOO N572“?
`
`QUE
`
`Apple Exhibit 1012
`Page 9 of 29
`
`

`
`U.S. Patent
`
`5,473,380
`
`.::om_oNezaoomaom
`
`
`
`HrIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIH
`
`_
`
`
`s_<mEm_.H,HH._...__%H3c_-mm.._..5me.:z<:oczaoomo.Dmm=.5EHmmmm>z_3>Hzo_Emomm
`
`nmo:ooEmmHH:¢._mz_
`.5._SoHvm_HmmLHrIIIIInNHmHm._<ommHHzo_»<~Fz<:oH
`
`
`
`0<:_u_zoFo_nmE1HHHH.mHm_..=:o_._$.25...HHHo._m:ms_<EHH.HwHzo:.oauE H_HH.HH_HHI..u<._u_Boo._u_...m_2<E4:HH_mczHHlllllllllllllInHHBHxuqmxmHHH.
`
`
`
`
`
` zo_5amEHHH.1mw¢o%.H_,HHHwHn:<2Eon_om<_sxo<mH_HHnnnnnnnnnnHHH_H%HH_HHmoo:J_HHHHHHHHHzo=o_omEHH_HHH_HHH_H__H_H:.._oEoHHHHHH5Hzo:<mzm._s_ooHHHHHmHzo:.os_HHHHHH
`
`
`Apple Exhibit 1012
`Page 10 of 29
`
`Apple Exhibit 1012
`Page 10 of 29
`
`
`

`
`US. Patent
`
`Dec. s, 1995
`
`Sheet 10 of 18
`
`5,473,380
`
`SNR +
`
`PPPPPP PPPPPP
`
`TIME
`
`Apple Exhibit 1012
`Page 11 of 29
`
`

`
`U.S. Patent
`
`Dec. 5, 1995
`
`Sheet 11 of 18
`
`5,473,380
`
`ON
`
`pamhao
`
`0._<zo.m
`
`_._...0_n_
`
`MONEE-
`
`N\N
`
`._.Dn_._.DO
`
`2._<zo_m
`
`EN
`
`N\N
`
`IN
`
`.0O._<Z<
`
`omn_>
`
`._<zo_m
`
`NONm.4N
`
`._<:a_o
`
`omn_>
`
`._<zc_w
`
`HI.
`
`‘ONIN
`
`5&2.
`
`m._<zw_m
`
`._<z.o_m
`
`omo_>
`
`co._<z<
`OON
`
`Apple Exhibit 1012
`Page 12 of 29
`
`Apple Exhibit 1012
`Page 12 of 29
`
`
`
`
`
`

`
`US. Patent
`
`Dec. 5, 1995
`
`Sheet 12 of 18
`
`5,473,380
`
`sun‘
`
`PPPPPP PPPPPP
`
`OUTPUT
`SIGNAL b
`
`B
`IBBBBBB| BBBBBBB
`
`PPPPPP PPPPPP
`
`OUTPUT
`SIGNAL 6
`
`BBBBBBBBBBBBBB
`
`TIME
`
`FIG.12
`
`Apple Exhibit 1012
`Page 13 of 29
`
`

`
`U
`
`a
`
`0
`
`5,473,380
`
`.owns3wo._<z<
`
`P5mH._<z.o_m.
`
`t§_>....um.._<.:c_a"uwmum__mcom
`
`_
`
`mo<uEm»z_3an:-.5
`§_<mEmtn.
`
`NB
`
`
`
`
`mo<uEm»z_—mo<uEm:z_M.M.2<.mw__,_»%_m_m.._<.:_o_n.=_E.o_a
`
`%ommmmmasooD1Now,
`5,an32m._<:.c_n,D:m
`
`m%fi_%__»_z_am
`
`B1W...%smammanm.mom1
`
`._.Dn_._.DO
`
`o._<z.o_m
`
`26.".
`
`Apple Exhibit 1012
`Page 14 of 29
`
`Apple Exhibit 1012
`Page 14 of 29
`
`
`
`

`
`SU
`
`
`\O\.oo._<z<P:momo_>
`
`unim._<z.u_mm§u__>mMn9M9_2<mEm
`.._<.:.o_oOS09
`
`TE
`
`,.\.czaoomnczaoo.
`
`fifimmm3.§._%\\3.Momn.__»=_mw_..m<z<HN28mnmxm._._=.S_2%\:o:2555N9
`mm..__.ww_n_
`
`
`
`
`
`
`&m.__.,_._.?
`
`81
`
`5,473,380
`
`
`
`m‘._oo._<z<T
`
`mO\
`
`mo‘
`
`E
`
`v_..G_u_
`
`In.1:
`
`mE.Fo_._Smat fo._<zo_mo._.:n_...:OM.._<z_u_maomo_>a3<z<....Izo:<mEmwmomxm._._:..5s_
`
`
`
`
`
`
`NEDFOE:_<Z.0_w
`
`Apple Exhibit 1012
`Page 15 of 29
`
`Apple Exhibit 1012
`Page 15 of 29
`
`
`

`
`US. Patent
`
`Dec. 5, 1995
`
`Sheet 15 of 18
`
`5,473,380
`
`SNR
`
`PPPPPP PPPPPPOUTPUT
`I
`SlGNALb
`
`I
`
`PPPPPP PPPPPP
`
`BBBBBBB BBBBBIBOUTPUT
`B
`SIGNAL c
`
`TIME
`
`FIG.15
`
`Apple Exhibit 1012
`Page 16 of 29
`
`

`
`U.S. Patent
`
`Dec. 5, 1995
`
`Sheet 16 of 18
`
`5,473,380
`
`\®‘
`
`D3
`
`mE:.o_._
`
`mat.
`
`._<z.o_m
`
`§<mE.m:3«.9
`
`._.:n_._.3O
`
`..._<zo_m
`
`._<.._.o_o
`
`mo<nEmpz_
`
`._<.—.__U_D
`
`omo_>
`
`._<z.o_m
`
`mED._.O_n_
`
`NV‘
`
`0O._<2<
`
`omo_>
`
`._<z.o_m
`
`._.3nE.3O
`
`o._<zo.m
`
`m;
`
`._<._._G_D
`
`omo_>
`
`._<zc_w
`
`._<.—._0_D
`
`mo<"EE.z_
`
`2.0.".
`
`m9
`
`zo_.::.._<._mm
`
`._<za_m
`
`
`
`ommmmmmsoomm_«um._om_mz_
`
`
`
`omn_>..<.:o_o
`
`wzaoomomi%
`
`omxm._.__H5=
`
`ma>»
`
`
`
`$0..mm:._.o_..
`
`mm.
`
`._<:.o_a
`
`omo_>
`
`._<z.o_m
`
`‘V9
`
`._<:¢_o
`
`ND.-62.000Z1mo<uEm»z_
`mmaozo
`
`mn_>._.
`
`.9
`
`a._<zo_m
`
`Ow;
`
`mi
`
`Apple Exhibit 1012
`Page 17 of 29
`
`.oo._<z<
`
`omo_>
`
`ommmmmmzoo
`
`._<z.o_w
`
`:1
`
`
`§_<mm.5to.N3
`
`xtlé._<z0_m.
`...:._z_
`
`Apple Exhibit 1012
`Page 17 of 29
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`
`US. Patent
`
`Dec. 5, 1995
`
`Sheet 17 of 18
`
`5,473,380
`
`
`
`Eosms msEE N. P. w
`
`
`
`WEE. mmEoE éfOk
`
`Apple Exhibit 1012
`Page 18 of 29
`
`

`
`U.S. Patent
`
`Dec. 5, 1995
`
`Sheet 18 of 18
`
`5,473,380
`
`uzaoumaom
`:=om_o\
`
`
`
`IIIIIIIIIII|'\4l.|]n'|I-I||l'|l||||'lI|lI|.’||l||IIlI'll|I
`
`._.Dn:.:O
`
`amoanomnmm
`
`mm=..._.U_n_
`
`mmvmrIIIIIII
`
`-:z<:c
`
`zo:<~
`
`m._<om
`
`m._m<_m<>
`
`IhO2m4
`
`wzaoomo
`
`\
`
`m
`
`
`
`oz_>_momm_2<mm_._.m
`
`
`
`mmuuam..
`
`._.Dn_2_
`
`.:m
`
`Dm<>>mOu_
`
`m>:o_nmm..
`
`mmD._.0_n_
`
`om<2o_o<m
`
`m>:o_nmE
`
`
`
`mE:.o_..__
`
`v_z<m
`
`.uz__._oEsm
`
`ms_<E
`
`>mos_m_2
`
`m>_5_omEn_
`
`mo.5m>
`
`o._m_n_ms_<E
`
`
`
`G<._..._u>_»o_omE
`
`o._m_n=m_2<E
`
`0<;_n_poo
`
`m_..5»o_n_
`
`Apple Exhibit 1012
`Page 19 of 29
`
`Apple Exhibit 1012
`Page 19 of 29
`
`
`
`
`
`
`
`
`
`
`
`

`
`5,473,380
`
`1
`PICTURE SIGNAL TRANSMITTING
`METHOD AND APPARATUS
`
`BACKGROUND OF THE INVENTION
`
`2
`ing P-pictures, while FIG. 2B shows the reference pictures
`used for encoding B-pictures.
`As shown in FIGS. 3A and 3B, there are four methods for
`encoding the macro-blocks (discussed below) of a picture.
`When multiple methods are suitable, the method which will
`give the smallest amount of encoded data is employed on a
`macro-block by macro-block basis. Blocks F1 to F5 in FIG.
`3A represent data for frames of moving picture signals,
`whereas blocks FlX to FSX in FIG. 38 represent data for
`encoded frames. The solid line arrows in FIG. 3A show the
`frames to which motion vectors x1 .
`.
`. x6 relate.
`The ?rst method, shown as SP1, is to not use predictive
`encoding, that is, to use only intra-frarne correlation. This is
`suitable for any macro-blocks of an I-picture a P-picture and
`a B-picture. In other words, if less encoded data is produced
`without predictive encoding, then this method is selected.
`The second method, shown as SP2, is to predictively
`encode relative to a picture which temporally succeeds the
`current picture, referred to as backward predictive encoding.
`The third method, shown as SP3, is to predictively encode
`relative to a picture which temporally precedes the current
`picture, referred to as forward predictive encoding. The
`second method is suitable for macro-blocks of only B-pic
`tures. The third method is suitable for macro-blocks of
`P-pictures and B-pictures.
`The fourth method, shown as SP4, is to predictively
`encode relative to the mean value of two pictures, one
`temporally preceding and one temporally succeeding the
`current picture. This method is suitable for macro-blocks of
`only B-pictures.
`The encoding sequence will now be described.
`The ?rst frame F1 is encoded as an I-picture using the ?rst
`method SP1 so that it is directly transmitted over a trans
`mission channel as encoded data FlX.
`The third frame F3 is encoded as a P-picture. When the
`third method SP3, forward predictive coding, is used for a
`macro-block, difference signals from the temporally preced
`ing frame F1 used as the reference picture, as indicated by
`a broken-line arrow SP3, and a motion vector x3 between
`the reference picture F1 and the current picture F3, are
`calculated and encoded as data F3X for that macro-block.
`Alternatively, in this or another macro~block of the P picture,
`if a smaller amount of encoded data is produced for a
`macro-block of the P picture being encoded, the ?rst method
`SP1 can be used wherein the data of the original frame F3
`are directly utilized as the transmission data F3X for that
`macro-block.
`The second frame F2 is encoded as a B-picture.
`When the fourth method SP4 is used to encode a macro
`block of the frame F2, a difference between the mean value
`of the temporally preceding frame F1 and the temporally
`succeeding frame F3 is calculated, on a pixel by pixel basis.
`The difference data and the motion vectors x1 and x2 are
`encoded as data F2X.
`When the ?rst processing method SP1 is used to encode
`a macro-block of the frame F2, the data of the original frame
`F2 forms the encoded data F2X.
`When one of the second or third methods SP2, SP3 is used
`to encode a macro-block of the frame F2, one of the
`difference between the temporally succeeding frame F3 and
`the current frame F2, and the difference between the tem
`porally preceding frame F1 and the current frame F2 is
`calculated. The difference data and one of the motion vectors
`x1, x2 are encoded as the data F2X.
`The frame F4 for the B-picture and the frame F5 for the
`
`The present invention relates to coding and decoding of a
`picture signal for transmission, and, more particularly, is
`directed to matching the type of predictive coding applied to
`pictures of the picture signal.
`In, for example, a teleconferencing system or a video
`telephone system, moving picture signals are compressed
`and encoded by taking advantage of intra-frame and inter
`frame correlation so that they can be more e?iciently trans
`mitted over a communication channel to a remote location.
`Intra-frame correlation can be utilized by an orthogonal
`transformation, such as a discrete cosine transformation
`(DCT).
`Inter-frame correlation can be utilized by predictive
`encoding between successive pictures. As used herein, a
`picture generally refers to an image represented by a frame.
`When the ?elds of a frame are coded in a non~interlaced
`manner, that is, separately, each ?eld may be referred'to as
`a picture.
`As shown in FIG. 1A, for example, frame pictures PCI,
`PC2 and PC3 are generated at time points t1, t2 and t3. As
`shown by shading in FIG. 1B, the difference between the
`frame pictures PCl and PC2 is obtained as diiTerence picture
`data PC12, and the difference between the frame pictures
`PC2 and PC3 is obtained as diiference picture data PC23.
`Since there is a fairly small change between signals of
`temporally neighboring frames, transmission of only the
`difference picture data utilizes the transmission channel
`more e?iciently than transmission of the original pictures.
`That is, using the difference picture data as encoded picture
`signals reduces the amount of data to be transmitted.
`However, if only the difference signals are transmitted,
`the original picture cannot be restored. It is advantageous to
`occasionally transmit a picture which is not predictively
`encoded as a reference for difference picture data, and
`because it is sometimes more e?icient than transmitting the
`picture as a predictively encoded picture.
`Pictures which are encoded utilizing only intra-frame
`correlation and not inter-frame correlation, are referred to
`herein as intra-pictures or l-pictures.
`Pictures which are encoded with predictive encoding
`relative to one previously encoded picture are referred to
`herein as predictive pictures or P-pictures. The previously
`encoded picture may be an I-picture or a P-picture, and
`temporally succeeds the P-picture.
`Pictures which are encoded with predictive encoding
`relative to at most two pictures, a temporally preceding and
`a temporally succeeding picture, are referred to herein as
`bi-directionally predictive coded pictures or B-pictures. The
`two pictures may each be an I-picture or a P-picture. When
`both are used, the mean value of the two pictures is obtained
`and used as a reference picture for the picture to be encoded.
`A series of pictures may be considered as groups of
`pictures having a predetermined number of frames such as
`F1 .
`.
`. F17. The luminance and chrominance picture signals
`of the leading frame F1 are encoded as an I-picture, the
`picture signals of the second frame F2 are encoded as a
`B-picture, and the picture signals of the third frame F3 are
`encoded as a P-picture. The fourth and the following frames
`F4 to F17 are encoded alternately as B-pictures and P-pic
`tures. FIG. 2A shows the reference pictures used for encod
`
`l0
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`65
`
`Apple Exhibit 1012
`Page 20 of 29
`
`

`
`3
`
`4
`
`5,473,380
`
`P-picture are processed in a similar manner as described
`above to generate transmitted data F4X and F5X.
`FIG. 4 illustrates an arrangement for encoding and decod-
`ing moving picture signals in accordance with the above-
`described predictive encoding scheme. As shown in FIG. 4,
`an encoding device 1 encodes input picture signals and
`transmits the encoded signals to a recording medium 3 as a
`transmission channel for recording. A decoding device 2
`reproduces the signals recorded on the recording medium 3
`and decodes these as output signals.
`The encoding device 1 includes an input terminal 10, a
`pre-processing circuit 11, A/D converters 12 and 13, a frame
`memory 14 including a lurrrinance signal frame memory 15
`and a color difference signal frame memory 16, a format
`converting circuit 17 and an encoder 18.
`Input terminal 10 is adapted to receive a video signal VD
`and to supply the signal VD to pre-processing circuit 11
`which functions to separate the video signal VD into lurni—
`nance signals and color signals, herein chrominance or color
`difference signals, that are applied to analog-to-digital (A/D)
`converters 12 and 13, respectively. The video signals, digi-
`tized by analog-to-digital conversion by the A/D converters
`12 and 13, are supplied to frame memory 14 having memo-
`ries 15, 16 which function to store the luminance signals and
`the color difference signals, respectively, and to read out the
`signals stored therein to format converting circuit 17.
`The converter 17 is operative to convert frame format
`signals stored in the frame memory section 14 into block
`format signals. As shown in FIG. SA, pictures are stored in
`the frame memory section 14 as frarne-forrnat data having V
`lines each consisting of H dots. The converting circuit 17
`divides each frame into N slices, each slice comprising a
`multiple of 16 lines. As shown, in FIG. 5B, the converter 17
`divides each slice into M macro-blocks. As shown in FIG.
`5C, each macro-block represents luminance signals Y cor-
`responding to l6><16 pixels or dots, and associated chrorni-
`nance Cr, Cb signals. These luminance signals are sub-
`divided into blocks Y1 to Y4, each consisting of 8x8 dots.
`The 16x16 dot luminance signals are associated with 8x8
`dot Cb signals and 8x8 dot Cr signals. The converter 17 is
`also operative to supply the block format signals to the
`encoder 18, which is described in detail below with refer-
`ence to FIG. 6.
`
`The encoder 18 operates to encode the block format
`signals and to supply the encoded signals as a bitstrearn over
`a transmission channel for recording on the recording
`medium 3.
`
`The decoding device 2 includes a decoder 31, a format
`converting circuit 32, a frame memory section 33 including
`a luminance signal frame memory 34 and a color difference
`signal frame memory 35, digital-to-analog converters 36 and
`37, a post-processing circuit 38 and an output terminal 30.
`The decoder 31 is operative to reproduce encoded data
`from the recording medium 3 and to decode the encoded
`data, as described in detail below with reference to FIG. 9,
`and to supply decoded data signals to format converting
`circuit 32 which is operative to convert the decoded data
`signals into frame format data signals and to supply the
`frame format data signals as luminance signals and color
`difference signals to the memory 33. The memories 34, 35
`of the memory 33 function to store the luminance and
`chrominance signals, respectively, and to apply these signals
`to D/A converters 36 and 37, respectively. The analog
`signals from converters 36, 37 are synthesized by a post-
`processing circuit 38 which functions to form output picture
`signals and to output them to output terminal 30, and thence
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`to a display unit, such as a CRT, not shown, for display.
`FIG. 6 illustrates the encoder 18 shown in FIG. 4.
`
`Generally, the encoder 18 stores three pictures, the current
`picture and the pictures temporally preceding and succeed-
`ing the current picture. Based on the sequential position of
`the current picture in the group of pictures, the picture
`coding type (I, P or B) is selected for each picture. The
`picture type sequence is determined by a user using picture
`type input device 65, independent of the pictures applied to
`an input terminal 49.
`The encoder 18 also chooses one of frarne-based and
`field-based predictive encoding as will be explained with
`reference to FIG. 7, and further chooses one of frarne-based
`and field-based DCT encoding as will be explained with
`reference to FIG. 8. For each picture, appropriate motion
`vectors are obtained and the picture is predictively encoded
`relative to zero, one or two previously encoded pictures
`which have been locally decoded and which are referred to
`as reference pictures to form a difference data signal. The
`dilference data signal
`is orthogonally transformed into
`blocks of coefficient data which are quantized, variable
`length encoded and transmitted as encoded data.
`At the encoder 18, the quantized data are dequantized,
`inverse orthogonally transformed, and stored as the refer-
`ence pictures. The predictive encoding applies the motion
`vector(s) obtained for the current picture to the reference
`picture(s) to produce a prediction picture which is subtracted
`from the current picture to yield the difference data.
`The elements of the encoder 18 will now be explained in
`detail.
`
`Picture data for encoding is supplied macro-block by
`macro-block to the input terminal 49 and thence to a motion
`vector detection circuit 50 which is operative to process the
`picture data of respective frames as I-pictures, P-pictures or
`as B-pictures, in accordance with a predetermined sequence
`for each group of pictures, as shown for example, in FIGS.
`2A, 2B. The circuit 50 applies the picture data of the current
`frame to a frame memory 51 having frame memories 51a,
`51b, 51c used for storing a temporally preceding picture, the
`current picture and a temporally succeeding picture, respec-
`tively.
`More specifically, the frames F1, F2, F3 are stored in the
`memories 51a, 51b, 51c, respectively. Then the picture
`stored in memory 51c is transferred to memory 51a. The
`frames F4, F5 are stored in the memories 51b, 51c, respec-
`tively. The operations of transferring the picture in memory
`51c to memory 51a and storing the next two pictures in
`memories 51b, 51c are repeated for the remaining pictures
`in the group of pictures.
`After the pictures are read into the memory and tempo-
`rarily stored, they are read out and supplied to a prediction
`mode changeover circuit 52 which is adapted to process the
`current picture for one of frame based and field based
`predictive encoding. After processing the first frame picture
`data in a group of pictures as an I-picture and before
`processing the second frame picture as a B-picture, the
`motion vector detection circuit 50 processes the third frame
`P-picture. The processing sequence is different from the
`sequence in which the pictures are supplied because the
`B-picture may involve backward prediction, so subsequent
`decoding may require that the P-picture temporally succeed-
`ing the B-picture have been previously decoded.
`The motion vector detection circuit 50 calculates as an
`estimated value for intra-coding for each macro-block, the
`sum of absolute values of prediction errors for the frame
`prediction mode for each macro-block and the sum of
`
`Apple Exhibit 1012
`Page 21 of 29
`
`Apple Exhibit 1012
`Page 21 of 29
`
`

`
`5,473,380
`
`5
`
`absolute values of prediction errors for the field prediction
`mode for each macro-block and supplies these sums to the
`prediction decision circuit 54 which compares these sums
`and selects frame prediction mode or field prediction mode
`in accordance with the smallest of these values and provides
`the selected mode to the prediction mode changeover circuit
`52.
`
`If the frame prediction mode is selected, the prediction
`mode changeover circuit 52 outputs the four luminance
`blocks Y1 to Y4 and the two chrominance or color difference
`blocks Cb, Cr of each macro-block received from the motion
`vector detection circuit 50 without processing. As shown in
`FIG. 7A, odd or first field line data, indicated by solid lines,
`and even or second field line data, indicated by dashed lines,
`alternate in each luminance and color difference block as
`received from the motion vector detection circuit 50. In FIG.
`
`7A, a indicates units for motion compensation. In the frame
`prediction mode, motion compensation is performed with
`four luminance blocks (macro-blocks) as a unit and a single
`motion vector is associated with the four luminance blocks
`Y1 to Y4.
`
`If the field prediction mode is selected, the prediction
`mode changeover circuit 52 processes the signals received
`from the motion vector detection circuit 50 so that each of
`
`the four luminance blocks comprises data from a single field
`and the two color difference blocks have non-interlaced odd
`and even field data. Specifically, as shown in FIG. 7B, the
`luminance blocks Y1 and Y2 have odd-field data and the
`luminance blocks Y3 and Y4 have even-field data, while the
`upper halves of the color difference blocks Cb, Cr represent
`odd field color diiference data for the luminance blocks Y1
`and Y2 and the lower halves of the color difference blocks
`
`Cb, Cr represent even field color difference data for the
`luminance blocks Y3 and Y4. In FIG. 7B, b indicates units
`for motion compensation. In the field prediction mode,
`motion compensation is performed separately for the odd-
`field blocks and even-field blocks so that one motion vector
`is associated with the two luminance blocks Y1 and Y2 and
`another motion vector is associated with the two luminance
`blocks Y3 and Y4.
`
`The prediction mode changeover circuit 52 supplies the
`current picture, as processed for frame based or field based
`predictive encoding, to arithmetic unit 53 of FIG. 6. The
`arithmetic unit 53‘ functions to perform one of intra-picture
`prediction, forward prediction, backward prediction or bi-
`directional prediction. A prediction decision circuit 54 is
`adapted to select the best type of prediction in dependence
`upon the prediction error signals associated with the current
`picture signals.
`The motion vector detection circuit 50 calculates, for the
`current picture, the sum of absolute values of the diiferences
`between each Aij and the average value of the Aij in each
`macro-block 2lAij—(average of Aij)! and supplies the sum as
`an estimated value for intra-coding to the prediction decision
`circuit 54.
`The motion vector detection circuit 50 calculates the sum
`of absolute values (or sum of squares) of the dilference (Aij
`—Bij) between signals Aij of the macro-blocks of the current
`picture, and signals Bij of the macro-blocks of the prediction
`picture )3|Aij—Bij l in each of frame prediction mode and field
`prediction mode. As explained above, the motion vector(s)
`for the current picture are applied to the reference picture(s)
`to generate the prediction picture. When the reference pic-
`ture temporally precedes the current picture, the quantity
`2|Aij— Bij! is referred to as a forward prediction error signal,
`and when the reference picture temporally succeeds the
`
`6
`current picture, the quantity 2‘.IAij—Bij| is referred to as a
`backward prediction error signal. When the prediction pic-
`ture is the mean of a temporally preceding and a temporally
`succeeding reference picture, as motion-compensated, the
`quantity 2lAij—Bij| is referred to as a bi-directional predic-
`tion error signal.
`The circuit 50 supplies the forward frame prediction, the
`forward field prediction, the backward frame prediction, the
`backward field prediction, the bi-directional frame predic-
`tion and the bi-directional field prediction error signals to the
`prediction decision circuit 54.
`The prediction decision circuit 54 selects one of intra-
`coding, forward inter-picture prediction, backward inter-
`picture prediction or bi-directional inter-picture prediction
`and one of frame and field prediction mode in accordance
`with the smallest of the estimated value for intra-coding and
`the forward frame, the forward field, the backward frame,
`the backward field,
`the bi-directional frame and the bi-
`directional field prediction error signals. The arithmetic unit
`53 predictively encodes the current picture, as processed by
`the frame or field changeover circuit 52, in accordance with
`the prediction mode selected by the prediction decision
`circuit 54.
`The motion vector detection circuit 50 serves to calculate
`
`and supply the motion vector(s) associated with the selected
`prediction mode to a variable length encoding circuit 58 and
`a motion compensation circuit 64, explained later.
`The sums of the absolute values of the inter-frarne dif-
`
`ferences (prediction errors) on the macro-block basis are
`supplied from the motion vector detection circuit 50 to the
`prediction mode changeover circuit 52 and to the prediction
`decision circuit 54, in the manner as described above.
`The arithmetic unit 53 supplies predictively encoded data,
`also referred to as difference data, for the current picture to
`a DCT mode changeover circuit 55 which is adapted to
`process the current picture for one of frame based and field
`based orthogonal transformation.
`The DCT changeover circuit 55 functions to compare the
`encoding efliciency when the DCT operations for the macro-
`blocks in a picture are performed with the odd field data
`alternating with the even field data, that is, for frame based
`orthogonal transformation, as shown in FIG. 8A, with the
`encoding efliciency when the DCT operations for the macro-
`blocks in a picture are performed with the odd field data
`separated from the even field data, that is, for field based
`orthogonal transformation, as shown in FIG. 8B. The circuit
`55 functions to select the mode with the higher encoding
`efliciency.
`To evaluate the encoding efiiciency for frame based
`orthogonal transfonnation, the DCT mode changeover cir-
`cuit 55 places the luminance macro-block data into inter-
`laced form, as shown in FIG. 8A, and calculates the differ-
`ences between the odd field line signals and even field line
`signals vertically adjacent to each other, and finds the sum
`of absolute values of the diiferences EFM, or the sum of
`squared values of the differences.
`
`10
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`16 15
`EFM= _2‘.
`_}L‘
`1:1 1:1
`
`|o(i,j)— e(i,j)l +
`
`Eq. 1
`
`l6 l5
`_2 _2 le(i,j)—o(i+ l,j)l
`]=1t=l
`
`65
`
`To evaluate the encoding efficiency for field based
`orthogonal transformation, the DCT mode changeover cir-
`
`Apple Exhibit 1012
`Page 22 of 29
`
`Apple Exhibit 1012
`Page 22 of 29
`
`

`
`5,473,380
`
`7
`cuit 55 places the luminance macro-block data into non-
`interlaced form, as shown in FIG. 8B, and calculates the
`differences between vertically adjacent odd field line signals
`and the differences between vertically adjacent even field
`line signals, and finds the sum of absolute values of the
`differences EFD, or the sum of squared values of the
`differences.
`
`16 15
`EFD= .2 _Z (|o(i,]) — o(i+ l,j)l + |e(i,j) — e(i+ l,})l)
`1:1 t=1
`
`Eq.2
`
`l0
`
`The DCT changeover circuit 55 compares the difference
`between the frame based and field based sums of the
`absolute values with a predetermined threshold and selects
`frame based DCT transformation if the difference EFM —
`EFD is less than the predetermined threshold.
`If the frame prediction mode is selected in the prediction
`mode changeover circuit 52, the probability is high that the
`frame DCT mode will be selected in the DCT mode
`
`changeover circuit 55. If the field prediction mode is
`selected in the prediction mode changeover circuit 52, the
`probability is high that the field DCT mode will be selected
`in the DCT mode changeover circuit 55. However, since this
`is not necessarily the case, the prediction mode changeover
`circuit 52 sets the mode which will give the least value of the
`sum of the absolute values of prediction errors, while the
`DCT mode changeover circuit 55 sets the mode which will
`give the optimum orthogonal transformation encoding efli-
`ciency.
`transformation mode, also
`If frame based orthogonal
`referred to as frame DCT mode, is selected, the DCT mode
`changeover circuit 55 functions to ensure that the four
`luminance blocks Y1 to Y4 and two color difference blocks
`Cb, Cr represent alternating or interlaced odd and even field
`lines, as shown in FIG. 8A.
`
`transformation mode, also
`If field based orthogonal
`referred to as field DCT mode, is selected, the DCT mode
`changeover circuit 55 functions to ensure that each of the
`luminance blocks represents only one field, and that each of
`the color difference blocks has segregated or non-interlaced
`odd and even field lines, as shown in FIG. 8B.
`
`The DCT mode changeover circuit 55 functions to output
`the data having the configuration associated with the
`selected DCT mode, and to output a DCT flag indicating the
`selected DCT mode to the variable length encoding circuit
`58 and the motion compensation circuit 64.
`The DCT mode changeover circuit 55 supplies appropri-
`ately configured diiference picture data to a DCT circuit 56
`shown in FIG. 6 which is operative to orthogonally trans-
`form it using a discrete cosine transformation into DCT
`coeflicients, and to supply the DCT coeflicient data to a
`quantization circuit 57 that functions to quantize the