United States Patent [191
`Ueno et a1.
`
`lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll
`US005150432A
`Patent Number:
`[11]
`5,150,432
`[45] Date of Patent:
`Sep. 22, 1992
`
`[54] APPARATUS FOR ENCODING/DECODING
`
`4,951,140 8/1990 Ueno et a1. ........................ .. 358/136
`
`
`
`VIDEO SIGNALS TO IMPROVE QUALITY OF A SPECIFIC REGION
`
`
`
`4,975,978 12/1990 Ando et a1. 4,991,009 2/1991 Suzuki et a1. ..................... .. 358/135
`
`[75] Inventors: Hideyuki Ueno, Fujisawa; Kenshi
`Dachikua Tokyo, both of Japan
`[73] Assignec: Kabushiki Kaisha Toshiba, Kawasaki,
`Japan
`
`_
`[21] Appl. No.. 673,416
`22 F']
`:
`.
`l
`1
`1 ed
`Mar 22’ 1991
`[30]
`Foreign Application Priority Data
`Mar. 26, 1990 [JP]
`Japan .................................. .. '2-73275
`Jun. 29, 1990 [JP]
`Japan ................................ .. 2-173077
`
`OTHER PUBLICATIONS
`SPIE vol. 804 Advances in Image Processing (1987)
`R-H-J‘M' Plompen er a1" pp- 379-384
`ITEJ Tech. Rep., Y. Nagashima et al., 1987 “Detection
`and Tracking Method of Human Area in Video Signal”,
`pp 1744
`
`Primary Examiner-Jose Couso
`Attorney, Agent, or Firm—-Oblon, Spivak, McClelland,
`Male’ 8‘ Neusmdt
`[57]
`
`ABSTRACI
`
`Int. Cl.5 ...................... ..
`] US. Cl. .................................... ..
`[
`[58] Field of Search
`
`""""""""
`
`[55]
`
`References Cited
`US. PATENT DOCUMENTS
`
`/ , 335861356,
`382/48 50 56_
`135 i36 ’105’
`’
`’
`
`’
`
`4,706,260 11/1987 Fedele et a1. ....................... .. 382/56
`4,754,487 6/1988 Newmuis ......... ..
`382/56
`
`4,802,006 l/1989 linuma et a1. . . . . . .
`. . . .. 358/135
`4,847,677 7/1989 Music etal. ...................... .. 358/135
`
`An image encoding apparatus Comprises a region de_
`mating circuit for detectingaspeci?c region from input
`image signals and outputting the region specifying sig
`ml for discriminating the speci?c region from other
`regions, a low-pass ?lter for selectively ?ltering and
`outputting the image signals of regions other than the
`speci?c region in the input image signals, and an encod
`ing circuit for encoding the image signal output from
`thelow'pass?lter‘
`
`19 Claims, 13 Drawing Sheets
`
`N102
`glégigl?
`DETECTING
`cIRcuIT
`
`V119
`
`101
`
`MEMORY
`
`100
`
`3
`I09
`
`108
`J
`
`g
`_
`we
`58
`55
`
`0Q
`
`1
`
`{07
`
`“6
`
`‘9
`E
`n:
`11,-‘
`x
`meet 1:
`:1
`125522? a I
`Q3 5585 m
`—
`20
`
`‘(,7
`
`‘(,8
`
`CODING
`CONTROL
`cIRcuIT
`1
`105
`1
`I
`5
`INTRA / INTER VALID/ INVALID
`PREDICTION
`DECISION
`r- SELECTOR
`CIRCUIT
`I
`'
`I03
`{ I06
`S
`I)
`01
`a212-
`-|°*
`FILTER \ l
`I
`{o4
`:
`]
`H5 I
`[I
`o]
`
`110x INVERSE
`QUANTIZING
`CIRCUIT
`1
`
`H1\
`
`IDcT
`
`-—--_-<>
`“4
`1
`
`MOTION
`
`H3
`3
`
`F'LTER
`I
`
`PREDICTION
`CIRCUIT
`
`Page 1 of 22
`
`SAMSUNG EXHIBIT 1021
`Samsung v. Image Processing Techs.
`
`

`

`US. Patent
`
`Sep. 22, 1992
`
`Sheet 1 of 13
`
`
`
`
`
`0....<>Z_\O.u.<>EMFZ.\(KHZ.
`
`833308
`
`
`ianHIO
`9N l 003
`Hl9N3-l
`BWGVIHVA
`
`
`
`m:.
`
`h:
`
`.LIOOHIO
`9NlX3'Idli‘mW
`
`E .LanHIO
`
`9N lZliNVflO
`
`02.N_...Z<DO
`
`..._30m.0
`
`wmmw>2.
`
`
`
`ZO.m.0mo20....05mmn.
`
`
`
`F.30m:0m0...0m|.mm
`
`
`
`mm....n.
`
`mmS...
`
`.0.
`
`>mozu2.m2<mu
`
`00¢
`
`om...<m2m.n..>.00
`
`.20:0Emma
`m2<mmmw...z.
`
`....30m..0
`
`20:05.
`
`
`
`mm....n.
`
`n50...
`
`SAMSUNG EXHIBIT 1021
`
`Page 2 of 22
`
`
`
`mo-..._Dow..0
`
`Jomhz00NO.20.0mm
`
`P505002_.r0m....mo
`
`m I
`
`02.000
`
`
`
`1.4.04“.
`
`SAMSUNG EXHIBIT 1021
`Page 2 of 22
`
`
`
`
`
`
`
`
`
`
`

`

`US. Patent
`
`Sep. 22, 1992
`
`Sheet 2 of 13
`
`5,150,432
`
`24
`
`. \/
`
`1 (LUMINANCE)
`SIGNAL
`
`C
`
`C
`
`\III
`
`16
`
`Y!
`
`20
`
`vFIG. 2A
`
`24
`
`FIG. 2B
`
`FIG. 20
`
`FIG. 2D
`
`)2
`
`SAMSUNG EXHIBIT 1021
`Page 3 of 22
`
`

`

`US. Patent
`
`Sep. 22, 1992
`
`_
`
`Sheet 3 of 13
`
`5,150,432
`
`(0,0)
`
`A
`
`YS" ———— --
`
`——"-—
`
`A
`
`34
`
`,
`
`(X , Y)
`
`HISTOGRAM
`I
`
`HISTOGRAM
`
`X5
`
`X8
`
`A,
`
`F l G.~ 3
`
`4O"\
`
`YS__
`
`SAMSUNG EXHIBIT 1021
`Page 4 of 22
`
`

`

`US. Patent
`
`Sep. 22, 1992
`
`Sheet 4 of 13
`
`5,150,432
`
`y
`
`y
`
`—-
`
`g
`,:
`6m
`
`55 m6
`
`r-o_
`
`I
`
`(0,0)
`
`Xs
`i
`1
`Ys ""
`
`I
`
`xe
`i
`'.
`
`5ov
`\/
`
`Ye-—--
`
`(X,Y)
`ERAME DIFFERENCE
`IMAGE
`F l G. 5
`
`‘
`
`B
`
`FRAME
`A/DIFFERENCE
`IMAGE
`SIGNAL
`
`,
`
`B’
`
`FIG-7C
`
`FIG. 7A
`
`A m AA’
`.
`i
`i
`,
`X
`TRANSITION POINT
`
`‘
`
`A A ’
`XB
`
`B
`
`FIG. 7B
`
`SAMSUNG EXHIBIT 1021
`Page 5 of 22
`
`

`

`US. Patent
`
`Sep. 22, 1992
`
`Sheet 5 of 13
`
`5,150,432
`
`"IVNEJIS
`SNIAdIOEdS
`NOIDBH
`
`N
`~—
`N
`
` .LI 038 ID
`
`9N IlV'InD'IVO
`NOISBH
`
`
`
`
`
`2H 5
`
`
`
`.LanHIO
`imam
`9NLLOELLBCI
`9NLL03J.3CI
`.LNIOd
`.LNIOd
`NOLLISNVHJ.
`NOLLISNVHJ.
`
`
`LO
`
`o
`a
`
`03
`
`oN
`
`0
`g _
`(\l
`
`LI.
`
`<r
`8
`
`
`
`
`
`
`.Lmoalo
`3mm
`mew
`
`
`
`.LanHIO
`9leaos
`WVHSOLSIH
`SlXV-K
`
`aowavawoo
`
`.LantHO
`arm/A
`mew
`
`.LanHIO
`SNIWHOA
`WVHSOLSIH
`SIXV-X
`
`201
`
`BE
`“2"0
`<5
`E2
`
`200
`
`SAMSUNG EXHIBIT 1021
`
`Page 6 of 22
`
`O3
`0
`N
`
`,0
`o
`N
`
`SAMSUNG EXHIBIT 1021
`Page 6 of 22
`
`

`

`US. Patent
`
`Sep. 22, 1992
`
`Sheet 6 0f 13
`
`5,150,432
`
`TRANSITION
`POINT DATA
`‘COORDINATE
`~ é8£$5€i
`,
`CUIT
`CIR
`
`N304
`
`r
`MEMORY
`
`3
`303
`
`0
`
`E
`a:
`0
`T.
`0‘)
`—
`I
`
`302
`S
`
`‘
`
`{E
`l-
`z
`D
`o
`0
`
`“1
`a)
`mm —'
`EELL‘DJEI
`ELI-‘21w
`u_o--cn
`2
`
`_ __
`
`301
`3
`w
`z
`5
`-'
`g.
`....
`g]:
`0:0
`I‘5
`l-O
`
`403
`s
`o—- COUNTER
`
`402
`\ FRAME
`MEMORY
`
`408
`
`1
`\
`i
`SEL-i !
`6
`
`_
`
`THRESHOL
`
`, CPU
`
`54%2
`MAP
`~——°REO
`__.MAP
`DATA
`
`\
`ADDRESS
`
`‘
`M404
`4* i
`407M’) 4 O
`X
`I
`HT: 2
`|
`‘
`5
`I
`'
`QSELAJIIR
`cOMPARATO? sEL.2:- ,sEL.3!
`i
`i
`5
`7
`j
`i
`|
`I
`405
`409T
`s401
`\
`
`°-'—-—‘
`
`406 THRESHOLD
`
`HISTOGRAM
`MEMORY
`
`F l G. 9
`
`SAMSUNG EXHIBIT 1021
`Page 7 of 22
`
`

`

`US. Patent ’
`
`Sep. 22, 1992
`
`Sheet 7 of 13
`
`5,150,432
`
`0
`~22
`
`5i6
`'—*o
`
`+
`
`OF
`
`501
`3
`PIXEL
`MEMORY
`
`_ LINE
`MEMORY ‘
`
`L502
`
`F l 6. i0
`
`C
`
`9 212
`‘ M
`
`604
`
`+
`
`>—-—---@
`
`_ 602
`
`603
`
`f3
`
`+
`
`601
`FRAME 'V
`MEMORY "———
`
`,0
`
`Q
`A V212
`
`604
`
`+
`
`,____o
`
`602
`
`(S505
`NONLINEAR
`CIRCUIT
`*
`
`+
`
`v
`
`FRAME ~60‘
`MEMORY *
`
`F I G. 12
`
`SAMSUNG EXHIBIT 1021
`Page 8 of 22
`
`

`

`US. Patent
`
`Sep. 22, 1992
`
`Sheet 8 of 13
`
`5,150,432
`
`OUTPUT
`
`T ~——-—
`
`INPUT
`
`F I G. 13A
`
`OUTPUT
`
`F I G. I38
`
`SAMSUNG EXHIBIT 1021
`Page 9 of 22
`
`

`

`US. Patent
`
`Sep. 22, 1992
`
`_
`
`Sheet 9 of 13
`
`5,150,432
`
`O
`x212
`
`C X‘
`
`602
`
`603
`
`-
`
`_ B
`
`604
`mv ‘
`
`506
`+
`'—-O
`
`+
`
`A
`
`cc1<>
`
`501 v503
`S
`
`_ PIXEL
`MEMORY
`
`505
`
`_ LINE
`MEMORY
`L502
`
`V601
`
`FRAME
`MEMORY F
`
`F | G. {4
`
`A
`II
`
`/
`/
`
`,
`
`c/
`BX
`
`PREVlOUS
`FRAME
`
`CURRENT
`FRAME
`
`FIG. {5
`
`SAMSUNG EXHIBIT 1021
`Page 10 of 22
`
`

`

`US. Patent
`
`Sep. 22, 1992
`
`Sheet 10 of 13
`
`5,150,432
`
`833.408
`
`
`.LanHIO
`ONIOOO
`HiSNB'I
`BWBVIHVA
`
`m:
`
`h:
`
`ianHIO
`
`r9N|X3'|dIJJ‘nW
`a .LIflOHIC)
`
`ZOE—own.
`
`._._Dom_0
`
`202.0Ewan.
`
`mOFomgmm
`
`9N|Z|1NVFIO
`
`«GLUE
`
`mmdm
`
`O:05. 20.6mm
`
`m:
`
`OZ.000
`
`JOEFZOU
`
`.EDUEO
`
`ON_
`
`OZ.>n:0m.n_m
`
`m3k<m<mm<
`
`
`
`.230m.002.Nc.2430
`
`mmmm>z_
`
`Duh<m2mm200z
`
`m2<mmmm§Z.
`
`20....05mmn.
`
`.EDUEQ
`
`EMF...Emoo...
`
`.0.
`
`
`
`Eozms..m2<E
`
`OO—
`
`SAMSUNG EXHIBIT 1021
`
`Page 11 of 22
`
`SAMSUNG EXHIBIT 1021
`Page 11 of 22
`
`
`
`
`
`
`
`
`
`
`

`

`US. Patent
`
`Sep.22, 1992
`
`Sheet 11 of 13
`
`5,150,432
`
`8OI
`§
`
`802
`
`COORDINATE
`INPUT
`APPARATUS
`
`* REGION
`CALCULATING m’
`i CIRCUIT
`
`F | G. IT
`
`A X/
`
`/9I
`
`B
`
`V
`
`F I G. 18
`
`92
`
`F I GJI9
`
`SAMSUNG EXHIBIT 1021
`Page 12 of 22
`
`

`

`US. Patent
`
`Sep. 22, 1992
`
`Sheet 12 of 13
`
`5,150,432
`
`Q.6ON
`
`MON.NON.
`
`mwo3NSM
`
`_Il.|_n.\.own.MUN—30m
`
`
`mmooozmmmooo
`
`
`
`$m3<m<am<m02.>n_GuamNON_20.0mm
`
`Muduc
`
`qllu:h—\_0mmMUKDOWOUO_>lmmem
`
`
`
`
`
`
`ENDOUUDmwooomoawn—002m0250%):
`
`F5950
`
`ON.0_u_
`
`
`
`avowAAww.oz.w4V9.no»
`
`SAMSUNG EXHIBIT 1021
`
`Page 13 of 22
`
`SAMSUNG EXHIBIT 1021
`Page 13 of 22
`
`
`
`

`

`US. Patent
`
`Sep. 22, 1992
`
`Sheet 13 0f 13
`
`5,150,432
`
`N
`
`N
`
`Q s
`
`Maud
`
`EM»
`
`m zéwmm
`om.)
`
`m No» 02
`
`Al mwoouzw $58
`w _ “8.
`
`nmm. “I _ Ht 1 M858 5
`3W W 6» A
`
`no» %
`
`m Hf?
`
`"0
`
`
`
`m m9. mob mQ. m2.
`
`
`
`
`
`3% W A A A _
`
`5 .0 _ n_
`
`
`
`1.3 “I _ Ht wumpow 89> ommmmxm
`
`
`1... 508mm $5085 I mwoouzm :DQEQ
`
`SAMSUNG EXHIBIT 1021
`Page 14 of 22
`
`

`

`1
`
`5,150,432
`
`2
`BRIEF DESCRIPTION OF THE DRAWINGS
`The accompanying drawings, which are incorpo
`rated in and constitute a part of the speci?cation, illus
`trate presently preferred embodiments of the invention,
`and together with the general description given above
`and the detailed description of the preferred embodi
`ments given below, serve to explain the principles of the
`invention.
`FIG.'1 is a block diagram showing an embodiment of
`the image encoding apparatus according to the present
`invention;
`FIGS. 2A to 2D are views for explaining the format
`of macro block and that of the signal input to low-pass
`?lter;
`FIG. 3 is a view for explaining the principle for detec
`tion of facial region;
`FIG. 4 is a view for explaining the principle for detec
`tion of facial region according to another method;
`FIG. 5 is a view showing that the detected area is
`speci?ed by a rectangle;
`FIG. 6 is a block diagram showing a con?guration of
`the facial region detecting circuit in FIG. 1;
`FIGS. 7A to 7C are views for explaining facial re
`gions detecting the operation of the facial region detect
`ing circuit in FIG. 6;
`'
`FIG. 8 is a block diagram showing a con?guration of
`the histogram forming circuit used for the face detect
`ing circuit in FIG. 6;
`FIG. 9 is a block diagram showing another con?gura
`tion of the facial region detecting circuit in FIG. 1;
`FIG. 10 is a block diagram of an example of the low
`pass ?lter in FIG. 1;
`FIG. 11 is a block diagram of another example of the
`low-pass ?lter in FIG. 1;
`FIG. 12 is a block diagram of still another example of
`the low-pass ?lter in FIG. 1;
`FIGS. 13A and 13B are views showing input-output
`characteristic of the nonlinear circuit in FIG. 12;
`FIG. 14 is a block diagram of still another example of
`the low-pass ?lter in FIG. 1;
`FIG. 15 is a view for explaining the operation of the
`low-pass ?lter in FIG. 14;
`FIG. 16 is a block diagram showing another embodi
`ment of the image encoding apparatus according to the
`present invention;
`FIG. 17 is a block diagram showing the con?guration
`of the region specifying apparatus used for the appara
`tus in FIG. 16;
`FIG. 18 is a view for explaining a method to specify
`an input image area in the apparatus in FIG. 16;
`FIG. 19 is a view for explaining another method to
`specify an input image area in the apparatus in FIG. 16;
`FIG. 20 is a block diagram of an embodiment of the
`videophone system using the image encoding apparatus
`according to the present invention; and
`FIG. 21 is a block diagram of another embodiment of
`the video telephone system using the image encoding
`apparatus according to the present invention.
`
`APPARATUS FOR ENCODING/DECODING
`VIDEO SIGNALS TO IMPROVE QUALITY OF A
`SPECIFIC REGION
`
`10
`
`BACKGROUND OF THE INVENTION
`1. Field of the Invention
`The present invention relates to an apparatus for
`encoding video signals used for a teleconference or
`videophone.
`2. Description of the Related Art
`Video signal encoding systems have more positively
`been studied in recent years because they will be stan
`dardized soon. Also, intelligent encoding has been pro
`posed as a future encoding system and encoding systems
`using knowledge of image in any form have been dis
`cussed as part of the study. As one of the encoding
`systems, the system is proposed to detect facial region
`from motion picture signals of human image and distrib
`ute many quantizing bit number to image signals of
`facial region.
`For example, in the Preture Coding symposium Japan
`PCS] 89, 7-15 in 1989, the technique was disclosed to
`relatively improve the quality of decoded images of
`25
`facial region by making the quantizing step size of re
`gions other than facial region larger than the quantizing
`step size determined by the capacity of the buffer to
`store encoded data when quantizing DCT (Discrete
`Cosine Transform) coef?cient obtained by DCT-encod
`ing movement images. This technique, however, has the
`problem that visually-remarkably distortion peculiar to
`DCT-encoding such as block distortion is generated in
`decoded image because there are many quantization
`noise of DCT coef?cient in regions other than facial
`region.
`'
`
`35
`
`40
`
`45
`
`50
`
`55
`
`SUMMARY OF THE INVENTION
`It is an object of the present invention to provide a
`movement image encoding apparatus capable of im
`proving the quality of decoded images of speci?c region
`without generation of large distortion in regions other
`than speci?c region such as facial region.
`According to the present invention, a speci?c region
`in image signals sequentially input for every frame is
`detected which corresponds to a movable image having
`the speci?c region and the region specifying signal to
`separate the speci?c region from other regions is out
`put. Image signals of regions other than the speci?c
`region among input image signals ar selectively ?ltered
`by a low-pass ?lter according to the region specifying
`signal. And the image signals output from the low-pass
`filter are encoded by an encoding circuit.
`According to the present invention, any speci?c re
`gion is speci?ed by a region specifying apparatus and,
`thereby, the region specifying signal same as the above
`is output. Image signals of regions other than the spe
`ci?c region are selectively ?ltered according to the
`region specifying signal and, moreover, the image sig
`nals output from the low-pass ?lter means are encoded
`by an encoding circuit.
`Additional objects and advantages of the invention
`will be set forth in the description which follows, and in
`part will be obvious from the description, or may be
`learned by practice of the invention. The objects and
`advantages of the invention may be realized and ob
`tained by means of the instrumentalities and combina
`tions particularly pointed out in the appended claims.
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENTS
`For simple explanation, an example using the
`Px64kbps standard video coding system (see CCITT
`recommendation H. 261) as an encoding system is de
`scribed in the following embodiment. As a matter of
`course, the present invention can be applied to other
`encoding systems.
`
`65
`
`SAMSUNG EXHIBIT 1021
`Page 15 of 22
`
`

`

`20
`
`5,150,432
`3
`4
`A multiplexing circuit 116 multiplexes the decision
`In the embodiment shown in FIG. 1, a terminal 100
`signal sent from the intra/interframe prediction selector
`for inputting movement image signals is connected to a
`105, the signal output from quantizing circuit 108, valid
`frame memory 101 for storing image signals for one
`/invalid decision signal sent from the valid/invalid deci
`frame (this is called frame data). The readout terminal
`sion circuit 109, motion vector information sent from
`of the frame memory 101 is connected to the input
`the motion compensated interframe prediction circuit
`terminal of a facial region detecting circuit 102 for de
`113, and ?lter on/off signal sent from the loop ?lter 114.
`tecting the human facial region of the frame data and
`The output terminal of the multiplexing circuit 116 is
`that of a low-pass ?lter 103 for ?ltering the frame data.
`connected to the input terminal of a variable length
`The output terminal of the facial region detecting cir
`encoding circuit 117. The output terminal of the vari
`cuit 102 is connected to the control terminal of the
`able length encoding circuit 117 is connected to the
`low-pass filter 103.
`write terminal of a buffer memory 118 for matching the
`The con?guration after the low-pass ?lter 103 is the
`amount of information generated by the encoding appa
`same as that of a normal image encoding apparatus
`ratus with the transmission rate. The readout terminal
`using said standard encoding system.
`of the buffer memory 118 is connected to a transmission
`That is, the output of the low-pass ?lter 103 is con
`line. The buffer memory 118 has a function for generat
`nected to a subtracter 104, intra/interframe prediction
`ing the information showing the buffer capacity and the
`selector 105, switch 106, and motion compensated inter
`information is supplied to an coding control circuit 119.
`frame prediction circuit 113. The output terminal of the
`The coding control circuit 119 controls the quantizing
`motion compensated interframe prediction circuit 113
`circuit 118 and the valid/invalid decision circuit 109
`connects with a loop ?lter 114. The motion compen
`according to the buffer amount.
`sated inter frame prediction circuit 113 comprises a
`Operations of the encoding apparatus of this embodi
`frame memory, motion vector detecting circuit, and
`ment are described below according to FIGS. 1 and 2A
`variable delay circuit to generate motion compensated
`through 2D.
`interframe prediction signals in macro-blocks. The in
`Video signals are sequentially input to the image
`terframe prediction signal is sent to the subtracter 104
`input terminal 100 in frames. The video signal is, for
`through the loop ?lter 114 and the difference from
`example, obtained by encoding analog image signals
`macro-block data output by the low-pass ?lter 103 is
`obtained through picking-up of a human with a TV
`detected by the subtracter 104. The intra/interframe
`camera (not illustrated) into luminance signal Y and two
`prediction selector 105 compares the power of the dif
`types of color signals (or color-difference signals) CB
`30
`ferential signal output by the subtracter 104 with that of
`and CR and converting these encoded signals into digi
`the macro-block data signal output by the low-pass
`tal data called CIF or QCIF by an A-D converter and
`?lter 103 to decide whether to execute interframe or
`format converter (not illustrated). The image signal is
`interframe prediction. The switch 106 receives the pre
`written in the frame memory 101 as frame data one
`diction result from the intra/interframe prediction se
`frame by one frame. The frame data written in the frame
`lector 105 to select the output signal of the subtracter
`memory 101 is read out every a plurality of blocks and
`104 which is the interframe prediction signal or that of
`supplied to the facial region detecting circuit 102 and
`the low-pass ?lter 103 which is the interframe predic
`low-pass ?lter 103. The data can also be read out every
`tion signal.
`frame. The data readout unit is larger than the block
`The output terminal of the switch 106 is connected to
`size (8 pixels><8 pixels) which is the conversion unit
`the input terminal of a DCT (Discrete Cosine Trans
`(encoding unit) in the DCT circuit 107. The face detect
`form) circuit 107. The output terminal of the DCT
`ing circuit 102 detects the facial region in the current
`circuit 107 is connected to a quantizing circuit 108 and
`frame data _by obtaining the difference between the
`valid/invalid decision circuit 109. The DCT circuit 107
`current frame data newly input from the frame memory
`outputs DCT coef?cient data. The quantizing circuit
`101 and the previous frame data stored in the internal
`108 quantizes each DCT coef?cient data value to out
`frame memory in macroblocks and outputs the region
`put, for example, a quantizing-table number and quan
`specifying signal to identify whether each macro-block
`tizing index. The valid/invalid decision circuit 109 cal- -
`in the current frame data belongs to the facial region or
`culates the signal power for each block with DCT coef
`other regions. The facial region detecting circuit 102
`ficient data to decide whether the block is valid or
`updates the data in the memory frame by the current
`invalid.
`frame data every macro-block for which facial region
`The output terminal of the quantizing circuit 108 is
`detection is completed.
`connected to the input terminal of an inverse quantizing
`The low-pass ?lter 103, as mentioned in detail later,
`circuit 110. The output terminal of the inverse quantiz
`having spatial and/or temporal low-pass characteristic.
`, ing circuit 110 is connected to the input terminal of an
`The following is the description of spatial ?ltering in
`inverse DCT circuit 111 which executes processing
`the space direction by the low-pass filter 103. Since a
`inversely to the DCT circuit 107. The output terminal
`plurality of blocks and pixels around the blocks are
`of the inverse DCT circuit 111 is connected to one input
`required for spatial filtering the blocks, the data in the
`terminal of an adder 112. The other input terminal of
`frame memory 101 is read out for each unit having the
`the adder 112 connects with the output terminal of the
`blocks and pixels. For the size of 24 pixelsX 16 pixels
`switch 115. The input terminal of the switch 115 con
`consisting of the luminous signal Y and color-difference
`nects with the output terminal of the loop ?lter 114. The
`signals C5 and CR whose readout unit is equal to the size
`adder 112 adds the interframe prediction signal input
`of the macro-block speci?ed in H. 261 as shown in FIG.
`from the loop ?lter 114 through the switch 115 with the
`2A, frame data is input to the low-pass ?lter 103 from
`output signal of the inverse DCT circuit 111 to generate
`the frame memory 101 in regions having the size of 32
`locally-decoded signals. The output terminal of the
`pixels><24 pixels including the luminous signal Y and
`adder 112 is connected to the input terminal of the
`color-difference signals CB and CR as shown in FIGS.
`motion compensated interframe prediction circuit 113.
`2B through 2D. The low-pass ?lter 103 converts the
`
`55
`
`65
`
`25
`
`35
`
`45
`
`SAMSUNG EXHIBIT 1021
`Page 16 of 22
`
`

`

`20
`
`30
`
`35
`
`5,150,432
`5
`currently-input macro-block into the previously-speci
`?ed macro-block format data (macro-block data) with
`out ?ltering if the macro-block belongs to the facial
`region but converts the macro-block into macro-block
`data after ?ltering if it belongs to regions other than the
`face region.
`The output of the low pass ?lter 103 is supplied to one
`input terminal of the subtracter 104 and also to the
`inter/interframe prediction selector 105. The subtracter
`104 produces an interframe difference signal by subtrac
`tion of the output signal of the low-pass ?lter 103 and
`the interframe prediction signal sent from the loop ?lter
`114. The intra/interframe prediction selector 105 com
`pares the power of the differential signal output by the
`subtracter 104 with that of the signal output by the
`low-pass ?lter 103 in macro-blocks and decides inter
`frame prediction if the former is smaller and intraframe
`prediction if the latter is smaller. The switch 106' selects
`the differential signal sent from the subtracter 104 if the
`decision by the intra/interframe prediction selector is
`interframe prediction and the output signal of the low
`pass ?lter 103 if it is intraframe prediction.
`The signal selected by the switch 106 is DCT
`encoded by the DCT circuit 107 and DCT coef?cient
`data is generated. DCT coef?cient data is quantized by
`25
`the quantizing circuit 108 and quantizing-table numbers
`and quantizing indexes, for example, are output as quan
`tization results. The output of the quantizing circuit 108
`is inversely quantized by the inverse quantizing circuit
`110 and also inversely discrete-cosine-transformed by
`the inverse DCT circuit 111. The locally-decoded data
`obtained through inverse DCT is stored in the frame
`memory built in the motion compensated interframe
`prediction circuit 113 and referenced so that the next
`frame data will be encoded. That is, the motion com
`pensated interframe prediction circuit 113 compares the
`locally-decoded data of the previous frame stored in the
`built-in frame memory with the current frame data sent
`from the low-pass ?lter 103 to generate motion vector
`information. The motion compensated interframe pre
`40
`diction circuit 113 also selects the macro-block data
`corresponding to the motion vector information among
`locally-decoded data using the variable delay circuit
`and supplies the macro-block data to the loop ?lter 114
`as interframe prediction signal. The loop ?lter 114 is
`45
`composed of a spatial ?lter, which eliminates noises
`contained in the interframe prediction signal sent from
`the motion compensated interframe prediction circuit
`113 to supply the signal to the subtracter 104.
`DCT coef?cient data is also input to the valid/ invalid
`decision circuit 109. The valid/invalid decision circuit
`109 calculates the signal power for each block with the
`DCT coef?cient data and compares the signal power
`with the threshold value to decide whether the block is
`valid or invalid.
`The decision signal sent from the intra/interframe
`prediction selector 105, quantizing table number and
`quantizing index sent from the quantizing circuit 108,
`valid/invalid decision signal sent from the valid/ invalid
`decision circuit 109, motion vector information sent
`from the motion compensated interframe prediction
`circuit 113, and ?lter on/off signal sent from the loop
`?lter 114 are multiplexed by the multiplexing circuit 116
`according to the speci?ed format. The output of the
`multiplexing circuit 116 is converted into variable
`length code (e.g. Huffman code) by the variable length
`encoding circuit 117. The output of the variable length
`encoding circuit 117 is written in the buffer memory 118
`
`6
`and read out at the speed to be matched with the trans
`mission rate. Read-out signals are sent to the transmis
`sion line.
`The buffer memory 118 also generates the informa
`tion indicating the buffer amount and the information is
`supplied to the coding control circuit 119. The coding
`control circuit 119 estimates the currently generated
`information amount of the encoding apparatus with the
`information of the buffer amount and controls the quan
`tizing circuit 108 and valid/invalid decision circuit 109
`according to the estimated result. Concretely, for a
`large amount of generated information, the circuit 119
`decreases the generated information amount by increas
`ing the quantizing step size in the quantizing circuit 108
`or increasing the threshold value for decision with the
`valid/invalid decision circuit 109. For a small amount of
`generated information, the circuit 119 increases the
`generated information amount through inverse opera
`tion. It is possible to use adaptive frame-drop control
`instead of changing the quantizing step size.
`In the image encoding apparatus of this embodiment
`mentioned above, macro-blocks in regions other than
`the facial region among image signals input to the termi
`nal 100 are ?ltered by the low-pass ?lter 103 and their
`high-frequency components are removed or sup
`pressed. The operation can decrease the quantizing bit
`number to be distributed to regions other than the facial
`region because the information amount generated due
`to the high-frequency components is decreased. This
`method can realize improvement of the image quality of
`the facial region or the subjective image quality without
`making remarkable distortion such as block distortion is
`prevented from occurring in regions other than the face
`region.
`In the facial region detecting circuit 102, detection is
`executed in macro-block as previously mentioned. Fil
`tering by the low-pass ?lter 103 is applied to macro
`blocks in the facial region and pixels signals around the
`region according to the region specifying signal from
`facial region detecting circuit 102. Therefore, the reso
`lution of the facial region detecting circuit 102 is
`enough if the circuit is able to identify whether the
`entire noticed macro-block belongs to the face region or
`other regions. However, if higher resolution can be
`obtained, it is possible to include the information indi
`cating the position of the boundary between the facial
`region and other regions in the region specifying signal
`to be sent from the facial region detecting circuit 102 to
`the low-pass ?lter 103. As a result, the low-pass ?lter
`103 can apply ?ltering only to regions other than the
`facial region in a macro-block when the boundary
`crosses the macro-block. According to the above
`method, the encoding ef?ciency is further improved
`and more natural images can be obtained because the
`boundary of a region with a different band width is not
`generated in regions other than the facial region.
`The encoding apparatus of the present invention can
`easily be embodied because only the facial region de
`tecting circuit 102 and the low-pass ?lter 103 for selec
`tively ?ltering face-region image signals for the region
`specifying signal sent from the facial region detecting
`circuit 102 are added to a conventional image encoding
`apparatus and the circuit con?guration after encoding
`and the encoding control may be basically the same as
`those of normal encoding apparatuses.
`The present invention can also be applied to the sys
`tem using, for example, DPCM (Differential PCM)
`other than DCT encoding system as the encoding sys
`
`50
`
`55
`
`65
`
`SAMSUNG EXHIBIT 1021
`Page 17 of 22
`
`

`

`IS
`
`30
`
`35
`
`5,150,432
`8
`7
`mined, for example, by the method in FIG. 3. The facial
`tem In other words, the present invention can be ap
`region is speci?ed by the rectangle 50 designated by
`plied to every encoding system using the correlation in
`the space direction.
`coordinates Xs, Xe, Ys, and Ye also through this
`The following is the description of practical examples
`method as shown in FIG. 5.
`of how to detect facial region in the facial region detect
`FIG. 6 shows a facial region detecting circuit using
`ing circuit 102 in FIG. 1 according to FIG. 3.
`the principle in FIG. 3. In this facial region detecting
`FIG. 3 shows an interframe difference image input to
`circuit, the x-axis histogram shown in FIG. 7B and the
`the facial region detecting circuit 102. Information for
`histogram shown in FIG. 7C are formed by the frame
`the interframe difference image is converted into binary
`difference image shown in FIG. 7A. By extracting the
`data of “O” or “1” by the preset ?rst threshold level.
`coordinates of the transition points of these histograms,
`Then the number of pixels having the binary data value
`the face region is speci?ed by a rectangle as shown in
`of “l” or the value equal to or more than the ?rst
`FIG. 5. It is considered that many signal components of
`threshold value is counted in the vertical and horizontal
`the frame difference image are generated at such a mov
`directions of the screen and histograms of the pixels
`able portion as shoulder. Therefore, the x-axis histo
`(x-and y-axis histograms) are formed. Facial detection is
`gram including the shoulder portion is formed which is
`executed according to the histograms.
`shown above the line B-B’ in FIG. 7B. Thus, the verti
`Face detection starts with the top of a head 30. The
`cal region of the inter-block difference image is limited
`top of the head 30 is detected by searching the y-axis
`in which the x-axis histogram is formed using the y-axis
`histogram from the top and selecting the point Ys
`histogram. As a result, the x-axis histogram of the facial
`which ?rst exceeds the present second threshold value.
`20
`region excluding the shoulder portion shown above the
`After the top of the head 30 is detected, the left end
`line A-A' in FIG. 7B is obtained.
`31 and right end 32 of the head 30 are detected by
`In FIG. 6, image signals sent from the frame memory
`searching the x-axis histogram from the left and right
`101 in FIG. 1 are input to a terminal 200. A frame mem
`and selecting the points Xs and Xe which ?rst exceed
`ory 201 delays the input image signal by the time for one
`the second threshold value. In this case, the y-axis histo
`25
`frame. A subtracter 202 outputs frame difference image
`gram is formed only for the region with the width of A
`signals by subtracting the image signal output by the
`from the top of the head 30 so that the region with the
`frame memory 201 from said input image signal. Histo
`width of a shoulder 34 will not be detected. Thus, the
`gram forming circuits 203 and 204 form y-axis and x
`y-axis histogram with the width of the head can be
`axis histograms with the interframe difference image
`obtained. The width A is, for example, determined by
`signal respectively. It is also possible to form x-axis and
`the following formula by assuming the image size as
`y-axis histograms using an inter ?eld difference image
`XXY.
`instead of the interframe difference image. Mean value
`circuits 205 and 206 obtain mean values from the
`formed y-axis and x-axis histograms. Comparators 207
`and 208 use these mean values as threshold values to
`compare these values with histograms formed by the
`transition point detecting circuit 203 and 204 respec
`tively.
`Transition point detecting circuits 209 and 210 detect
`the coordinates of x-axis and y-axis histograms and
`output coordinate data. The output of the transition
`point detecting circuit 209 corresponding to the y-axis is
`supplied to the x-axis histogram forming circuit 204