Ex. GOOG 1032
`
`
`
`
`
`

`
`United States Patent [19]
`Tatsumi et al.
`
`I|ll|l|||l|||lllllllllllllllllllllllllllllllllllllllIllllllllllllllllllllll
`5,594,736
`Jan. 14, 1997
`
`US005594736A
`Patent Number:
`[11]
`[45] Date of Patent:
`
`[54]
`
`IIVIAGE-INFORMATION STORAGE
`REGENERATING DEVICE
`
`[75]
`
`Inventors: Masahiro Tatsumi; Noritaka Koizumi,
`both of Chiba; Keiichi Hibi, Matsudo;
`Tsuneaki Iwano, Tokyo; Hiroyuki Ito,
`Chiba; Osamu Nakamura, Hanno;
`Hirotaka Nakano, Musashino, all of
`Japan
`
`[73] Assignee: Sharp Kabushiki Kaisha, Osaka, Japan
`
`Appl. No.: 364,891
`Filed:
`Dec. 23, 1994
`Foreign Application Priority Data
`
`[21]
`[22]
`[30]
`Dec. 24, 1993
`Dec. 24, 1993
`Dec- 24, 1993
`Dec- 24’ 1993
`
`[JP]
`[JP]
`[JP]
`[JP]
`
`’
`
`Japan .................................. .. 5-328365
`Japan
`. 5-328555
`Japan
`- 5323717
`Japan -
`53288”
`. HMJ 3,24
`[51] Im_ GL6 _______________ __
`[52] US. Cl. ........................ .. 370/474; 370/535; 348/466;
`348/232
`348/466 467
`[58] Field of Search
`419 376/941’
`348/423 17
`97 ’ 60’ 6O 1’ 942’ 112’ 84 ’85 6’ 118 169’
`168 1’10 ‘1_’371'/:;7 1 ’38 1’ 39‘ 1’ 30 ’37 2’
`37
`37's’ 40 2 6'9
`I206 ’371’ 36'3’
`‘
`‘
`’
`'
`’ 21‘4f 455/18_ 5.79/5’3 96’
`’
`’
`’
`References Cited
`
`[56]
`
`U.S. PATENT DOCUMENTS
`
`5,347,305
`5,392,284
`5,481,543
`
`9/1994 Bush et a1. ..
`.... .. 348/17
`.... .. 348/17
`2/1995 Sugiyama
`1/1996 Veltrnan ................................ .. 370/112
`
`FOREIGN PATENT DOCUMENTS
`
`62-188579
`3-132128
`4-079655
`
`8/1987 Japan .
`6/1991 Japan .
`3/1992 Japan .
`
`OTHER PUBLICATIONS
`
`Ichizo Kimura et al., Prototyping of PC-Videophone with
`Built-In Codec based on Open Architecture, 1993, The
`Journal of the Institute of Image Electronics Engineers of
`Japan, vol. 22, N0. 4, pp. 355-367.
`Primary Examiner—D0uglas W. Olms
`Assistant Examiner—Dang Ton
`
`[57]
`
`ABSTRACT
`
`The present invention provides such a data storing format
`that is eifective for solving problems which may arise when
`regenerating video data in a video data storage and com
`municati‘m System- A channel data. “multiplexing PM”
`splits a multiplexed input into audio/video and other com
`ponents. An error correction portion corrects an error of the
`video data output from the channel data demultiplexing
`portion. A video frame extracting portion extracts a video
`frame output from the error correction portion.
`packet
`assembhng portion assembles a packet from the v1deo data
`output from the video frame extractmg portion and the audio
`data output from the data demultiplexing portion. A storage
`portion stores therein the packet output from the packet
`assembling portion. Consequently, video data can be stored
`by video frames.
`
`4/1991 Davies .................................. .. 348/467
`5,012,338
`5,159,452 10/1992 Kinoshita et al. .................... .. 348/466
`
`11 Claims, 33 Drawing Sheets
`
`/71
`
`SYSTEM CONTROL /77
`PORTION
`
`l
`
`73
`
`74
`\
`
`I
`
`‘I
`
`i
`
`7%
`VIDEO
`-
`CODING/DECODING
`POHT'ON
`
`75
`
`
`
`79 \
`
`
`
`; ---------------- " V
`
`AUDIO
`CODING/DECODING
`PORTION
`
`76
`
`i
`
`?°
`
`MULEK’TLEXED
`TRANSMITTING
`IRECEIVING
`PORTION
`
`COMMUN,
`-CAT|ON
`NETWORK
`
`72
`
`!
`1.
`
`r
`I SYSTEM CONTROL/ 131
`
`PORTION A
`/83 1
`1
`!
`MULTIPLEXED
`STORAGE !
`DATA
`TRANSMITTING 4-» DEV‘CE
`!
`IRECEIVING
`!
`
`PORTION \
`.
`
`82
`
`!
`|
`
`Ex. GOOG 1032
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 1 of 33
`
`5,594,736
`
`fiwi
`
`
`
`Cm<IO_En:
`
`
`
`oz_._m2mmm<<29oz:o<Exm
`
`
`
`.......................................!_................!._.»mxo<¢_Womxm#E.5_2<29mzaofimwz_mo.5_\r--.N.
`
`
`
`zo_Eoo_zo_Eon_
`
`
`
`m\_.---m.........................................................ni_
`
`
`
`
`
`Exoiom_xm#._.::2._mzz<xo_om»<mmzmomm
`
`zo_Eon_
`
`ZO:.m_On_
`
`
`
`oz_._ms_mmm<wzfiémzmw
`
`-m_o<23<:..o.
`
`EX. GOOG 1032
`
`Ex. GOOG 1032
`
`
`
`
`
`

`
`US. Patent
`
`Jan. 14, 1997
`
`Sheet 2 0f 33
`
`5,594,736
`
`v
`
`(cf
`
`9.; w
`
`as
`
`3% gm .
`
`
`
`FE. Egg V q NGE
`
`i m”
`
`3
`
`Ex. GOOG 1032
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 3 of 33
`
`5,594,736
`
`omxm.E_S3_2
`
`<._.<D
`
`
`
` >>mms_<E?-_._.._:s_m5m:
`
`8
`
`m.0_n_
`
`CE.moan:
`
`<._.<Q
`
`omxm._n__5:_2
`wms_<mn_.:.._:_2m:m
`ZO_._.<S_mOn_Z_
`
`n_Ommmssz
`
`E.
`
`mmo<m_._
`
`Tll1_
`-_.S:_2m:m
`
`Fmm§<m“_
`
`EX. GOOG 1032
`
`Ex. GOOG 1032
`
`
`
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 4 of 33
`
`5,594,736
`
`B
`
`mo<mo+m
`
`mo_>mo
`
`.V.G_n_
`
`C5.moan:
`
`mm
`
`mmn_mz<E
`
`ZO_._.mOn_
`
`oz_>_momm
`
`zO_._.mOn_
`
`._<._._0_Q
`
`zo_mm__2mz<E.
`
`xmozpmz
`
`Em
`
`NY’
`
`0)
`._<z_s_mm»
`
`._<z__>_mw._.
`
`0
`V—
`
`U
`‘F-
`
`._<z_s_mm»
`
`Em
`
`._<z_sEm:
`
`._<z_s_mmh
`
`EX. GOOG 1032
`
`Ex. GOOG 1032
`
`
`
`
`

`
`US. Patent
`
`Jan. 14, 1997
`
`Sheet 5 of 33
`
`5,594,736
`
`FIG.5
`( PRIOR ART)
`
`CIF'
`
`QCIF
`
`3
`
`FRAME
`
`11
`
`1O
`1 2
`
`1
`
`234567891011
`
`12 13 14 15 1s 17 1s 19 2o 21 22 GROUP OF BLOCKS
`2a 24 2s 26 27 2s 29 so 3132 as
`
`MACRO BLOCKS
`
`BLOCK
`
`Ex. GOOG 1032
`
`

`
`US. Patent
`
`Jan. 14,1997
`
`Sheet 6 0f 33
`
`5,594,736
`
`Q01
`
`TF2 mQE v
`
`mOEDOw
`
`
`
`ZOFEOm 02-000
`
`Ex. GOOG 1032
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 7 of 33
`
`5,594,736
`
`NOE
`
`Cm<EO_En:
`
`
`
`m=2<E..._own.
`
`mommm
`
`zofiommmoo
`
`>:m<m
`
`._._mBmtmmm...
`
`
`A_._._s
`
` :m._.__u_E2.zo:<o_u_:.zmo_._._EE_<._.<Qomo_>H
`
`EX. GOOG 1032
`
`Ex. GOOG 1032
`
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 8 of 33
`
`5,594,736
`
`mmoo<
`
`
`
`wzaoomomommm
`
`zo_Eom
`
`zo_Eon_zo:o_ommn_
`
`AE<EO_En_V
`
`zo_Eon_
`
`z:_BE:2:mmooo_.m_s:Em<
`
`._<Fzmmm&_o
`
`
`
`.u._0<_>=._.Dn_z_
`
`._<zo_m
`
`
`
`
`
`>mos_ms_w_2<mn_wD._<>
`
`
`
`
`
`mo_.om_>zo:os_m>:<mzmn_s_oo-zo_»o2
`
`oz_.8_ammn_.m_2<mu_mm»z_
`
`zo_Eom
`
`zo_Eon_
`
`EX. GOOG 1032
`
`Ex. GOOG 1032
`
`
`
`
`
`
`

`
`US. Patent
`
`Jan. 14, 1997
`
`Sheet 9 of 33
`
`5,594,736
`
`m m m
`
`
`m womaow 2052535
`
`
`
`m zo?mom ozaooma ‘ 2056.... mzaoomo
`
`_. ......................................................... |-|
`
`
`
`_ 6N: mwocomo
`
`A E< mQE v
`
`Ex. GOOG 1032
`
`

`
`US. Patent
`
`Jan. 14, 1997
`
`Sheet 10 of 33
`
`5,594,736
`
`A IMAGE 1 B I IMAGE 2
`
`)
`
`
`
`FIG.1O A (PRIOR ART)
`
`Ex. GOOG 1032
`
`

`
`US. Patent
`
`Jan. 14, 1997
`
`Sheet 11 of 33
`
`5,594,736
`
`t.
`
`m?
`
`NF
`
`€m< moan:
`
`.
`
`ZGE
`
`mw
`
`F
`
`m -zmmmow M
`
`m -ZwmmOw
`M05):
`
`\\\\\\\\\\\\ \\
`
`m
`
`@ <55 89>
`
`Ex. GOOG 1032
`
`

`
`US. Patent
`
`Jan. 14, 1997
`
`Sheet 12 of 33
`
`5,594,736
`
`M
`
`
`SE28 SE96
`‘ zo_Eon_
`
`+ +
`
`_
`
`K\
`
`
`
`.r1 .................................. !|_
`
`
`
`
`
`E R\ .6528 2596
`
`_ 8 + + + 2 .z
`
`E0352 m wz_>_womm\ m m
`2056- k T @zEEwEE m zo_Eon_ F
`232200 r <55 “ m‘ _~ .
`
`M 56425:: m ozaowwwwzaoo \ m
`
`
`
` _. ........................ -l. ........... li? _ zoimom E a v ozaoomQwzaoo M . m 053. m
`
`_ zo_Eon_ “ ................ 1. 2 m \
`
`. zQEom . R
`
`_ \ u + / w b
`
`_ + ./ w E
`
`Ex. GOOG 1032
`
`

`
`US. Patent
`
`Jan. 14, 1997
`
`'Sheet 13 of 33
`
`5,594,736
`
`Alli 02.52%? .5255
`
`
`
`
`
`“ EVE/E \ o @5256 M m zo?mom “ <20
`
`
`
`
`
`m. ................................................................................... liw m _ _ _
`
`
`
`m“ ................................................................................... 1-1m _ _
`
`
`
`
`
`“ zo?mom 55 052 29 E01 <20 o5n< zo?mom
`
`_ v _
`
`_ _
`
`
`
`8; mm\ 8\1 B\ _ _.-|. ................................................................................. l-L
`
`
`
`M <F<a owQ> 5E own; m
`
`
`
`
`
`mm PI] I I | I l l l | l ‘ | I I 1 l i I I A I l I | I | I I ' I l I I | I l | | l I i | I I l l l l I | I l ‘ l l l i l I I i l I I l I i l I l I l l l I I I | I I l I ‘IL
`
`
`
`_ zoimom 6m“; .
`
`
`
`
`
`m 89> mommw 89> . m
`
`
`m zQEom 02cm: _
`
`
`_ M . _ _ <55 05?. zo_Eon_ _
`
`Ex. GOOG 1032
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 14 of 33
`
`5,594,736
`
`E:CD
`:5
`
`<):
`,
`V‘
`? '1' Z
`in:
`G :' 0%’:
`
`
`
`—;
`
`LL
`
`o
`V
`
`1"
`
`LI.
`
`03
`E
`
`-
`
`“-
`
`z
`59%
`5:6
`—?
`
`
`
`z
`E9
`E:
`3-?
`
`0
`
`2 S
`
`4
`Z<
`Z
`
`E9(D
`‘2’
`~>-
`(0
`
`,
`
`
`
`2
`
`8*:
`
`“3‘°
`2
`'
`
`z
`[E0
`2:5
`
`3%
`
`
`
`EX. GOOG 1032
`
`.
`
`,
`
`z
`“:0
`0;}-
`%<
`--2
`I
`
`E
`<
`Q:
`u.
`o
`Lu
`
`9>
`
`o
`
`Ex. GOOG 1032
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 15 of 33
`
`5,594,736
`
`0:
`
`<._.<D0.92
`
`mtrm>
`
`m:LE1.
`
`7mF>mXT_>>m_2<E__m_2<mn_
`
`v:
`
`
`
`m..m__F——
`
`
`Am;.5E2m...mm_2<m...
`m_«%..m_m89>no
`
`89>t:z<:
`O<55o_o:<<55owe
`
`
` EI._._0Zm._I._.OZm._zo_._.<sEon_z_
`
`mmo<m:
`
`EX. GOOG 1032
`
`Ex. GOOG 1032
`
`

`
`US. Patent
`
`Jan. 14, 1997
`
`Sheet 16 of 33
`
`5,594,736
`
`m 5205 dzz<zo
`
`
` w mm -552 _ 55
`
`
`
`_ zQEom w zockom U_ _ ozEmwz wzEEwzwo <25
`
`2: N
`
`
`
`.1 ................................................................ IL
`
`_ a? N8 8; \ v2
`m \ \ own=> \
`. E ii wsEE owl
`
`zzmsm <
`
`
`
`<55
`
`
`
`zorEom 052 ww?mmm
`
`V
`
`Ex. GOOG 1032
`
`

`
`US. Patent
`
`Jan. 14, 1997
`
`Sheet 17 of 33
`
`5,594,736
`
`____I:IG.17 B
`
`
`
`VIDEO FRAME F |G_1 7 A
`
`INFOR -MATION BIT
`
`INFOR -MATION
`
`BIT
`
`\\\\\\\\\\\‘
`
`INFOR —MATION
`
`BIT
`
`PARITY
`
`INFOR -MATION BIT
`
`INFOR -MATION BIT
`
`INFOR -MATION BIT
`
`SYNCHFIONIZING SIGNAL
`
`
`
`VIDEO FRAME
`
`INFOR -MAT|ON 5n"
`
`O
`(I)
`I1.
`
`"
`
`W 14:3.’ ..... -
`
`FILL BIT
`
`FILL BIT
`
`Ex. GOOG 1032
`
`

`
`US. Patent
`
`Jan. 14, 1997
`
`Sheet 18 of 33
`
`5,594,736
`
`m
`
`m zo?mom
`
`
`
`_ ‘E3052 <55
`
`m mm P \
`
`m2
`
`
`
`_l- -|-|\2 ....................................................................... l-|_
`
`m m m
`
`m zockom " ozwazwwméa m 520i
`
`_ 5,5 m 052
`
`M o 2056.‘ I zQEOm
`
`
`
`m :m ._._E wsZE 0mm
`
`E:
`
`ZOFEOm
`
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`2
`J
`
`Ex. GOOG 1032
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 19 of 33
`
`5,594,736
`
`F|\G.1‘9B
`
`.___E|G.19C INFOR3
`
`f
`
`
`-MATIONBIT
`
`
`
`. .\\\\\\
`
`
`
`I K\\\\\
`
`..»f'_'_,
`
`.' .\\\\\‘
`
`
`BIT INFOR-MATION
`
`
`5"’ ,'
`
`
`
`
`
`VIDEOFRAMEFIG_‘|9A
`
`
`
`VIDEOFRAME
`
`
`
`‘.'_'.'.‘_'_'.'_'.‘_'_'_'_
`
`
`
`
`
`
` m\\\\\\\V """""""-------- " \\\\\V
`
`
`SYNCHROMZINGSIGNAL
`
`EX. GOOG 1032
`
`Ex. GOOG 1032
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 20 of 33
`
`5,594,736
`
`.2
`
`
`
`._.Z30_2<<._.<Q
`
`wz_m_>mm_n5m
`
`zo_Eon_
`
`
`cz_._M_Q%mm<-_.r.5_2
`...mxo<n_ZO_._.IOn_ZO_._.mOn_4m&4._.7...&_I0
`
`
`
`zo_mm.s_mz<Etm._._Em2<Eom:mwmm,
`
`
`Gzm__+.u.@_m_<_2oz_._.mmmz.ozfiémzmo
`
`
`
`
`
`ZO_._.mOn._
`
`oz_xm:n_
`
`ZO_._.mOn_
`
`lI.|tI.|IulIl||.l..‘.lIlI|n|||l:uIII-I|.tIIn.IIuIIIIIIIIIIIOIIIII|I|.IlII|I|I.lI|l'I|.I|ln.|I|.Il|II-I.lIl.lI|on|1II|oIu|I.||I|I
`
`_-_-___________J
`
`EX. GOOG 1032
`
`Ex. GOOG 1032
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Jan. 14,1997
`
`Sheet 21 of 33
`
`5,594,736
`
`.___E__[G.21C
`
`
`
`. E
`2%
`A Q
`
`9
`S_,
`5;
`%<2
`gm
`>-
`cn
`
`III!
`
`>\\\\\\
`
`
`
`
`
`11(2)
`BEE
`32
`
`
`
`EX. GOOG 1032
`
`_____ _-
`
`
`
`0:3
`BEE
`2:
`
`[Z
`09+-
`tgga
`— I
`
`m 1
`
`*‘
`
`g
`
`,'l
`
`Z
`591-
`LLLE5
`E5
`.
`
`I
`
`,v
`
`'
`
`I
`
`.-'
`
`:'
`."
`:
`:'
`.-
`
`.-
`.-'
`
`.«-'
`
`.~'
`..-'
`:'
`
`.-‘
`.:'
`:'
`'
`
`Z
`
`$9:
`
`"5.
`
`[I51-
`
`32
`'
`
`<:
`
`N
`'
`0
`-L:
`
`g
`E
`LI.
`0
`u1
`9
`>
`
`3.‘:
`
`3
`3
`
`1.\\\\\\
`
`Ex. GOOG 1032
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 22 of 33
`
`5,594,736
`
`zofimom02.000
`
`ZO_._.mOn_
`
`_.oN._._
`
`m_2<E<m.rz_
`
`—oN.I
`
`GZEOOMD
`
`mmmmz<m»
`
`ZO_._.m_On_
`
`oz_>_momm
`
`zo_Eom
`
`EX. GOOG 1032
`
`
`
`
`
`zo_Eon_._oEzoozo:<o_z:_2s_oo
`
`N.
`
`..
`V
`
`mm.0_“_
`
`Ex. GOOG 1032
`
`
`
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 23 of 33
`
`5,594,736
`
`F|G.23
`
` RECEIVING CODED
`
`IMAGE DATA
`
`STORING START
`(AN INITIAL SCREEN)
`
`
`
`THE INITIAL SCREEN
`(ACCORDING TO H.261)
`
` INTRAFRAME CODING OF
`
`
`
`
`
`TAFIANSFERRING IMAGE
`DATA TO STORAGE
`DEVICE
`
`
`
`EX. GOOG 1032
`
`Ex. GOOG 1032
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 24 of 33
`
`5,594,736
`
`._..-—.___-._.-:-i-..—....:_.—-_.
`
`!__
`
`.|nlII|lnIlI||.I..|vl|IIIIlIIl.ll||IlIlla:I|n'..sullI¢lIIl|II.I|.IIll|II||I|.I|Ia|I|I|lIol|ulI.|lIII
`
`._mZZ<IO
`
`405200
`
`ZO_._.mOn_
`
`ZO.ww_S_wZ<E._.
`
`
`
`<._.<Qomo_>
`
`
`
`zo_Eomoz_o<mm
`
`wS_<In_
`
`
`
`muZ_mmmOOIn_Nm_.
`
`ZO_._.mOn_
`
`mo<s__-zmmmom
`
`
`
`ZO_._.mOn_oz_o<mE
`
`
`
`mmms5zzmmmow
`
`0zEm>>mE
`
`
`
`ZO_._.m_On_4oE.zoo
`
`mm_.2.1
`
`! E ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! E ! ! ! ! ! ! ! !
`
`<r
`
`N 9 u
`
`.
`
`l_-_-__-_____-___-
`
`,____________ ____-_____|
`
`EX. GOOG 1032
`
`Ex. GOOG 1032
`
`
`
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 25 of 33
`
`5,594,736
`
`._mzz<:o
`
`ZO_._.<O_ZD_2S_OO
`
`zo_.Emmzmomm
`
`‘m_o_>mo._Om_._.ZOO
`
`mm..m_u_
`
`<5589>
`
`mo<mo5
`
`mo_>mo
`
`EX. GOOG 1032
`
`Ex. GOOG 1032
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 26 of 33
`
`5,594,736
`
`F|G.26 A
`
`VIDEO DATA 1 (SCREEN-IMAGE)
`
`
`
`
`
`
`
`
`
`
`
`
`
`EX. GOOG 1032
`
`SCREEN NUMBER<wHE~
`
`1 (1)
`
`3 (3)
`
`)
`
`SCREEN-IMAGE
`
`SCREEN !MAGE
`
`SCREEN—|MAGE
`
`
`
`
`
`F|G.26 B
`
`VIDEO DATA 2
`
`
`
`
`
`
`
`Ex. GOOG 1032
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 27 of 33
`
`5,594,736
`
`
`
`
`
`
`
`
`
`
`
`FIG. 27
`
`A
`
`SCREENIMAGE
`-
`
`VIDEO DATA
`
`VIDEO DATA
`
`VIDEO DATA
`
`_
`
`VIDEO DATA
`
`VIDEO DATA
`
`VIDEO DATA
`
`VIDEO DATA
`
`VIDEO DATA
`
`EX. GOOG 1032
`
`SCREENNUMBER
`<wHe~
`
`1 (1)
`<~"=>
`5<5>
`7 (10
`
`)
`
`9
`
`(12)
`
`
`
`
`
`
`
`
`12 (15)
`
`14 (17)
`
`15 (18)
`
`17 (20)
`
`18 (21)
`
`2o (23)
`
`Ex. GOOG 1032
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 28 of 33
`
`5,594,736
`
`<H<Domo_>
`
`
`
`¢.._.<Qomo_>
`
`om<._.<Domo_>
`
`
`8+:8<5589>
`
`
`
`<._.<Qomo_>
`
`mflE<+<oomo_>
`
`mug“.
`
`A||IIl|||.|llII|I.|.|l|||
`
`
`
`<H<Qomo_>
`
`%mmms=._zm_s_<Ev
`
`ombammzmomm
`zmmmom<ozEm2m_m
`
`
`
`<._.<Qom_o_>
`
`omE__2wz<E
`
`EX. GOOG 1032
`
`Ex. GOOG 1032
`
`
`
`
`

`
`U.S. Patent
`
`_
`
` m9:_w_4,_ZO_._.m_OQ05200M__0zEm>>mm0u_mmmssz
`zmmmomM
`_mm_\wtmt
`
`
`
`03
`C‘!
`S_'3
`U.
`
`Mz0_E0n_
`
`
`
`
`z0_mm_s_wz<E. 20:mJompmnmm0z_mmmo0mn_z0_.E0n_._0Ez0o_m_s_<mn_>:.__m<n_<o-4<z_sEm»_dzz<:o
`
`
`
`
`
`
`
`0w
`
`5,594,736
`
`m<500m0_>
`
`
`
`ZO_._.m_On_0z_m0»m
`
`"Iii.l'l'|lIN‘ii‘llll|'|-|‘li||'lJI"'IIli|l‘iiI|llii.iii
`
`EX. GOOG 1032
`
`Ex. GOOG 1032
`
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 30 of 33
`
`5,594,736
`
`F|G.3O
`
`FRAME1
`
`FRAME2
`
`FRAME3
`
`FRAME4
`
`IIII
`II
`
`IIIIIIIIIIIIIIIII
`
`FRAME32
`
`EX. GOOG 1032
`
`Ex. GOOG 1032
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 31 of 33
`
`5,594,736
`
`(STORED)
`
` SCREEN NUMBER
`
`EX. GOOG 1032
`
`Ex. GOOG 1032
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 32 of 33
`
`5,594,736
`
`Exoé
`
`oz:m2mmm<
`
`
`
`.omo_>
`
`m2<E
`
`oz_5<Exm
`
`zo_Eon_
`
`zo_Eon_Gm“:
`
`
`
`
`
`ZO_._.EOn_n=2<._.mm=2_._.
`
`wzzkmmzmm
`
`zo_Eon_
`
`OZ_S=._.E50
`
`
`
`<._.<QO_QD<
`
`._mzz<:o
`
`<55
`
`-:..5sm_o
`
`wz_xm:n_
`
`ZO_._.mOn.
`
`
`
`
`
`oz.._ms_mmm<o_o:<ozmvmmonwmmo_o:<oz_m_momm<oz_xm#_:..5_>_
`
`._.w¥O<n_.mzz<:o
`
`
`-m_o<»<o>Smo<»<o
`
`zo_Eon_
`
`EX. GOOG 1032
`
`Ex. GOOG 1032
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Jan. 14, 1997
`
`Sheet 33 of 33
`
`5,594,736
`
`m
`
`»>m>.
`
`m+>mx _m2<mm
`
`Nm§<mkTi
`
`mmT\hm._
`
`_|ull.|I.L
`
`mm.w_n_
`
`mm?
`
`8_
`
`vm.mm.mm_
`
`wm_
`
`n=o:<
`
`<H<D
`
`ms:
`
`n_S_<._.m
`
`E.
`
`
`
`m_2<m.u_<55om_o_>no55<»<o
`
`
`89>mm_m,%m_E2:>Ez<:oo_o:<89>
`
`
`
`2E8Eozm:n_s_<»mmm_s_<EEozm:Ebzm._
`
`mmo<m:
`
`zo:<2mo¢Z
`
`EX. GOOG 1032
`
`Ex. GOOG 1032
`
`
`

`
`1
`HVIAGE-INFORMATION STORAGE
`REGENERATING DEVICE
`
`BACKGROUND OF THE INVENTION
`
`invention relates to a image—inforrnation
`The present
`storage regenerating device, and more particularly to a video
`communication system which can collect and store multi-
`plexed data of audio/video and other information from video
`data communication terminals (e.g., visual telephone sets)
`according to the recommendation by ITU (International
`Telecommunication Union) and which allows users to
`regenerate the stored data through their terminal sets. More
`particularly, the present invention is related to a video data
`storage control device for regenerating a video data file
`stored in a storing/regenerating center (storage device) for
`video data, and more particularly to a video data regenera-
`tion control device which is capable of transmitting stored
`coded video data from a video data storage and regeneration
`center (storage device) to an audio-visual (AV) terminal and
`enabling the terminal to decode the receivedcoded video
`data and reproduce images thereat, and which is also capable
`of transmitting stored therein coded video data to a receiving
`terminal and enabling the terminal to decode the received
`coded video data and reproduce therefrom the video data.
`
`Recently, besides conventional interactive communica-
`tion services using telephones and facsimiles, such a com-
`munication service has been provided that audio data stored
`at a center of communication network can be always regen-
`erated through users’ terrninal sets. On the other hand, with
`improvement and development of digital communications
`networks, e.g., ISDN (Integrated Services Digital Network),
`it has been possible to use video communication services,
`e.g., visual telephone service. On such background, there is
`an increasing demand for video delivery services using a
`database of video information. Such communication ser-
`vices have been locally realized for practical use.
`Recently, with improvement and development of digital
`transmission charmels and advancement of image process-
`ing technique, it has been expected to develop a method
`allowing one to effectively use image-information services
`over high-speed digital transmission networks, a represen-
`tative of which is ISDN (Integrated Services Digital Net-
`work), by applying advanced high-speed digital-signal pro-
`cessing and related LSI technology. Representative image
`communication services such as visual telephone services
`and video conference services have been already realized
`and practically utilized. Lately, such an information delivery
`service for supplying information to many unspecified users
`at unspecified times has been expected to appear in practice
`as a new business.
`
`To supply information to many unspecified users, it is
`necessary to store information in a storage (recording
`device), e.g., a host center, and to provide a control system
`for regenerating information at the many unspecified audio-
`visual terminals and storing information from the audio-
`visual terminals into the storage. This system is represented,
`for example, by an image-information regeneration control
`device that controls regeneration of image-information from
`the host center to the audio-visual terminals.
`
`In a conventional image-information regeneration control
`device, coded image-information is stored with screen
`(frame) numbers and is transmitted with the same screen
`(frame) numbers to an audio-visual terminal at which the
`coded image-information is then decoded and regenerated
`according to the screen numbers.
`
`10
`
`20
`
`25
`
`30
`
`35
`
`45
`
`50
`
`55
`
`60
`
`65
`
`5,594,736
`
`2
`
`The image-information communication service may often
`transmit a interactive-mode screen (e.g., a service menu or
`guidance for user) and coded data of different object images
`in succession to a user terminal and continuously regenerate
`them at the terminal.
`
`In a conventional image-information transfer system, a
`center control unit negotiates with a terminal as to decoding
`capability of the terminal before transmitting coded image-
`information thereto and, only in case of the terminal having
`a sufficient decoding capability, it reads and transmits the
`coded image-information stored together with their screen
`(frame) numbers to the terminal.
`Japanese laid-open patent publication No. 62-188579
`describes a stored video data regenerating device for use in
`a packet communication network for transmitting moving
`picture information from a moving picture database, which
`converts video data into video signals adapted to be easily
`received by user’s terminal sets by absorbing variations of
`delay in transmission of packets over the packet communi-
`cations network and allows user to get desired video infor-
`mation from the data base at any desired time.
`
`Japanese laid-open patent publication No. 3-132128
`describes a stored video data regenerating device that allows
`for storing messages of video and audio data composed of
`still-pictures and recorded pictures inputted from users
`through their terminal devices and allows each user to
`reproduce a message corresponding to his request on a
`display screen of his terminal.
`Japanese laid-open patent publication No. 4-79655
`describes a stored video data regenerating device that has
`means to register conditions of collecting information from
`a user’s terminal having functions for inputting and output-
`ting video/audio and characters, and that calls the terminal
`according to the registered information and automatically
`receives information and stores the collected information as
`requested by a client.
`As a conventional stored image regenerating device there
`may be designated a device that is described in a reference
`paper titled “Prototyping of PC-Videophone with Built-irr
`Codec based on Open Architecture” (by Ichizo KIMURA
`and 7 et al.), the journal of the Institute of Image Electronics
`Engineers of Japan, 1993, Vol. 22, No. 4, pp. 355-367.
`The above-mentioned reference describes how to store
`and regenerate audio/video data, indicating two possible
`methods: the first method is directed to store the multiplexed
`audio/video data as unchanged and the second method is
`directed to disassemble multiplexed data and store audio/
`video data as separated from each other. However,
`the
`conventional systems prefer the first method from the View
`point of easier processing of multiplexed data at a center
`device and describes necessary functions of this method and
`how to realize said functions.
`
`The present invention pointed out problems of the con-
`ventional method of storing audio/video as multiplexed data
`and provides a novel method that separates media, particu-
`larly video (image-information), and stores them as sepa-
`rated from each other in order to solve the problems
`involved in the conventional systems. Furthermore,
`the
`present invention discloses that an excellent video commu-
`nication system can be constructed by applying therein
`several new techniques. Practically, the data storage and
`regenerating device according to the present invention is
`featured by new functions such as data management by
`frames, synchronizing by time stamps and adjustment of
`information amount by inserting fill bits, and which realizes,
`thereby, so flexible and so high leveled functions that can not
`
`EX. GOOG 1032
`
`Ex. GOOG 1032
`
`

`
`3
`
`4
`
`5,594,736
`
`be attained by the conventional multiplexed data storing
`system.
`Accordingly, it may be clearly understood that the present
`invention in its basic conception quite differs from the
`conventional systems, and it also discloses novel and useful
`art to solve the problems of the conventional systems.
`As described above, the conventional system stores data
`as multiplexed by the method adopted as an international
`standard. The data display at the terminal device is, there-
`fore, conducted usually after synchronization of a FEC-
`frame (forward error correction frame)
`is established.
`Namely, an image can not immediately be displayed. The
`initial. Image is not always encoded by intra-frarne mode
`and may not be self-contained for decoding. Furthermore,
`the terminal may not well reproduce images because of
`occurrence of underflow or overflow of its buffer, which may
`result from that the Storing buffer condition differs from the
`regenerating buffer condition.
`The video data are stored under the control of encoded
`information amount according to buffer condition of a
`terminal. The conventional method can not assure the same
`condition of buffer of a terminal device when storing data
`and when regenerating data. Any terminal may encounter
`overflow or underflow of its buffer when regenerating stored
`video data. Furthermore, in the conventional video regen-
`erating device, such a problem may arise that stored video
`data can not be regenerated at a terminal if the charmel
`transmission rate to which the terminal is adapted is not
`compatible with the channel rate at which the data were
`stored. For instance, said problem may occur when video
`data stored by a terminal adapted only to a channel rate of
`B using a basic interface of ISDN is regenerated at a
`terminal adapted only to a charmel rate of 2B.
`In video data storing, a transmitting AV-terrninal accord-
`ing to the recommendation H.261 encodes successive objec-
`tive frames by motion compensative inter-frame prediction
`and sends the encoded video data. When an video data
`storing center starts storing operation, coded video data by
`screens, for example, B0-B,, are read-out and stored in a
`storage such as a host machine. In this case, the screen B0
`is encoded by prediction with reference to the frame B_1.
`In image reproduction, the coded video data, e.g., frames
`A_3-A1 and Bo-B" stored subsequently to the screen A_1 are
`successively read from the video data storage center by the
`host machine and sent to an AV-terminal. These data main-
`tain an integrity as successive coded video frames and they
`are, therefore, normally decoded at the AV—terminal.
`While the screen B0 is, however, predicted with reference
`to the screen B_1 at the time of data storing (i.e., coding), it
`will be decoded with reference to the screen A_1 that relates
`to a preceding object in the order of coded video data.
`Namely, the screen B0 is predicted according to different
`screens at coding and decoding. As the result, the screen B0
`can not normally be reproduced and displayed at the AV-
`terrninal. Furthermore,
`this may affect
`the subsequent
`screens since screens B1-B" have been dependently coded in
`succession by inter-frame prediction. The desplayed screen
`videos may be damaged.
`In the conventional image-information regeneration con-
`trol device, as described above, a screen number (frame
`number) of video data (frame data) to be transmitted and
`received by an audio-visual terminal is the same as that of
`the corresponding video data stored in a storage. Namely, the
`stored video data with its unchanged screen (frame) number
`are transmitted to the terminal. For example, when video
`data are stored in the order of screen (frame) numbers “I0,
`
`l0
`
`IS
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`corresponding video data to be transmitted
`.
`.
`12, l5, l7 .
`are given the same frame numbers “l0, 12, 15, 17 .
`.
`. ”
`Accordingly, when such video data are regenerated in suc-
`cession to a currently displayed image or video data of
`different object, a discrepancy of screen numbers may occur
`at a junction between two different image sequences.
`For instance, in case of continuously displaying the stored
`video data in succession to a sequence of screen images,
`jointed screen (frame) numbers are described as “screen-
`image (1), screen-image (3), screen-image (5), video data
`(10), video data (12), video data(15)
`.
`.
`. ” which is
`transmitted without any modification. This causes the audio-
`visual terminal to regenerate the received video data in the
`order of their screen (frame) numbers with a discrepancy of
`screen (frame) numbers between “screen-image (5)” and
`“video data (10)”, i.e., a frame interval corresponding to a
`split of image sequences. In this case, video data behind the
`frame interval is regenerated with a delay time.
`As described above, the conventional image-information
`regeneration control device has such a drawback that when
`a screen-image and/or different kinds of video data are
`successively transmitted, discontinuity of screen numbers
`occurs at a junction of the transmitted video data between
`the last screen (frame) of a preceding video data and the top
`screen (frame) of a subsequent video data, causing a delay
`time between the preceding video data and the succeeding
`video data when they are regenerated at the terminal.
`The conventional image-information regeneration control
`device has to first negotiate with a receiving terminal
`whether the terminal has suflicient capability to decode the
`coded video data stored in the device or not. When the
`
`decoding capability of the terminal is insufiicient, i.e., a
`minimal frame interval of video data to be decoded by the
`terminal
`is,
`for example,
`2/29.97
`seconds
`(hereinafter
`described as 2/30 see.) which is more than a frame interval of
`the coded video data stored in a storage of the device, which
`stores the video data encoded at a frame interval 1/29.97
`seconds (hereinafter described as 1/30 sec.) according to a
`coding method based upon recommendation H.261 of ITU
`(International Telecommunications Union), screen numbers
`(frame numbers) of the video data(frame data) to be received
`by the terminal must be “O, 2, 4, 6 .
`.
`. ” which differ from
`screen numbers “O, l, 2, 3 .
`.
`. ” of the stored coded Video
`data. Consequently, the center control device can not supply
`the terminal with the coded video data as be unchanged.
`The center control device may transmit stored video data
`only by odd number frames (1, 3, 5 .
`.
`. ) to get frame
`numbers “O, 2, 4, 6 .
`.
`. ”, but the terminal can not correctly
`regenerate images because information of each frame is
`coded as closely related with information of precedent and
`subsequent frames by inter-frame prediction coding.
`As described above, the conventional video data regen-
`eration control device involves a problem that its center
`control device can not communicate with a terminal not
`having suflicient decoding capability.
`As described above, the conventional multiplexed data
`storing and regenerating device separates multiplexed data
`into audio/video and other information, processes respective
`data and stores or regenerates the data, resulting in arising of
`a differential delay between audio/video data at the time of
`their regeneration.
`
`SUMMARY OF THE INVENTION
`
`It is an object of the present invention to provide an
`image-information storage regenerating device which,
`
`Ex. GOOG 1032
`
`Ex. GOOG 1032
`
`

`
`5
`
`6
`
`5,594,736
`
`unlike the above-mentioned conventional system in which
`audio/video data are stored as multiplexed data, separates
`the received data into audio data and video data, processes
`and stores the video data as video frames (fields) or con-
`tinuous frame (field) having a data format allowing for
`image processing to solve problems that may arise at the
`time of data regeneration; and which is capable of solving
`the problem of an image that can not
`immediately be
`displayed that may occur at regeneration of stored video
`data; and which is capable of supervising an amount of
`information and controlling insertion of fill bits to surely
`prevent occurrence of overflow or underflow of a buffer at
`a regenerating terminal; and which is capable of matching a
`data transmission rate of a data sending side at the time of
`data storing to that of a receiving side so that a receiving
`terminal can receive and regenerate stored video data even
`if a charmel rate to which the terminal is adapted is not
`compatible with a channel rate at which the video data were
`stored.
`
`It is another object of the present invention to provide an
`image-infonnation storage regenerating device by which
`video data of an initial screen to be stored are coded in
`intra-frarne coding mode (called intra-mode) to obtain video
`data coded in intra-mode (intra-coded screen image) of an
`initial screen that may be correctly reproduced by the
`receiving terminal.
`It is another object of the present invention to provide an
`image-inforrnation storage regeneration device which, in
`case of reading coded video data from a storage device and
`successively transmitting different objects of video data to a
`terrrrinal
`to successively regenerate the data thereat,
`is
`capable of previously rewriting a screen (frame) number of
`a top video data just" after the end of preceding different
`video data so that the screen (frame) number is consecutive
`with that of the last screen (frame) number of the preceding
`video data to eliminate a delay time of video data due to a
`frame interval, i.e., a discrepancy between screen (frame)
`numbers of two diiferent video data sequences and to
`successively decode the video data, and which, in image
`storage and delivery service using coded video data stored in
`a center control device, is capable of communicating with
`any terminal having a low decoding capability by slowly
`reproducing images thereat.
`It is another object of the present invention to provide an
`image-lnforrnation storage
`regenerating device which
`attaches a time stamp to every component previous to
`storing therein the received multiplexed data, i.e., audio/
`video data and, when regenerating the multiplexed data,
`absorbs differential delay of different components by using
`the time stamps so that the audio/video signals may not have
`any difierential delay in their regeneration.
`device
`An image-inforrnation
`storage
`regenerating
`according to the present invention splits a multiplex input of
`audio/video data into separate signals by using data demul-
`tiplexing portion, corrects an error of the video data by error
`correcting portion based upon the method of FEC (forward
`error correction), extracts a video frame by using video
`frame extracting portion, assembles a packet with inforrna-
`tion on multiplexing rate and encoding method, by which
`audio/video data are stored, by using packet assembling
`portion, and stores thus prepared data in a storage device,
`providing thereby the possibility of easily processing the
`data to solve any problem that may arise at regeneration of
`video data.
`
`The present invention can provide an image-information
`storage regenerating device which disassembles a packet
`
`10
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`according to attached thereto information, establishes syn-
`chronization of FEC operation at a receiving terminal for
`image regeneration by sending a fill frame as need be, and
`then transmits stored video data to the terminal, assuring
`thereby smooth and quick representation of stored video
`data at the terminal; and which r